00:00:36  <chilts>yay \o/ https://twitter.com/NodeJSNZ/status/369971806817353728
00:00:46  <chilts>now I have to do my homework to make a good presentation :D
00:01:34  * dominictarrquit (Quit: dominictarr)
00:03:33  <chapel>awesome chilts
00:03:44  <chapel>what have you been working on lately?
00:07:03  <chilts>chapel: not much to be honest, but I do learn a lot by osmosis here
00:07:19  <chapel>chilts: I hear ya, I've been too focused on work
00:07:22  <chilts>well, not much in relation to LevelDB anyway ... other stuff, new work has been busy :)
00:07:26  <chilts>yeah
00:07:46  <chilts>and (dare I say it) MongoDB too :)
00:08:54  <chapel>we use mongodb a lot at work
00:08:58  <chapel>and in my previous jobs
00:09:07  <chapel>so I very used to it
00:12:18  * wolfeidaujoined
00:13:26  * VeXiijoined
00:17:21  * meschjoined
00:26:49  * mikealquit (Quit: Leaving.)
00:28:11  * mikealjoined
00:32:50  * jcrugzzjoined
00:36:36  * wilmoore-dbjoined
00:41:38  * i_m_cajoined
00:47:34  * paulfryzeljoined
00:51:55  * paulfryzelquit (Ping timeout: 246 seconds)
00:57:38  * kenansulaymanquit (Quit: ∞♡∞)
01:02:27  * i_m_caquit (Ping timeout: 256 seconds)
01:04:13  * jxsonquit (Remote host closed the connection)
01:08:54  * mikealquit (Quit: Leaving.)
01:32:23  * tmcwjoined
01:39:12  * tmcwquit (Remote host closed the connection)
01:40:23  * tmcwjoined
01:48:07  * paulfryzeljoined
01:52:30  * paulfryzelquit (Ping timeout: 245 seconds)
01:54:35  * thlorenzjoined
01:55:44  * timoxleyjoined
02:00:56  * dguttmanquit (Quit: dguttman)
02:04:30  * jxsonjoined
02:09:04  * jxsonquit (Ping timeout: 256 seconds)
02:16:13  * mikealjoined
02:38:39  * tmcwquit (Remote host closed the connection)
02:40:16  * levelbotquit (Remote host closed the connection)
02:40:34  * levelbotjoined
03:00:56  * thlorenzquit (Remote host closed the connection)
03:05:15  * dguttmanjoined
03:18:19  * prettyrobotsquit (Quit: ZNC - http://znc.in)
04:05:28  * jxsonjoined
04:10:03  * jxsonquit (Ping timeout: 260 seconds)
04:15:22  * i_m_cajoined
04:37:04  * i_m_caquit (Read error: Connection reset by peer)
04:37:29  * i_m_cajoined
04:41:00  * dguttmanquit (Quit: dguttman)
04:49:48  * dguttmanjoined
05:06:01  * jxsonjoined
05:06:26  * jxsonquit (Read error: Connection reset by peer)
05:06:43  * jxsonjoined
05:07:43  * mikealquit (Quit: Leaving.)
05:08:14  * mikealjoined
05:11:07  * jxsonquit (Ping timeout: 256 seconds)
05:25:20  * levelbotquit (Remote host closed the connection)
05:25:39  * levelbotjoined
05:42:51  * i_m_caquit (Ping timeout: 256 seconds)
06:05:09  * dguttmanquit (Quit: dguttman)
06:28:21  * julianduquequit (Ping timeout: 248 seconds)
06:56:30  <levelbot>[npm] level-set@0.0.3 <http://npm.im/level-set>: Saves JSON to your LevelDB (@maiah)
06:59:27  * wilmoore-dbquit (Remote host closed the connection)
07:00:01  * wilmoore-dbjoined
07:04:37  * wilmoore-dbquit (Ping timeout: 248 seconds)
07:11:04  * wilmoore-dbjoined
07:15:34  * dominictarrjoined
07:21:30  <levelbot>[npm] level-set@0.0.4 <http://npm.im/level-set>: Saves JSON to your LevelDB (@maiah)
07:35:32  <levelbot>[npm] search-index@0.2.16 <http://npm.im/search-index>: A text search index module for Node.js. Search-index allows applications to add, delete and retrieve documents from a corpus. Retrieved documents are ordered by tf-idf relevance, filtering on metadata, and field weighting (@fergie)
07:37:47  * dominictarrquit (Quit: dominictarr)
07:38:20  <chilts>dominictarr: mind if I steal (lots of) stuff from your scotland.js talk on LevelDB ... I'm doing a presentation to Node Night in Wgtn next week. http://www.meetup.com/Node-js-Wellington/events/131124832/
07:38:31  * chiltsjust watched the vimeo video
08:05:16  * dominictarrjoined
08:11:53  * ralphtheninjaquit (*.net *.split)
08:11:54  * jondelamottequit (*.net *.split)
08:12:00  * ralphtheninjajoined
08:13:47  * jondelamottejoined
08:36:54  * kenansulaymanjoined
08:41:09  <substack>chilts: http://www.meetup.com/Node-js-Wellington/events/131124832/
08:41:12  <substack>whoops
08:41:30  <substack>chilts: https://npmjs.org/package/bytewise would be a good thing to cover in a leveldb talk
08:42:10  <substack>I think the details about how leveldb works are less important for audiences and the tricks about how to cope with a flat key/value namespace with fast sorted ranges are much more important
08:43:49  <substack>because once you have a good feel for how to generate keys that will sort properly for efficient queries the "how do I use this" picture becomes much more clear
09:00:19  * wilmoore-dbquit (Remote host closed the connection)
09:12:03  <levelbot>[npm] level-set@0.0.5 <http://npm.im/level-set>: Saves JSON to your LevelDB (@maiah)
09:18:00  <levelbot>[npm] level-push@0.0.1 <http://npm.im/level-push>: Pushes JSON to your LevelDB with auto-generated UUID (@maiah)
09:22:30  <levelbot>[npm] level-set@0.0.6 <http://npm.im/level-set>: Saves JSON to your LevelDB (@maiah)
09:28:30  <levelbot>[npm] level-set@0.0.7 <http://npm.im/level-set>: Saves JSON to your LevelDB (@maiah)
09:29:27  * kenansulaymanquit (Quit: ∞♡∞)
09:30:25  <chilts>substack: thanks ... yeah, just saw that on your twitter feed :)
09:30:26  <chilts>thanks
09:31:28  <chilts>I think the talk is going to be titles "Creating your own Database", so I'll just touch on LevelDB first to let them know what it's about, then I'll talk about the whole ecosystem popping up around it :)
09:31:39  <chilts>that bytewise looks cool though
09:53:00  <levelbot>[npm] level-push@0.0.2 <http://npm.im/level-push>: Pushes JSON to your LevelDB with auto-generated UUID (@maiah)
09:58:12  * ramitosquit (Ping timeout: 260 seconds)
10:02:12  * mcollinajoined
10:05:31  <levelbot>[npm] levelgraph@0.6.4 <http://npm.im/levelgraph>: A graph database for Node.js and the browser built on top of LevelUp (@matteo.collina)
10:24:33  <levelbot>[npm] level-assoc@0.5.0 <http://npm.im/level-assoc>: relational foreign key associations (hasMany, belongsTo) for leveldb (@substack)
10:50:25  * mcollinaquit (Ping timeout: 246 seconds)
11:17:38  * jcrugzzquit (Ping timeout: 264 seconds)
11:27:19  * thlorenzjoined
11:44:08  * jcrugzzjoined
11:52:40  * jcrugzzquit (Ping timeout: 264 seconds)
12:03:25  * Acconutjoined
12:08:11  * Acconutquit (Ping timeout: 256 seconds)
12:11:00  * ramitosjoined
12:14:11  * Acconutjoined
12:30:49  * timoxley_joined
12:35:03  * timoxleyquit (Ping timeout: 276 seconds)
12:41:25  * ednapiranhajoined
12:47:07  * fb55joined
12:47:57  * Acconutquit (Quit: Acconut)
12:48:21  * kenansulaymanjoined
13:02:54  * pgtejoined
13:06:23  <pgte>@dominictarr I'm now using level-trigger for keeping a persistent job queue, but I need something that allows me to serialize job execution (or to define a maximum job concurrency). Do you know any module / suggestion?
13:10:19  * thlorenzquit (Remote host closed the connection)
13:33:59  <kenansulayman>pgte "serialize job execution"?
13:34:38  <pgte>kenansulayman: yup, run one job at a time, limit the maximum concurrency to 1
13:35:19  <kenansulayman>wouldn't that be easier if you implement a lock system?
13:36:28  <kenansulayman>Do you want the overall concurrency (all queue workers) to be 1 — or per worker?
13:37:44  * VeXiiquit (Quit: Lost terminal)
13:38:41  * julianduquejoined
13:44:21  * tmcwjoined
13:48:23  <dominictarr>pgte: juliangruber wrote a module for that i think: level-schedule
13:48:52  <pgte>@dominictarr thanks, I'll check it out
13:49:44  <pgte>@kenansulayman no lock, I don't want to queue in memory, I want the next job to be retrieved from the database as one finishes
13:49:56  * julianduquequit (Quit: leaving)
13:50:02  <dominictarr>pgte: you don't need @ in irc
13:50:30  <kenansulayman>pgte I suggested a lock since you said you want a maximum concurrency of 1
13:50:31  <pgte>dominictarr: of course not, been tweeting too much ;)
13:50:38  <dominictarr>:)
13:52:46  * julianduquejoined
13:54:40  <pgte>dominictarr: hmmm, this module doesn't limit the concurrency, I guess I'll have to roll out my own..
13:55:00  <dominictarr>pgte: or pull request
13:55:27  <dominictarr>hmm, or maybe they do quite different things
13:55:38  <pgte>dominictarr: it does scheduling, which is more than I need, but I'll consider it
13:55:58  <pgte>kenansulayman: yup, of course, thanks for your help though :)
13:56:17  <kenansulayman>ok
14:04:28  * alanhoffjoined
14:15:55  * julianduquequit (Quit: leaving)
14:19:49  * timoxleyjoined
14:22:05  * timoxley_quit (Ping timeout: 245 seconds)
14:22:30  * fallsemojoined
14:48:28  * jerrysvjoined
14:53:21  * timoxleyquit (Read error: Connection reset by peer)
14:58:48  * pgtequit (Remote host closed the connection)
15:01:51  * paulfryzeljoined
15:07:51  * fb55quit (Remote host closed the connection)
15:08:18  * fb55joined
15:12:35  * fb55_joined
15:12:37  * fb55quit (Ping timeout: 248 seconds)
15:47:21  * mikealquit (Quit: Leaving.)
15:47:47  * ednapiranhaquit (Remote host closed the connection)
15:50:50  * mikealjoined
15:53:18  * dguttmanjoined
16:11:06  * jxsonjoined
16:15:00  * mikealquit (Quit: Leaving.)
16:15:44  * jxsonquit (Read error: Operation timed out)
16:18:50  * ednapiranhajoined
16:26:34  * dguttmanquit (Quit: dguttman)
16:27:42  * ednapiranhaquit (Ping timeout: 264 seconds)
16:27:59  * dguttmanjoined
16:31:29  * dominictarrquit (Quit: dominictarr)
16:37:57  * fb55_quit (Remote host closed the connection)
16:55:02  * ednapiranhajoined
17:06:24  * dominictarrjoined
17:14:14  * tmcwquit (Remote host closed the connection)
17:17:11  * niftylettucequit (Ping timeout: 246 seconds)
17:19:53  * kenansulaymanquit (Quit: ≈♡≈)
17:20:29  * wilmoore-dbjoined
17:27:44  * jxsonjoined
17:31:56  * mikealjoined
17:32:13  * mikealquit (Client Quit)
17:33:41  * mikealjoined
17:38:30  * tmcwjoined
18:03:32  * esundahl_joined
18:23:42  * julianduquejoined
18:27:38  * jcrugzzjoined
18:43:04  * esundahl_quit (Remote host closed the connection)
18:43:30  * esundahljoined
18:47:38  * esundahlquit (Ping timeout: 245 seconds)
18:53:17  * paulfryzelquit (Remote host closed the connection)
18:53:55  * paulfryzeljoined
18:57:58  * paulfryzelquit (Ping timeout: 246 seconds)
19:00:28  * esundahljoined
19:24:35  * jcrugzzquit (Ping timeout: 278 seconds)
19:26:28  * niftylettucejoined
19:48:34  * pgtejoined
19:48:56  <pgte>https://github.com/pgte/level-jobs#readme
19:50:09  <mbalho>steve jobs
19:54:29  * paulfryzeljoined
19:58:29  * levelbotquit (Ping timeout: 248 seconds)
19:59:02  * paulfryzelquit (Ping timeout: 264 seconds)
20:12:20  <juliangruber>pgte: am I right that the only difference between level-jobs and level-schedule is that level-jobs supports concurrency?
20:12:36  <juliangruber>pgte: also, for peek, check out level-peek
20:15:35  <pgte>juliangruber: it's a bit more than that IIRC. level-jobs doesn't support scheduling, jobs are always async and retries on error.
20:16:04  <pgte>juliangruber: tried level-peek, but it's a bit too much for what I needed
20:16:18  <pgte>juliangruber: I just needed a read stream of limit 1 and that's it :)
20:16:29  <juliangruber>pgte: yeah I wanted to put solid error recovery logic into level-schedule
20:16:50  <juliangruber>pgte: the reason for level-peek's complexity is old leveldown behavior
20:17:20  <pgte>juliangruber: error recovery and concurrency was the most difficult part, lots of synchronization required..
20:17:55  <juliangruber>yeah
20:18:13  <juliangruber>level-schedule was so easy beacuse of the one-task limit
20:18:41  <pgte>juliangruber: also, I found out that, for that I needed to have 2 sublevels, one for new work and another for pending work
20:19:04  <pgte>juliangruber: it would probably mean rewriting level-schedule
20:19:33  <juliangruber>mmh yeah
20:20:47  * gwenbelljoined
20:22:25  <pgte>juliangruber: I considered doing it, but then it would make level-sublevel even more complicated than level-jobs is because of the scheduling...
20:22:43  <juliangruber>mhm
20:22:48  <juliangruber>maybe eventually we can have one module
20:22:53  <juliangruber>but for now this seems fine
20:23:02  <juliangruber>as long as you don't need scheduling and retries and concurrency :D
20:23:47  <pgte>juliangruber: exactly
20:50:24  * jxsonquit (Remote host closed the connection)
20:51:20  * gwenbellquit (Quit: Lost terminal)
20:54:11  * paulfryzeljoined
21:20:43  * jxsonjoined
21:28:45  * jxsonquit (Ping timeout: 245 seconds)
21:31:27  * redidasjoined
21:31:34  * ednapiranhaquit (Remote host closed the connection)
21:32:40  <redidas>So I was doing some tests with levelup on windows...
21:33:43  <redidas>after about 1.3 million puts things get crazy slow. the node process starts using 2+ gb of ram
21:34:20  <redidas>is leveldb on windows just less stable?
21:34:56  <redidas>my node process dies shortly after the slowdown with a process out of memory error
21:39:18  <redidas>I should probably switch to linux... but I'm kind of stuck in a microsoft shop
21:40:23  <redidas>All this made me kind of curious though - what are some of the largest leveldbs you all have worked with?
21:42:08  <mbalho>ive only done aroudn a million in tests but ill be trying to break it pretty soon for a project im working on
21:42:28  <mbalho>redidas: also the process out of memory error could very much be a JS issue and not leveldb related
21:43:29  <redidas>I'm writing a script to query a SQL Server table in batches of 10,000 records
21:43:40  <redidas>after each query of 10,000 I batch write the result to leveldb
21:44:13  <redidas>I commented out the leveldb work, and I run through all 3.5 million records no problem, getting them from SQL Server
21:44:29  <redidas>when I decide to start writing them to leveldb though, that's where I start having issues
21:45:12  <redidas>leveldb is left open the whole time... I'm considering maybe opening it and closing it after each batch
21:45:14  * jcrugzzjoined
21:45:19  <mbalho>redidas: is your code online?
21:46:35  <redidas>not at this time
21:47:21  <redidas>its a work related thing, not an open source project
21:47:36  <redidas>I could paste a snippet of it somewhere though I suppose
21:48:21  <redidas>Its not a huge deal at all, this is something I'm just doing out of curiosity to see if we can do anything with leveldb
21:48:22  <mbalho>a good hack is to write lots of node modules and then publish the generic bits to npm
21:51:13  * jxsonjoined
21:52:51  <redidas>I'm gonna try the opening and closing idea quick and report back
21:53:37  <rescrv>redidas: opening and closing will only slow you down
21:53:48  <rescrv>you might want to try hyperleveldb for higher write throughput
21:55:25  <rescrv>Your primary writer thread is being held up in the "sleep" used to throttle when compaction really starts hitting. I suspect that if you measured it, you'd see yourself limited to just under 1,000 "Write" calls per second.
21:56:57  * wilmoore-dbquit (Remote host closed the connection)
21:57:38  <rescrv>redidas: http://hackingdistributed.com/2013/06/17/hyperleveldb/
21:57:47  * fb55joined
22:08:54  <redidas>Thanks I'll have to check that out
22:09:29  <rescrv>redidas: please let me know how it goes
22:09:39  <redidas>the open/close for each batch didn't slow down too much at first, but I'm still hitting that wall at 1.3 million puts
22:09:52  <rescrv>we also provide commercial support for HyperLevelDB, in case that appeals to your organization
22:12:27  * mikealquit (Quit: Leaving.)
22:18:45  * jcrugzzquit (Ping timeout: 276 seconds)
22:23:54  * esundahlquit (Remote host closed the connection)
22:24:21  * esundahljoined
22:28:34  * esundahlquit (Ping timeout: 240 seconds)
22:31:18  * esundahljoined
22:31:40  * paulfryzelquit (Remote host closed the connection)
22:32:18  * paulfryzeljoined
22:37:02  * paulfryzelquit (Ping timeout: 256 seconds)
22:39:13  * paulfryzeljoined
22:41:06  * pgtequit (Remote host closed the connection)
22:42:46  * paulfryzelquit (Remote host closed the connection)
22:43:22  * paulfryzeljoined
22:43:44  * fb55quit (Remote host closed the connection)
22:45:34  * mikealjoined
22:48:14  * paulfryzelquit (Ping timeout: 264 seconds)
22:51:56  * esundahlquit (Ping timeout: 256 seconds)
22:52:50  * esundahljoined
23:12:38  * tmcwquit (Remote host closed the connection)
23:18:57  * fallsemoquit (Quit: Leaving.)
23:24:00  * jcrugzzjoined
23:29:19  * jcrugzzquit (Ping timeout: 264 seconds)
23:31:10  * paulfryzeljoined
23:34:03  * mikealquit (Quit: Leaving.)
23:35:55  * mikealjoined
23:43:00  * paulfryzelquit (Remote host closed the connection)
23:43:34  * paulfryzeljoined
23:47:38  * paulfryzelquit (Ping timeout: 245 seconds)
23:51:27  * jerrysvquit (Remote host closed the connection)
23:56:04  * esundahlquit (Remote host closed the connection)