00:48:39  * jerrysv_quit (Quit: Leaving...)
01:13:47  * kenansulaymanchanged nick to kenan|afk
01:31:04  * paulfryzelquit (Remote host closed the connection)
01:31:23  * thlorenzjoined
01:34:30  <thlorenz>rvagg: starting to see these readable stream errors in node 0.8: https://travis-ci.org/thlorenz/transfigurify/jobs/17732999#L582
01:34:50  <thlorenz>same here: https://travis-ci.org/thlorenz/apply-transform/jobs/17718165#L201
01:34:58  <thlorenz>do you want me to file an issue?
01:35:15  * ednapiranhaquit (Quit: Leaving...)
01:38:15  * paulfryzeljoined
01:48:52  * paulfryzelquit (Read error: Connection reset by peer)
01:49:24  * paulfryzeljoined
01:53:07  * ednapiranhajoined
02:05:14  * paulfryzelquit (Remote host closed the connection)
02:05:22  * ednapiranhaquit (Quit: Leaving...)
02:06:53  <rvagg>thlorenz: what version of readable-stream are you using?
02:07:02  <rvagg>you should be using ~1.0.x
02:07:16  <thlorenz>rvagg: I suppose the latest I just created one of those modules today
02:07:31  <rvagg>~1.1 is streams3 al. la. node 0.11
02:07:37  <rvagg>~1.0 is streams2, stable 0.10
02:07:49  * rvaggcan't load travis, stupid rails app
02:07:54  <thlorenz>rvagg: https://github.com/thlorenz/apply-transform/blob/master/package.json#L19
02:08:16  <rvagg>thlorenz: take that down to ~1.0.24
02:08:31  <thlorenz>but that should actually only load if there is no stream: https://github.com/thlorenz/apply-transform/blob/master/string-readable.js#L6
02:08:40  <thlorenz>so I'm not even sure why it applies on the server
02:08:57  <thlorenz>0.8 has stream.Readable right?
02:09:02  <thlorenz>rvagg: ok I'll take it down
02:09:05  <thlorenz>thanks
02:09:12  <rvagg>thlorenz: don'
02:09:23  <rvagg>thlorenz: don't require('stream') at all, just require('readable-stream')
02:09:35  <rvagg>thlorenz: I see the error now, interesting
02:09:43  <thlorenz>rvagg: why, wouldn't I want to favor the built in?
02:09:58  <rvagg>thlorenz: not at all, then it's the luck of the draw what you get
02:10:07  <rvagg>with readable-stream you get pure control over which version of streams you're uysing
02:10:08  <thlorenz>ok, makes sense
02:10:18  <rvagg>stick with 1.0.x for now and even on node 0.11 and 0.8 you get pure streams2
02:10:27  <rvagg>then when you're comfortable with streams3 and it's stable, use 1.1
02:10:37  * rvaggis considering taking 1.1 off @latest but I need to talk to isaac about that
02:11:03  <thlorenz>rvagg: ok this is weird
02:11:10  <thlorenz>locally tests pass with 0.8
02:11:23  <thlorenz>even with the newest readable stream
02:12:22  <thlorenz>same version of node locally though (only difference, I'm on a Mac)
02:14:49  <thlorenz>rvagg: a weird, actually dialing down version repro'd the problem
02:15:19  <thlorenz>I believe it's because I'm using through2 in my tests and that is using ~1.1.9
02:15:32  <thlorenz>(compatibility issue I guess)
02:19:45  <thlorenz>rvagg: now seeing this locally as well - I got latest through2 which depends on ~1.0.17 of rs, so I installed this toplevel as well
02:20:12  * jcrugzzquit (Ping timeout: 265 seconds)
02:21:56  <thlorenz>rvagg: if you wanna check it out (not urgent) pull latest here and do npm test : https://github.com/thlorenz/apply-transform
02:24:55  <rvagg>thlorenz: remove your readable-stream and string_decoder and npm install again
02:25:10  <rvagg>I've updated string_decoder to use a substitute Buffer.isEncoding() where it's missing
02:25:12  <thlorenz>rvagg: I did rm -rf node_modules
02:25:18  <thlorenz>ah now ok
02:25:21  <rvagg>rad
02:25:46  <thlorenz>rvagg: bingo, that fixed EVERYTHING
02:26:20  <thlorenz>thanks, but still stick with lower stream-readable until further notice?
02:26:24  <rvagg>fixes a bunch of failing tests in readable-stream that I need to fix for Node 0.8 too actually
02:26:30  <rvagg>thlorenz: stick with it until 0.12 at least IMO
02:26:37  <rvagg>personal preference but I'm not using 1.1 anywhere
02:26:39  <rvagg>too .. new
02:26:40  <thlorenz>rvagg: but why is an alpha published as the latest version?
02:26:50  <rvagg>had some compatibility issues with it myself
02:27:10  <thlorenz>whoever does npm install -S stream-readable will get that
02:27:10  <rvagg>thlorenz: the nature of the versioning beast! I'm going to try and tag it as @unstable or somethign but I'll have to talk to Isaac
02:27:21  <thlorenz>ok, sounds good
02:27:27  <rvagg>I guess the idea is to push streams3 out into the real world for testing before Node 0.12 goes live
02:27:33  <rvagg>but that means you're a guinea pig
02:27:40  <thlorenz>:) you are welcome
02:28:12  <thlorenz>rvagg: you know you can change what the latest version is right?
02:28:41  <thlorenz>i.e. you could make 1.0.17 the latest so that is what ppl would get unless they do npm install stream-readable@1.1
02:28:55  <rvagg>yeah ... but until I talk to isaac I don't want to go ahead and do that
02:28:58  <rvagg>for now I'm just following suit
02:29:05  <thlorenz>makes sense
02:29:26  <thlorenz>thanks for your help
02:29:41  <rvagg>gently, gently with a package who's download counts look like this: https://nodei.co/npm-dl/readable-stream.png
02:30:45  <thlorenz>yep - lots of ppl will run into these problems hopefully isaacs will understand that once clear problems emerge it is wise to pull this version or at least not make it default
02:32:50  * TehShrikejoined
03:06:32  * paulfryzeljoined
03:10:59  * paulfryzelquit (Ping timeout: 240 seconds)
03:12:56  * TehShrikequit (Quit: Leaving.)
03:17:52  * TehShrikejoined
03:32:15  * Sorellaquit (Quit: It is tiem!)
03:53:24  * eugenewarequit (Remote host closed the connection)
03:53:54  * eugenewarejoined
03:54:14  * jcrugzzjoined
03:56:22  * jerrysvjoined
03:57:50  * paulfryzeljoined
03:58:27  * eugenewarequit (Ping timeout: 252 seconds)
03:58:45  * TehShrikequit (Quit: Leaving.)
03:59:26  * paulfryzelquit (Read error: Connection reset by peer)
03:59:40  * paulfryzeljoined
04:03:59  * paulfryzelquit (Ping timeout: 240 seconds)
04:09:10  * jerrysvquit
04:09:38  * TehShrikejoined
04:39:20  * thlorenzquit (Remote host closed the connection)
04:39:52  * thlorenzjoined
04:44:18  * thlorenzquit (Ping timeout: 245 seconds)
05:00:17  * paulfryzeljoined
05:04:29  * paulfryzelquit (Ping timeout: 240 seconds)
05:48:41  * thlorenzjoined
05:51:06  * thlorenzquit (Remote host closed the connection)
05:53:33  * eugenewarejoined
05:55:51  * guybrushquit (Excess Flood)
05:56:06  * guybrushjoined
06:02:53  * paulfryzeljoined
06:07:29  * paulfryzelquit (Ping timeout: 240 seconds)
06:08:01  * pfrazequit (Ping timeout: 252 seconds)
06:33:23  * blessYahuquit (Remote host closed the connection)
06:33:38  * ralphtheninjajoined
07:03:28  * jcrugzzquit (Ping timeout: 245 seconds)
07:03:51  * paulfryzeljoined
07:07:59  * paulfryzelquit (Ping timeout: 240 seconds)
07:18:59  * ralphtheninjaquit (Ping timeout: 260 seconds)
07:26:21  * daviddiasjoined
07:32:18  * jcrugzzjoined
07:38:22  * eugenewarequit (Remote host closed the connection)
07:38:53  * jcrugzzquit (Ping timeout: 245 seconds)
07:38:55  * eugenewarejoined
07:42:59  * eugenewarequit (Ping timeout: 240 seconds)
08:04:26  * paulfryzeljoined
08:08:59  * paulfryzelquit (Ping timeout: 240 seconds)
08:10:56  * daviddiasquit (Read error: Connection reset by peer)
08:11:44  * daviddiasjoined
08:18:21  * wolfeidauquit (Ping timeout: 265 seconds)
08:19:48  * rudquit (Ping timeout: 265 seconds)
08:20:37  * rudjoined
08:20:56  * rudquit (Changing host)
08:20:56  * rudjoined
08:21:52  * wolfeidaujoined
08:36:14  * ncthom91joined
08:36:28  * ncthom91quit (Max SendQ exceeded)
08:37:04  * ncthom91joined
08:39:30  * ncthom91quit (Client Quit)
09:01:36  * jjmalinajoined
09:05:08  * paulfryzeljoined
09:09:29  * paulfryzelquit (Ping timeout: 240 seconds)
09:11:32  * kenan|afkchanged nick to kenansulayman
09:28:11  * eugenewarejoined
09:44:21  * eugenewarequit (Ping timeout: 252 seconds)
10:05:58  * paulfryzeljoined
10:09:21  * eugenewarejoined
10:10:29  * paulfryzelquit (Ping timeout: 240 seconds)
11:06:43  * paulfryzeljoined
11:09:10  * jjmalinaquit (Quit: Leaving.)
11:09:29  * jjmalinajoined
11:09:34  * jjmalinaquit (Max SendQ exceeded)
11:09:57  * jjmalinajoined
11:10:59  * paulfryzelquit (Ping timeout: 240 seconds)
11:18:33  * jjmalinaquit (Quit: Leaving.)
11:29:55  * daviddiasquit (Read error: Connection reset by peer)
11:50:29  * eugenewarequit (Remote host closed the connection)
11:50:57  * eugenewarejoined
11:54:45  * eugenewa_joined
11:55:35  * eugenewarequit (Remote host closed the connection)
12:07:28  * paulfryzeljoined
12:11:59  * paulfryzelquit (Ping timeout: 240 seconds)
12:42:17  * cwmmajoined
13:05:58  * thlorenzjoined
13:08:16  * paulfryzeljoined
13:12:29  * paulfryzelquit (Ping timeout: 240 seconds)
13:32:00  * thlorenzquit (Remote host closed the connection)
13:52:01  * AndreasMadsenjoined
13:55:17  * tarrudajoined
13:58:16  * AndreasMadsenquit
14:09:10  * paulfryzeljoined
14:13:45  * paulfryzelquit (Ping timeout: 252 seconds)
14:18:18  * Sorellajoined
14:19:10  * Sorellaquit (Max SendQ exceeded)
14:20:03  * Sorellajoined
14:32:43  * brianloveswordsquit (Excess Flood)
14:34:13  * brianloveswordsjoined
14:52:40  * pfrazejoined
15:09:45  * paulfryzeljoined
15:13:59  * paulfryzelquit (Ping timeout: 240 seconds)
15:19:04  * thlorenzjoined
15:42:29  * Punnajoined
15:52:29  * Punnaquit (Ping timeout: 240 seconds)
15:56:40  * paulfryzeljoined
16:00:22  * ryan_ramagejoined
16:02:57  * ednapiranhajoined
16:03:14  * paulfryzelquit (Read error: Connection reset by peer)
16:03:52  * paulfryzeljoined
16:16:40  * jerrysvjoined
16:31:35  * jcrugzzjoined
16:53:32  * daviddiasjoined
17:19:43  * eugenewa_quit (Remote host closed the connection)
17:20:10  * eugenewarejoined
17:24:35  * eugenewarequit (Ping timeout: 245 seconds)
17:32:53  <kenansulayman>rvagg is there already a solution for batch get? we would benefit greatly from it
17:32:59  <kenansulayman>I mean no JS sugar
17:33:08  <jerrysv>...
17:33:14  <kenansulayman>jerrysv what?
17:33:51  <jerrysv>kenansulayman: https://github.com/rvagg/node-leveldown/pull/68
17:34:14  <jerrysv>the full discussion is there
17:34:23  <kenansulayman>ah nice, I shamefully remember rvagg linking me to it
17:35:36  <jerrysv>i opened a pull request ... and a can of worms
17:51:06  * daviddiasquit (Read error: Connection reset by peer)
17:57:31  * eugenewarejoined
18:02:38  * eugenewarequit (Ping timeout: 264 seconds)
18:23:15  <rescrv>kenansulayman: a multi-get will (should?) not improve performance of the C++ LevelDB. There may be room for optimization in leveldown, but I would be very surprised if taking a snapshot and doing iterated Get was significantly different from a MutliGet
18:23:31  <rescrv>you're basically moving the "for" loop from outside a function call to inside a function call
18:24:32  * jxsonjoined
18:25:38  * jxsonquit (Remote host closed the connection)
18:26:00  * jxsonjoined
18:28:18  <kenansulayman>rescrv Imagine we have a user stored at "id" and values stored at id/downloads, id/views and id/transactions this would be three callbacks in node => db.get(......db.get(.......db.get(......; if we can delegate that into the lowlevel as a simply db.get([a, b, c]) this would save code and the abstraction cost for all the highlevel callbacks, if implemented in c++ land
18:29:19  <kenansulayman>I have to leave just now, please answer anyway, my bouncer will catch it. I'll send the mail later today
18:30:41  * kenansulaymanchanged nick to kenan|afk
18:36:32  * dguttmanjoined
18:38:42  * ralphtheninjajoined
18:52:49  * tarrudaquit (Quit: WeeChat 0.4.2)
18:54:51  * mikealjoined
19:19:51  * pfraze_joined
19:21:28  * pfrazequit (Ping timeout: 265 seconds)
19:52:00  * jxsonquit (Remote host closed the connection)
19:55:44  <ogd>prettyrobots: good news! the dat tests now pass on top of locket
20:04:25  <rescrv>kenan|afk: that makes sense. In C++ land, the iterator to do that is cheap, and doing it in LevelDB would be expensive. Maybe you want a "get range" that makes an iterator, scans the range, and then returns all results in one cross of the interface boundary
20:05:24  <ogd>rescrv: yea i think the hypothesis was whether or not that would have a significant impact on different types of workloads, i dont think anyones done the legwork yet
20:11:55  * ednapiranhaquit (Remote host closed the connection)
20:23:33  * pfraze_quit (Ping timeout: 248 seconds)
20:25:52  * pfrazejoined
20:27:54  * jxsonjoined
21:04:54  * blessYahujoined
21:05:12  * blessYahuquit (Remote host closed the connection)
21:07:47  * jcrugzzquit (Ping timeout: 260 seconds)
21:33:00  * ednapiranhajoined
21:41:29  * ralphtheninjaquit (Ping timeout: 240 seconds)
22:04:46  * eugenewarejoined
22:16:22  * jcrugzzjoined
22:16:41  * blessYahujoined
22:16:47  * blessYahuquit (Remote host closed the connection)
22:17:26  * ralphtheninjajoined
22:24:39  * cwmmaquit (Quit: cwmma)
22:37:26  * eugenewarequit (Ping timeout: 264 seconds)
22:42:20  * kenan|afkchanged nick to kenansulayman
22:44:09  * ralphtheninjaquit (Ping timeout: 251 seconds)
22:46:20  <rescrv>ogd: I strongly suspect that unless the constant cost for crossing the js/c++ boundary is high, it will make no difference
23:01:14  * paulfryz_joined
23:03:59  * paulfryzelquit (Ping timeout: 240 seconds)
23:10:32  * paulfryz_quit (Remote host closed the connection)
23:10:40  <kenansulayman>well rescrv you have to cross js/c++ three times in my example; the multiget would only once call c++ land and wait for the callback
23:11:04  <rescrv>kenansulayman: and what's the cost of that crossing? I'm betting no more than 100ns total
23:11:09  * paulfryzeljoined
23:11:23  <rescrv>if it costs more than that I'd look at optimizing node/v8
23:11:27  <rescrv>that's just me though
23:11:40  <rescrv>and multi-get would be a "good enough" workaround
23:12:43  <kenansulayman>rescrv That's totally true for my three-key example. Imagine, though, gathering a full user profile shred over the database with 40, 50 or much more keys
23:16:07  * paulfryzelquit (Ping timeout: 272 seconds)
23:23:03  * jcrugzzquit (Read error: Connection reset by peer)
23:27:43  * nnnnathannjoined
23:32:17  * eugenewarejoined
23:39:55  * thlorenzquit (Remote host closed the connection)
23:40:41  <kenansulayman>rescrv the mail is out :)
23:43:20  <rvagg>kenansulayman: yeah, a native multiget would certainly be much more efficient cause of the barrier crossing and also the thread switching
23:43:33  <rvagg>but I'm pretty sure it's not going in leveldown, we need to enable native addon plugins
23:47:17  <kenansulayman>rvagg What's the issue of it being in leveldown? The native addon would be leveldb specific, too (non-lmdb, et al. that is)
23:48:06  <kenansulayman>I'd say it should be simply added as a new option in the batch (so that we have put, del, get)
23:48:32  <ogd>kenansulayman: are there any benchmarks proving it actually helps performance?
23:48:46  * eugenewarequit (Remote host closed the connection)
23:48:52  <kenansulayman>ogd [00:12:44] kenansulayman: rescrv That's totally true for my three-key example. Imagine, though, gathering a full user profile shred over the database with 40, 50 or much more keys
23:48:57  <ogd>kenansulayman: is that a no?
23:49:18  <kenansulayman>I can't actually benchmark without any code in place. By logic it should be faster
23:49:37  <ogd>kenansulayman: i think a good place to start would be implementing it and seeing if it is faster in reality
23:50:04  <rvagg>kenansulayman, ogd: that's true, some changes have made it in to levelup/down purely by providing convincing benchmarking
23:50:10  <rvagg>otherwise they would have got nowhere
23:50:37  <rvagg>but still, I *feel* a multiget is outside of the scope of core functionality, it's not a required primitive, it's a nicety
23:51:01  <rvagg>this is an argument that can be had on github though and I'm not the only one to convince
23:52:12  <kenansulayman>I see troubles tho, because I look at it from a production-perspective; it simply is a huge overhead with having hundreds of db.get's to leveldb to gather a shred profile together
23:52:43  <kenansulayman>it's not just for fun that facebook implemented it in Rocks, it's the same issue
23:53:20  <rvagg>ok, so outline your case in a github issue on leveldown and discuss it there, you'll need to win over some people and benchmarks tend to speak volumes
23:54:27  <kenansulayman>Hm, ok. Why do you think that it is nothing that should be in core, after all?
23:57:19  * eugenewarejoined
23:59:41  * eugenewarequit (Remote host closed the connection)