00:03:01  * kenansulaymanjoined
00:03:03  * kenansulaymanquit (Remote host closed the connection)
00:04:41  * mikealquit (Quit: Leaving.)
00:09:00  * ednapiranhajoined
00:13:43  * werlequit (Ping timeout: 264 seconds)
00:14:08  * ryan_ramagejoined
00:15:45  * ryan_ramagequit (Client Quit)
00:25:12  * tmcwquit (Remote host closed the connection)
00:31:40  * jxsonquit (Remote host closed the connection)
00:36:33  * ednapiranhaquit (Remote host closed the connection)
00:41:37  * ryan_ramagejoined
00:43:42  * werlejoined
00:48:32  * timoxleyjoined
00:50:02  * timoxley_joined
00:53:02  * timoxleyquit (Ping timeout: 264 seconds)
00:55:13  * timoxley_quit (Ping timeout: 268 seconds)
00:55:54  * ryan_ramagequit (Quit: ryan_ramage)
01:01:58  * jxsonjoined
01:10:14  * jxsonquit (Ping timeout: 240 seconds)
01:28:58  * thlorenz_zzjoined
01:30:34  * mikealjoined
01:36:38  * mikealquit (Quit: Leaving.)
01:37:44  * ramitosquit (Quit: Textual IRC Client: www.textualapp.com)
01:44:17  <levelbot>[npm] level-agile@0.2.0 <http://npm.im/level-agile>: multilevel client and server with various data transforms for inserting into leveldb (@jcrugzz)
01:47:00  <thlorenz_zz>mbalho: I fixed the bug we talked about in the meantime (until we separate those out) https://github.com/maxogden/level-batcher/commit/df648ec176b0290a5783ac8a154d0a81a7538952
01:47:05  * thlorenz_zzchanged nick to thlorenz
01:47:32  <thlorenz>do you wanna review this or should I just go ahead and publish - output from test is identical as before
01:49:30  * timoxleyjoined
02:00:54  * mikealjoined
02:05:10  * ryan_ramagejoined
02:10:34  * mikealquit (Quit: Leaving.)
02:11:42  * werlequit (Quit: leaving)
02:25:27  * ryan_ramagequit (Quit: ryan_ramage)
02:40:53  * jcrugzzquit (Ping timeout: 250 seconds)
02:41:28  * jmartinsquit (Quit: Konversation terminated!)
02:44:38  <mbalho>thlorenz: i fixed it also, just havent published yet cause i was gonna put it in the new module which isnt done
02:44:55  <thlorenz>ok - well I'm still seeing the problems though
02:45:05  <mbalho>thlorenz: what problems?
02:45:09  <thlorenz>high memory and slow writes
02:45:37  <mbalho>is your code online that uses it?
02:45:52  <mbalho>i wrote 11 million items with level-batcher today
02:46:06  <thlorenz>yes I'm now writing items one by one: https://github.com/thlorenz/valuepack-mine-github/blob/master/lib/store-github-repos.js
02:47:05  <thlorenz>maybe I'm doing something wrong - not sure
02:47:38  <thlorenz>I'm thinking of throttling my requests to gitub in order to avoid creating a backlog of data that needs to be stored
02:48:01  <thlorenz>right now I'm doing 20 concurrent requests, maybe I'll dial that down a bit
02:48:11  <mbalho>ah yes that would do it
02:48:17  <levelbot>[npm] grunt-s3-sync@0.3.0 <http://npm.im/grunt-s3-sync>: A streaming interface for uploading multiple files to S3 (@calvein, @bockit, @hughsk)
02:48:51  <thlorenz>mbalho: I'm also not sure how the auto indexing works out: https://github.com/thlorenz/valuepack-mine-github/blob/master/lib/store-github-repos.js#L40
02:49:20  <mbalho>thlorenz: you arent doing anything with the return value of chunke.rwrite, it should be true if it is resumed and false if it is paused
02:49:22  <mbalho>chunker.write*
02:49:24  <thlorenz>since that will add a put into the batch automagically - should be fairly small though and have not much impact
02:49:41  <mbalho>thlorenz: e.g. if you do httprequest.pipe(chunker).pipe(db) then node will handle tcp backpressure
02:50:06  <mbalho>thlorenz: but in your use case there is no backpressure since you are doing a buncha http requests yourself but theres no throttling
02:50:27  <thlorenz>ah - I need to handle backpressure myself? wasn't aware of that
02:51:03  <mbalho>nowhere in your code does chunker hook up to the upstream http requests, so yes
02:51:13  <mbalho>its complicated in your use case cause you are writing N http requests to 1 chunker
02:51:26  <mbalho>in a situation where it is 1:1 and not N:1 its a lot more straightforward cause you can just pipe
02:52:37  <thlorenz>mbalho: thanks that makes more sense, so I guess when write returns false I'm gonna pause and sit around for a bit before trying to write again?
02:53:28  <mbalho>thlorenz: something like that, not sure exactly what the best solution is for that scenario
02:53:59  <thlorenz>mbalho: ideally it'd be streams all the way - maybe I could refactor this into that somehow
02:54:39  <thlorenz>I mean the driver of all of it are logins that I can stream out of level - I could just map stream those into requests and continue that pipeline
02:54:54  <thlorenz>it may be easier than manually handling backpressure
02:59:18  <thlorenz>mbalho: actually maybe not that easy at all since I'm doing so many writes per request
02:59:42  <thlorenz>will have to come up with some mechanism to stop github requests while pausing due to backpressure
03:01:25  <mbalho>thlorenz: just pipe the http request to the batcher
03:02:04  <thlorenz>you mean res.pipe(..) right?
03:02:18  <levelbot>[npm] grunt-s3-sync@0.3.1 <http://npm.im/grunt-s3-sync>: A streaming interface for uploading multiple files to S3 (@calvein, @bockit, @hughsk)
03:02:22  <mbalho>thlorenz: yea
03:02:23  <thlorenz>but I have multiple writes to different sublevels for each response
03:02:38  <mbalho>ah
03:05:18  <thlorenz>hey I could use dominictarr's from and batch up all the puts and then generate a stream from that which handles backpressure
03:06:02  <thlorenz>mbalho: I'd just need to find a way to communicate to the module pushing the requests that I'm backed up
03:08:01  * ryan_ramagejoined
03:08:09  * ryan_ramagequit (Client Quit)
03:24:35  <thlorenz>mbalho: got a solution in the works, thanks for your help - hopefully can finish it tomorrow morning - will post the code to get your feedback
03:28:02  * thlorenzquit (Remote host closed the connection)
03:49:08  * jcrugzzjoined
03:56:30  * disordinaryquit (Ping timeout: 264 seconds)
04:01:44  * julianduquequit (Quit: leaving)
04:07:08  * esundahljoined
04:25:17  * thlorenzjoined
04:55:41  * dguttmanquit (Quit: dguttman)
04:58:32  * esundahlquit (Remote host closed the connection)
04:59:06  * esundahljoined
05:00:58  * esundahl_joined
05:02:46  * thlorenzquit (Remote host closed the connection)
05:03:42  * esundahlquit (Ping timeout: 264 seconds)
05:06:08  * dguttmanjoined
05:08:18  * jxsonjoined
05:12:30  * jxsonquit (Ping timeout: 245 seconds)
05:45:51  * wolfeidauquit (Ping timeout: 260 seconds)
05:46:44  * dguttmanquit (Quit: dguttman)
05:46:45  * wolfeidaujoined
06:40:00  * Acconutjoined
06:41:19  * st_lukejoined
06:56:36  * Acconutquit (Quit: Acconut)
06:57:39  * esundahl_quit (Remote host closed the connection)
07:15:31  * st_lukequit (Remote host closed the connection)
07:28:44  * esundahljoined
07:31:42  * dominictarrjoined
07:37:15  * esundahlquit (Ping timeout: 260 seconds)
08:03:51  * jcrugzzquit (Ping timeout: 260 seconds)
08:22:36  * fb55joined
08:27:49  * dominictarrquit (Quit: dominictarr)
08:50:19  * fb55quit (Remote host closed the connection)
08:54:35  * dominictarrjoined
08:54:53  * fb55joined
09:01:44  * fb55quit (Remote host closed the connection)
09:38:21  * fb55joined
10:41:06  * wolfeidauquit (Ping timeout: 240 seconds)
10:41:46  * timoxleyquit (Remote host closed the connection)
10:48:06  * rudquit (Quit: rud)
10:50:21  * dominictarrquit (Quit: dominictarr)
10:59:50  * fb55quit (Remote host closed the connection)
11:39:02  * kenansulaymanjoined
12:21:31  * rudjoined
12:21:37  * rudquit (Changing host)
12:21:38  * rudjoined
12:30:03  * esundahljoined
12:34:31  * esundahlquit (Ping timeout: 268 seconds)
12:36:22  * jcrugzzjoined
12:56:57  * dominictarrjoined
13:04:17  * fb55joined
13:30:36  <substack>https://github.com/substack/level-track
13:30:46  <levelbot>[npm] level-track@0.0.0 <http://npm.im/level-track>: receive updates from a set of keys and ranges in leveldb (@substack)
13:35:59  * ryan_ramagejoined
13:50:31  * ryan_ramagequit (Quit: ryan_ramage)
13:51:52  * thlorenzjoined
14:08:39  * ryan_ramagejoined
14:10:55  * tmcwjoined
14:22:11  * jjmalinajoined
14:26:00  * ryan_ramagequit (Quit: ryan_ramage)
14:26:51  * fb55quit (Remote host closed the connection)
14:28:38  * fb55joined
14:31:02  * jcrugzzquit (Ping timeout: 240 seconds)
14:44:36  * esundahljoined
14:50:54  * dguttmanjoined
14:51:44  * ramitosjoined
15:05:10  * jerrysvjoined
15:12:14  * dguttmanquit (Read error: Connection reset by peer)
15:12:40  * dguttmanjoined
15:14:20  * fallsemojoined
15:17:14  * rudquit (Quit: rud)
15:26:25  <thlorenz>rvagg: does levelup apply backpressure?
15:27:02  <thlorenz>I looked through the writestream code and write doesn't return false unless I gave a buffer length or it isn't writable
15:28:10  <thlorenz>but writable only ever becomes false when there was an error or the stream ended - or am I missing something?
15:40:39  * kenansulaymanquit (Quit: โ‰ˆโ™กโ‰ˆ)
15:51:27  * rudjoined
15:51:38  * rudquit (Changing host)
15:51:38  * rudjoined
16:37:38  * timoxleyjoined
16:55:41  * ednapiranhajoined
16:56:49  * ednapiranhaquit (Remote host closed the connection)
16:57:03  * ednapiranhajoined
17:04:35  * ryan_ramagejoined
17:05:43  * wolfeidaujoined
17:14:08  * jcrugzzjoined
17:19:00  * rudquit (Quit: rud)
17:21:04  * dguttmanquit (Quit: dguttman)
17:38:09  * timoxleyquit (Remote host closed the connection)
17:38:22  * timoxleyjoined
17:43:09  * timoxleyquit (Remote host closed the connection)
17:46:12  * ednapiranhaquit (Remote host closed the connection)
17:49:18  * jxsonjoined
18:09:57  * ednapiranhajoined
18:42:13  * tmcwquit (Remote host closed the connection)
18:42:46  * tmcwjoined
18:47:06  * tmcwquit (Ping timeout: 245 seconds)
18:53:13  * tmcwjoined
18:57:02  * ednapiranhaquit (Remote host closed the connection)
18:58:11  * jxsonquit (Remote host closed the connection)
18:58:19  * jxsonjoined
19:01:33  * fb55quit (Remote host closed the connection)
19:02:01  * fb55joined
19:06:35  * fb55_joined
19:06:50  * fb55quit (Ping timeout: 264 seconds)
19:14:17  * Acconutjoined
19:16:36  * Acconutquit (Client Quit)
19:24:02  * ryan_ramagequit (Quit: ryan_ramage)
19:25:35  * ryan_ramagejoined
19:26:46  * dominictarrquit (Quit: dominictarr)
19:28:10  * ednapiranhajoined
19:33:03  * Acconutjoined
19:36:22  * ednapiranhaquit (Ping timeout: 256 seconds)
19:41:31  * jxsonquit (Remote host closed the connection)
19:42:19  * jxsonjoined
19:46:46  * Acconutquit (Quit: Acconut)
19:51:21  * esundahlquit (Ping timeout: 276 seconds)
20:08:20  * esundahljoined
20:11:26  * Acconutjoined
20:13:38  * julianduquejoined
20:15:03  * jxsonquit (Remote host closed the connection)
20:19:52  * Acconutquit (Quit: Acconut)
20:21:27  * ednapiranhajoined
20:39:16  <levelbot>[npm] fash@2.1.1 <http://npm.im/fash>: A consistent hashing library for node (@yunong)
20:46:27  * disordinaryjoined
21:00:18  * Acconutjoined
21:00:53  * Acconutquit (Client Quit)
21:13:03  * Acconutjoined
21:15:22  * jxsonjoined
21:16:35  * Acconutquit (Client Quit)
21:19:36  * jxsonquit (Ping timeout: 245 seconds)
21:29:40  * Acconutjoined
21:36:48  * fb55_quit (Remote host closed the connection)
21:36:50  * jcrugzzquit (Ping timeout: 264 seconds)
21:37:14  * jxson_joined
21:47:50  * Acconutquit (Quit: Acconut)
21:56:11  * disordinaryquit (Quit: Konversation terminated!)
22:00:51  * Acconutjoined
22:00:52  * Acconutquit (Client Quit)
22:04:45  * tmcwquit (Remote host closed the connection)
22:07:05  * thlorenzquit (Remote host closed the connection)
22:07:12  * fb55joined
22:23:37  * fb55quit (Remote host closed the connection)
22:24:20  * esundahlquit (Remote host closed the connection)
22:24:47  * esundahljoined
22:24:51  <Raynos>im trying to update as get() merge() put()
22:24:59  <Raynos>and realised I just introduced data loss and race conditions
22:27:29  * dominictarrjoined
22:27:34  * esundahl_joined
22:29:12  * esundahlquit (Ping timeout: 256 seconds)
22:37:39  * thlorenzjoined
22:40:25  * ryan_ramagequit (Quit: ryan_ramage)
22:47:30  * thlorenzquit (Ping timeout: 276 seconds)
22:47:50  * jjmalinaquit (Quit: Leaving.)
22:52:31  <rescrv>Raynos: hold a lock across all three
22:52:42  * jjmalinajoined
22:52:45  * jjmalinaquit (Max SendQ exceeded)
22:52:58  * jjmalinajoined
22:53:40  * rudjoined
22:54:04  <Raynos>rescrv: that makes sense
22:55:42  <rescrv>if you're in a multithreaded app, have an array of locks and hash the key into the array.
23:01:21  * fallsemoquit (Quit: Leaving.)
23:05:34  * jerrysvquit (Read error: Connection reset by peer)
23:07:32  * fb55joined
23:11:58  * fb55quit (Ping timeout: 246 seconds)
23:14:56  * dominictarrquit (Quit: dominictarr)
23:17:51  * dominictarrjoined
23:21:34  * jjmalinaquit (Quit: Leaving.)
23:28:52  * esundahl_quit (Remote host closed the connection)
23:29:18  * jjmalinajoined
23:29:19  * esundahljoined
23:30:19  * thlorenzjoined
23:30:39  * esundahl_joined
23:34:18  * esundahlquit (Ping timeout: 276 seconds)
23:35:36  * jjmalinaquit (Quit: Leaving.)
23:37:02  * ednapiranhaquit (Remote host closed the connection)
23:40:36  * thlorenzquit (Remote host closed the connection)
23:42:45  * thlorenzjoined
23:43:58  * dominictarrquit (Quit: dominictarr)
23:47:57  * thlorenzquit (Ping timeout: 276 seconds)
23:57:04  * esundahl_quit (Remote host closed the connection)
23:57:30  * esundahljoined