00:05:41  * thlorenz_quit (Ping timeout: 265 seconds)
00:22:02  * DTrejojoined
00:54:13  * jcrugzzjoined
00:58:20  * jcrugzzquit (Ping timeout: 245 seconds)
01:07:04  * kenansulaymanquit (Ping timeout: 264 seconds)
01:08:15  * kenansulaymanjoined
01:11:32  * DTrejoquit (Remote host closed the connection)
01:11:58  * DTrejojoined
01:16:21  * DTrejoquit (Ping timeout: 248 seconds)
02:00:16  * ramitosquit (Quit: Textual IRC Client: www.textualapp.com)
02:02:35  * thlorenzjoined
02:07:01  * thlorenzquit (Ping timeout: 248 seconds)
02:27:58  * ednapiranhajoined
02:46:42  * mikealjoined
03:03:15  * thlorenzjoined
03:07:20  * thlorenzquit (Ping timeout: 241 seconds)
03:19:00  * jcrugzzjoined
03:23:49  * ehdquit (Ping timeout: 248 seconds)
03:24:21  * book`quit (Ping timeout: 248 seconds)
03:25:38  * ehdjoined
03:27:10  * book`joined
03:34:39  * kenansulaymanquit (Quit: ≈ and thus my mac took a subtle yet profound nap ≈)
03:48:05  * dguttmanquit (Quit: dguttman)
04:55:12  * brycebariljoined
04:58:15  * ednapiranhaquit (Remote host closed the connection)
05:04:27  * thlorenzjoined
05:07:07  * mikealquit (Quit: Leaving.)
05:09:31  * thlorenzquit (Ping timeout: 272 seconds)
05:12:14  * DTrejojoined
05:14:56  * mikealjoined
05:16:45  * DTrejoquit (Ping timeout: 272 seconds)
05:39:55  * DTrejojoined
05:41:46  * eugenewarechanged nick to zz_eugeneware
05:44:26  * zz_eugenewarechanged nick to eugeneware
05:49:33  * DTrejoquit (Remote host closed the connection)
06:05:11  * thlorenzjoined
06:10:07  * thlorenzquit (Ping timeout: 265 seconds)
06:18:45  * niftylettucepart
07:05:49  * thlorenzjoined
07:10:29  * thlorenzquit (Ping timeout: 272 seconds)
07:36:01  * eugenewarechanged nick to zz_eugeneware
08:22:50  * frankblizzardjoined
09:07:04  * thlorenzjoined
09:11:52  * thlorenzquit (Ping timeout: 265 seconds)
09:25:23  * zz_eugenewarechanged nick to eugeneware
09:29:58  * insertcoffeejoined
10:05:01  <levelbot>[npm] level-sum@0.0.0 <http://npm.im/level-sum>: Calculate sums in a LevelDB and get live updates (@juliangruber)
10:05:47  * insertcoffeequit (Ping timeout: 248 seconds)
10:07:39  * thlorenzjoined
10:12:15  * thlorenzquit (Ping timeout: 272 seconds)
10:18:21  * insertcoffeejoined
10:28:08  * rudquit (Quit: rud)
11:53:23  * eugenewarechanged nick to zz_eugeneware
12:01:18  * twilson63joined
12:08:53  * thlorenzjoined
12:13:20  * thlorenzquit (Ping timeout: 245 seconds)
12:14:30  * kenansulaymanjoined
12:25:09  * twilson63quit (Ping timeout: 248 seconds)
12:43:45  * rudjoined
12:43:46  * rudquit (Changing host)
12:43:46  * rudjoined
13:02:10  * thlorenzjoined
13:11:58  * thlorenzquit (Remote host closed the connection)
13:13:14  * thlorenzjoined
13:19:52  * thlorenzquit (Remote host closed the connection)
13:20:25  * thlorenzjoined
13:24:38  * thlorenzquit (Ping timeout: 240 seconds)
13:34:55  * tmcwjoined
13:38:09  * thlorenzjoined
13:51:18  * thlorenz_joined
13:52:41  * brycebar1ljoined
13:57:26  * rud_joined
13:57:36  * thlorenzquit (Remote host closed the connection)
13:59:55  * rudquit (Ping timeout: 248 seconds)
13:59:56  * rud_changed nick to rud
14:03:34  * dguttmanjoined
14:13:49  * jcrugzzquit (Ping timeout: 272 seconds)
14:19:26  * jjmalinajoined
14:19:30  * jjmalinaquit (Max SendQ exceeded)
14:20:52  * jjmalinajoined
14:48:59  * tarruda_joined
14:50:59  * Acconutjoined
14:51:06  * brycebar1lquit (Quit: leaving)
14:53:07  * dominictarrjoined
14:58:25  * tarruda_quit (Quit: WeeChat 0.3.7)
14:58:48  * tarruda_joined
15:06:45  * dominictarrquit (Quit: dominictarr)
15:09:57  * Acconutquit (Quit: Acconut)
15:16:02  * ednapiranhajoined
15:17:14  * frankblizzardquit (Remote host closed the connection)
15:38:04  * Acconutjoined
15:38:12  * Acconutquit (Client Quit)
15:48:55  * mikealquit (Quit: Leaving.)
15:54:40  * esundahljoined
15:57:35  * Acconutjoined
16:05:46  * Acconutquit (Quit: Acconut)
16:11:59  * esundahlquit (Remote host closed the connection)
16:12:25  * esundahljoined
16:13:56  * tarruda_quit (Quit: WeeChat 0.3.7)
16:16:40  * esundahlquit (Ping timeout: 245 seconds)
16:18:23  * esundahljoined
16:21:27  * Acconutjoined
16:23:10  * mikealjoined
16:23:23  * Acconutquit (Client Quit)
16:28:23  * julianduquejoined
16:36:04  * tarruda1joined
16:37:04  * tarruda1quit (Client Quit)
16:41:27  * tarruda_joined
16:45:14  * Acconutjoined
16:52:11  * scttnlsnjoined
16:52:12  * Acconutquit (Quit: Acconut)
17:00:03  * Acconutjoined
17:00:07  * Acconutquit (Client Quit)
17:05:01  * dguttmanquit (Quit: dguttman)
17:07:04  * dominictarrjoined
17:08:11  * pgtejoined
17:09:02  * tarruda_quit (Quit: WeeChat 0.4.2)
17:09:25  * tarruda_joined
17:15:28  * esundahlquit (Remote host closed the connection)
17:15:55  * esundahljoined
17:20:08  * esundahlquit (Ping timeout: 240 seconds)
17:20:11  * tarruda_quit (Quit: WeeChat 0.4.2)
17:20:40  * tarruda_joined
17:22:10  * esundahljoined
17:34:00  * insertcoffeequit (Ping timeout: 256 seconds)
17:36:13  * tarruda_quit (Quit: WeeChat 0.4.2)
17:36:20  * dguttmanjoined
17:45:03  * Acconutjoined
17:45:22  * tarruda_joined
17:46:25  * Acconutquit (Client Quit)
17:48:36  * pgtequit (Remote host closed the connection)
17:49:12  * pgtejoined
17:50:38  * tarruda_quit (Quit: WeeChat 0.4.2)
17:51:13  * tarrudajoined
17:52:41  * pgtequit (Read error: Operation timed out)
18:01:24  * dominictarrquit (Quit: dominictarr)
18:15:09  * tarrudaquit (Quit: WeeChat 0.4.2)
18:16:19  * tarrudajoined
18:23:39  * tarrudaquit (Quit: WeeChat 0.4.2)
18:33:36  * esundahlquit (Remote host closed the connection)
18:34:02  * esundahljoined
18:38:31  * esundahlquit (Ping timeout: 245 seconds)
18:39:41  * esundahljoined
18:40:36  * timoxleyquit (Remote host closed the connection)
18:41:10  * timoxleyjoined
18:41:13  * timoxleyquit (Read error: Connection reset by peer)
18:41:45  * timoxleyjoined
18:44:20  * jcrugzzjoined
18:49:37  * julianduquequit (Quit: leaving)
18:59:19  * scttnlsnquit (Remote host closed the connection)
19:08:51  * Acconutjoined
19:09:00  * Acconutquit (Client Quit)
19:13:25  * scttnlsnjoined
19:13:29  * scttnlsnquit (Remote host closed the connection)
19:15:28  * kenansulaymanquit (Ping timeout: 264 seconds)
19:17:27  * kenansulaymanjoined
19:22:59  * jjmalinaquit (Quit: Leaving.)
19:23:20  * jjmalinajoined
19:23:58  * brycebaril1joined
19:25:05  * twilson63joined
19:25:42  * brycebarilquit (Ping timeout: 272 seconds)
19:26:44  * brycebaril1changed nick to brycebaril
19:27:10  * fallsemojoined
19:31:02  * jmartinsjoined
19:32:36  * jmartinsquit (Remote host closed the connection)
19:36:30  * julianduquejoined
19:37:39  * jxsonjoined
19:43:20  * esundahlquit (Remote host closed the connection)
19:43:47  * esundahljoined
19:46:02  * jmartinsjoined
19:48:06  * esundahlquit (Ping timeout: 245 seconds)
19:54:32  * julianduquequit (Ping timeout: 246 seconds)
20:00:56  * jcrugzzquit (Ping timeout: 260 seconds)
20:15:22  * pgtejoined
20:16:26  * pgtequit (Remote host closed the connection)
20:17:01  * pgtejoined
20:21:07  * pgtequit (Ping timeout: 246 seconds)
20:29:11  * tarrudajoined
20:31:41  * tarrudaquit (Client Quit)
20:38:06  * jmartinsquit (Ping timeout: 245 seconds)
20:45:19  * jxsonquit (Remote host closed the connection)
20:49:27  * pgtejoined
20:50:21  * jxsonjoined
20:51:57  * pgtequit (Read error: Connection reset by peer)
20:52:21  * pgtejoined
21:08:33  * jxsonquit (Remote host closed the connection)
21:09:31  * thlorenz_quit (Remote host closed the connection)
21:12:05  * jxsonjoined
21:21:01  * twilson63quit (Ping timeout: 268 seconds)
21:30:45  * tarrudajoined
21:31:45  * twilson63joined
21:35:12  * tarrudaquit (Client Quit)
21:35:26  * tarrudajoined
21:39:43  * thlorenzjoined
21:45:18  * joshcarterjoined
21:45:20  * jcrugzzjoined
21:45:24  * pgtequit (Remote host closed the connection)
21:48:26  * thlorenzquit (Ping timeout: 256 seconds)
21:56:13  * tarrudaquit (Quit: WeeChat 0.4.2)
21:56:58  * thlorenzjoined
21:57:33  * thlorenzquit (Remote host closed the connection)
22:02:07  * tarrudajoined
22:02:46  * dominictarrjoined
22:05:20  * esundahljoined
22:07:28  * esundahlquit (Remote host closed the connection)
22:08:54  * esundahljoined
22:13:26  * twilson63quit (Ping timeout: 268 seconds)
22:16:59  * ralphtheninjajoined
22:19:32  * ralphtheninjaquit (Client Quit)
22:19:57  * joshcarterquit (Quit: joshcarter)
22:22:22  * tmcwquit (Remote host closed the connection)
22:23:37  * pgtejoined
22:28:41  * thlorenzjoined
22:32:02  * dominictarrquit (Quit: dominictarr)
22:34:05  * tarrudaquit (Quit: WeeChat 0.4.2)
22:34:26  * tarrudajoined
22:36:29  * tarrudaquit (Client Quit)
22:36:48  * tarrudajoined
22:37:00  * thlorenzquit (Ping timeout: 252 seconds)
22:41:14  * twilson63joined
22:44:27  * tarrudaquit (Quit: WeeChat 0.4.2)
22:46:07  * tarrudajoined
22:47:28  * tarrudaquit (Client Quit)
22:48:35  * twilson63quit (Ping timeout: 268 seconds)
23:00:31  * ednapiranhaquit (Remote host closed the connection)
23:03:47  * thlorenzjoined
23:08:04  * thlorenzquit (Ping timeout: 246 seconds)
23:18:06  * jjmalinaquit (Quit: Leaving.)
23:18:36  * thlorenzjoined
23:19:16  * pgtequit (Remote host closed the connection)
23:19:40  * kenansulaymanquit (Ping timeout: 264 seconds)
23:19:43  * kenansul_joined
23:21:23  * thlorenz_joined
23:22:13  * tmcwjoined
23:23:17  * thlorenzquit (Ping timeout: 248 seconds)
23:26:01  * jcrugzzquit (Ping timeout: 245 seconds)
23:26:39  * esundahlquit (Remote host closed the connection)
23:30:00  * briancjoined
23:30:47  <brianc>hey rvagg I used lmdb.js a bit
23:31:08  <brianc>got a sec to chat about it?
23:31:18  <brianc>or anyone else who understands lmdb
23:32:24  <brianc>I have another question related to leveldb - it seems when I'm using createReadStream() and using that read stream to iterate over 1,000,000 items, the memory usage grows a lot
23:32:31  <brianc>anyone else experience this? is it expected?
23:32:58  <rvagg>brianc: not expected, does it go down afterwards?
23:33:24  <brianc>I'm not sure - still running. :) I'll know in about 10 min
23:33:50  <rvagg>ok, that's odd cause it should be only dealing with a single item at a time, i.e. proper streaming
23:34:30  <brianc>one thing I had to do was set the highWaterMark on the stream
23:34:37  <brianc>i'm piping it to a slow consumer
23:34:50  <brianc>if I didn't do that it just kept on reading
23:35:11  <rvagg>ah, right yes; are you doing much writing to the store while this is happening?
23:35:21  <brianc>not writing to leveldb at all
23:35:25  <brianc>only a single read stream
23:35:26  <rvagg>leveldb needs to keep a snapshot around but that shouldn't be impacting on memory usage at all
23:35:40  <brianc>i'm piping it into a copyIn command to postgres
23:36:06  <brianc>it seems to have capped out at about 40% memory usage
23:36:09  <brianc>that's really high though
23:36:38  <brianc>i have 16gigs on the box
23:36:50  * tmcwquit (Remote host closed the connection)
23:36:53  <rvagg>yeah, that is high
23:37:03  <brianc>strangeness
23:37:09  <rvagg>it's possible that v8 gc is involved here, there's a lot of object creation and destruction going on
23:37:27  <brianc>i noticed when using lmdb it seemed to grow unbounded when I was doing inserts
23:37:42  <rvagg>yeah, well lmdb is all about memory, for mmap
23:37:49  <brianc>my workflow is basically to export a postgres table into a local leveldb instance and do a bunch of streaming processing on it and then put it back into postgres
23:38:22  <rvagg>mm, that *should* be a good usecase
23:38:42  <brianc>but i put together a readableStream() from postgres and a writeable stream today so i'm thinking if i get some time make a postgres-down backend for levelup. :p
23:38:54  <brianc>yeah it's worked fantastically well
23:39:11  <brianc>lmdb seemed a lot faster but I didn't wanna figure out what would happen if i had more data than ram
23:39:14  <rvagg>yes, postgresdown would be great! mysqldown may have some pointers to offer, perhaps
23:39:22  <brianc>yeah I'll check that out for sure
23:39:51  * esundahljoined
23:40:21  <brianc>yeah it's just sitting at 40% memory doing leveldb.createReadStream().pipe(transform).pipe(postgresWriteStream)
23:40:52  <rvagg>if you have time, would you be able to narrow it down to a test case? something that doesn't involve postgres, perhaps just a slow-stream consumer that goes nowhere so it can be run stand-alone.
23:40:58  <brianc>w/ lmdb can you have multiple writers in many processes or only multiple readers?
23:41:04  <brianc>yeah i'll do that right when the script finishes
23:41:05  <rvagg>that way we can run it through mdb to check out what objects are floating around in there
23:41:26  <rvagg>kenansul_: ping
23:41:33  <kenansul_>rvagg yo!
23:41:34  <rvagg>kenansul_ is using lmdb a bit I believe
23:41:39  <brianc>sweeeet
23:41:44  <kenansul_>yup lmdb in production
23:41:49  <rvagg>kenansul_: brianc wants to talk lmdb and multiproc, are you doing that?
23:41:51  <brianc>kenansul_: mind if i pick your brain?
23:42:14  <kenansul_>multiproc = clustering?
23:42:48  <kenansul_>brianc sure
23:43:02  <kenansul_>rvagg ya we're doing that
23:43:33  <brianc>kenansul_: I have 1.5 million rows in a postgres table
23:43:39  <brianc>wanted to export it local to work on it faster
23:43:48  <brianc>thought leveldb would be good fit but saw lmdb.js and wanted to try that
23:44:04  <brianc>so i did 1.5 million inserts in 1000 item bulk puts
23:44:12  <brianc>the memory grew seemingly unbounded the entire time
23:44:24  <kenansul_>Yes it does
23:44:28  <kenansul_>Since it's memory-mapped
23:44:37  <brianc>being as I'm a noob with lmdb I was like "hmm...that scares me, switching to level-down backend"
23:44:49  <brianc>what happens if you put 20gb in lmdb but don't have 20gb of memory?
23:45:09  <kenansul_>AFAIK it swaps to the disk
23:45:15  * kenansul_changed nick to kenansulayman
23:45:28  <brianc>cool - so it'll slow down but not explode like redis
23:45:36  <brianc>i'll switch back to lmdb and try it out some more
23:45:40  <kenansulayman>It won't really slowdown
23:45:43  <brianc>ah okay
23:45:44  <brycebaril>rvagg brianc I am leaving right now but I have done a lot of work with readStream and have some insight to share, but I'll have to share when I come back
23:45:54  <brianc>okay cool thanks. :)
23:46:02  <kenansulayman>brycebaril cyy
23:46:16  <kenansulayman>cya*
23:46:22  <brianc>since i'm doing non-mission critical stuff I wanna try out all the cutting edge goodness
23:46:22  * tmcwjoined
23:46:28  <kenansulayman>uhm
23:46:30  <brycebaril>essentially yes it will use a lot of memory because objectMode & stuff. ok bbl
23:46:40  * esundahlquit (Remote host closed the connection)
23:46:45  <brianc>i mean it's mission critical but i'm using levelup as a caching layer
23:46:48  <kenansulayman>How important is performance for you?
23:46:51  <brianc>so if it gets corrupted i only lose a day's work
23:47:06  <brianc>it ends up making a difference
23:47:15  <brianc>I'm dealing with 50 million records
23:47:23  <kenansulayman>What is your primary field of application? Web?
23:47:43  <brianc>this is kinda like a data collection + ETL backend thing
23:47:52  * esundahljoined
23:47:56  <kenansulayman>thing is
23:48:16  * tmcwquit (Remote host closed the connection)
23:48:39  <kenansulayman>usually the difference between hyper-leveldb and lmdb isn't too obvious when your bandwidth is bad
23:49:14  <brianc>hmm
23:49:18  <kenansulayman>That is, are you using your database more offline then networked?
23:49:23  <brianc>i never heard of hyper-leveldb
23:49:34  <kenansulayman>checkout hyperlevel oder rvagg's level-hyper
23:49:35  <brianc>i'm using levelup right now as basically a sane file system
23:49:48  <brianc>everything is local to the same box and i'm looping through sets checking for outliers and stuff
23:49:54  <kenansulayman>it's a crazily polished version of rescrv
23:49:58  <kenansulayman>*by
23:50:28  <kenansulayman>Okay then I'd really recommend lmdb since it's uber fast
23:50:32  <brianc>can it support multi-process access?
23:50:36  <kenansulayman>No
23:50:40  <brianc>bummers
23:50:43  <kenansulayman>That's why we use lmdb ;)
23:50:48  <brianc>yeah
23:50:59  <brianc>i feel like the multi-process thing jives well w/ node's cluster module stuff
23:51:05  <brianc>i know peeps have different opinions there
23:51:22  <kenansulayman>We usually spawn 6 - 8 instances per node
23:51:23  * fallsemoquit (Quit: Leaving.)
23:51:32  <brianc>so can you have multiple writers from separate processes?
23:51:37  <kenansulayman>yup
23:51:42  <brianc>sweet.
23:51:52  <brianc>i guess i'll just ignore the fact that I'm using 100% of memory
23:51:57  <kenansulayman>hah
23:51:58  <brianc>i guess that's why it's there in the first place, to be used.
23:52:04  <kenansulayman>RAM isn't too expensive, right?
23:52:13  <brianc>haha nah
23:52:26  <kenansulayman>We're deploying some crazy 128gb of ram per node
23:52:31  <kenansulayman>so that's not much of an issue
23:53:19  <kenansulayman>What kind of data are you analyzing btw?
23:54:30  <brianc>dang
23:54:33  <brianc>financial data
23:54:37  <kenansulayman>woah
23:54:56  <kenansulayman>I saw people over at NASDAQ deploying Kyoto-Cabinet for financial data
23:55:13  <kenansulayman>which is pretty binary tho
23:55:24  <brianc>ah that's cool
23:55:46  <kenansulayman>but afaik lmdb > kyoto
23:55:51  <kenansulayman>and kyoto is a pain to install^
23:55:52  <brianc>rvagg: i just tested piping the database through the transform stream and out to stdout and it only used 2% of memory so I think there's something up in the postgres part
23:56:29  <rvagg>brianc: did it slow it down further tho?
23:56:38  <brianc>nah it stayed at about 40% memory
23:56:43  <brianc>when doing postgres import
23:56:47  <brianc>and 2% when piping to stdout
23:56:51  <brianc>lemme try one more time
23:56:56  <kenansulayman>rvagg I'll +1 as comment on your default option @ lmdb
23:57:40  <brianc>oh yeah lemme pipe it to something slow
23:57:50  <rvagg>kenansulayman: ah yes, I should probably merge that in eh?
23:58:05  <kenansulayman>yeah please. that additional options sucks
23:59:31  <levelbot>[npm] lmdb@0.1.2 <http://npm.im/lmdb>: A Low-level, LevelDOWN-compatible, Node.js LMDB binding (@rvagg)
23:59:33  <rvagg>^^ ✔
23:59:45  <kenansulayman>rvagg ty. lemme update the clusters