00:03:55  <rvagg>you know.... sticking lru-cache in front of levelup wouldn't be such a bad idea, saves even the JS->C++ round-trip to the leveldb cache
00:04:04  <rvagg>there's a module idea, perhaps I should have a crack at it!
00:07:42  <chilts>that's a great idea
00:20:37  * levelbotquit (Remote host closed the connection)
00:20:58  * levelbotjoined
00:21:36  <rvagg>would certainly make benchmarks look pretty good
00:22:11  <dominictarr>and everyone knows, that is what benchmarks are for
00:22:26  <dominictarr>encouraging playful oneupsmanship
01:07:21  * dominictarrquit (Quit: dominictarr)
01:20:31  * dominictarrjoined
01:48:47  * dominictarrquit (Ping timeout: 255 seconds)
01:57:40  * glacialheartjoined
01:59:55  <glacialheart>for some reason i don't understand, i can only do one db.put when i have a list of 10 of them in a row
02:00:08  <glacialheart>i'm reading in a file with 10 lines in it
02:00:32  <glacialheart>with readline
02:01:10  <glacialheart>i can console.log() each value, but the db.put() only saves data the first it's called, the other 9 do not work
02:01:21  <glacialheart>*first line
02:02:01  * ChrisPartridgejoined
02:02:38  <glacialheart>any ideas?
02:02:46  <glacialheart>i'm stumped
02:04:49  <glacialheart>http://pastebin.com/Kg3hXAcS
02:08:54  * colinwrenjoined
02:10:18  <rvagg>glacialheart: and you're sure those 'line' events are getting through as you expect?
02:10:27  <rvagg>oh, you have a console.log
02:10:58  <glacialheart>yes, when i console.log i get 10 lines just as expected
02:10:59  <rvagg>glacialheart: perhaps you should not start processing until your 'db' instance emits a 'ready' event
02:11:12  <rvagg>otherwise it's getting queued until the database is actually open
02:11:18  <glacialheart>ah
02:11:45  <rvagg>and perhaps you're closing too early, before they actually get anywhere
02:12:26  <rvagg>this is something I was hoping to fix for 0.9, but I'm not sure I'll find the time--queued writes should be completed gracefully before you close(), if that close() happens before the db is actually open
02:13:20  <glacialheart>so, should i have my rd.on('line' inside of an rd.on('ready'?
02:17:16  <glacialheart>how do i keep the db from closing before the last put saves to it?
02:20:57  <rvagg>glacialheart: db.on('ready', function () { GO })
02:21:15  <rvagg>it's the levelup instance you need to care about the 'ready' event from
02:21:36  <rvagg>or, use the nested syntax instead: require('levelup')('/path/to.db', function (err, db) { GO })
02:22:06  <glacialheart>thanks, i will try that
02:41:34  * st_lukejoined
03:37:57  * st_lukequit (Remote host closed the connection)
04:01:48  * timoxleyjoined
04:21:17  * timoxleyquit (Quit: Computer has gone to sleep.)
05:33:10  * glacialheartquit (Quit: Page closed)
05:49:44  * st_lukejoined
06:28:53  * Pwnnaquit (Read error: Operation timed out)
07:03:23  <levelbot>[npm] multilevel@4.0.1 <http://npm.im/multilevel>: A levelUp instance exposed over a streaming interface (@juliangruber)
07:06:54  * colinwrenquit (Quit: colinwren)
07:08:36  <juliangruber>rvagg: cool, I'll try with a higher cache size
07:11:25  <juliangruber>rvagg: doesn't seem to make a difference. I have 32 * 1024 * 1024 as cache size now
07:12:15  <juliangruber>does levelup/db work with the cluster module?
07:12:30  <juliangruber>I imagine the master owning the db and each fork hosting a multilevel server
07:57:41  <rvagg>yeah, that'd be the only way to do it since they really are separate processes
08:20:26  * dominictarrjoined
08:44:36  * ChrisPartridgequit (Ping timeout: 264 seconds)
08:52:51  * levelbotquit (Remote host closed the connection)
08:53:08  * levelbotjoined
09:01:58  * dominictarrquit (Quit: dominictarr)
09:45:25  <juliangruber>rvagg: but you haven't tried that out?
09:45:39  <rvagg>nope
09:46:04  <juliangruber>ok, I will
09:53:38  <juliangruber>although you can't share variables
09:53:58  <juliangruber>so I'll try ipc between master and clients, only the master having a reference to the db
10:18:33  <juliangruber>4k ops/s, ha :D
10:19:24  <juliangruber>i didn't know ipc was so slow
10:19:51  <rvagg>yeah, it's basically like a network stream that has to stringify everything
10:22:25  <juliangruber>ok
10:24:23  <juliangruber>so, it really looks like ~15k ops/s is the fastest rpc we can write for leveldb in node
10:24:38  <juliangruber>I'm super tempted to try the same in go or C
10:29:50  <juliangruber>with dominictarr's level-master replication we can have way higher throughput with multilevel of course
10:30:29  <st_luke>juliangruber: I think I just wrote the same thing you were writing, gist?
10:31:35  <juliangruber>st_luke: https://gist.github.com/juliangruber/5524370
10:31:54  <juliangruber>just does rpc over ipc
10:32:35  <st_luke>ahh i was doing it a little differently
10:32:41  <st_luke>so this is kinda poorly documented in the cluster docs
10:32:47  <juliangruber>gist? :)
10:33:15  <juliangruber>what throughput did you get?
10:37:13  <st_luke>i just wrote it up really quick and didnt do any testing on it
10:37:28  <st_luke>https://gist.github.com/st-luke/5524385
10:37:44  <st_luke>could be really bad perf for all i know :)
10:38:49  <juliangruber>hmmm
10:39:03  <juliangruber>it's doing all the work on the master
10:39:21  <juliangruber>ah
10:39:33  <juliangruber>the slave does all the tcp stuff perhaps?
10:40:00  <juliangruber>no i don't think so
10:41:22  <juliangruber>with a language like c or go you could have multiple threads that each do the multilevel thing and access a leveldb instance that's protected by a mutex lock
10:42:10  <juliangruber>it would be super awesome to be able to expose a leveldb over multilevel from c but do all the other things in javascript, including adding plugins
10:42:15  <juliangruber>^ this
10:43:19  <juliangruber>should I try this in Go or C first?
10:45:23  <st_luke>java
10:45:40  <juliangruber>:D
10:45:50  <juliangruber>visual basic
11:01:31  <wolfeidau>juliangruber: what is the issue?
11:01:42  <juliangruber>wolfeidau: speed speed speed
11:02:13  <wolfeidau>juliangruber: Where is the bottleneck?
11:04:43  <wolfeidau>juliangruber: Just read back over the conversation
11:04:46  <juliangruber>wolfeidau: serializing perhaps
11:05:00  <juliangruber>it was kinda hard finding the bottleneck with dtrace
11:05:21  <wolfeidau>juliangruber: Yeah kinda like a needle in a haystack
11:05:46  <wolfeidau>have you tried zeroing in on each part?
11:06:14  <wolfeidau>so take leveldb out of the picture, how much data can you get between node processes using cluster
11:06:54  * werlejoined
11:07:06  <wolfeidau>I aways find graphing stuff helps, like memory use and cpu through the life cycle of a test
11:07:18  <wolfeidau>You start to see patterns
11:09:41  <st_luke>taking leveldb out of the picture is difficult because your decisions when using cluster in node are so impacted by level's one process connection
11:12:25  <juliangruber>hm true
11:12:35  <juliangruber>test multilevel by providing a fake db to it
11:12:36  <juliangruber>will do
11:13:46  <rvagg>test msgpak too, I bet that thing can't be too efficient, even if it saves bytes
11:14:00  <wolfeidau>mspack is pretty damn fast
11:14:21  <juliangruber>it uses the js version of msgpack
11:14:21  <wolfeidau>it is one of the fastest binary encodings around, it takes a lot of short cuts
11:14:41  <wolfeidau>but again worth checking :)
11:15:19  <wolfeidau>juliangruber: have you played iwth node-profiler?
11:15:40  <wolfeidau>https://github.com/bnoordhuis/node-profiler
11:15:42  <juliangruber>wolfeidau: not yet
11:16:03  <wolfeidau>the output was featured in that v8 tuning video floating around
11:16:14  <wolfeidau>if you do make sure you grab my patch :)
11:16:35  <wolfeidau>as v8 changed some of the attributes in teh logs and broke it
11:17:44  <wolfeidau>it will tell you about what routines are using the most time, a classic is regex's
11:18:07  <wolfeidau>or overruns of numbers, which as far as i understand impact v8
11:18:30  <wolfeidau>as in overruning 7bit signed or whatever it is
11:21:11  <juliangruber>with a fake db multilevel does ~20k ops/s (vs. 15k ops/s with the real db)
11:21:41  <juliangruber>wolfeidau: sweet, I'll do that
11:22:06  <juliangruber>wolfeidau: I'll benchmark multilevel with the fake db
11:22:23  <wolfeidau>juliangruber: so fake as in no operations ?
11:22:37  <juliangruber>the fake db is just { get : function (key, cb) { cb(null, 'bar') }, put : function (key, value, cb) { cb() } }
11:22:44  <wolfeidau>yepo cool
11:23:40  <wolfeidau>juliangruber: Using the node-profiler can be a bit anoying
11:24:00  <wolfeidau>I will do a gist for you
11:24:35  <juliangruber><3!!
11:45:36  <wolfeidau>juliangruber: Sorry took me a min to work out installing npm module from a github branch https://gist.github.com/wolfeidau/5524680
11:46:00  <wolfeidau>juliangruber: Was installing master and cussing cause it wasn't working :P
11:48:49  <juliangruber>wolfeidau: sweeet
11:48:55  <juliangruber>I'll do this later today
11:49:55  <wolfeidau>np if it works for you make sure you +1 the pull request :)
12:38:57  * st_lukequit (Remote host closed the connection)
13:04:09  <juliangruber>wolfeidau: which pull request?
13:08:58  <rvagg>juliangruber: he has one to the node-profiler project
13:09:08  <rvagg>juliangruber: https://github.com/bnoordhuis/node-profiler/pull/12
13:34:39  <juliangruber>rvagg: thanks
13:54:24  * werlequit (Quit: Leaving.)
14:25:22  * werlejoined
15:09:13  * owen1quit (Ping timeout: 245 seconds)
15:53:01  * Pwnnajoined
15:53:57  * owen1joined
16:00:04  * mreviljoined
16:06:31  <levelbot>[npm] levels@0.0.1 <http://npm.im/levels>: leveldb full text search for node.js (@nharbour)
16:14:13  * dominictarrjoined
16:27:31  <levelbot>[npm] levels@0.0.2 <http://npm.im/levels>: leveldb full text search for node.js (@nharbour)
16:59:18  * colinwrenjoined
17:10:09  * dominictarrquit (Quit: dominictarr)
18:07:04  <wolfeidau>juliangruber: Did you get a chance to try out that gist?
18:08:16  <juliangruber>wolfeidau: nope not yet
18:15:18  * colinwrenpart
18:22:04  * dominictarrjoined
18:29:54  * mrevilquit (Remote host closed the connection)
18:34:01  * mreviljoined
18:42:33  <hij1nx>juliangruber: so, im building a client/server logger based on leveldb, and i was debating putting read capabilities into the client. but i think this might be an antipattern -- leading to misuse of the logging client and turning it into a storage device
18:43:12  <hij1nx>i think im going to leave createReadStream and get off of the api
18:49:13  * mrevilquit (Remote host closed the connection)
18:57:32  <dominictarr>hij1nx: I can't believe that people are still arguing about the callback thing...
18:59:47  <hij1nx>dominictarr: or calling things powerful! haha
19:00:25  <hij1nx>"a powerful abstraction for eating hotdogs"
19:00:57  <dominictarr>if people think this callback/promise thing is important….
19:01:12  <dominictarr>they are doing too many things one after each other
19:01:37  <dominictarr>like REST req/res shaped apis
19:02:02  <dominictarr>promises only optimise the easy case
19:02:26  <dominictarr>but streams are a ladder to a new layer of LHF
19:04:18  <hij1nx>some people are pattern crazy, some people are crazy for meta data. they need classes and patterns for everything to guide them though their day so they can go home at 5pm.
19:07:09  <dominictarr>the true test isn't showing elegant syntax in a few examples
19:07:31  <dominictarr>it's building something awesome that you could never practically achieve the other way...
19:09:48  <dominictarr>instead of arguing, do something that no one can deny.
19:09:52  <hij1nx>dominictarr: and no one is building anything awesome with p/A+, its an psudo-academic circle jerk
19:11:13  <dominictarr>promises are mostly used in specs, as far as I can tell…
19:19:43  * mreviljoined
19:28:05  * mrevilquit (Ping timeout: 255 seconds)
19:43:08  <hij1nx>dominictarr: your reconnect module should be called restream, and my restream module should be called reconnect, trade? :)
19:43:34  <hij1nx>dominictarr: hey, what do you think about merging the async2 branch in level-hooks?
19:44:17  <hij1nx>i've been using it, and it seems to work pretty well, i have some tests surrounding it if you look at gh/loqui-server
19:46:26  <dominictarr>hij1nx: I need to have another think about it after our discussion
19:46:37  <dominictarr>also, I want to discuss with rvagg
19:46:46  <dominictarr>since he has a usecase for that too.
19:46:58  <dominictarr>that is different...
19:47:22  <dominictarr>hmmm...
19:47:36  <dominictarr>prehaps the intersection point is at lock...
19:47:54  <dominictarr>you can consider pushing onto an array to be a reduce
19:48:02  <dominictarr>and merge is a reduce too.
19:48:36  <dominictarr>if you did multiple puts, and then merge them, you'd still want the cbs to fire at the point that all have been saved.
19:57:55  <Raynos>hij1nx: your going to be at nodeconf right?
19:59:54  * mreviljoined
20:01:00  <hij1nx>Raynos: hell yeah, going to be playing some live music with max and substack
20:01:17  <Raynos>nice!
20:01:31  * mrevil_joined
20:01:44  <Raynos>https://twitter.com/hij1nx/status/331470682229706753 <
20:01:46  <Raynos>That is epic
20:02:19  <hij1nx>dominictarr: ok so, we could consider that a point version, but for now it seems pretty ?stable
20:02:59  * mrevil_quit (Remote host closed the connection)
20:03:31  <hij1nx>Raynos: hehe, it's such a silly conversation for people to be having, but... that's typical twitter...
20:04:20  * mrevilquit (Ping timeout: 246 seconds)
20:05:07  <hij1nx>...very qualitative rather than quantitative
20:05:38  <Raynos>its interesting, I feel that I make the same mistakes as promises people by overusing FP.
20:05:52  <Raynos>"FP is powerful, in the same way that coffeescript is powerful" puts that in perspective
20:11:54  <dominictarr>omg… did you see this https://github.com/maxtaco/mkb/blob/master/bin/upload.iced#L61-L71
20:12:03  <dominictarr>that is so bad on a number of levels
20:12:23  <dominictarr>should just use streams rather than custom async calls to fs.read!
20:15:29  <dominictarr>Raynos: a powerful way of chasing your own tail.
20:21:45  <Raynos>dominictarr: why do we always make fun of AWS?
20:22:04  <Raynos>MOTHER OF FUCK
20:22:07  <Raynos>thats iced coffeescript
20:22:20  <Raynos>i though that was some crazy shellscript stuff
20:22:36  <Raynos>but its node. that code 100% does not look like node
20:30:16  * mreviljoined
20:37:14  * st_lukejoined
20:44:15  <dominictarr>chrisdickinson: are you using through to make readable streams?
20:44:38  <chrisdickinson>dominictarr: they're mostly transforms
20:44:43  <chrisdickinson>+ a few duplexes
20:44:55  <dominictarr>can you show me an example?
20:45:25  <chrisdickinson>so, current usage: https://github.com/chrisdickinson/js-git-demo/blob/master/frontend/index.js#L55-L61
20:45:46  <chrisdickinson>example transform: https://github.com/chrisdickinson/git-list-pack/blob/master/index.js
20:46:54  <chrisdickinson>example outer duplex (the `a` in `a.pipe(b).pipe(a)`) https://github.com/chrisdickinson/git-fetch-pack/blob/master/index.js
20:47:19  <chrisdickinson>and inner duplex: (the `b` in `a.pipe(b).pipa(a)`) https://github.com/chrisdickinson/git-transport-protocol/blob/master/index.js
20:47:24  <chrisdickinson>sorry for the wall of links
20:48:39  <dominictarr>hmm, actually this might not work nicely.
20:49:05  <dominictarr>the only simple way would be to use stream-to-pull-stream
21:04:56  <werle>dominictarr: https://github.com/jwerle/draft
21:05:23  <werle>dominictarr: I'm creating the schema level stuff now for what will be an orm to levelup and indexedDB
21:06:34  <dominictarr>looks good
21:06:48  <werle>cool thanks
21:08:04  <dominictarr>werle: how does it decide what the id should be?
21:08:33  <werle>dominictarr: it isn't concerned with that
21:09:04  <werle>dominictarr: when it is consumed that functionality would be added by the consumer
21:09:12  <werle>dominictarr: it is only a schema to model module
21:09:50  <dominictarr>oh, I see. I thought it was a leveldb thing
21:10:00  <dominictarr>but it's not bound to level -
21:10:00  <werle>it will be
21:10:10  <werle>I'm working on a level-schema thing right now
21:10:18  <werle>that consumes that module
21:11:34  <dominictarr>cool, keep me posted!
21:11:52  <werle>will do man!
21:28:52  * st_lukequit (Remote host closed the connection)
21:30:30  * werlequit (Quit: Leaving.)
21:31:26  * timoxleyjoined
22:31:58  * werlejoined
22:37:49  * no9joined
23:25:58  <hij1nx>there's some odd contion with rpc-stream where i get "ERROR: unknown callback id"
23:26:24  <hij1nx>i only get it with my test suite because i'm doing stuff abnormally fast
23:26:26  <hij1nx>maybe
23:26:30  <hij1nx>...robots
23:26:44  <hij1nx>s/contion/condition/
23:27:47  <hij1nx>hotdogs are gross. america is gross. most dive bars in america are gross, but fun.
23:28:28  <hij1nx>dominictarr:
23:28:38  <hij1nx>dominictarr: sweet as
23:29:15  <hij1nx>i prolly shouldnt be IRCing at a bar after several vodkas
23:29:46  <no9>hij1nx that is the only time to IRC
23:29:48  <dominictarr>can you make a pull request with the code?
23:30:01  <hij1nx>dominictarr: can do
23:30:25  <dominictarr>all the ladies must be like, wowa is that dude using irc?
23:30:45  <dominictarr>he must be a 1337 ha4x0r
23:30:49  <hij1nx>dominictarr: the band is like, dude, you're stealing our thunder
23:31:46  * Pwnnaquit (Quit: mrrow~)
23:32:09  * Pwnnajoined
23:32:26  * ChrisPartridgejoined
23:51:20  * mrevilquit (Remote host closed the connection)
23:52:53  <rvagg>I'm keen to see what a bar + vodkas pull request looks like