00:04:09  * ChrisPartridgejoined
00:06:11  * ashihjoined
00:13:27  * lithiumjakequit (Quit: No Ping reply in 180 seconds.)
00:13:48  * paulfryzeljoined
00:13:54  * lithiumjakejoined
00:14:18  <ogd>rvagg: any tips on tracking down a segmentation fault? I'm getting it when I call db.close on a levelup instance backed by leveldown-hyper, but only in a kind of complicated setup, working on a reduced test case now
00:15:04  * fritzyquit (Remote host closed the connection)
00:16:35  * guybrushquit (Excess Flood)
00:17:04  * guybrushjoined
00:18:11  * paulfryzelquit (Ping timeout: 246 seconds)
00:18:17  <ogd>rvagg: is it possible that it's being caused by me working with two separate dbs, one using leveldown-hyper and another using leveldown, in the same process?
00:18:20  * lithiumjake_joined
00:18:31  * lithiumjakequit (Ping timeout: 240 seconds)
00:19:22  * mhernandez1quit (Remote host closed the connection)
00:20:20  * ednapiranhajoined
00:20:38  <dominic_>rvagg: okay I got this out of it: http://i.imgur.com/8ahQsiX.png
00:20:48  <dominic_>how do I interpret this?
00:31:24  <levelbot>[npm] localstorage-down@0.4.2 <http://npm.im/localstorage-down>: A Node.js and browserify leveldown API implementation that maps to localstorage in the browser (@no9)
00:32:52  <levelbot>[npm] no-plebs@0.0.6 <http://npm.im/no-plebs>: leveldb commenting module (@ednapiranha)
00:34:49  <dominic_>rvagg: where did you get the idea to display the graph like this - with the noisy graph of the write time?
00:41:00  * wolfeidaujoined
00:41:07  * wolfeidauquit (Remote host closed the connection)
00:42:04  <ogd>rvagg: if you wanna check out the segfault clone https://github.com/maxogden/dat/tree/hyperlevel-tests, run `npm install` and then do `node test/run.js test/experimental_tests/hyperlevel-clone.js`
00:42:23  <ogd>rvagg: its segfaulting somewhere inside leveldown-hyper
00:50:44  * lithiumjake_quit (Quit: No Ping reply in 180 seconds.)
00:51:08  * lithiumjakejoined
00:51:23  * thlorenzjoined
00:52:13  * thlorenzquit (Remote host closed the connection)
00:54:18  * nnnnathannquit (Ping timeout: 240 seconds)
01:04:49  * thlorenzjoined
01:32:20  * tarrudaquit (Remote host closed the connection)
01:40:49  * rudjoined
01:40:49  * rudquit (Changing host)
01:40:49  * rudjoined
01:43:12  * dguttmanquit (Quit: dguttman)
01:49:12  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
01:49:59  * daviddiasjoined
01:54:26  * daviddiasquit (Ping timeout: 246 seconds)
02:01:56  * mhernandez1joined
02:11:12  * mhernandez1quit (Remote host closed the connection)
02:12:06  * ramitosquit (Quit: Textual IRC Client: www.textualapp.com)
02:15:10  * thlorenzquit (Remote host closed the connection)
02:15:44  * thlorenzjoined
02:20:07  * thlorenzquit (Ping timeout: 240 seconds)
02:22:37  <rvagg>dominic_: re noisy graph, I'm not sure, seemed logical way to plot so many points
02:23:08  <rvagg>dominic_: actually, if you look at matthew von z<whatever>'s talk at recon east on basho leveldb you'll see something similar, I was probably aiming for something along those lines
02:23:24  <rvagg>dominic_: I have the scripts he used to generate those but it uses the riak benchmark suite which is not useful
02:24:15  <rvagg>ogd: tracking down a segfault -- use gdb: `gdb --args node test/run.js test/experimental_tests/hyperlevel-clone.js` then when you get the segfault do a backtrace and other gdb magic to get details
02:24:23  * fritzyjoined
02:29:33  <dominic_>rvagg: yeah, it shows the variation a lot. if you plot it time/writes then you just get a straightish line at some gradient.
02:30:23  <dominic_>I'm gonna do benchmarks too look for things like how perf varies as you vary value size and concurrency.
02:30:24  * nnnnathannjoined
02:32:14  * fritzyquit (Read error: Connection reset by peer)
02:32:52  * fritzyjoined
02:34:42  * nnnnathannquit (Ping timeout: 240 seconds)
02:38:27  * fritzyquit (Read error: Connection reset by peer)
02:39:04  * fritzyjoined
02:48:06  * stagasquit (Read error: Connection reset by peer)
02:51:18  * daviddiasjoined
02:55:20  * daviddiasquit (Ping timeout: 246 seconds)
02:57:38  * paulfryz_joined
03:01:59  * paulfryz_quit (Ping timeout: 246 seconds)
03:10:05  * itsMontoyaquit (Remote host closed the connection)
03:34:30  * kenansulltopic: http://logs.nodejs.org/leveldb/latest — http://r.va.gg/2013/11/leveldown-v0.10-managing-gc-in-native-v8-programming.html
03:37:23  * nnnnathannjoined
03:41:26  * eugenewarequit (Quit: Connection closed for inactivity)
03:49:11  * fritzyquit (Read error: Connection reset by peer)
03:49:49  * fritzyjoined
03:51:43  * daviddiasjoined
03:55:54  * daviddiasquit (Ping timeout: 240 seconds)
03:58:17  * paulfryz_joined
04:00:13  * nnnnathannquit (Remote host closed the connection)
04:02:51  * paulfryz_quit (Ping timeout: 260 seconds)
04:21:34  * aaronlidmanjoined
04:36:15  * dguttmanjoined
04:36:50  * dguttmanquit (Client Quit)
04:37:59  * fritzyquit
04:47:17  * aaronlidmanquit (Quit: Leaving...)
04:53:19  * dguttmanjoined
04:59:09  * paulfryz_joined
05:01:15  <ogd>rvagg: hmm i guess gdb isnt on macos anymore, replaced by lldb but it seems to have a different API. would you mind giving it a shot if you have a minute? `node test/run.js test/experimental_tests/hyperlevel-clone.js` should end in a segfault
05:03:06  * dguttmanquit (Quit: dguttman)
05:03:26  * paulfryz_quit (Ping timeout: 246 seconds)
05:12:07  <ogd>rvagg: omg wtf its magically not segfaulting now... nevermind....
05:24:19  * mikealjoined
05:46:10  * ashihjoined
05:53:01  * daviddiasjoined
05:57:19  * daviddiasquit (Ping timeout: 240 seconds)
05:59:48  * paulfryzeljoined
06:03:59  * paulfryzelquit (Ping timeout: 246 seconds)
06:05:22  <ogd>interesting https://github.com/basho/leveldb/wiki/mv-tiered-options
06:07:25  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
06:26:22  <levelbot>[npm] levelup-riak@0.0.2 <http://npm.im/levelup-riak>: A levelup-esque interface to riak (@fritzy)
06:31:04  * fritzyjoined
06:50:42  * plumajoined
07:00:38  * paulfryzeljoined
07:04:53  * paulfryzelquit (Ping timeout: 246 seconds)
07:09:22  * fritzyquit (Remote host closed the connection)
07:11:24  * fritzy_joined
07:11:25  * fritzy_quit (Remote host closed the connection)
07:11:59  * fritzy_joined
07:16:18  * fritzy_quit (Ping timeout: 240 seconds)
07:29:18  * ednapiranhaquit (Remote host closed the connection)
07:46:20  <dominic_>rvagg: running benchmarks against leveldown-* modules leveldown-hyper is quite slow/spiky is it up to date?
07:46:49  <rvagg>dominic_: nah, they are all a bit old, really need to get a release out
07:48:31  <dominic_>rvagg: okay cool
07:51:46  * paxos^offchanged nick to paxos2k
07:52:21  <rvagg>dominic_: shouldn't be slow though!
07:52:37  <rvagg>mind you, it's optimised for linux, perhaps it's not so awesome on mac?
07:54:10  <dominic_>rvagg: I'm on linux - maybe some stuff has changed because what I'm seeing is google-leveldb is the fastest now
07:54:38  <dominic_>my guess is because you've kept that one up to date?
07:55:11  <dominic_>(hyper is only slower than google leveldb, it's still probably fast in absolute terms :)
07:57:46  * daviddiasjoined
08:01:20  * ednapiranhajoined
08:01:21  * paulfryzeljoined
08:05:47  * paulfryzelquit (Ping timeout: 246 seconds)
08:05:54  * ednapiranhaquit (Ping timeout: 240 seconds)
08:21:05  * daviddiasquit (Remote host closed the connection)
08:21:31  * daviddiasjoined
08:25:44  * daviddiasquit (Ping timeout: 246 seconds)
08:32:18  * ChrisPartridgequit (Ping timeout: 240 seconds)
08:44:04  * daviddiasjoined
08:50:12  * dfreirejoined
08:52:41  <rescrv>rvagg dominic_: we're working on a comprehensive LevelDB benchmark suite (concurrency, write pattern, object size, etc.)
08:53:05  <rescrv>I suspect that any benchmark where google-leveldb is fastest is loading a small amount of data
08:53:15  * tarrudajoined
09:00:32  * ednapiranhajoined
09:00:36  <dominic_>rescrv: right - also I'm doing this from within the node.js binding because that is how leveldb is useful to me.
09:01:05  * eugenewarejoined
09:01:11  <dominic_>it would certainly be interesting to know what overhead node.js has though.
09:01:40  <dominic_>rescrv: btw, how big is 'small'?
09:03:20  * stagasjoined
09:04:42  * ednapiranhaquit (Ping timeout: 240 seconds)
09:19:48  * daviddia_joined
09:20:35  * daviddiasquit (Read error: No route to host)
09:20:53  * daviddiasjoined
09:23:54  * daviddia_quit (Ping timeout: 240 seconds)
09:35:58  * stollengajoined
09:36:12  <stollenga>hello, where is a good place to read on good practices for leveldb?
09:39:55  <dominic_>stollenga: there are many issues on leveldb discussing ways of doing certain things
09:40:17  <dominic_>also, look at the code in level-* modules
09:41:05  <stollenga>i now created a large database of images, but the disk usage is huge
09:41:44  <stollenga>average image size is around 2mb
09:41:59  <stollenga>but just iterating over the keys takes ages, is there a way to speed it up?
09:51:01  <stollenga>what are for example recommendations on blocksize
09:56:56  <dominic_>stollenga: I think that leveldb is designed for much smaller values than that - how many images do you have?
09:58:58  <dominic_>stollenga: really - I think that your particular usecase could be solved better with a slightly different approach, watch this: https://www.youtube.com/watch?v=lYAzCqe2cEY
09:59:29  <dominic_>you could still use leveldb for metadata - image name -> hash + file + offset
09:59:57  <dominic_>but then store the actual files appended into large immutable files.
10:00:24  <dominic_>This has been on my cool projects todo list for a while, except I don't actually need it for anything.
10:01:40  <stollenga>thanks ill look at that
10:01:43  <stollenga>they are 400000 images
10:02:04  <stollenga>which take up 160 gb normally
10:02:13  <stollenga>in leveldb it becomes 640gb:O
10:02:21  <dominic_>yeah - you should definately watch this talk.
10:02:51  * paulfryzeljoined
10:03:05  <dominic_>stollenga: let me know how it goes!
10:03:59  <dominic_>stollenga: it's probably being copied during compactions - but since the images are probably immutable that isn't really very helpful.
10:04:21  <dominic_>or, "immutablish"
10:05:54  * guybrushquit (Excess Flood)
10:06:08  <stollenga>any reason why the overhead is so big?
10:06:08  <stollenga>brb
10:06:10  * guybrushjoined
10:07:14  * paulfryzelquit (Ping timeout: 246 seconds)
10:07:52  <dominic_>stollenga: I don't know enough about level internals to answer that.
10:08:43  <dominic_>I have been doing some benchmarks - I'll do one testing varying value sizes.
10:17:37  <rescrv>dominic_: small is < 3GB or so. Given the activity of the node.js level community, my benchmark will probably be native and node to quantify the overhead and provide people with usable guidance
10:18:24  <rescrv>stollenga: how are you storing the images? Are they the key? The value?
10:21:37  <rescrv>that being said, I'd recommend a different approach. Put the files on the filesystem and refer to them from LevelDB as dominic_ said. Even if you have one file per image, you'll be doing strictly better than loading them into LevelDB
10:40:07  * tarrudaquit (Ping timeout: 240 seconds)
10:43:16  * tarrudajoined
10:58:42  * tarrudaquit (Ping timeout: 246 seconds)
11:00:12  <stollenga>hmm i thought leveldb would be smarter with disk usage somehow
11:00:24  <stollenga>i use protobuf to store the images with metadata
11:00:28  <stollenga>in the values
11:00:39  * ednapiranhajoined
11:03:42  * paulfryzeljoined
11:04:42  * ednapiranhaquit (Ping timeout: 240 seconds)
11:08:09  * paulfryzelquit (Ping timeout: 246 seconds)
11:08:55  <levelbot>[npm] mosca@0.19.1 <http://npm.im/mosca>: The multi-transport MQTT broker for node.js. It supports AMQP, Redis, ZeroMQ, MongoDB or just MQTT. (@matteo.collina)
11:17:08  * tarrudajoined
11:20:48  * daviddiasquit (Remote host closed the connection)
11:21:14  * daviddiasjoined
11:25:19  * daviddiasquit (Ping timeout: 240 seconds)
11:25:23  * levelbotquit (Remote host closed the connection)
11:28:19  <stollenga>does changing the blocksize on an existing database matter?
11:37:01  * levelbotjoined
11:37:01  * levelbotquit (Remote host closed the connection)
11:42:56  * levelbotjoined
11:43:00  <levelbot>[npm] mosca@0.19.1 <http://npm.im/mosca>: The multi-transport MQTT broker for node.js. It supports AMQP, Redis, ZeroMQ, MongoDB or just MQTT. (@matteo.collina)
12:00:35  * ednapiranhajoined
12:02:12  * ednapiranhaquit (Read error: Connection reset by peer)
12:02:44  * ednapiranhajoined
12:04:24  * paulfryzeljoined
12:07:06  * ednapiranhaquit (Ping timeout: 240 seconds)
12:08:41  * paulfryzelquit (Ping timeout: 246 seconds)
12:12:04  * dfreirequit (Quit: Computer has gone to sleep.)
12:39:58  <stollenga>which filesystem is best for using leveldb?
12:43:11  * mhernandez1joined
12:46:52  <rescrv>stollenga: leveldb is smarter with disk usage when objects are small. its own files are 2MB in size, and everything is designed for many objects per file.
12:47:03  <rescrv>as for which filesystem is best, ext2 would be my bet
12:47:44  <rescrv>I recommend ext3/ext4 with these tuning options: http://hyperdex.org/doc/latest/TuningHyperDex/#sec:tuning:filesystem
12:48:46  <rescrv>you can get even better performance by setting the data=writeback option, but it can be dangerous for non-LevelDB apps, so be careful there.
12:49:43  * kenan|afkchanged nick to kenansulayman
12:52:52  * daviddiasjoined
12:57:21  * daviddiasquit (Ping timeout: 250 seconds)
12:57:48  * kenansulaymanchanged nick to kenan|afk
13:00:42  * ednapiranhajoined
13:03:52  * kenan|afkchanged nick to kenansulayman
13:04:56  * thlorenzjoined
13:05:07  * ednapiranhaquit (Ping timeout: 240 seconds)
13:05:15  * paulfryzeljoined
13:09:39  * paulfryzelquit (Ping timeout: 240 seconds)
13:14:30  * savardcquit (Quit: ZNC - http://znc.in)
13:27:00  * mhernandez1quit (Remote host closed the connection)
13:30:03  * mhernandez1joined
13:34:13  * dfreirejoined
13:46:31  * mikealquit (Quit: Leaving.)
13:49:23  * tarrudaquit (Read error: Connection reset by peer)
13:52:54  * Pwnnaquit (Ping timeout: 240 seconds)
13:53:33  * daviddiasjoined
13:53:58  * mikealjoined
13:54:52  * Pwnnajoined
13:58:13  * aaronlidmanjoined
14:00:38  * ednapiranhajoined
14:02:30  * daviddiasquit (Remote host closed the connection)
14:02:59  * daviddiasjoined
14:04:38  * ednapiranhaquit (Ping timeout: 240 seconds)
14:06:02  * paulfryz_joined
14:07:18  * daviddiasquit (Ping timeout: 240 seconds)
14:07:26  * dguttmanjoined
14:10:30  * paulfryz_quit (Ping timeout: 246 seconds)
14:12:23  * ashihjoined
14:13:22  * ashih_joined
14:16:39  * ashihquit (Ping timeout: 240 seconds)
14:18:50  * jjmalinajoined
14:19:39  * patrickarltjoined
14:21:26  * stagasquit (Read error: Connection reset by peer)
14:22:53  * redidasjoined
14:25:53  * ramitosjoined
14:26:14  * mikealquit (Quit: Leaving.)
14:27:38  * blessYahujoined
14:31:37  * patrickarltquit (Remote host closed the connection)
14:32:34  * thlorenzquit (Remote host closed the connection)
14:32:48  * thlorenzjoined
14:39:30  * jjmalinaquit (Read error: Connection reset by peer)
14:41:34  * ashih_quit (Quit: My MacBook has gone to sleep. ZZZzzz…)
14:48:01  * jjmalinajoined
15:02:16  * daviddiasjoined
15:05:09  * savardcjoined
15:06:48  * jerrysvjoined
15:07:28  * ashihjoined
15:09:32  * daviddia_joined
15:11:19  * daviddiasquit (Ping timeout: 240 seconds)
15:11:37  * daviddiasjoined
15:14:13  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
15:14:38  * daviddia_quit (Ping timeout: 240 seconds)
15:14:57  * daviddiasquit (Client Quit)
15:15:03  * paulfryzeljoined
15:17:56  * mikealjoined
15:23:23  * Sorellajoined
15:23:40  * ednapiranhajoined
15:23:43  * dguttmanquit (Ping timeout: 240 seconds)
15:28:40  * dguttmanjoined
15:56:01  * ashihjoined
16:18:26  * rudquit (Quit: rud)
16:18:46  * rudjoined
16:22:50  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
16:25:02  * fritzyjoined
16:26:25  * ashihjoined
16:36:39  * fritzyquit
16:37:07  * stagasjoined
16:42:42  * dfreirequit (Quit: Computer has gone to sleep.)
16:44:21  * fritzyjoined
16:45:33  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
16:46:26  * mcavagejoined
16:47:53  * ashihjoined
17:00:22  * paulfryzelquit (Remote host closed the connection)
17:01:15  * paulfryzeljoined
17:05:38  * paulfryzelquit (Ping timeout: 240 seconds)
17:08:04  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
17:08:11  * mhernandez1quit (Remote host closed the connection)
17:08:19  * paulfryzeljoined
17:09:25  * kenansulaymanchanged nick to kenan|afk
17:12:48  * thlorenzquit (Remote host closed the connection)
17:13:16  * mhernandez1joined
17:28:46  * mhernandez1quit (Remote host closed the connection)
17:36:23  * mhernandez1joined
17:41:35  * mikealquit (Quit: Leaving.)
17:46:28  * mikealjoined
18:04:57  * thlorenzjoined
18:06:42  * mhernandez1quit (Remote host closed the connection)
18:07:17  * 17SAABXDJjoined
18:17:47  * paulfryzelquit (Remote host closed the connection)
18:18:28  * paulfryzeljoined
18:20:39  * ashihjoined
18:21:41  * paulfryz_joined
18:22:18  * paulfryzelquit (Ping timeout: 240 seconds)
18:22:18  * dfreirejoined
18:22:29  * paulfryz_quit (Remote host closed the connection)
18:23:07  * paulfryzeljoined
18:26:03  * mikealquit (Quit: Leaving.)
18:27:23  * paulfryzelquit (Ping timeout: 246 seconds)
18:28:39  <hij1nx>hmm, someone came to the room and asked if leveldb could be scale horizontally, then "arvindeep" answered no: "because it was a library" and then the person asking the question left. The answer should have been "hyperdex-warp" or "node.js", we should possibly have some ops in here.
18:29:09  <hij1nx>./cc rvagg
18:29:18  <hij1nx>./cc rescrv
18:30:30  <rescrv>hij1nx: I agree. I didn't see that question otherwise I would have suggested those options
18:31:17  * levelbotquit (Remote host closed the connection)
18:31:22  <rescrv>hij1nx: if you're going to push for structural changes, perhaps modify levelbot to forward people iff they're looking for the levelup community. There's a few people who are blindly forwarded
18:32:41  <hij1nx>rescrv: off-topic; im about to do a talk at the philly science center and im talking about http://hyperdex.org/papers/warp.pdf and nodejs+hyper-level :)
18:34:31  <hij1nx>rvagg: im thinking we should add a section to leveldb.org that has all the level forks and their related commercial offerings
18:34:53  <rescrv>hij1nx: If you forward me a copy of any slides, I'm happy to provide feedback (no obligation).
18:35:24  <rescrv>Note that we're actually updating warp with a different version of the protocol (the high level details about chaining still hold), so I'd recommend not diving too deeply.
18:35:39  <rescrv>and this summer we'll be releasing Node.js bindings for hyperdex-warp
18:36:39  <hij1nx>rescrv: im only covering it from a high level, basically how you provide ACID and why people should care, the meat of the talk is about using nodejs+hyperlevel
18:37:38  * levelbotjoined
18:37:40  * levelbotquit (Remote host closed the connection)
18:37:44  <rescrv>hij1nx: sounds great! I'd love to hear about how the talk goes for you.
18:51:23  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
18:51:59  * ashihjoined
18:56:23  * paulfryzeljoined
18:56:41  * levelbotjoined
18:56:42  * levelbotquit (Remote host closed the connection)
19:04:08  * mikealjoined
19:08:34  * ramitosquit (Read error: Connection reset by peer)
19:08:54  * ramitosjoined
19:08:54  * ramitosquit (Client Quit)
19:09:06  * ramitosjoined
19:10:23  * ramitosquit (Read error: Connection reset by peer)
19:10:47  * ramitosjoined
19:14:13  * levelbotjoined
19:14:13  * levelbotquit (Remote host closed the connection)
19:19:36  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
19:21:30  * ashihjoined
19:21:32  * levelbotjoined
19:23:55  * ramitos_joined
19:24:31  * ramitosquit (Ping timeout: 240 seconds)
19:29:31  * dfreirequit (Ping timeout: 250 seconds)
19:33:23  * levelbotquit (Remote host closed the connection)
19:33:38  * dfreirejoined
19:37:04  * levelbotjoined
19:37:18  * nnnnathannjoined
19:43:59  <levelbot>[npm] nitrogen-cli@0.1.23 <http://npm.im/nitrogen-cli>: Provides a command line interface to the Nitrogen service. (@tpark)
19:48:30  <levelbot>[npm] level-mapreduce@0.3.0 <http://npm.im/level-mapreduce>: Stored map engine with query engine. (@mikeal)
19:52:27  <scrivy>are there performance differences when using node 0.10 vs 0.11?
19:52:34  <scrivy>with leveldb?
19:54:18  * lithiumjakequit (Ping timeout: 240 seconds)
19:55:39  * mcavagequit
19:55:41  <dominic_>scrivy: measure it and find out
19:56:00  <scrivy>:-P
19:56:03  <scrivy>starting to
19:56:12  <scrivy>wondered if it was obvious
20:08:03  * paulfryzelquit (Read error: Connection reset by peer)
20:08:24  * paulfryzeljoined
20:15:57  * 17SAABXDJquit (Remote host closed the connection)
20:16:17  * mhernandez1joined
20:16:21  * mhernandez1quit (Remote host closed the connection)
20:16:39  * tarrudajoined
20:16:48  * lithiumjakejoined
20:28:47  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
20:30:50  * dfreirequit (Quit: Computer has gone to sleep.)
20:35:16  * tarrudaquit (Ping timeout: 276 seconds)
20:42:46  * dfreirejoined
20:46:19  * thlorenzquit (Remote host closed the connection)
20:47:37  <mikeal>man
20:47:45  <mikeal>"databases as streams" is fuckin great
20:48:10  <mikeal>oh, more importantly, "indexers as streams"
20:53:41  * ramitos_quit (Quit: Textual IRC Client: www.textualapp.com)
20:53:53  * ramitosjoined
20:54:33  <hij1nx>scrivy: there are, you can find out easily exactly what the differences are by doing git fetch and running the benchmarks on each branch
20:57:50  * fuze_workingjoined
20:57:52  * fuze_workingquit (Remote host closed the connection)
20:58:14  * redidasquit (Remote host closed the connection)
20:58:26  * fuze_workingjoined
21:02:44  <dominic_>mikeal: but what about durability?
21:03:58  <mikeal>that all comes later
21:04:11  <mikeal>for creating new indexes and handling back pressure this is fantastic
21:04:28  <mikeal>and it's a lot simpler to layer a protocol to handle stuff like that on top of the stream abstraction than without it
21:04:42  <mikeal>we're overthinking replication
21:04:54  <mikeal>there's a lot of use cases that are really simple that we make way too complicated
21:05:18  <mikeal>i'm doing a bunch of stuff right now where it doesn't even make sense to try and have a sequences index
21:05:45  <jerrysv>brianloveswords: are you in for jsconf?
21:05:54  <mikeal>i can just overwrite the whole index each time since I'm only going to create it once, unless i modify it, in which case i wanna just blow it all away again
21:06:52  <scrivy>hij1vx O.o
21:09:39  <dominic_>mikeal: okay got it, yeah it's very simple for batch indexing.
21:10:23  <mikeal>especailly once i moved to just being the regular levelup format
21:10:36  <mikeal>so you can do lev.createReadStream().pipe(indexer)
21:10:49  <mikeal>and then do indexer.pipe(otherIndexer)
21:11:04  <mikeal>super simple index trees
21:11:11  <dominic_>what should indexer output? does it just echo it's input?
21:11:22  <mikeal>it outputs the new indexes values
21:11:30  <mikeal>so, this is a stored map index
21:11:35  <dominic_>so what does that look like/
21:11:38  <dominic_>?
21:11:46  <mikeal>so each indexer has a function that returns [[key, value],[key,value]]
21:12:16  <dominic_>why twice?
21:12:16  <mikeal>so if the map is function (entry) { return [[entry.value.test, entry.key]] }
21:12:34  <mikeal>because a single entry could created N secondary index entries
21:12:48  <mikeal>anyway, that's the map
21:12:53  <dominic_>oh right
21:13:20  <mikeal>that indexer will emit {key: entry.key, value: [[entry.value.test, entry.key]]}
21:13:20  <dominic_>because you can emit multiple mappings per thing
21:13:41  <mikeal>that way, if something gets changed, or deleted, the indexes know to update or remove all the indexes for that entry, so the key gets preserved all the way
21:14:12  <mikeal>yeah, i didn't want to replicate the emit() api from couchdb because it would require me to inject or pass in an extra method
21:14:17  <dominic_>yeah you need to retrive the old mappings, so that you can propagate deletes
21:14:26  <mikeal>exactly
21:14:43  <mikeal>the first thing the indexer does is pull the meta information for the key and prepare the deletes
21:15:08  <dominic_>but if you are doing this in batch mode, when does it get rerun? or is this just from a live feed?
21:15:11  <mikeal>because if it's an update it needs to remove all old entries
21:15:26  <mikeal>it doesn't actually care
21:15:40  <dominic_>sure, but how do you do it in practice?
21:15:50  <mikeal>it could be a levelup.createReadStream() or it could be another database that gets piped
21:16:22  <mikeal>right now I'm using a readStream() from levelup or a readStream() from an indexer and generate a whole index at once
21:16:23  <dominic_>but a normal readstream won't have any deletes
21:16:34  <dominic_>because it will be a snapshot of the database.
21:17:00  <mikeal>yeah, that's something i added on in this kind of hacky way from couchdb, i add some meta to the entry if the doc was deleted
21:17:28  <dominic_>replace the doc with {_delete: true} ?
21:17:42  <mikeal>i just store the doc with _deleted:true
21:18:28  <mikeal>the delete story is still not great, but this data doesn't actually change
21:18:59  <mikeal>some databases don't store a record of deletes, so they can't support deletes in their indexes if they aren't always piped
21:19:11  <mikeal>in practice you could just pipe them always because you put up the server
21:19:38  <mikeal>that's not too big of a problem, the bigger issue is "i wasn't piped for a while and now i need to catch up"
21:19:51  <mikeal>which means you probably want a sequence index and a record of deletes
21:20:03  <mikeal>but these are all just upgrades, just some new meta on the entires that get emitted
21:25:46  * mikealquit (Quit: Leaving.)
21:27:29  <levelbot>[npm] len@0.2.0 <http://npm.im/len>: Calendar database for of resource bookings in leveldb (@binocarlos)
21:49:29  <levelbot>[npm] len@0.2.1 <http://npm.im/len>: Calendar database for of resource bookings in leveldb (@binocarlos)
21:52:50  * copongcopongquit (Ping timeout: 246 seconds)
21:56:29  * mikealjoined
22:00:30  * copongcopongjoined
22:04:55  * mikealquit (Ping timeout: 240 seconds)
22:06:59  * thlorenzjoined
22:07:30  * dominic_part ("Leaving")
22:07:37  * thlorenzquit (Remote host closed the connection)
22:13:58  * blessYahu_joined
22:14:55  * blessYahuquit (Ping timeout: 240 seconds)
22:19:43  * paulfryzelquit (Read error: Connection reset by peer)
22:20:15  * paulfryzeljoined
22:22:48  * thlorenzjoined
22:23:29  <levelbot>[npm] len@0.2.2 <http://npm.im/len>: Calendar database for of resource bookings in leveldb (@binocarlos)
22:27:36  * paulfryzelquit (Remote host closed the connection)
22:28:02  * paxos2kchanged nick to paxos^off
22:28:14  * paulfryzeljoined
22:28:30  <levelbot>[npm] len@0.2.3 <http://npm.im/len>: Calendar database for of resource bookings in leveldb (@binocarlos)
22:28:33  * plumaquit (Remote host closed the connection)
22:31:08  * thlorenzquit (Ping timeout: 252 seconds)
22:32:31  * paulfryzelquit (Ping timeout: 240 seconds)
22:35:39  * ashihjoined
22:35:53  * thlorenzjoined
22:43:02  <levelbot>[npm] dat@3.0.1 <http://npm.im/dat>: real-time replication and versioning for large tabular data sets (@maxogden)
22:54:22  * thlorenzquit (Remote host closed the connection)
22:57:42  * dfreirequit (Quit: Computer has gone to sleep.)
23:04:06  * fuze_workingquit (Remote host closed the connection)
23:19:18  * jerrysvquit (Remote host closed the connection)
23:22:13  * aaronlidmanquit (Quit: Leaving...)
23:25:30  * ramitosquit (Remote host closed the connection)
23:25:49  * ednapiranhaquit (Quit: Leaving...)
23:26:00  * ramitosjoined
23:31:23  * blessYahujoined
23:33:59  * blessYahu_quit (Ping timeout: 252 seconds)
23:34:44  * tarrudajoined
23:36:41  * ChrisPartridgejoined
23:41:35  * jjmalinaquit (Quit: Textual IRC Client: www.textualapp.com)
23:43:39  * ChrisPartridgequit
23:46:44  * blessYahu_joined
23:49:23  * tarrudaquit (Ping timeout: 252 seconds)
23:49:42  * blessYahuquit (Ping timeout: 240 seconds)
23:51:37  * stagasquit (Read error: Connection reset by peer)