00:13:11  * mrevilquit (Remote host closed the connection)
00:13:45  * thl0joined
00:18:19  * thl0quit (Ping timeout: 252 seconds)
00:24:59  * thl0joined
00:29:55  * thl0quit (Ping timeout: 264 seconds)
00:43:27  <substack>somebody tell these folks about leveldb http://www.reddit.com/r/node/comments/1gmwex/open_source_introducing_nodebb_the_discussion/
00:43:45  <substack>they're using redis as permanent, non-emphemeral data storage >_<
00:43:54  <ChrisPartridge>heh
00:47:08  * mreviljoined
00:52:07  * mrevilquit (Ping timeout: 264 seconds)
00:59:55  * ralphtheninjaquit (Ping timeout: 264 seconds)
01:00:29  * no9quit (Ping timeout: 252 seconds)
01:16:45  * levelbotjoined
01:19:11  * julianduquequit (Ping timeout: 252 seconds)
01:20:30  * thl0joined
01:23:11  * julianduquejoined
01:56:14  * julianduquequit (Quit: leaving)
02:36:09  <levelbot>[npm] valuepack-mine-github@0.1.1 <http://npm.im/valuepack-mine-github>: Mines github for user and repository data used by valuepack. (@thlorenz)
02:43:59  * werlequit (Quit: Leaving.)
02:45:31  * timoxleyquit (Ping timeout: 264 seconds)
02:56:16  * thl0quit (Remote host closed the connection)
03:03:50  * levelbotquit (Remote host closed the connection)
03:04:08  * levelbotjoined
03:04:11  <levelbot>[npm] valuepack-mine-github@0.1.1 <http://npm.im/valuepack-mine-github>: Mines github for user and repository data used by valuepack. (@thlorenz)
03:13:00  <levelbot>[npm] level-ttl@0.2.0 <http://npm.im/level-ttl>: Adds a 'ttl' option to LevelUP for puts and batches (@rvagg)
03:15:28  * mreviljoined
03:25:00  <rvagg>rescrv: great article btw (http://hackingdistributed.com/2013/06/17/hyperleveldb/), excellent detail, tho I'd love to know more about how you choose SST sets for compaction to reduce write amplification, is that a tricky thing to do?
03:41:16  * julianduquejoined
04:09:46  * eugenewarejoined
04:28:42  * mrevilquit (Remote host closed the connection)
04:31:51  * piklu_joined
04:32:24  * pikluquit (Ping timeout: 252 seconds)
04:36:31  * piklu_quit (Ping timeout: 264 seconds)
05:05:08  * st_lukejoined
05:42:44  * wolfeidauquit (Remote host closed the connection)
05:49:16  * wolfeidaujoined
05:57:30  <levelbot>[npm] doorknob@0.1.0 <http://npm.im/doorknob>: convenience module for adding Mozilla Persona user login + LevelDB based session storage to node web apps (@maxogden)
05:57:30  <levelbot>[npm] socket-sync@0.1.0 <http://npm.im/socket-sync>: bidirectional replication of large data sets (binary or ascii) between node and web browsers using websockets (@maxogden)
06:37:07  * julianduquequit (Ping timeout: 264 seconds)
07:07:48  * st_lukequit (Remote host closed the connection)
07:08:43  * eugeneware1joined
07:08:46  * eugeneware1quit (Client Quit)
07:20:22  * no9joined
08:04:23  * no9quit (Ping timeout: 246 seconds)
08:18:36  * no9joined
08:27:17  * eugenewarequit (Remote host closed the connection)
08:48:29  * ChrisPartridgequit (Ping timeout: 256 seconds)
09:00:23  * wolfeidauquit (Ping timeout: 245 seconds)
09:13:12  * mcollinajoined
09:37:55  * eugenewarejoined
09:45:09  * eugenewarequit (Ping timeout: 256 seconds)
09:54:26  * ralphtheninjajoined
09:57:56  <substack>experimenting with tacodb, fighting strange errors
09:58:07  <substack>such as: LevelUPError: Could not locate LevelDOWN, try `npm install leveldown`
09:58:44  <substack>why is require('leveldown') wrapped inside of a try/catch in levelup/lib/util.js?
10:00:20  <substack>then I get Error: Module version mismatch. Expected 11, got 1.
10:00:26  <substack>if I take away the try/catch
10:10:01  * mcollina_joined
10:13:23  * mcollinaquit (Ping timeout: 240 seconds)
10:16:03  * wolfeidaujoined
10:27:52  <no9>substack are you running it from git-repo or npm install tacodb -g?
10:28:03  <no9>tacodb start --port 8000
10:29:04  <rvagg>substack: your module mismatch means that you need to compile leveldown for your current version of node
10:29:15  <rvagg>.. should take module version mismatch errors into account
10:29:30  <rvagg>go into node_modules/leveldown and `node-gyp rebuild`
10:31:24  <rvagg>substack: or, reinstall leveldown and npm will do the compile for you
10:32:00  <rvagg>you must have installed it with an older version of node than you're running now
10:38:35  <substack>"reinstall leveldown"?
10:39:15  <substack>figured it out!
10:39:24  <substack>I was running tacodb from a directory with a leveldown dep in it
10:39:28  <substack>but it was picking up that dep
10:39:38  <substack>must be something tricky going on with module resolution
10:39:44  <substack>bins shouldn't do that normally
10:49:39  <rvagg>binary modules resolve just like normal node modules
10:51:00  <rescrv>rvagg: https://github.com/rescrv/HyperLevelDB/blob/master/db/version_set.cc lines 1300-1387 and related calls
11:00:54  * thl0joined
11:02:24  * chirinojoined
11:02:25  <rvagg>rescrv: is that the size of the *range* in terms of keys or the size of the files involved?
11:08:25  <rvagg>going to take a bit to digest that code
11:14:27  * timoxleyjoined
11:26:29  <levelbot>[npm] valuepack-mine-github@0.2.0 <http://npm.im/valuepack-mine-github>: Mines github for user and repository data used by valuepack. (@thlorenz)
11:29:39  * missinglinkjoined
11:51:16  * eugenewarejoined
11:55:06  * ralphtheninjaquit (Quit: leaving)
12:01:13  <rescrv>rvagg: it's the size in terms of number of bytes on disk
12:01:51  <rescrv>so the sum of the files involved
12:05:53  <rvagg>ok, clever
12:05:57  <rvagg>well it seems to be working!
12:05:57  <rescrv>essentially LA/LB are arrays of SSTs (metadata about them), and we do a sliding window through LA, computing the overlap in LB. As we go, we track the one that maximizes the ratio of data from LA to data in LB
12:06:58  <rvagg>I'm still running my 1.5B random writes benchmark for Google's LevelDB, taking ages
12:07:22  <rvagg>it's been 35 hours or something so far
12:07:56  <rescrv>If I'm estimating correctly, you've another 15 days to go
12:08:26  <rvagg>nono, should be finished before 48 hours are up
12:08:56  <rvagg>then hopefully yours will run in a fraction of that time, basho's may run in ~2/3rds of the time
12:08:59  <rvagg>will be interesting
12:09:09  <rescrv>oh, 4 threads. so total of 3-4 days
12:09:44  <rescrv>48 hours may be closer for your workload (you'd know better)
12:09:57  <rescrv>is this on spinning disk or flash?
12:10:35  <rvagg>spinning disk, and I only had a RAID1 set available, would have preferred a plain disk but this'll have to do
12:14:25  <rescrv>rvagg: btw, I'm working on a few patches to HyperLevelDB that'll improve the latency. If the test takes that many hours, it might be worth doing the basho run first so that you can run our fork with the patches.
12:14:26  <rvagg>oh, and the value size is quite small, makes a bit of difference
12:15:14  <rvagg>rescrv: sure, I'm going to be talking LevelDB in SF on Tuesday, I leave here (Aus) on Sunday so I wanted to get things organised before I left
12:16:14  <rvagg>just a casual Node/LevelDB meetup, I wanted to talk a bit about the implications of some of the changes both you and basho have made for the way that Node works with LevelDB
12:16:54  <rescrv>rvagg: sounds great! I'm on the east coast here, and wish I could get out there for the meetup, but as a grad student, I'm a little tight on cash.
12:17:14  <rvagg>aye, I understand! perhaps you could find a grant for it!
12:17:14  <rvagg>heh
12:17:21  * Acconutjoined
12:19:11  <rescrv>rvagg: Unfortunately, grants take time to get.
12:19:21  <rescrv>I think the quickest turnaround I've seen was a month
12:21:16  * Acconutquit (Client Quit)
12:21:23  <rvagg>hij1nx: when (if?) you head back to NY, you should catch up with rescrv and talk leveldb some time, will probably be relevant for the work you're doing as well as just interesting
12:28:15  * st_lukejoined
12:31:55  * mreviljoined
12:32:54  * Pwnnaquit (Ping timeout: 240 seconds)
12:36:21  * mrevilquit (Ping timeout: 252 seconds)
12:40:33  * Pwnnajoined
12:48:23  * timoxleyquit (Quit: Computer has gone to sleep.)
12:55:47  * thl0quit (Remote host closed the connection)
13:02:44  * ednapiranhajoined
13:24:19  * chirinoquit (Quit: Computer has gone to sleep.)
13:24:26  * Acconutjoined
13:24:34  * Acconutquit (Client Quit)
13:26:49  * chirinojoined
13:40:04  * st_lukequit (Remote host closed the connection)
13:41:03  * chirinoquit (Quit: Computer has gone to sleep.)
13:43:49  * chirinojoined
13:58:47  * thl0joined
14:01:48  * chirinoquit (Quit: Computer has gone to sleep.)
14:02:23  * ednapiranhaquit (Remote host closed the connection)
14:05:04  * ralphtheninjajoined
14:13:24  * ednapiranhajoined
14:13:36  * st_lukejoined
14:16:03  * chirinojoined
14:34:07  * no9quit (Ping timeout: 264 seconds)
14:40:38  * werlejoined
14:43:53  * mcollinajoined
14:45:48  * no9joined
14:46:44  * mcollina_quit (Ping timeout: 268 seconds)
15:01:41  <mbalho>https://github.com/louischatriot/nedb#
15:08:03  <st_luke>dat logo
15:13:01  <levelbot>[npm] admin@0.0.1 <http://npm.im/admin>: A generic, modular admin interface for node (@hij1nx)
15:14:20  * ednapiranhaquit (Remote host closed the connection)
15:17:34  * thl0_joined
15:17:34  * thl0quit (Read error: Connection reset by peer)
15:18:57  * werlequit (Quit: Leaving.)
15:22:00  * ednapiranhajoined
15:25:39  * mreviljoined
15:27:54  * julianduquejoined
15:29:33  * mcollina_joined
15:31:26  * mcollinaquit (Read error: Operation timed out)
15:35:29  <levelbot>[npm] admin@0.0.1 <http://npm.im/admin>: A generic, modular admin interface for node (@hij1nx)
15:39:29  <levelbot>[npm] admin@0.0.1 <http://npm.im/admin>: A generic, modular admin interface for node (@hij1nx)
15:55:12  * julianduquequit (Ping timeout: 256 seconds)
16:15:59  <levelbot>[npm] admin@0.0.1 <http://npm.im/admin>: A generic, modular admin interface for node (@hij1nx)
16:23:55  * chirinoquit (Ping timeout: 264 seconds)
16:25:47  * chirinojoined
16:29:10  * redidasjoined
16:34:32  * no9quit (Ping timeout: 260 seconds)
16:35:24  <redidas>Anyone available for a level-up question ?
16:36:05  <redidas>I'm curious about when/if I should close leveldb - the github readme for levelup says to close it whenever its not needed to save resources
16:36:45  <redidas>is it recommended to close it after every request? Or would that produce too much overhead?
16:39:11  * mcollina_quit (Ping timeout: 260 seconds)
16:46:28  * no9joined
16:49:34  * mcollinajoined
17:11:42  * ralphtheninjaquit (Quit: leaving)
17:36:59  * Acconutjoined
17:43:15  * ednapiranhaquit (Remote host closed the connection)
17:44:32  * ednapiranhajoined
17:49:33  * Acconutquit (Quit: Acconut)
17:51:58  * timoxleyjoined
17:58:05  * Acconutjoined
18:07:29  <levelbot>[npm] admin@0.0.1 <http://npm.im/admin>: A generic, modular admin interface for node (@hij1nx)
18:07:31  * thl0_quit (Remote host closed the connection)
18:08:08  * thl0joined
18:09:19  * thl0quit (Remote host closed the connection)
18:28:00  * mrevilquit (Remote host closed the connection)
18:29:06  * julianduquejoined
18:39:43  * thl0joined
18:40:32  * timoxleyquit (Quit: Computer has gone to sleep.)
18:47:54  * thl0quit (Ping timeout: 240 seconds)
18:52:16  * Acconutquit (Quit: Acconut)
18:54:17  * thl0joined
19:04:08  * no9quit (Ping timeout: 246 seconds)
19:07:25  * mcollinaquit (Read error: Connection reset by peer)
19:07:47  * julianduquequit (Ping timeout: 240 seconds)
19:11:54  * missinglinkquit (Ping timeout: 268 seconds)
19:17:31  * no9joined
19:18:15  * werlejoined
19:28:37  * mreviljoined
19:28:49  * werlequit (Quit: Leaving.)
19:33:41  <chapel>redidas: I don't see why you would close it after each request, that would mean opening it on each request
19:33:56  * mrevilquit (Remote host closed the connection)
19:34:11  * mreviljoined
19:44:44  <st_luke>you're microoptimizing
19:45:00  * st_lukequit (Remote host closed the connection)
19:51:59  <levelbot>[npm] meatspace-leveldb@0.0.3 <http://npm.im/meatspace-leveldb>: Decentralized micro[b]logging with leveldb (@ednapiranha)
19:55:29  <redidas>thanks chapel (and st_luke even though you're gone)
19:55:38  <redidas>I kind of figured that was a micro-optimization
19:55:57  <redidas>but I don't really know when I'd want to close it then
19:56:37  <chapel>redidas: if you're using level up I think you can ignore that, if you use leveldown then you need to close it properly
19:56:58  <redidas>good to know
19:57:18  <redidas>I was almost contemplating breaking up my leveldb into a bunch of dbs... almost like partitioning it by client
19:57:29  <redidas>and then just opening the one that has the client data the app is looking at
19:57:54  <redidas>but then that meant that if a bunch of clients made requests, I'd be opening a bunch of leveldbs
19:58:10  <redidas>So 1 leveldb always open it is!
20:00:29  <levelbot>[npm] meatspace-leveldb@0.0.3 <http://npm.im/meatspace-leveldb>: Decentralized micro[b]logging with leveldb (@ednapiranha)
20:09:08  * julianduquejoined
20:24:30  * juliandu1uejoined
20:24:38  * julianduquequit (Disconnected by services)
20:24:43  * juliandu1uechanged nick to julianduque
20:24:51  * julianduquequit (Changing host)
20:24:51  * julianduquejoined
21:05:53  * mcollinajoined
21:14:42  * ednapiranhaquit (Remote host closed the connection)
21:16:04  * no9quit (Read error: Operation timed out)
21:27:15  * ralphtheninjajoined
21:31:08  * no9joined
21:33:14  * ednapiranhajoined
21:53:19  * ednapiranhaquit (Remote host closed the connection)
22:06:00  * piklujoined
22:13:05  * pikluquit (Remote host closed the connection)
22:31:23  * thl0quit (Remote host closed the connection)
22:45:38  * ednapiranhajoined
22:47:19  <rvagg>redidas: look at http://npm.im/level-sublevel as an alternative to having multiple separate stores, if you can fit them in the same one then that's generally a better choice
22:57:57  * mcollinaquit (Remote host closed the connection)
23:04:38  <redidas>rvagg: interesting - I was sort of planning on doing something like that manually - all of my keys were going to be prefixed with "client~theclientid~"
23:05:05  <rvagg>redidas: yeah, use sublevel, it'll do it safer for you and give you a full levelup interface for each subdb
23:05:13  <redidas>sublevel will make implementing it a lot cleaner
23:05:22  <rvagg>redidas: lots of work going into sublevel at the moment, it's becoming something of a defacto standard for extending levelup
23:05:32  <redidas>cool
23:08:35  <redidas>rvagg: some of my data is kind of hierarchical - I'm kind of tempted to have part of that hierarchy in my key for range queries.
23:08:52  <redidas>rvagg:¬†Would it be better to handle that in a separate index type structure? Or are wider keys okay?
23:09:17  <rvagg>redidas: totally, you need to make sure your keys are as useful as possible for range queries
23:09:37  <rvagg>redidas: leveldb is quite good with long keys, it does some minimal duplication-reduction so don't worry about key length
23:09:46  <rvagg>it's common to append multiple variables to a key
23:10:02  * juliandu1uejoined
23:10:05  <redidas>Good to hear
23:10:07  * julianduquequit (Disconnected by services)
23:10:15  * juliandu1uechanged nick to julianduque
23:10:20  * julianduquequit (Changing host)
23:10:20  * julianduquejoined
23:10:41  <redidas>Leveldb is my first actual "nosql" database I'm using to build something
23:10:54  <rvagg>redidas: just be careful with your choice of separators
23:11:08  <redidas>I've looked a lot at others and read about them, but leveldb sucked me in
23:11:10  <rvagg>redidas: http://dailyjs.com/2013/05/03/leveldb-and-node-2 see the bottom section of this article for a discussion about that
23:11:38  <rvagg>redidas: it's the sorting in leveldb that's the main win, makes queries really powerful
23:11:42  * ednapiranhaquit (Remote host closed the connection)
23:11:58  * thl0joined
23:12:08  <rvagg>redidas: and don't be afraid to make secondary indexes if you need, just additional keys pointing to the original value, it's not too expensive to do double-lookups
23:12:26  <rvagg>redidas: https://github.com/rvagg/node-level-mapped-index that kind of thing, tho it's not too hard to write your own
23:12:45  <redidas>Thanks - I saw that article a while ago
23:12:53  <redidas>I'm totally going with¬†\x00
23:14:34  <redidas>Thanks rvagg - this is all helpful info
23:14:44  <rvagg>cool
23:14:55  <redidas>I wasn't anticipating having to give this much thought to my keys, but they're turning out to be quite powerful
23:15:19  <rvagg>redidas: this is a good article by luke too: http://luke.xxx/post/52916123542/lexicographical-key-sorting-in-leveldb
23:15:28  <redidas>yep saw that too
23:15:41  <rvagg>you've done your research then!
23:15:45  <redidas>been stalking levelup for a while :)
23:16:12  <redidas>but I'm on windows... so I had to wait a bit
23:18:05  <redidas>its been really great so far. Extremely pleased and amazed with all the work everyone's been putting into it
23:46:38  * ChrisPartridgejoined
23:48:21  * no9quit (Ping timeout: 248 seconds)
23:53:22  * thl0quit (Remote host closed the connection)