00:09:19  * thl0joined
00:12:54  * rescrvquit (Read error: Connection reset by peer)
00:13:31  <eugeneware>Tokutek posted this press release saying that they've forked and fixed mongodb. Called TokuMX. Anyone played this this? http://www.tokutek.com/products/tokumx-for-mongodb/?org=321&lvl=1&ite=435&lea=68113&ctr=0&par=1
00:15:45  * eugenewarequit (Quit: Leaving.)
01:05:51  <wolfeidau>Seems people are checking out my leveldb connect session store which is awesome
01:20:50  <chilts>wolfeidau: cool ... I can't see download numbers at the moment though
01:21:06  <chilts>I guess this doesn't quite work if you load balance across servers or use cluster though?
01:21:14  <chilts>since LevelDB is one process only?
01:21:26  <wolfeidau>chilts: Yeah i need to do a post on that
01:21:42  <chilts>yeah
01:21:46  * werlequit (Quit: Leaving.)
01:21:53  <chilts>might be worth a couple of lines in the README?
01:22:06  <wolfeidau>Yeah I will do that
01:22:30  <wolfeidau>TBH i would prefer to run the cluster with shared nothing
01:22:37  <wolfeidau>And sticky load balancer
01:22:51  <wolfeidau>And deal with the reauth on failure
01:23:11  <chilts>interesting :) I prefer it the other way where you share the session (somewhere) but the servers contain no state!
01:23:41  <wolfeidau>yeah there are trade offs with both approaches, but the shared session store is a single point of failure
01:23:56  <wolfeidau>If one of those processes goes rogue all your users can't login
01:24:25  <chilts>wolfeidau: btw, this means no shared state and no sticky sessions : https://npmjs.org/package/client-sessions
01:25:05  <chilts>I've done shared sessions with a cluster of memcache instances before, so if one instance goes down (say 1 of 4), then just 25% of people need to reauthenticate
01:25:11  <chilts>worked quite well
01:25:11  <wolfeidau>yeah I have tried a few approaches, I am looking forward to the clustered leveldb :)
01:25:29  <chilts>ah ... so when I've finished MoDB, that'll work for you :D
01:25:31  <chilts>heh
01:25:38  <wolfeidau>haha :)
01:25:45  <wolfeidau>dynamo me :)
01:26:12  <wolfeidau>But yeah at the moment I just wanted a small embedded session store
01:26:33  <wolfeidau>Which can do N+1 failure
01:26:46  <wolfeidau>So two processes, if one fails the other takes over
01:26:59  <wolfeidau>Leveldb can cater for that
01:27:53  <chilts>yeah, I love how things like sessions (or anything for that matter) is somewhere where we have lots of choices to be able to make great choices in whatever circumstance we're in
01:28:47  <chilts>once you know the trade-offs
01:34:50  <rvagg>wolfeidau: did you notice there were already a few session stores in npm? https://github.com/kevinohara80/connect-leveldb https://github.com/davidbanham/connect-level https://github.com/rvagg/node-level-session
01:35:53  <wolfeidau>rvagg: well the first one isn't in npm
01:36:00  <wolfeidau>As i have that name now
01:36:08  <wolfeidau>And the other ones are called level
01:36:16  <wolfeidau>And i didn't search for that lol
01:36:20  <chilts>:)
01:37:19  <wolfeidau>npm ftl
01:37:40  <wolfeidau>Search for connect leveldb
01:37:54  <wolfeidau>nothing but my module comes up
01:38:04  <rvagg>anyway, put it on https://github.com/rvagg/node-levelup/wiki/Modules that's where you should find em
01:38:42  <wolfeidau>O well we have another one now :)
01:38:52  <rvagg>yeah, that's cool, the more the merrier!
01:39:15  <wolfeidau>I learnt a lot writing it
01:39:31  <wolfeidau>I ran into the same issue as i did a while ago
01:39:40  <wolfeidau>Key expiration
01:40:34  <chilts>yeah, 'coz you gotta do it yourself rather than your store doing it for you?
01:40:46  <wolfeidau>Need to see if there is an elegant way of managing it
01:40:52  <wolfeidau>chilts: Yeah that is what i did
01:41:06  <chilts>yeah, I figured from your comment on the readme (lazy expiration)
01:41:07  <wolfeidau>But I didn't want to add a backgound "job" at this stage
01:41:11  <chilts>true
01:41:25  <chilts>easy enough to add if you get shitloads of sessions
01:41:33  <chilts>esp. short ones (robots for example)
01:41:41  <wolfeidau>Yeah key is you need to have churn of users to make it bad
01:41:43  <chilts>I don't like how connect creates a session on every request
01:41:51  <wolfeidau>aha yeah
01:41:57  <chilts>I want to write some session middleware which doesn't do that
01:42:11  <chilts>(once I have time and $7m so I can do what I like)
01:42:15  <chilts>$8m is also fine
01:44:04  <wolfeidau>haha
01:44:57  <wolfeidau>are you back in NZ?
01:49:17  <rvagg>wolfeidau: http://github.com/rvagg/level-ttl
01:49:23  <chilts>yeah, in Wgtn - got back yesterday morning
01:49:28  <rvagg>err
01:49:29  <rvagg>https://github.com/rvagg/node-level-ttl
01:49:48  <chilts>heh, you have a level plugin for every occassion! :)
01:50:00  <rvagg>read the wiki is the new rtfm
01:50:14  <wolfeidau>rvagg: Sweet will add that tonight
01:52:09  * werlejoined
01:52:21  <chilts>rvagg: so with the way htese level-* plugins work, they all seem to do things like override .get(), .put() etc ... so does that mean if you use a few of them, they're all essentially wrapping the next (which eventually wraps the LevelDB one)?
01:52:42  <chilts>if that's right, I wonder if we'll start seeing subtle bugs appearing with interactions between some of them
01:52:51  <rvagg>chilts: yerp, indeed, it's not ideal
01:52:59  <rvagg>see the issues list for a bunch of discussions about "fixing" it
01:53:09  <chilts>wonder what a better way is ... ahah, you're onto it
01:53:25  <chilts>in some ways I like say adding new functionality, rather than wrapping existing
01:53:39  <chilts>e.g. level.setWithExpiry() (or something)
01:54:19  * ralphtheninjaquit (Ping timeout: 246 seconds)
01:56:16  <brycebaril>rvagg so the internal scan limits the TTL resolution?
01:56:19  * werlequit (Ping timeout: 252 seconds)
01:57:09  <brycebaril>The way Redis does it is pretty interesting, the internal scan only checks a portion of the keys with TTL set, but every lookup checks the TTL
01:57:37  <brycebaril>So that when you inspect a key with a TTL the TTL is always respected, even if the internal expiration cycle didn't clean it.
01:57:45  <brycebaril>And the internal cycle doesn't have to work as hard
02:07:26  <rvagg>brycebaril: I considered that but decided that I didn't want to slow down get()s just to honour exact ttl, it's not important enough for anything I need it for; I'd be interested to hear if anyone else has more precise needs tho
02:07:36  <rvagg>I guess that could be an option you could turn on if you needed it
02:08:22  <rvagg>chilts: the way I've been writing my extensions allows for that: https://github.com/rvagg/node-level-ttl/blob/master/level-ttl.js#L221-L223
02:08:24  <brycebaril>Yeah, it probably comes down to how many keys one has set with a TTL and how often they look at them.
02:08:37  <rvagg>if you pass in a 'methodPrefix' option then you can have your cake and eat it too
02:08:41  <rvagg>... or something like that
02:09:21  <rvagg>brycebaril: the timer resolution could be set pretty low though and it wouldn't impact too much on performance, the ttl keys are sorted by time so the iterator does all the work of finding the ones that need to expire and the ones that don't need to expire are completely ignored
02:09:34  <rvagg>brycebaril: most of the time it's an empty iterator, which isn't trivial but isn't too expensive either
02:12:24  * eugenewarejoined
02:12:45  <brycebaril>true -- Redis doesn't have that luxury, I don't believe it indexes TTLs, just knows which keys have a TTL set
02:12:51  <chilts>rvagg: that's awesome :)
02:25:54  * thl0quit (Remote host closed the connection)
03:44:36  * ChrisPartridgeClchanged nick to ChrisPartridge
03:52:23  * werlejoined
03:56:28  * werlequit (Ping timeout: 246 seconds)
04:48:48  <ChrisPartridge>anyone have any thoughts on storing small files in leveldb?
04:49:15  <rvagg>ChrisPartridge: just do it direct, or https://github.com/juliangruber/level-store if you want to stream them
04:49:32  <ChrisPartridge>ah sweet
04:50:08  <ChrisPartridge>rvagg: exactly what i need
04:50:27  <rvagg>yeah, that's a cool extension
04:50:51  <ChrisPartridge>working on storing mail from haraka -> leveldb
04:50:55  <ChrisPartridge>https://github.com/cjpartridgeb/node-haraka-leveldb
04:51:30  <rvagg>what's haraka?
04:51:38  <ChrisPartridge>node smtp server
04:51:52  <rvagg>ohhhh, nice
04:52:55  <ChrisPartridge>working quite well, probably going to put something in production with postgres for the host/mailbox/acls - then leveldb for storing messages and now attachments
04:58:24  * no9quit (Read error: Operation timed out)
05:01:26  <ChrisPartridge>rvagg: will level-store operate with level-sublevel?
05:06:27  <chilts>ChrisPartridge: that sounds cool! I like the idea of Haraka+LevelDB
05:07:15  <mbalho>nice
05:07:20  <mbalho>i wrote a haraka couchdb thing a long time ago
05:07:40  <mbalho>couldnt find anywhere to host it though but maybe openshift lets you receive incoming mail
05:10:03  <ChrisPartridge>mbalho: this one https://github.com/maxogden/haraka-couchdb ?
05:10:40  <mbalho>yep
05:11:12  <ChrisPartridge>yeah, that helped me immensely
05:11:27  <mbalho>ah cool!
05:12:13  <ChrisPartridge>initially wrote is postgres https://github.com/cjpartridgeb/node-haraka-postgres
05:12:27  <ChrisPartridge>although, postgres isn't really a good fit for storing the messages
05:12:35  <mbalho>ah
05:12:53  <mbalho>ChrisPartridge: if you figure out a nice deployment solution let me know, i was keen on ditching gmail for a while but it wasnt practical
05:13:11  <mbalho>ChrisPartridge: but i would still jump ship if there was a paas that let you receive email
05:13:17  <mbalho>ChrisPartridge: i just dont wanna maintain a vps
05:16:14  <chilts>ChrisPartridge: surely Postgres is more ideal than LevelDB, since you can store different fields and query on them more easily
05:16:32  <chilts>(and indeed, many other storage solutions :)
05:19:18  <ChrisPartridge>chilts: I was going to treat leveldb purely as the store, and keep a mailbox table in postgres with important headers(to,from,subject), and a full text index for the bodies
05:21:36  <chilts>cool
05:45:55  * dominictarrjoined
05:47:59  <eugeneware>ChrisPartridge: what's your experience been with Haraka? Do they handle ISP mail sending limits automatically for email deliverability?
05:52:49  * timoxleyjoined
05:55:55  <mbalho>for sending mail reliably you need to use a deliverability service
05:56:00  <mbalho>but for receiving mail haraka is great
05:58:13  <chapel>someone needs to make a full on node.js based imap/pop/smtp server
05:58:32  <chapel>well really, multiple components that can be combined into a full email service
05:59:45  <ChrisPartridge>eugeneware: i havent used it in production yet, but either way I'm sure if you went down that path, you could simply grab the response from your ISP and tell haraka to queue up for X amount of time
06:00:08  <eugeneware>we're looking at sendgrid/Amazon SES/mailgun etc… but would be awesome if we could do it ourselves. There's a lot of blackmagic with email deliverability that I'd rather not have to deal with.
06:00:14  <ChrisPartridge>chapel: i found an imap-server project on github, thought yay, opened it, empty
06:00:35  <eugeneware>ChrisPartridge: Yeah. The plugin system looks awesome.
06:00:50  <eugeneware>ChrisPartridge: Was wondering if they had some plugins that managed the dark arts for me :-)
06:59:12  * timoxleyquit (Ping timeout: 252 seconds)
07:16:37  * dominictarrquit (Quit: dominictarr)
07:21:02  * wolfeidauquit (Remote host closed the connection)
07:47:59  * timoxleyjoined
08:02:44  * wolfeidaujoined
08:17:48  * dominictarrjoined
08:22:27  * mcollinajoined
08:32:40  * no9joined
08:38:09  * babof1tosjoined
08:38:45  * b4bofitosquit (Ping timeout: 252 seconds)
08:42:11  * nathan7quit (Ping timeout: 252 seconds)
09:00:53  * no9quit (Ping timeout: 248 seconds)
09:14:18  * no9joined
09:26:01  <dominictarr>rvagg: you there?
09:31:58  * no9quit (Ping timeout: 276 seconds)
09:34:49  * eugeneware1joined
09:37:00  * eugenewarequit (Ping timeout: 240 seconds)
09:39:00  * eugeneware1quit (Ping timeout: 240 seconds)
09:45:03  * no9joined
09:47:13  <dominictarr>hij1nx: hey, whats up?
09:56:33  * dominictarrquit (Quit: dominictarr)
09:57:01  * dominictarrjoined
09:59:55  * dominictarrquit (Client Quit)
10:04:09  * ralphtheninjajoined
10:11:49  * wolfeidauquit (Remote host closed the connection)
10:20:07  * wolfeidaujoined
10:21:31  * timoxleyquit (Quit: Computer has gone to sleep.)
10:26:31  * thl0joined
10:27:18  * timoxleyjoined
10:31:50  * nathan7joined
10:33:26  * rescrvjoined
10:35:50  * mcollinaquit (Remote host closed the connection)
10:48:28  * dominictarrjoined
10:51:54  * mcollinajoined
11:04:33  * timoxleyquit (Quit: Computer has gone to sleep.)
11:05:30  * timoxleyjoined
11:28:08  * werlejoined
11:28:49  <thl0>dominictarr: I guess there is no way to insert in one transaction across sublevels?
11:29:13  <thl0>I mean it's still same database, but sublevel currently has no feature like that
11:30:04  <thl0>dominictarr: https://github.com/thlorenz/valuepack-mine-github/blob/86352c5f0dcf5381f32f4f84d4531b826a7d50e2/lib/store-github-repos.js#L75
11:30:31  <dominictarr>thl0: no, it does!
11:30:41  <thl0>here I'm basically queuing up batch and puts across different sublevels, but ideally they should all be one transaction
11:30:58  <dominictarr>in a batch you can do {…,prefix: subDb},
11:31:02  <thl0>dominictarr: ah yes, I forgot, can't I override sublevel
11:31:04  <thl0>yes
11:31:09  <thl0>thanks - forgot
11:31:10  <thl0>:)
11:31:40  <dominictarr>if it wasn't for that, sublevel wouldn't be half as useful
11:32:20  <thl0>I'll change that code right now, lets see how it works :)
11:37:24  * wolfeidauquit (Remote host closed the connection)
11:40:21  <thl0>dominictarr: much better now! https://github.com/thlorenz/valuepack-mine-github/blob/b22862f2b221d145bc154184dbbe845061f43833/lib/store-github-repos.js#L95-L108
11:40:31  <thl0>batch ftw \o/
11:49:33  <dominictarr>yup!
11:51:42  * chirinojoined
11:54:44  * chirinoquit (Client Quit)
12:11:55  * wolfeidaujoined
12:12:25  * timoxleyquit (Quit: Computer has gone to sleep.)
12:15:05  * timoxleyjoined
12:18:17  * Acconutjoined
12:21:46  * Acconutquit (Client Quit)
12:54:00  * rescrvquit (Read error: Connection reset by peer)
12:54:22  * thl0quit (Remote host closed the connection)
13:05:17  <rvagg>dominictarr: pong, need anything or did you want me about the issue you filed on level-ttl?
13:08:04  <dominictarr>yeah, I've just been thinking about how to progress on that stuff
13:09:05  <rvagg>ok, not in the right headspace atm, just got back from 6 hours on the road
13:09:20  <rvagg>dominictarr: when are you wanting to do a level* meetup in SF? I'm coming to nodeconf now
13:09:30  <dominictarr>oh, cool!
13:09:35  <dominictarr>when do you arrive?
13:09:49  <rvagg>in on the 24th, out on the 1st
13:10:35  <dominictarr>that is 24th pst?
13:10:43  <rvagg>yeah, I think
13:11:29  <rvagg>actually, a receipt I have says the 23rd
13:11:34  * rvaggdouble-checks
13:11:48  <dominictarr>I nearly missed my flight back on my first trip to California, because I was thinking I left on tuesday
13:12:00  <dominictarr>but actually, I arrived on tuesday, but left on sunday
13:12:11  <dominictarr>must take into account the date line!
13:12:53  <rvagg>mm, getting over there is easier cause you arrive about the same time that you leave
13:14:48  <rvagg>I'm starting a new project very soon and I'm going to use level as the database I think, so I should have my head back in it a little more I hope
13:15:11  <rvagg>particularly the replication stuff, it's a small service but I need the redundancy
13:18:19  <rvagg>ah, it's the 23rd, I get to SF at 1.30pm
13:23:45  <levelbot>[npm] levelgraph@0.5.2 <http://npm.im/levelgraph>: A graph database built on top of LevelUp (@matteo.collina)
13:37:37  <rvagg>dominictarr: let me know when/where/how we can do a level meetup and we can start to promote it to interested people
13:42:42  * niftylettucequit (Quit: Connection closed for inactivity)
13:45:52  * timoxleyquit (Ping timeout: 246 seconds)
13:58:40  <werle>rvagg: that would be so cool
14:01:26  * no9quit (Ping timeout: 256 seconds)
14:08:23  <dominictarr>rvagg: cool, thursday is a good day for a meetup
14:09:02  <dominictarr>so if you arrive on the 23rd, that is a tuesday, so you'll have two days to get over the jetlag
14:09:29  <dominictarr>so, how about the 25th?
14:10:25  <dominictarr>which is also two days before nodeconf, so there is a chance to get over any tax-deductable-hangover before you go get another one
14:13:05  * eugenewarejoined
14:13:58  * no9joined
14:16:23  * thl0joined
14:19:11  * thl0quit (Remote host closed the connection)
14:32:16  * thl0joined
15:01:30  * thl0quit (Remote host closed the connection)
15:09:32  * thl0joined
15:12:30  * mreviljoined
15:15:43  * dominictarrquit (Quit: dominictarr)
15:21:13  <mbalho>rvagg: i can help organize an sf leveldb meetup if you want
15:26:41  * thl0quit (Remote host closed the connection)
15:28:01  * owen1quit (Ping timeout: 248 seconds)
15:28:03  * mrevilquit (Remote host closed the connection)
15:32:48  * owen1joined
15:42:54  * brianloveswordsquit (Excess Flood)
15:45:10  * brianloveswordsjoined
15:51:56  * mreviljoined
16:02:32  * mcollina_joined
16:04:44  <levelbot>[npm] byteup@0.1.1 <http://npm.im/byteup>: Add bytewise as a levelup leveldb encoding (@nharbour, @eugeneware)
16:08:14  <levelbot>[npm] changeset@0.0.4 <http://npm.im/changeset>: Library to diff JSON objects into atomic put and delete operations, and apply change sets to objects. Useful with Levelup/LevelDB object synchronization. (@nharbour, @eugeneware)
16:11:17  * mcollinaquit (Ping timeout: 246 seconds)
16:14:43  <levelbot>[npm] firedup@0.0.11 <http://npm.im/firedup>: A node.js implementation of firebase based on levelup (@nharbour, @eugeneware)
16:16:14  <levelbot>[npm] homer@0.0.7 <http://npm.im/homer>: Dynamic DNS server (@nharbour, @eugeneware)
16:21:13  <levelbot>[npm] jsonquery@0.0.3 <http://npm.im/jsonquery>: MongoDB query language implemented as a node.js Stream (@nharbour, @eugeneware)
16:22:09  * thl0joined
16:22:12  <eugeneware>forgive the levelbot updates all - change my github/npm username to line up with everything else
16:23:44  <levelbot>[npm] levelplus@0.0.4 <http://npm.im/levelplus>: Adds atomic updates to levelup database (@nharbour, @eugeneware)
16:25:44  <levelbot>[npm] levels@0.0.6 <http://npm.im/levels>: leveldb full text search for node.js (@nharbour, @eugeneware)
16:28:44  <levelbot>[npm] msgpackup@0.0.2 <http://npm.im/msgpackup>: Add msgpack as a levelup leveldb encoding (@nharbour, @eugeneware)
16:48:54  * no9quit (Ping timeout: 256 seconds)
16:55:06  * thl0quit (Remote host closed the connection)
16:58:20  * thl0joined
17:07:14  * thl0quit (Remote host closed the connection)
17:09:05  * thl0joined
17:09:49  * thl0quit (Remote host closed the connection)
17:18:36  * mrevil_joined
17:19:03  * mrevilquit (Ping timeout: 252 seconds)
17:24:11  * mrevil_quit (Ping timeout: 260 seconds)
17:44:04  * mreviljoined
17:45:26  * mrevil_joined
17:48:25  * mrevilquit (Ping timeout: 246 seconds)
17:54:59  * mrevil_quit (Ping timeout: 255 seconds)
17:57:10  * Acconutjoined
17:57:50  * Acconutquit (Client Quit)
18:00:10  * thl0joined
18:03:25  * thl0quit (Remote host closed the connection)
18:07:05  * mreviljoined
18:09:52  * thl0joined
18:19:16  * no9joined
18:24:11  * mrevilquit (Remote host closed the connection)
18:29:55  * thl0quit (Remote host closed the connection)
18:31:03  * thl0joined
18:35:27  * thl0quit (Remote host closed the connection)
18:48:35  * mcollina_quit (Remote host closed the connection)
18:52:12  * rescrvjoined
18:56:06  * thl0joined
18:59:10  * mreviljoined
19:10:06  * thl0quit (Remote host closed the connection)
19:12:31  * mrevilquit (Remote host closed the connection)
19:31:49  * b4bofitosjoined
19:33:48  * babof1tosquit (Ping timeout: 252 seconds)
19:40:31  * thl0joined
19:42:47  * thl0quit (Remote host closed the connection)
19:42:54  * thl0joined
19:49:29  * mreviljoined
20:14:17  * Acconutjoined
20:14:21  * Acconutquit (Client Quit)
20:28:45  * werlequit (Quit: Leaving.)
20:32:07  * mcollinajoined
20:43:44  * mrevilquit (Ping timeout: 255 seconds)
20:51:43  * thl0quit (Remote host closed the connection)
21:30:57  * no9quit (Ping timeout: 252 seconds)
21:40:23  * mreviljoined
21:43:26  * no9joined
22:03:09  * brianloveswordsquit (Excess Flood)
22:04:13  * brianloveswordsjoined
22:09:54  * mrevilquit (Remote host closed the connection)
22:57:33  * brianloveswordsquit (Excess Flood)
22:58:14  * brianloveswordsjoined
22:59:58  * thl0joined
23:11:47  * thl0quit (Remote host closed the connection)
23:13:35  * mcollinaquit (Remote host closed the connection)
23:18:43  * timoxleyjoined
23:20:26  * mreviljoined
23:24:48  * mrevilquit (Ping timeout: 252 seconds)
23:37:23  * dominictarrjoined
23:39:31  * CPartridgejoined
23:45:57  * owen1quit (Ping timeout: 256 seconds)
23:47:08  * CPartridgequit
23:47:18  * CPartridgejoined
23:48:43  * thl0joined
23:51:35  * thl0quit (Remote host closed the connection)
23:54:03  * mmckeggjoined