00:01:18  * thloren__quit (Ping timeout: 264 seconds)
00:02:47  * thlorenz_joined
00:03:58  * paulfryzelquit (Ping timeout: 256 seconds)
00:03:58  * thlorenz_quit (Read error: Connection reset by peer)
00:04:14  * thlorenzquit (Remote host closed the connection)
00:35:00  * jerrysvquit (Remote host closed the connection)
00:41:20  <rvagg>wut? https://github.com/basho/leveldb/wiki/Riak-tuning-1
00:42:13  <rvagg>maybe they should just bite the bullet and switch to hyperleveldb and admit defeat on the multi-thread thing
00:42:22  <rescrv>rvagg: it's more subtle than that
00:42:31  <rvagg>I'm sure it is!
00:42:47  <rvagg>and then there's the fact that they run many nodes on a single server
00:42:49  <rescrv>they made a mistake early on having multiple instances of LevelDB
00:43:23  <rescrv>that's been a thorn in their side that's led them toward worse and worse optimizations. They're doing great, but they'll hit a local maxima that's not the maximum perf.
00:44:23  <rescrv>I may be mistaken, but I believe they run all instances in the same process, and use a limited number of threads to service all DBs.
00:45:13  <rescrv>That's where limiting the erlang scheduler count comes in. I suspect it's either contention for CPU time, reduced contention, or both.
00:45:29  <rvagg>doesn't sound like they have too much of a clue either
00:45:48  <rvagg>the OTP certainly isn't a magic bullet for multi-threaded perf
00:46:17  <rescrv>They're sharp guys, but they're just hampered by decisions that seem obvious in hind-sight. OTP?
00:46:47  <rvagg>rescrv: this is a bit of fun: https://gist.github.com/mikeal/5847297 couchdb implemented (sort of) in Node on top of vanilla leveldb
00:46:55  <rvagg>OTP/Erlang
00:47:25  * thlorenzjoined
00:48:31  <rescrv>when researching languages/libraries awhile back I came to the conclusion that Erlang only hampered dist-sys design when you need more than "eventual" behavior
00:48:52  * fallsemoquit (Quit: Leaving.)
00:49:50  <rescrv>It looks like couchup is quite fast. What does it use for the backend? What's level-mutex buy you?
00:51:41  * jcrugzzjoined
00:53:30  * thlorenzquit (Ping timeout: 276 seconds)
00:54:54  <rvagg>level-mutex is a way of doing non-global locks
00:57:50  <rvagg>we benefit from only having a single thread in our node processes for JS but multiple worker threads for I/O, including leveldb, so locking is pretty simple
00:57:50  * thlorenzjoined
00:57:51  <rescrv>leveldb is multithreaded already. I'm just curious what more level-mutex adds
00:57:51  <rvagg>right, it just adds a locking mechanism above so that mikeal can do some crazy queueing stuff that he needs to do in order to stick to the way that couchdb does its crazy revisioning thing
00:58:40  <rescrv>ah
00:58:59  * thlorenz_joined
00:59:07  * thlorenzquit (Read error: Connection reset by peer)
01:01:07  * thlorenz_quit (Read error: Connection reset by peer)
01:01:22  * thlorenzjoined
01:01:42  <rescrv>is it so you can do an atomic read/write?
01:02:52  * thlorenz_joined
01:02:52  * thlorenzquit (Read error: Connection reset by peer)
01:05:04  * thlorenz_quit (Read error: Connection reset by peer)
01:12:20  * timoxleyjoined
01:12:42  * thlorenzjoined
01:15:58  <rvagg>something like that -- it's to do with the revision trees I think, write this entry if it's the latest revision
01:16:23  <rvagg>actually, him and dominictarr recorded a chat about locking recently that was interesting cause it covered a lot of what he's doing in couchup
01:16:24  * rvaggfinds
01:16:36  <rvagg>http://futurealoof.com/posts/fa-podcast-nodebases-chat-replication-w-maxogden-rvagg-mikea.html
01:16:45  <rvagg>it's actually about replication but he talks a bit about the locking stuff
01:17:05  <rescrv>rvagg: thanks
01:20:02  <rvagg>rescrv: have you seen anything like this? https://github.com/dominictarr/npmd/issues/10
01:20:08  <rvagg>"Corruption: missing start of fragmented record(2)"
01:20:47  <rvagg>as in, I'm pretty sure thats that's from the sst reader but we have no idea how you'd trigger that
01:20:56  <rvagg>discussion continues on https://github.com/rvagg/node-leveldown/issues/52
01:20:59  <rescrv>rvagg: is that issue 194/197 on leveldb's bug tracker?
01:21:04  <rescrv>oh, 52
01:21:23  <rescrv>there's a few LevelDB issues. I've yet to reproduce locally, otherwise I'd fix it.
01:22:00  <rescrv>rvagg: if you or others have a LevelDB instance with that error, I'm happy to take a copy of the DB and find a fix.
01:22:01  * wilmoore-dbjoined
01:22:05  <rvagg>I wondered if it was 197, but I really have no idea
01:22:55  <rvagg>cool, can you contribute on the npmd issue and say that? I'm pretty sure there's nothing sensitive going into those dbs so a copy ought to be easy enough to provide
01:28:13  <rescrv>rvagg: GitHub notifications are flakey for me. If someone follows up and you notice, a poke on IRC would be much appreciated.
01:28:23  <rescrv>In return, I'll do my best to come up with a fix.
01:28:58  <rvagg>ok, cool
01:40:12  * eugenewarejoined
01:40:43  * jxsonquit (Remote host closed the connection)
01:56:26  * dguttmanquit (Quit: dguttman)
02:12:52  * ralphtheninjajoined
02:26:32  <prettyrobots>I'm using a read/write lock to do autmic writes.
02:26:39  <prettyrobots>atomic writes.
02:26:53  <prettyrobots>https://github.com/bigeasy/sequester
02:27:14  * thloren__joined
02:27:53  <prettyrobots>It's read/write and with it I'm able to implement latches, countdown latches, mutexes (natch) and downgradable exclusive locks.
02:28:01  <prettyrobots>It's how I control concurrency in Strata.
02:28:07  * thloren__quit (Read error: Connection reset by peer)
02:28:26  <prettyrobots>But, not using Sequester exactly, yet. I've extracted Sequester. It is trivial and flexible.
02:33:06  <prettyrobots>https://github.com/bigeasy/sequester/tree/master/t/sequester
02:34:30  * dguttmanjoined
02:37:46  * timoxleyquit (Remote host closed the connection)
02:41:53  * prettyrobotspart ("Textual IRC Client: www.textualapp.com")
02:45:08  * prettyrobotsjoined
02:51:46  * timoxleyjoined
03:02:30  <rvagg>rescrv: http://futurealoof.com/posts/fa-podcast-nodebases-chat-locks-and-mutexes-w-dominictarr-mikea.html
03:02:44  * eugenewa_joined
03:02:47  <rvagg>rescrv: that's the link I meant to give you, that one IS about locks and mutexes, forget the other one, just me max and mikeal rambling about crap
03:02:52  <rvagg>sorry
03:03:51  * dguttmanquit (Quit: dguttman)
03:07:10  * eugenewarequit (Ping timeout: 268 seconds)
03:18:42  <chilts>heh, no wonder I couldn't find it in #polyhack ... it was here I clicked that previous one :)
03:31:23  * thlorenz_joined
03:36:00  * thlorenz_quit (Ping timeout: 276 seconds)
04:02:42  * thlorenz_joined
04:07:18  * thlorenz_quit (Ping timeout: 264 seconds)
04:15:46  * dguttmanjoined
04:19:19  * timoxleyquit (Remote host closed the connection)
04:22:40  * timoxleyjoined
04:29:57  * st_lukejoined
04:40:12  * st_lukequit (Remote host closed the connection)
04:50:31  * eugenewa_quit (Remote host closed the connection)
04:51:37  * eugenewarejoined
05:03:07  * thlorenz_joined
05:08:18  * thlorenz_quit (Ping timeout: 276 seconds)
05:55:55  * fallsemojoined
06:03:35  * thlorenz_joined
06:05:56  * mcollinajoined
06:08:20  * thlorenz_quit (Ping timeout: 256 seconds)
06:13:24  * fallsemoquit (Ping timeout: 268 seconds)
06:21:09  * jcrugzzquit (Ping timeout: 264 seconds)
06:56:02  * jxsonjoined
07:04:02  * thlorenz_joined
07:06:02  * jxsonquit (Read error: No route to host)
07:09:12  * thlorenz_quit (Ping timeout: 276 seconds)
07:15:45  * mcollinaquit (Remote host closed the connection)
07:18:01  * jcrugzzjoined
07:29:12  * dguttmanquit (Quit: dguttman)
07:35:03  * timoxleyquit (Remote host closed the connection)
07:45:47  * timoxleyjoined
07:56:15  * timoxley_joined
07:59:24  * timoxleyquit (Ping timeout: 256 seconds)
08:04:24  * thlorenz_joined
08:07:06  * timoxley_changed nick to timoxley
08:08:45  * thlorenz_quit (Ping timeout: 245 seconds)
08:23:30  * timoxleyquit (Read error: Connection reset by peer)
08:26:48  * mcollinajoined
08:31:32  * mcollinaquit (Ping timeout: 268 seconds)
09:04:48  * thlorenz_joined
09:09:10  * thlorenz_quit (Ping timeout: 245 seconds)
09:31:17  * alanhoff89joined
09:32:30  * alanhoffquit (Ping timeout: 245 seconds)
09:39:58  * dominictarrjoined
09:51:45  * wolfeidaujoined
09:55:39  * eugenewa_joined
09:55:51  * eugenewa_quit (Read error: Connection reset by peer)
09:58:58  * eugenewarequit (Ping timeout: 256 seconds)
10:05:12  * thlorenz_joined
10:08:21  * alanhoff89quit (Ping timeout: 268 seconds)
10:09:35  * thlorenz_quit (Ping timeout: 245 seconds)
10:22:31  * brycebarilquit (Read error: Connection reset by peer)
10:36:25  * eugenewarejoined
10:46:40  * eugenewarequit (Ping timeout: 245 seconds)
11:05:39  * thlorenz_joined
11:10:21  * thlorenz_quit (Ping timeout: 276 seconds)
11:12:42  * jcrugzzquit (Ping timeout: 264 seconds)
11:39:04  * jcrugzzjoined
11:46:54  * jcrugzzquit (Ping timeout: 264 seconds)
12:06:06  * thlorenz_joined
12:10:48  * thlorenz_quit (Ping timeout: 276 seconds)
12:32:37  * wilmoore-dbquit (Remote host closed the connection)
12:53:00  * eugenewarejoined
13:02:21  * Acconutjoined
13:02:41  * Acconutquit (Client Quit)
13:06:29  * thlorenz_joined
13:10:38  * thlorenz_quit (Ping timeout: 240 seconds)
13:15:31  * thlorenzquit (Remote host closed the connection)
13:26:57  * timoxleyjoined
13:30:15  * timoxleyquit (Remote host closed the connection)
13:35:38  * ednapiranhajoined
13:48:02  * thlorenzjoined
13:50:58  * fallsemojoined
13:59:15  * sveisveijoined
14:06:17  * fallsemoquit (Quit: Leaving.)
14:24:54  * fallsemojoined
14:26:36  * thlorenzquit (Remote host closed the connection)
14:31:56  * tmcwjoined
14:37:34  * brycebariljoined
14:45:57  * dguttmanjoined
15:04:05  * paulfryzeljoined
15:38:16  * wilmoore-dbjoined
15:44:20  * dominictarrquit (Quit: dominictarr)
15:56:41  * mcollinajoined
16:04:14  * mcollinaquit (Remote host closed the connection)
16:19:19  * jcrugzzjoined
16:35:04  * tmcwquit (Remote host closed the connection)
16:35:54  * mcollinajoined
16:46:06  * mcollinaquit (Ping timeout: 268 seconds)
16:46:06  * no9quit (Ping timeout: 268 seconds)
16:48:39  * tmcwjoined
16:55:21  * mikealjoined
16:57:46  * no9joined
17:09:10  * no9quit (Ping timeout: 276 seconds)
17:12:20  * mcollina_joined
17:19:18  * jerrysvjoined
17:21:39  * no9joined
17:26:01  * tmcwquit (Remote host closed the connection)
17:28:21  * tmcwjoined
17:31:02  * mcollina_quit (Remote host closed the connection)
17:53:22  * mcollinajoined
17:54:03  * mcollinaquit (Read error: Connection reset by peer)
17:54:41  * Acconutjoined
17:55:06  * Acconutquit (Client Quit)
17:58:22  * mcollinajoined
18:00:19  * jxsonjoined
18:01:07  * dominictarrjoined
18:09:08  * mcollinaquit (Ping timeout: 256 seconds)
18:12:19  * Acconutjoined
18:14:01  * mikealquit (Quit: Leaving.)
18:14:48  * mikealjoined
18:32:49  * mikealquit (Quit: Leaving.)
18:37:50  * thlorenzjoined
18:38:07  * paulfryz_joined
18:41:02  * paulfryzelquit (Ping timeout: 240 seconds)
18:53:29  * jxsonquit (Remote host closed the connection)
18:55:51  * jxsonjoined
18:55:58  * jxsonquit (Remote host closed the connection)
18:56:28  * jxsonjoined
18:57:32  * jxsonquit (Remote host closed the connection)
18:58:05  <prettyrobots>Mumur3 32-bit if anyone here is doing mad checksums like I am: https://github.com/bigeasy/hash/blob/master/hash.murmur3.32/index.js
18:58:06  <prettyrobots>s/Mumur/Murmur/
18:58:12  * jxsonjoined
19:01:28  <mbalho>that is awesome
19:03:08  <prettyrobots>mbalho: Glad you like. I'm going to try to do one a day.
19:03:51  <prettyrobots>Then benchmark the different algorithms and compare them to the openssl crypto hashes in Node.js.
19:05:56  <brycebaril>Is cityhash on your list? I've heard it is pretty slick
19:06:07  * mikealjoined
19:19:33  * jerrysvquit (Remote host closed the connection)
19:23:44  <prettyrobots>Yes.
19:23:55  <prettyrobots>CityHash, Jenkins and the Murmurs.
19:24:27  <prettyrobots>brycebaril:
19:28:20  <brycebaril>Cool, I'll keep an eye out for the benchmarks.
19:43:21  * Acconutquit (Ping timeout: 264 seconds)
19:43:58  * mikealquit (Quit: Leaving.)
19:51:24  * thlorenzquit (Remote host closed the connection)
19:53:23  * paulfryz_quit (Remote host closed the connection)
20:22:29  * thlorenzjoined
20:24:48  * Acconutjoined
20:30:48  * thlorenzquit (Ping timeout: 256 seconds)
20:31:52  <rescrv>dominictarr: saw the live specimen of a corrupt db. I'll dig into it when I get a chance. Probably not until the weekend.
20:32:15  <dominictarr> it had a different error to the one posted
20:54:15  * ednapiranhaquit (Remote host closed the connection)
20:57:33  * thlorenzjoined
21:02:30  * thlorenzquit (Ping timeout: 276 seconds)
21:17:17  * mikealjoined
21:18:39  * Acconutquit (Quit: Acconut)
21:30:25  * thlorenzjoined
21:31:51  * thlorenz_joined
21:31:52  * thlorenzquit (Read error: Connection reset by peer)
21:33:45  * thlorenzjoined
21:33:45  * thlorenz_quit (Read error: Connection reset by peer)
21:34:50  * thlorenzquit (Read error: Connection reset by peer)
21:35:14  * thlorenzjoined
21:36:32  * thlorenz_joined
21:36:32  * thlorenzquit (Read error: Connection reset by peer)
21:39:28  * Acconutjoined
21:39:37  * Acconutquit (Client Quit)
21:41:16  * thlorenz_quit (Ping timeout: 264 seconds)
21:48:18  * thlorenzjoined
21:51:34  * thlorenzquit (Read error: Connection reset by peer)
21:51:46  * thlorenzjoined
21:53:11  * thlorenz_joined
21:53:11  * thlorenzquit (Read error: Connection reset by peer)
21:55:55  * thlorenz_quit (Read error: Connection reset by peer)
21:56:12  * thlorenzjoined
21:58:43  * thlorenzquit (Read error: Connection reset by peer)
21:59:04  * thlorenzjoined
22:02:40  * thlorenzquit (Read error: Connection reset by peer)
22:02:42  * thlorenz_joined
22:05:05  * thlorenz_quit (Read error: Connection reset by peer)
22:05:24  * thlorenzjoined
22:07:04  * thlorenz_joined
22:07:05  * thlorenzquit (Read error: Connection reset by peer)
22:10:15  <rescrv>corruption is corruption...
22:10:23  * thlorenz_quit (Read error: Connection reset by peer)
22:10:24  <rescrv>no matter what, I want to eliminate the problem
22:12:28  * thlorenz_joined
22:14:11  * mikealquit (Quit: Leaving.)
22:14:18  * thlorenz_quit (Read error: Connection reset by peer)
22:14:48  * thlorenzjoined
22:17:05  * thlorenzquit (Read error: Connection reset by peer)
22:17:11  * thlorenz_joined
22:21:26  * thlorenz_quit (Ping timeout: 240 seconds)
22:35:50  * wolfeidauquit (Remote host closed the connection)
22:44:23  * mikealjoined
22:52:22  * tmcwquit (Remote host closed the connection)
22:55:25  * fallsemoquit (Quit: Leaving.)
22:58:12  * fallsemojoined
23:02:17  * fallsemoquit (Client Quit)
23:09:19  * jxsonquit (Remote host closed the connection)
23:11:26  * wilmoore-dbquit (Remote host closed the connection)
23:15:58  * jxsonjoined
23:16:30  * dominictarrquit (Quit: dominictarr)
23:17:48  * thlorenzjoined
23:18:50  * thlorenz_joined
23:18:50  * thlorenzquit (Read error: Connection reset by peer)
23:20:45  * jxsonquit (Remote host closed the connection)
23:39:12  * jxsonjoined
23:43:13  * jxsonquit (Ping timeout: 240 seconds)