00:00:00  * ircretaryquit (Remote host closed the connection)
00:00:08  * ircretaryjoined
00:03:54  * kriskowalquit (Quit: kriskowal)
00:24:04  * wiwilliajoined
00:24:19  <wiwillia>hey pkrumins how have you been??
00:28:47  * ITproquit (Ping timeout: 256 seconds)
00:38:21  * mcollinajoined
00:42:46  * mcollinaquit (Ping timeout: 256 seconds)
00:56:51  * mirkokiefer1joined
01:03:02  * mirkokiefer1quit (Remote host closed the connection)
01:03:19  * techjoined
01:03:43  * techchanged nick to Guest65919
01:05:43  <mbalho>isaacs: node question. if i am fs and/or http streamin a 5gb file, and piping the chunks into a database, and when the db is busy keeping a queue of stuff i get from the readstream (which i put into the db when its not busy anymore), AND sending proper backpressure to the read stream, should everything be just dandy?
01:06:04  <mbalho>isaacs: e.g. is there some common gotcha that i will probably experience if my filesize grows
01:06:30  <isaacs>how are you managing when to send backpressure and when to alow the queue to grow?
01:06:42  <isaacs>mbalho: are you using stream.PassThrough or something with a highWaterMark setting?
01:06:45  <mbalho>isaacs: if the db is busy i pause the stream (using through)
01:06:52  <isaacs>"busy"?
01:07:01  <mbalho>isaacs: in the middle of writing a batch of writes to disk
01:07:27  <mbalho>isaacs: with leveldb if you write too much it degrades throughput
01:07:31  <isaacs>i see
01:08:08  <isaacs>mbalho: why not use the appropriate stream base classes, and return false from writes that go over the water mark?
01:08:24  <isaacs>mbalho: if yo'ure using v0.10 or v0.12 anyway
01:08:50  <mbalho>isaacs: through is easier, and still returns false when i want it to
01:09:08  * thlorenzjoined
01:09:16  <isaacs>ok, then
01:09:19  <isaacs>sounds fine
01:09:38  <isaacs>the importnat thing is to return false when you don't want any morewrites to happen, and to emit('drain') when yor'e reading to have writes resume.
01:09:48  <mbalho>yea im doing that
01:09:51  <isaacs>make sure to always emit('drain') in the future if you've returned false from a write
01:09:54  <isaacs>otherwise you die.
01:10:51  <mbalho>isaacs: to what extent will tcp slow down if i have a really slow consumer?
01:10:55  * Guest65919quit (Remote host closed the connection)
01:11:30  <mbalho>isaacs: is it all dependent on the server?
01:11:34  <isaacs>huh?
01:11:54  <isaacs>TCP will go as fast as both parties and the network can handle.
01:12:21  <mbalho>isaacs: like if i always return false on a stream, what does that actually do to tcp throughput
01:12:47  <isaacs>mbalho: don't worry about it.
01:13:06  <isaacs>mbalho: the tcp socket has its own high water mark, so if you're just choppy, it'll smooth out however it needs to
01:13:24  <isaacs>mbalho: if you're actuall going at a slower average speed, then it'll slow down to accomodate
01:13:36  <mbalho>thats what i thought, just wanted to confirm
01:13:47  <mbalho>do all filesystems do the same thing?
01:14:23  <mbalho>its surprisingly hard to find detailed information about backpressure online
01:15:56  <isaacs>backpressure is a tcp concept, not a fs concept
01:16:04  <isaacs>in fs land, you just write() and then you block until it's done
01:16:28  <mbalho>isaacs: so if i dont consume data fast enough from an fs stream node will run out of memory?
01:18:44  <isaacs>no
01:18:55  <isaacs>node kinda coerces fs to look like tcp
01:19:08  <isaacs>which is a lot easier and saner than the reverse
01:24:44  * tech8989joined
01:25:03  * tech8989quit (Remote host closed the connection)
01:25:21  * tech8989joined
01:27:23  <mbalho>isaacs: okay so basically there are no situations in which i should worry about running out of memory if i'm being a good stream citizen
01:28:16  * tech8989quit (Remote host closed the connection)
01:28:31  * tech8989joined
01:28:48  * tech8989quit (Remote host closed the connection)
01:29:06  * tech8989joined
01:32:28  * owen1quit (Ping timeout: 246 seconds)
01:36:18  * tech8989quit (Remote host closed the connection)
01:43:06  * kriskowaljoined
01:44:20  * owen1joined
01:46:44  * mirkokieferquit (Quit: mirkokiefer)
01:48:26  * mirkokieferjoined
01:59:18  * mirkokieferquit (Quit: mirkokiefer)
02:03:53  * defunctzombiechanged nick to defunctzombie_zz
02:14:12  <isaacs>mbalho: correc
02:14:16  <isaacs>mbalho: that is the purpose of streams
02:14:23  * wiwilliaquit (Ping timeout: 250 seconds)
02:14:30  <isaacs>mbalho: to slow down and smooth out data flows so that fast streams don't overwhelm slow ones
02:14:45  <isaacs>(fast readers and slow writers, especially)
02:14:53  <isaacs>if you have a slow source and a fast dest, it's no big deal.
02:15:37  <mbalho>ya
02:15:38  * jcrugzzjoined
02:15:58  * ins0mniaquit (Read error: Operation timed out)
02:27:45  * kumavisjoined
02:43:49  * ednapiranhajoined
02:44:06  * ednapiranhaquit (Remote host closed the connection)
02:53:38  * kumavisquit (Quit: kumavis)
03:08:18  * timoxleyjoined
03:26:37  * kumavisjoined
03:36:30  * timoxleyquit (Remote host closed the connection)
03:40:42  * shamaquit (Remote host closed the connection)
03:43:12  * timoxleyjoined
04:33:21  * thlorenzquit (Remote host closed the connection)
04:42:37  * kumavisquit (Quit: kumavis)
05:02:44  * kumavisjoined
05:48:18  * kriskowal_joined
05:49:23  * kriskowalquit (Ping timeout: 260 seconds)
05:49:23  * kriskowal_changed nick to kriskowal
05:53:43  * kumavisquit (Quit: kumavis)
05:55:48  * kumavisjoined
05:58:45  * kumavisquit (Client Quit)
06:01:55  * kumavisjoined
06:34:55  * kriskowalquit (Quit: kriskowal)
06:44:29  * kumavisquit (Quit: kumavis)
07:06:55  * DTrejojoined
07:09:16  * dominictarrjoined
07:43:42  * DTrejoquit (Remote host closed the connection)
07:47:49  * timoxleyquit (Remote host closed the connection)
07:50:27  * timoxley_joined
08:03:58  * nichola__joined
08:03:59  * nicholas_quit (Read error: Connection reset by peer)
08:09:52  * vitorpachecoquit (Quit: Saindo)
08:10:51  * timoxleyjoined
08:12:38  * timoxley_quit (Ping timeout: 245 seconds)
08:18:17  * defunctzombie_zzchanged nick to defunctzombie
08:32:52  * defunctzombiechanged nick to defunctzombie_zz
08:35:18  * jcrugzzquit (Ping timeout: 264 seconds)
08:46:01  * mirkokieferjoined
09:28:13  * itprojoined
09:28:13  * itprochanged nick to ITpro
09:36:05  * mirkokieferquit (Quit: mirkokiefer)
09:40:00  * mcollinajoined
09:40:11  * mirkokieferjoined
09:41:23  * jcrugzzjoined
09:46:16  * jcrugzzquit (Ping timeout: 245 seconds)
09:47:58  * timoxley_joined
09:48:14  * mcollinaquit (Ping timeout: 240 seconds)
09:50:39  * timoxleyquit (Ping timeout: 260 seconds)
09:58:29  * ITproquit (Ping timeout: 252 seconds)
10:03:46  * mcollinajoined
10:04:47  * timoxleyjoined
10:07:13  * timoxley_quit (Ping timeout: 245 seconds)
10:15:23  * mcollinaquit (Ping timeout: 260 seconds)
11:12:27  * yorickjoined
11:18:46  * itprojoined
11:18:46  * itprochanged nick to ITpro
11:25:36  <daleharvey>any browserify module to pass options via process.argv or document.location.search?
11:46:21  * mirkokieferquit (Quit: mirkokiefer)
11:53:37  <daleharvey>just wrote a quite one @ https://github.com/daleharvey/browserify-getopts, if theres a better way then say
11:53:40  <daleharvey>*quick
12:13:36  * ins0mniajoined
14:00:03  <dominictarr>creationix: I need a framing scheme that works for utf8 strings and for binary
14:01:25  <dominictarr>daleharvey: make it same api as https://github.com/substack/node-optimist
14:02:54  <dominictarr>creationix: I'm thinking it's gonna have to have something to indicate whether it's binary or utf8 because otherwise the "length" is different.
14:04:59  <dominictarr>and you have to get the binary length...
14:08:42  <dominictarr>hmm, … maybe you could have 3 formats: binary, utf8 and ascii
14:09:04  <dominictarr>then, because you can trust the length if there are no extended characters
14:09:22  <dominictarr>so, in that case you could parse it quickly
14:09:43  <dominictarr>and just fallback to checking the characters when there is uft8 involved!
14:09:45  * fallsemojoined
14:20:22  * fallsemoquit (Quit: Leaving.)
14:21:34  <daleharvey>dominictarr: yeh would be nice, but not important for me and have a lot to do, will wait until someone else implements it and will switch :)
14:22:35  <dominictarr>daleharvey: fair enough
14:23:32  * timoxley_joined
14:25:50  * timoxleyquit (Ping timeout: 240 seconds)
14:26:50  * mirkokieferjoined
14:31:50  * kenperkinsjoined
14:46:32  * thlorenzjoined
15:02:25  <daleharvey>hmm, any ideas what would be up with http://pastebin.mozilla.org/2952927 ?
15:02:36  <daleharvey>getting a vague, 'syntax error'
15:05:36  * mirkokieferquit (Quit: mirkokiefer)
15:10:30  <daleharvey>must be some shell encoding issue
15:20:06  <daleharvey>easy fix, just dont echo it
15:27:36  * mirkokieferjoined
15:27:41  * kenperkinsquit (Quit: Computer has gone to sleep.)
15:31:17  * mirkokieferquit (Client Quit)
15:36:52  * missinglinkjoined
15:45:18  * missinglinkquit (Read error: Operation timed out)
15:50:56  * mcollinajoined
15:51:01  <dominictarr>daleharvey: bash strings are stupid
15:51:20  <dominictarr>\ does not escape
15:51:43  <dominictarr>put strings with " inside '
15:51:59  <dominictarr>single quote is "literal"
15:52:06  <dominictarr>with no escaping or substitution
15:52:12  * mcollina_joined
15:52:35  <dominictarr>to put a literal ' you can put it inside of "'" though
15:52:47  * fallsemojoined
15:53:06  <dominictarr>so you can do 'escape a single quote like this:'"'"' string continues'
15:53:23  <dominictarr>since there is no white space between the strings it's one string… (wtf)
15:54:02  * mcollinaquit (Read error: Connection reset by peer)
15:57:00  * fallsemoquit (Ping timeout: 256 seconds)
16:11:30  * kumavisjoined
16:54:32  * kenperkinsjoined
17:19:16  * shamajoined
17:28:47  * timoxley_quit (Remote host closed the connection)
17:30:31  * kenperkinsquit (Quit: Computer has gone to sleep.)
17:30:59  * timoxleyjoined
17:35:02  * timoxleyquit (Remote host closed the connection)
17:45:19  * thlorenzquit (Remote host closed the connection)
17:51:59  * kenperkinsjoined
17:52:21  * kenperkinsquit (Client Quit)
17:58:32  <daleharvey>substack: would you take patch to ignore reporting tests that have null as a name?
17:59:30  * defunctzombie_zzchanged nick to defunctzombie
18:06:42  * kriskowaljoined
18:09:01  * kriskowalquit (Client Quit)
18:10:27  <dominictarr>daleharvey: I doubt he'd take that
18:10:55  <dominictarr>daleharvey: do you want to skip some tests?
18:11:38  * defunctzombiechanged nick to defunctzombie_zz
18:12:03  <mbalho>chrisdickinson: ping, have a weird binary question
18:12:06  <daleharvey>nope, this is for the setup / teardown tests
18:23:21  * kriskowaljoined
18:59:05  * defunctzombie_zzchanged nick to defunctzombie
19:01:57  * i_m_cajoined
19:03:26  <defunctzombie>daleharvey: why not report those tests?
19:03:37  <defunctzombie>setup and teardown are times when you should be testing things too :)
19:04:52  <daleharvey>Its an extra like 4/6 reports per test (each report is 2 lines), I am running > 1500 tests
19:05:03  <daleharvey>gets in the way
19:09:11  * thlorenzjoined
19:09:42  * sveisveiquit (Ping timeout: 264 seconds)
19:12:58  * kumavisquit (Quit: kumavis)
19:17:53  * kumavisjoined
19:20:58  * kriskowalquit (Quit: kriskowal)
19:25:31  <jesusabdullah>nexxy: way to cause a ruckus with Dylan's Mom XD
19:25:46  <jesusabdullah>nexxy: that line about political cartoons is gold though
19:27:50  * kriskowaljoined
19:39:50  * jolissquit (Ping timeout: 240 seconds)
19:41:30  * mirkokquit (Ping timeout: 264 seconds)
19:42:45  * mirkokjoined
19:47:26  * sveisveijoined
19:48:21  * AvianFlujoined
19:48:24  * mikolalysenkojoined
19:53:07  * mikolalysenkoquit (Ping timeout: 256 seconds)
20:04:00  * nicholasfjoined
20:04:00  * nichola__quit (Read error: Connection reset by peer)
20:11:15  <nexxy>jesusabdullah, :)
20:27:44  <chrisdickinson>mbalho: what's up?
20:30:49  <mbalho>chrisdickinson: i was wondering if there would be performance benefits to writing a newline parser using pure buffers... e.g. foreach the buffers and check for newline (0a) and slice + emit the line as a buffer
20:31:36  <chrisdickinson>mbalho: theoretically yes
20:31:42  <mbalho>chrisdickinson: with leveldb in mind so, data comes into node as buffers, gets converted to string, then back to buffer to go into leveldb
20:31:48  * i_m_caquit (Ping timeout: 245 seconds)
20:31:49  <chrisdickinson>oh yeah
20:31:58  <mbalho>chrisdickinson: but if it never had to go into a JS object i bet it would be noticable? just speculation though
20:32:04  <chrisdickinson>keep that as a buffer! plus buffer.slice (or bops.subarray) is zero-copy
20:32:29  <chrisdickinson>my guess is that with proper type hinting you could get that to be *very* fast
20:33:32  <mbalho>chrisdickinson: im storing lots of single level deep objects lately
20:33:37  <mbalho>chrisdickinson: e.g. no objects as values
20:33:45  <mbalho>chrisdickinson: and have been thinking about how to optimize them on disk
20:34:26  <mbalho>chrisdickinson: and it would be nice if i could keep them as buffers but still do things like obj['_id'] = 'abc123'
20:34:52  <chrisdickinson>basically treating them like c structs
20:35:52  <mbalho>haha i guess so, i am a c noob
20:36:13  * chrisdickinsonsearches for the relevant es6/7 proposal + modules
20:36:30  <chrisdickinson>http://wiki.ecmascript.org/doku.php?id=harmony:typed_objects
20:36:38  <mbalho>oh yea
20:38:14  <mbalho>chrisdickinson: have you seen any libraries that map simple object-style data onto buffers with efficiency in mind? msgpack/bson etc all seem too high level
20:39:17  <chrisdickinson>https://npmjs.org/package/reified maybe?
20:39:41  <mbalho>interesting
20:41:30  <chrisdickinson>there's none on top of bops that i know of
20:42:49  <chrisdickinson>i had one that was particularly silly: https://github.com/chrisdickinson/pacman.js
20:43:02  <chrisdickinson>(predating bops)
20:44:19  <mbalho>lol nice
20:44:52  <mbalho>chrisdickinson: alot of what im doing right now is trying to optimize level bulk write performance, the test data ive been using is this 7.5 million row csv of flight on-time data
20:45:28  <chrisdickinson>mbalho: where's the bottleneck at the moment?
20:45:30  <mbalho>chrisdickinson: also in dat i assign uuids to rows and a sequence number
20:45:44  <mbalho>chrisdickinson: well one is the buffer -> string -> object -> buffer cycle
20:45:58  <mbalho>chrisdickinson: there are also leveldb specific ones but i'm making good headway there
20:46:13  <mbalho>https://github.com/maxogden/level-bulk-load/issues/1
20:47:29  <mbalho>chrisdickinson: so im interested in trying to keep the csv data in buffers, and minimize new buffer allocations, while still parsing them and adding new columns and putting them into leveldb
20:47:39  <mbalho>chrisdickinson: then when i read them out i can just turn them into json later
20:48:05  * blobaumjoined
20:48:12  * blobaumquit (Remote host closed the connection)
20:49:01  <chrisdickinson>yeah, you should see a big speedup (potentially) from keeping them as buffers and just doing Buffer#slice
20:49:26  <chrisdickinson>since you're not copying any data anymore, and you'll be generating less garbage
20:49:31  * mikolalysenkojoined
20:50:25  <mbalho>chrisdickinson: i could probably convert this to binary pretty easily https://github.com/AndreasMadsen/rfc-csv/blob/master/csv.js
20:51:12  <mbalho>cause sadly csv isnt as easy as split('\n').join(',')
20:51:37  <chrisdickinson>mbalho: creationix and i have had good results with function-based state machines as parsers lately
20:51:57  <mbalho>chrisdickinson: got an example?
20:52:13  <chrisdickinson>creationix probably has better ones, but, https://github.com/chrisdickinson/sparser/blob/master/index.js#L544-L548
20:52:33  <chrisdickinson>basically you return the state from the current state function called against the current byte
20:53:05  <chrisdickinson>each function is responsible for advancing the index
20:53:46  * mikolalysenkoquit (Ping timeout: 246 seconds)
20:55:54  <kriskowal>mbalho: chrisdickinson: see https://github.com/montagejs/argo/blob/master/argo.js
20:56:29  <kriskowal>particularly, https://github.com/montagejs/argo/blob/master/argo.js#L51
20:57:01  <kriskowal>monadic pushdown automata. it is an interesting idea.
20:58:55  <kriskowal>not shown in that example, because json is easy, but the concept supports arbitrary length lookahead
20:59:24  <kriskowal>creationix wrote a bytewise variant
21:00:23  <kriskowal>here: https://gist.github.com/creationix/5992451
21:01:05  <mbalho>ooh
21:01:43  <dominictarr>prettyrobots_: hi
21:02:14  <chrisdickinson>kriskowal: yep! that's what i was looking for
21:02:25  <kriskowal>mbalho: chrisdickinson a less trivial example is the old FRB parser https://github.com/montagejs/frb/blob/802e613bec4f33c764e848e63d6ba5d980207e14/parse.js
21:02:47  <chrisdickinson>yeah, i was using a non-entirely-there version of that for my inflate implementation
21:03:24  <kriskowal>and all the parsers in my tengwar transcriber use the same concept https://github.com/kriskowal/tengwarjs
21:04:05  <kriskowal>there are some interesting bigs, like a tokenizer or token stream preprocessor have the exact same form
21:04:24  <kriskowal>s/bigs/bits/
21:04:59  <chrisdickinson>kriskowal: so that pattern is called "monadic pushdown automata"?
21:05:15  <kriskowal>that's just a description of what it is. i didn't name it.
21:06:00  <chrisdickinson>cool, just wondering. I had been calling it a "function-based state machine"
21:06:38  <kriskowal>stick with that
21:07:27  <kriskowal>automaton is PL-nerdish for state machine
21:07:47  <kriskowal>pushdown just means that it can parse recursive languages
21:08:25  <kriskowal>monads are loosely speaking the kind of functions it employs
21:09:05  <chrisdickinson>ah, cool. thanks for the explanation!
21:09:22  <kriskowal>state(input) -> new state
21:10:19  <kriskowal>also uses destination-passing-style, which is another pattern i'm seeing a lot in FRP
21:15:03  <dominictarr>mbalho: what is the lower bound for just writing data into node, let a lone level
21:15:26  <dominictarr>I was running your dataset into node yesterday - and not saving it, just writing it
21:15:35  <dominictarr>and it still took quite a while
21:17:20  <mbalho>dominictarr: there are perf hurdles on both sides
21:17:59  <dominictarr>I remember ry tweeted for a JSON.parse(Buffer) a while back
21:18:00  <mbalho>dominictarr: you can get horrible write perf really easily with crypto.randomBytes buffers + leveldb
21:18:12  <dominictarr>he'll love you for ever if you write that
21:18:13  * mirkokieferjoined
21:18:34  <dominictarr>"a while back" == in the old days
21:22:54  * dimadimaquit (Ping timeout: 264 seconds)
21:23:12  * dimadimajoined
21:25:35  * jesusabd1llahjoined
21:25:47  * jesusabdullahquit (Quit: Reconnecting)
21:26:19  * jesusabd1llahchanged nick to jesusabdullah
21:28:02  <mbalho>dominictarr: so for that test data its 120mb bzipped, 700 mb unzipped, 7.5 million rows, takes ~20 seconds to unzip
21:28:24  <mbalho>dominictarr: when unzipped it takes 27 seconds to run all 7.5 million rows through the streaming csv parser im using
21:29:24  <mbalho>dominictarr: and it took 15 minutes to unzip, parser and insert in batches into leveldb in my test yesterday
21:29:51  <mbalho>dominictarr: i think i can improve certain aspects but the bulk of the time is due to leveldb
21:31:05  <dominictarr>hmm, I'm seeing much longer time to parse the json
21:31:32  <mbalho>dominictarr: https://gist.github.com/maxogden/7893606f384dfc6c1d55
21:32:00  <dominictarr>what is true for?
21:32:14  <mbalho>emit headers
21:32:18  <mbalho>doesnst impact perf
21:32:25  <mbalho>only happens once at the beginning of the file
21:32:43  <dominictarr>some say that options: {headers: true}
21:32:50  <dominictarr>are always better than a binary argument
21:33:03  <dominictarr>because the indicate what it actually does
21:33:17  <mbalho>agreed, but i didnt write rfc-csvb
21:34:26  <dominictarr>far enough
21:34:50  * jolissjoined
21:34:52  <dominictarr>s/far/fair/
21:34:56  * mirkokieferquit (Quit: mirkokiefer)
21:35:31  <dominictarr>mbalho: is the flight data in order?
21:35:38  <mbalho>dunno
21:35:58  <mbalho>yea looks liek it is
21:38:03  <dominictarr>hmm, you are really getting into the nuts and bolts then, but you could use an SST
21:38:08  <dominictarr>(sorted string table)
21:38:18  <dominictarr>directly, with no LSMT
21:38:57  <mbalho>dat is general purpose, data wont always be sorted, this data just happens to be
21:39:26  <dominictarr>mbalho: I'm just thinking out loud - bulk loads might be?
21:39:43  <mbalho>yea for replication i can just insert in reverse
21:39:53  <dominictarr>It takes me 1.5 minutes to read the 1988 flights into node
21:40:18  * mcollina_quit (Ping timeout: 264 seconds)
21:40:20  <mbalho>using that script i gave you?
21:41:32  <dominictarr>using an almost identical one
21:42:11  <dominictarr>oh… I'm using ldjson-stream
21:42:17  * mcollinajoined
21:42:44  <mbalho>that just zips the headers and values together and makes new objects
21:43:11  <mbalho>dominictarr: also did you unbzip ahead of time or streaming
21:43:28  <dominictarr>I'm just reading the json lines
21:43:46  <dominictarr>oh, the lines of json.
21:44:34  <dominictarr>not the csv
21:44:34  <mbalho>dominictarr: the file is a csv.bzip2, im wondering if you did bzip2 -d 1988.csv.bzip2 > 1988.csv or cat 1988.csv.bzip2 | bzip2 -d | node something.js
21:44:55  <dominictarr>right - so I was streaming the JSON version
21:45:28  <dominictarr>which is 5 times larger than the csv version
21:45:33  <dominictarr>2.5G!
21:46:14  * mcollinaquit (Read error: Connection reset by peer)
21:46:52  * mcollinajoined
21:47:06  <dominictarr>now it only takes 24 seconds
21:47:09  <dominictarr>okay, cool.
21:49:44  * nicholasfquit (Remote host closed the connection)
21:50:33  <dominictarr>rvagg: I'm wondering about the memtable in leveldown
21:51:09  * mcollinaquit (Ping timeout: 240 seconds)
21:51:48  * mcollinajoined
21:51:58  <mbalho>dominictarr: yea json is big, im thinking about a way to store json on disk that is smaller, since im willing to constrain it to 1 level JSON (e.g. no objects as values)
21:53:59  <dominictarr>the overhead is really saving the headers again
21:54:27  <dominictarr>that is what csv eleminates
21:54:53  * nicholas_joined
21:57:18  <mbalho>yea
21:57:28  <mbalho>and { + } + "
21:58:00  <chrisdickinson>mbalho: well known keys?
21:58:31  <mbalho>chrisdickinson: nope
21:59:02  * jcrugzzjoined
21:59:42  <mbalho>chrisdickinson: well.... i should think about that
22:03:22  <mbalho>chrisdickinson: its essentially excel tables/csvs soooo
22:03:36  <mbalho>chrisdickinson: i bet i can get away with well known keys given that you can add keys later
22:04:30  <dominictarr>mbalho: you could use a number for a column name, like, how erlang does "atoms"
22:04:48  <dominictarr>provided that you have a ~ fixed number of columns
22:04:49  <mbalho>oooh im not familiar
22:05:08  <dominictarr>atoms are like constant strings
22:05:28  <dominictarr>and they are just id'd by address,
22:05:32  <dominictarr>not by value
22:05:51  <mbalho>but it makes sense that for a table there would be an append only list of columns that are indexed
22:05:53  <dominictarr>so, they are only one world - 2 bytes? or maybe 4?
22:07:25  <daleharvey>any easy way to run a single test with tape?
22:07:29  <mbalho>most data would be under 1024 columns but i bet there are important outliers that would come up
22:39:13  * i_m_cajoined
22:51:43  <dominictarr>daleharvey: only a single file, oh! I think you can put test(name, {skip: true},function (t) {…})
22:51:51  <dominictarr>but I've never used that.
22:52:25  <dominictarr>mbalho: 2 bytes is 65k
22:52:54  <mbalho>oh right
22:53:40  <dominictarr>that would work for sparse tables
22:54:32  <dominictarr>the further you go down this road… the closer get to sql
22:55:11  <mbalho>haha
22:55:31  <dominictarr>mbalho: what about atomic cells?
22:55:44  <dominictarr>would that have good effects on size?
22:56:44  <mbalho>dominictarr: yes also might have good write throughput impacts since the cells could be quite small
22:59:20  * yorickquit (Remote host closed the connection)
23:04:03  <mbalho>dominictarr: i discuss tradeoffs here https://github.com/maxogden/dat/issues/10#issuecomment-23287110
23:07:17  * dominictarrquit (Quit: dominictarr)
23:19:13  * dominictarrjoined
23:30:25  * kenperkinsjoined
23:30:49  * kenperkinsquit (Client Quit)
23:31:36  * shamaquit (Read error: Connection reset by peer)
23:32:05  * shamajoined
23:34:25  * thlorenzquit (Remote host closed the connection)
23:38:52  * dominictarrquit (Quit: dominictarr)
23:42:29  * jcrugzzquit (Ping timeout: 246 seconds)
23:45:11  * thlorenzjoined
23:57:34  * kumavisquit (Quit: kumavis)