00:00:01  * ircretaryquit (Remote host closed the connection)
00:00:09  * ircretaryjoined
00:00:14  <terinjokes>guys, i still can't successfully connect via a {fd: 3} handler passed to net.server.listen
00:00:34  <terinjokes>i also can't step into the listen function, making this debugging very annoying
00:00:40  * tilgoviquit (Remote host closed the connection)
00:04:51  <thealphanerd>substack: indeed the command-line version :D
00:05:02  <thealphanerd>such a great bare bones test runner
00:13:10  * contrahaxjoined
00:17:08  * jxson_joined
00:20:24  * jxsonquit (Ping timeout: 256 seconds)
00:21:32  * jxson_quit (Ping timeout: 256 seconds)
00:21:38  * dguttmanquit (Quit: dguttman)
00:29:07  * thlorenzquit (Remote host closed the connection)
00:40:25  * contrahaxquit (Quit: Sleeping)
00:52:34  <mikolalysenko>I would love to see this get emscriptened: https://github.com/Z3Prover/z3
00:54:13  <substack>mikolalysenko: do you know any techniques off-hand for balancing a kd-tree on the fly without pre-sorting?
00:56:15  <mikolalysenko>in principle, it is no different than balancing any b-tree and you can use 2-4 trees or whatever
00:56:29  <mikolalysenko>but like all dynamic tree data structures, that strategy gets messy
00:56:50  <mikolalysenko>the other option would be scape goating, which keeps the tree mostly balanced
00:56:58  <mikolalysenko>or you can use log structured merging
00:58:07  <substack>thanks, will read up!
01:00:00  * contrahaxjoined
01:00:06  * domanicquit (Ping timeout: 244 seconds)
01:00:13  * contrahaxquit (Client Quit)
01:02:25  * pilijoined
01:05:06  * shamaquit (Quit: (╯°□°)╯︵ɐɯɐɥs)
01:23:49  * Guest63357quit (Ping timeout: 264 seconds)
01:32:56  * yoshuawuyts1quit (Ping timeout: 256 seconds)
01:52:22  * contrahaxjoined
01:55:14  * contrahaxquit (Client Quit)
02:21:20  * thealphanerdquit (Quit: thealphanerd)
02:45:54  * fotoveritejoined
02:51:11  * sethvincentjoined
03:00:39  * sethvincentquit (Ping timeout: 265 seconds)
03:12:33  * domanicjoined
03:16:35  <domanic>timoxley, yo yo
03:17:13  <timoxley>domanic: ah hello.
03:17:39  <domanic>timoxley, hey you still at hotel? I'm gonna go get a bite
03:17:46  <domanic>to tide me over until dinner
03:17:51  <domanic>if you want to join!
03:18:24  <timoxley>I'm still groggy after red-eye flight. Having a nap to tide me over until speaker dinner
03:20:13  <domanic>timoxley, ha, no problem catch you later
03:20:18  <timoxley>ttyl
03:42:18  * domanicquit (Ping timeout: 252 seconds)
03:51:23  * phatedquit (Remote host closed the connection)
04:03:05  * piliquit (Read error: Connection reset by peer)
04:03:33  * pilijoined
04:14:42  * domanicjoined
04:17:45  * simcop2387quit (Read error: Connection reset by peer)
04:17:53  * perlbotquit (Read error: Connection reset by peer)
04:25:26  * anvakajoined
04:28:38  * phatedjoined
04:30:16  * Guest63357joined
04:31:33  * knownasilyaquit (Quit: Connection closed for inactivity)
05:16:49  * phatedquit (Remote host closed the connection)
05:17:25  * phatedjoined
05:19:37  * sethvincentjoined
05:21:52  * phatedquit (Ping timeout: 256 seconds)
05:25:28  <domanic>substack, https://medium.com/starts-with-a-bang/supertides-are-real-92543449e987
05:58:39  <substack>neato
06:06:52  * Guest63357quit (Ping timeout: 255 seconds)
06:07:58  * Guest63357joined
06:26:39  <substack>I'm headed to the dinner now
06:32:17  * pfrazequit (Remote host closed the connection)
06:43:15  * perlbotjoined
06:44:45  * simcop2387joined
06:44:46  <domanic>on my way too
06:48:56  * domanicquit (Ping timeout: 246 seconds)
07:01:45  <feross>On my way now
07:02:47  <terinjokes>and i'm going to bed
07:05:16  * fotoveritequit (Quit: fotoverite)
07:45:50  * thealphanerdjoined
07:50:29  * gozalaquit (Quit: Connection closed for inactivity)
07:56:05  * yoshuawuyts1joined
07:58:25  * sethvincentquit (Ping timeout: 256 seconds)
08:08:21  * thealphanerdquit (Quit: thealphanerd)
08:18:15  * yoshuawuyts1quit (Ping timeout: 252 seconds)
08:42:14  * yorickquit (Ping timeout: 252 seconds)
09:06:18  * peutetrejoined
09:11:57  * yoshuawuyts1joined
09:33:11  * pfrazejoined
09:37:57  * pfrazequit (Ping timeout: 265 seconds)
11:41:44  * peutetrequit (Quit: ...)
11:48:31  * peutetrejoined
12:39:37  * knownasilyajoined
13:00:26  * fotoveritejoined
13:38:27  * pfrazejoined
13:49:30  * pelletierjoined
14:05:52  * thlorenzjoined
14:09:08  * domanicjoined
14:09:21  * thlorenzquit (Remote host closed the connection)
14:31:08  * reqsharkquit (Quit: Be back later ...)
14:35:06  * thlorenzjoined
14:48:53  * domanicquit (Ping timeout: 248 seconds)
15:03:40  * domanicjoined
15:20:02  * reqsharkjoined
15:20:09  * reqsharkquit (Remote host closed the connection)
15:20:36  * reqsharkjoined
15:40:23  * peutetrequit (Quit: ...)
15:46:39  * peutetrejoined
16:04:03  * Guest63357quit (Read error: Connection reset by peer)
16:04:20  * Guest63357joined
16:26:21  * piliquit (Read error: Connection reset by peer)
16:26:54  * shamajoined
16:53:12  * yoshuawuyts1quit (Ping timeout: 256 seconds)
17:05:06  * peutetrequit (Ping timeout: 256 seconds)
17:09:53  * domanicquit (Ping timeout: 250 seconds)
17:17:01  * jxsonjoined
17:20:09  * DTrejojoined
17:31:34  * thealphanerdjoined
17:57:03  * tilgovijoined
17:58:35  * thlorenzquit (Remote host closed the connection)
18:19:01  * sethvincentjoined
18:23:24  * thlorenzjoined
18:23:51  * phatedjoined
18:33:50  * thlorenzquit (Remote host closed the connection)
18:34:38  * mmaleckiquit (Quit: leaving)
18:34:51  * mmaleckijoined
18:37:36  * DTrejoquit (Remote host closed the connection)
19:03:58  * thlorenzjoined
19:16:35  * contrahaxjoined
19:27:25  * DTrejojoined
19:41:10  * shamaquit (Quit: (╯°□°)╯︵ɐɯɐɥs)
20:01:21  * thealphanerdquit (Quit: thealphanerd)
20:12:30  * DTrejoquit (Remote host closed the connection)
20:12:33  * sz0joined
20:20:13  * DTrejojoined
20:24:09  * jxson_joined
20:27:32  * domanicjoined
20:27:33  * jxsonquit (Ping timeout: 265 seconds)
20:29:00  * jxson_quit (Ping timeout: 265 seconds)
20:29:55  * jxsonjoined
21:01:42  * jxsonquit (Remote host closed the connection)
21:04:01  * domanicquit (Ping timeout: 264 seconds)
21:06:30  * jxson_joined
21:08:32  * DTrejoquit (Read error: Connection reset by peer)
21:08:54  * DTrejojoined
21:11:10  <substack>hooray https://gist.github.com/substack/bd469fc54d0ea7a84ef4
21:12:03  <substack>so I might not need to balance the tree right away, it's smaller than the pbf file after I fixed a bug
21:12:59  * Guest63357quit (Remote host closed the connection)
21:13:20  * Guest63357joined
21:17:23  * domanicjoined
21:38:21  * pelletierquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
21:59:59  * thlorenzquit (Remote host closed the connection)
22:02:15  * ceejbotjoined
22:04:49  * ceejbotquit (Remote host closed the connection)
22:05:27  * sethvincentquit (Ping timeout: 252 seconds)
22:09:47  * Guest63357quit (Ping timeout: 246 seconds)
22:10:45  * Guest63357joined
22:21:26  * Guest63357quit (Ping timeout: 244 seconds)
22:23:13  * Guest63357joined
22:36:58  * DTrejoquit (Remote host closed the connection)
22:42:45  * ceejbotjoined
22:49:27  * domanicquit (Ping timeout: 252 seconds)
22:57:47  * reqshark_joined
23:00:42  * reqsharkquit (Ping timeout: 250 seconds)
23:00:53  * thlorenzjoined
23:00:54  <mikolalysenko>substack: since these indices are meant to be mostly static anyway, it is better just to do all the sorting up front
23:01:08  <mikolalysenko>or you can speed up the tree construction a bit using a fast median finding algorithm instead of a sort
23:01:33  * knownasilyaquit (Quit: Connection closed for inactivity)
23:01:37  * jxsonjoined
23:02:46  * reqshark_quit (Ping timeout: 272 seconds)
23:04:56  * jxson_quit (Ping timeout: 256 seconds)
23:05:19  <substack>mikolalysenko: it's working ok so far, the big thing I want to optimize for is memory consumption
23:05:23  * thlorenzquit (Ping timeout: 246 seconds)
23:05:33  <substack>because I want to index all of OSM on cheapo cloud hosting
23:05:45  <substack>and the indexing phase doesn't need to be especially fast
23:05:57  <substack>but I've got limited memory and disk to work with
23:06:04  * jxsonquit (Ping timeout: 256 seconds)
23:06:35  <substack>I could pre-sort and write to disk, but that would eat up even more disk
23:08:17  * jxsonjoined
23:09:47  <mikolalysenko>sorting up front is probably the way to go
23:10:39  <mikolalysenko>you could also do a few tricks to shard things
23:10:55  <mikolalysenko>like build the top few levels in memory, then for each lower tree just chop up the data set and build recursively
23:12:17  <mikolalysenko>one thing I've thought about is that for each kdtree level, instead of doing alternating x/y/z etc splits you could group them up
23:12:31  <mikolalysenko>like do sqrt(b) x splits then sqrt(b) y splits
23:13:12  <mikolalysenko>making it work a bit more like a grid file
23:14:51  * tilgoviquit (Remote host closed the connection)
23:16:30  <mikolalysenko>substack: also a cool trick with floats, they can be sorted as ints
23:16:38  <mikolalysenko>so sorting floats ~ sorting ints
23:16:48  <mikolalysenko>that is int order = lexicographic order = float order
23:16:52  <mikolalysenko>for unsigned ints anyway
23:17:37  <mikolalysenko>it might be possible to leverage sort to do most of the dirty work
23:24:39  * ircretaryquit (Ping timeout: 252 seconds)
23:26:02  * Guest63357quit (Ping timeout: 246 seconds)
23:30:30  <anandthakker>substack: strange things happening when i try osm-pbf-to-mddf
23:31:44  <anandthakker>substack: including my (mac)os thinking the disk is full, till I kill the process and nuke the file
23:34:22  * pilijoined
23:34:40  <substack>anandthakker: it works best on filesystems that can handle sparse blocks right now
23:34:50  <substack>because I haven't written the presorting or balancing yet
23:35:31  <anandthakker>substack: ah cool
23:36:19  <substack>also I'm just testing with a 7M file
23:36:28  <substack>osm extract of auckland
23:37:25  <anandthakker>substack: yah i originally started with a hilariously huge one (75M), but even the 7M delaware had problems
23:37:50  <anandthakker>substack: i should back up and try some stuff with mddf directly first
23:39:15  <substack>the goal is to eventually get up to the entire planet earth >:D
23:39:35  <substack>because with mddf you won't need extracts, you can just slice into the data wherever you want without downloading the entire thing
23:40:23  <anandthakker>yeah i love this idea
23:41:13  <anandthakker>(beautiful alignment of fascinating technical problem and worthy result!)
23:43:43  * thlorenzjoined
23:45:10  * DTrejojoined
23:55:04  <anandthakker>substack: yeah, the 100000 points ex
23:55:26  <anandthakker>...ample for mddf exhibits the same problem
23:56:13  <substack>the traversal isn't quite correct yet either
23:56:35  <substack>I need to back off when it gets to the leaf node to explore the nearby leaves up to the level dimension
23:58:28  <substack>or something like that, need to re-read some things
23:58:44  <substack>but anyways, it's coming along yay!
23:58:53  * DTrejoquit