00:00:19  <feross>and they'll think that it's not possible to make a website to show words without 100 software engineers that get paid $120k/yr
00:02:03  * mrb_bk_quit (Quit: Connection closed for inactivity)
00:26:31  <jfhbrook>it's harder than it sounds?
00:26:53  <jfhbrook>I mean, I don't know why the fuck medium has front-end javascript (we generate most of our content server-side)
00:27:07  <jfhbrook>do they have ads?
00:27:19  <jfhbrook>usually it's the ads and the analytixxx that lead to hella javascript
00:55:28  * dguttmanjoined
01:11:17  <ogd>feross: hahahah
01:11:37  <ogd>feross: i think that salary is conservative
01:12:46  <jfhbrook>feross: ogd I work for a major publisher and can confirm, that is relatively conservative
01:13:10  <ogd>i meant specifically for medium
01:13:22  <ogd>but also generally probably too
01:14:35  <jfhbrook>right, I just have to assume that salaries amongst people in publishing are similar
01:14:40  <jfhbrook>though medium's west coast huh?
01:14:47  <jfhbrook>I think most of the major publishers are east coast
01:14:51  <jfhbrook>I know tumblr has offices here
02:01:55  * pfrazejoined
02:07:09  * dguttmanquit (Quit: dguttman)
02:14:57  <ogd>substack: mk30 you need any warm snow clothes or are you good to go? i might have extras or can pick up cheap stuff at andy and bax if you need anything specific
02:16:45  <substack>we have warm clothes
02:21:49  <ogd>substack: cool! so were gonna pick you up from the airport then drive down
02:22:14  <ogd>substack: what is yours and mk30's shoe sizes so i can get snow shoes
02:23:22  <substack>I'm 10, marina is 8.5-9
03:08:30  <jfhbrook>man you two are YOUNG substack
03:08:43  <jfhbrook>also my feet are apparently huge
03:08:48  <jfhbrook>I'm a size 13
03:18:16  * darvonquit (Remote host closed the connection)
03:18:23  * shamaquit (Quit: (╯°□°)╯︵ɐɯɐɥs)
03:47:00  <noffle>oh, medium. they also break rss: sure they'll give you a feed, but it's just a sub-paragraph stub with a "read the full article" link
03:47:47  <noffle>feross: that's so damaging! to think that the new gen of devs will think that this is the Right Way(tm) to do web dev. they won't even know that Back In the Day(R) you could actually read websites and that this was hugely condusive to the spirit of the web!
03:50:04  <substack>it's much easier to learn how to make websites now
03:50:38  <substack>I don't think we've lost anything valuable because view->source doesn't turn up however a website was haphazardly constructed
03:54:48  <feross>substack: I didn't understand your last point. could you rephrase?
03:55:37  <substack>learning websites with view->source
03:55:51  <substack>which I did plenty of, 15 years ago
03:56:09  <substack>not a very good way to learn
03:57:21  <feross>interesting, I still learn mostly by reading other people's code
03:57:51  <feross>back when I learned, people's source code was available with view source
03:58:12  <feross>now it's hosted on GitHub
03:59:49  <feross>same thing essentially, except you can't accidentally stumble onto github as easily as right click -> view source
04:00:12  <feross>but I guess there's lot of awareness building stuff now, classes, etc
04:00:19  <feross>maybe it's not an issue anymore
04:06:30  <jfhbrook>what of server-side code?
04:06:44  <noffle>I still do a lot of digging /w view source too
04:06:47  <jfhbrook>I feel like Back In The Day people were mostly using js for chrome and the heavy lifting was in php
04:11:29  <noffle>jfhbrook: are you asking how can we get "view source" for server code? github's the best we have, I think?
04:12:58  <jfhbrook>I'm mostly suggesting that view-source wasn't actually that useful back in the day
04:13:52  <noffle>oh I see -- because JS was inconsequential for logic? I meant more for html/css/layout learning
04:15:26  <mikolalysenko>mk30: substack wonder if it is possible to mess with the bbc documentary on oakland? https://www.reddit.com/r/oakland/comments/3ypmt0/im_a_bbc_reporter_making_a_documentary_on_oakland/
04:23:18  <mikolalysenko>re learning js today: it always gets more hopeless
04:23:31  <mikolalysenko>people used to be able to see all the code their machines were running
04:23:56  <mikolalysenko>total awareness of all machine instructions wasn't a big deal back when computers had at most 64k of memory
04:24:03  <mikolalysenko>these days who knows what the hell computers do?
04:24:51  <mikolalysenko>we all probably have dozens of rootkits from various government agencies, criminal organizations and corporations monitoring and watching our every move
04:50:10  <substack>mikolalysenko: are the nodes in your interval-tree-1d balanced?
04:50:21  <mikolalysenko>substack: yeah
04:50:29  <mikolalysenko>*weight-balanced technically
04:50:41  <mikolalysenko>so it is an amortized thing
04:50:56  <mikolalysenko>that modules is kind of half assed though
04:51:05  <mikolalysenko>I know I can do it better, but it takes time
04:52:39  <substack>I'm attempting to use an interval tree as an example for a log structured merge tree module I'm working on
04:53:06  <substack>but I'm thinking maybe it's a bit too complicated, hard to figure out how to merge N trees into 1 without buffering them all into memory
04:53:30  <mikolalysenko>well, there are multiple ways to do interval trees
04:53:36  <mikolalysenko>some might be easier to merge than others
04:54:04  <mikolalysenko>another option is to try r-trees
04:54:27  <mikolalysenko>though they don't have as nice theoretical properties they are simpler and often work ok in practice
04:55:38  <mikolalysenko>re merging the trees: what you just do a full rebuild of the interval tree?
04:55:43  <mikolalysenko>that isn't too bad to do streaming
04:55:51  <substack>yes pretty much
04:56:05  <mikolalysenko>and that's all you need to do for log structured merging
04:56:22  <mikolalysenko>seems wasteful to throw out the old tree data structures, but at worst you are paying 2x
04:56:23  <substack>with lsm you have N trees, each size 2^N ascending from 1..M
04:56:31  <substack>times B
04:56:35  <mikolalysenko>right
04:57:00  <mikolalysenko>so just concat all the intervals together when you merge some set of trees and then do a full rebuild
04:58:30  <substack>that's what I was thinking
04:58:45  <mikolalysenko>you don't need to buffer them to do that
04:59:29  <substack>does that require any special tricks or is it pretty straightforward from how interval tree construction or insertion is usually described?
04:59:38  <mikolalysenko>no, don't do insertion
04:59:40  <mikolalysenko>do divide and conquer
05:00:02  <mikolalysenko>it takes as many passes as the height of the tree, which is O(log(n))
05:00:09  <mikolalysenko>and each pass is basically a partition operation
05:00:19  <substack>cool
05:00:20  <mikolalysenko>split the stream of intervals into 3 substreams
05:00:33  <mikolalysenko>one for intervals to the left/right/crossing the partition point
05:00:52  <mikolalysenko>then for those crossing you need to do one extra pass to sort the left/right end points
05:01:15  <mikolalysenko>incremental insertion/deletion is almost always a bad idea when building trees
05:01:29  <mikolalysenko>better to do things in batches like that, you get economies of scale and better cache performance
05:03:33  * PsionTheoryquit (Read error: Connection reset by peer)
05:04:19  <mikolalysenko>basically comparing O(n/B log(n)) vs O(n log_B(n))
05:05:12  <mikolalysenko>since log(n)/b is usually < log_b(n) for large enough B (which is always the case in practice)
05:16:33  <mikolalysenko>also if your tree construction is done using shell scripts you could get some parallelism by building the subtrees using separate processes
05:17:04  <mikolalysenko>so basically you build the top P levels sequentially and then you get a 1/P speed up on the rest of the tree
05:17:45  <mikolalysenko>giving a complexity of O((P * n + n log(n)/P) / B) for P processors
05:20:28  <substack>what about merging with streams for each tree from minimum to maximum values, since the nodes are already sorted?
05:20:52  <substack>I can easily see how the construction would work for that approach, not so much for building each level in a separate pass
05:35:50  * pfrazequit (Remote host closed the connection)
05:48:41  <mikolalysenko>once you know the split value you just classify left/right/center
05:48:48  <mikolalysenko>and then emit the nodes into each stream
05:49:02  <mikolalysenko>after those streams return you merge them together into the final tree
05:49:57  <mikolalysenko>look at how this subroutine does it: https://github.com/mikolalysenko/interval-tree-1d/blob/master/interval-tree.js#L265-L302
05:50:02  <mikolalysenko>that is the only interesting thing in that module
05:50:26  <mikolalysenko>the rest is just book keeping which figures out when to blow up subtrees and rebuild them due to the weight balance invariant being violated
05:50:41  <mikolalysenko>if you replace the arrays with streams, you get the idea
05:51:10  <mikolalysenko>this part here: https://github.com/mikolalysenko/interval-tree-1d/blob/master/interval-tree.js#L269-L275 is just to select the cut point
05:51:24  <mikolalysenko>it is wasteful since you could do it in O(n), but like I said earlier this code is half assed
05:51:42  <mikolalysenko>this part is what splits the intervals into 3 streams: https://github.com/mikolalysenko/interval-tree-1d/blob/master/interval-tree.js#L277-L289
05:52:00  <mikolalysenko>this is the extra book keeping for the center intervals: https://github.com/mikolalysenko/interval-tree-1d/blob/master/interval-tree.js#L291-L295
05:52:20  <mikolalysenko>and this is the divide and conquer recursive step: https://github.com/mikolalysenko/interval-tree-1d/blob/master/interval-tree.js#L297-L301
05:53:44  <substack>cool thanks!
05:55:57  <mikolalysenko>if the streams are sorted you can get the cut point in O(1) if you can seek
05:56:01  <substack>starting to get the idea I think
05:56:15  <mikolalysenko>or if you can't seek on the streams you can get it by scanning using the tortoise-hare trick
05:56:46  <substack>how does it work to get the cut point across N trees in O(1)?
05:57:01  <mikolalysenko>you have to scan all the intervals
05:57:04  <mikolalysenko>to find the mid point
05:57:14  <mikolalysenko>you can do it in O(n) time using a fast median algorithm like quick select
05:57:39  <mikolalysenko>or if they are sorted already you just take the middle element
05:58:12  <substack>they're each sorted, but not in the aggregate
05:58:37  <mikolalysenko>is it reasonable to assume that you can seek on these streams?
05:58:50  <substack>yes
05:58:54  <mikolalysenko>ok
05:59:39  <mikolalysenko>so you can think of this as finding the median of a set of k sorted lists
06:00:08  <mikolalysenko>I can think of a way to do this in k log(n), but feels like you could maybe do it in O(k) time...
06:02:43  <mikolalysenko>ok, you could do this again using interval trees
06:03:22  <mikolalysenko>so for each of the k lists, let [l_i, h_i] be the lowest and highest elements in each list
06:03:54  <mikolalysenko>then select the mid point of the 2k endpoints of these lists (can do it in O(k) time using quick select or whatever)
06:05:00  <mikolalysenko>ah wait this won't work...
06:07:30  <mikolalysenko>ok, only way I can think to do it is the O(log(k*n)) algorithm
06:07:43  <substack>I think it will be ok without this particular optimization
06:08:01  <mikolalysenko>yeah
06:08:12  <mikolalysenko>worst case merging everything is still linear so amortized it doesn't matter
06:08:23  <mikolalysenko>but you can skip that merging step if you can seek
07:26:23  * fotoverite_joined
07:26:26  * fotoveritequit (Ping timeout: 276 seconds)
07:26:26  * fotoverite_changed nick to fotoverite
08:58:45  * peutetrejoined
09:00:58  * fotoveritequit (Quit: fotoverite)
09:22:40  * djcoinjoined
09:23:59  * drptbljoined
10:17:19  * peutetrequit (Quit: ...)
10:17:39  * peutetrejoined
10:52:26  * peutetrequit (Quit: ...)
10:56:14  * peutetrejoined
11:05:52  * peutetrequit (Quit: ...)
11:31:06  * djcoinquit (Quit: WeeChat 1.0.1)
11:44:23  * peutetrejoined
13:01:07  * peutetrequit (Quit: ...)
13:15:03  * fotoveritejoined
13:55:21  * peutetrejoined
14:24:26  * darvonjoined
14:25:04  * darvonquit (Remote host closed the connection)
14:25:23  * darvonjoined
14:31:57  * pfrazejoined
14:40:23  * peutetrequit (Ping timeout: 264 seconds)
14:57:12  * fotoveritequit (Quit: fotoverite)
15:20:06  * knownasilyajoined
15:56:26  * dguttmanjoined
16:06:36  * pfrazequit (Remote host closed the connection)
16:25:41  * PsionTheoryjoined
16:35:21  <mk30>mikolalysenko: thanks for sharing! i'm not in oakland right now, but i've sent it to some good oakland people!
16:51:02  * dguttmanquit (Quit: dguttman)
16:52:09  * dguttmanjoined
16:54:44  * mrb_bk_joined
16:56:24  * dguttmanquit (Ping timeout: 245 seconds)
16:58:43  * dguttmanjoined
17:00:40  * dguttmanquit (Client Quit)
17:01:25  * pfrazejoined
17:01:57  * dguttmanjoined
17:09:18  * dguttmanquit (Quit: dguttman)
17:10:01  * bengljoined
17:17:19  * drptblquit (Quit: My MAC has gone to sleep. zZz..)
17:58:40  * pfrazequit (Remote host closed the connection)
18:01:22  * serapathquit (Ping timeout: 252 seconds)
18:02:08  * pfrazejoined
18:14:33  * dguttmanjoined
18:28:54  * fotoveritejoined
18:43:19  <ogd>mikolalysenko: mk30 hahahaha you should hide in bushes and film the filmmaker with a telephoto lens and narrate with a fake david attenborough nature documentary voice
18:46:49  * peutetrejoined
18:51:05  <ogd>mikolalysenko: this reminded me of you because they talk about qft a lot http://omegataupodcast.net/2015/12/191-string-theory/
18:56:03  <pfraze>has anybody here benched postgres against leveldb, or had experience using level with 100+ users?
18:57:12  <pfraze>it seems like level *ought* to outperform postgres any time you're doing a sequential read of an index you already built
18:57:55  <jfhbrook>that seems like apples and oranges pfraze
18:58:03  <pfraze>jfhbrook: you think?
18:59:11  <jfhbrook>well yeah, leveldb has no concept of relationships
18:59:31  <pfraze>jfhbrook: right, but they can still be used to solve the same kinds of problems
19:00:51  <pfraze>Im choosing an application's datastore, they may be apples/oranges, but that's fine, Im looking for a fruit
19:00:58  <jfhbrook>sure, you can solve the same problem(s) with any arbitrary kv store, but you have to RYO joins in leveldb
19:01:30  * ralphtheninjaquit (Ping timeout: 265 seconds)
19:01:33  <pfraze>yeah fine, but the question isnt about how usable the tech is, Im just trying to sort out how well level scales compared to pgsql
19:01:38  <jfhbrook>so if I had to guess, if you were just doing key-value lookups without any joins, level would win, but given anything more complex pg would probably win in the aggregate
19:01:53  <jfhbrook>it depends really heavily on how well you can optimize your query to the datastore
19:02:38  <pfraze>yeah that's why I mentioned "a sequential read of an index you already built", that's what Im trying to sort out - the optimal cases for both
19:02:47  * ralphtheninjajoined
19:03:40  <pfraze>it's really strange I cant find any published benches of level vs any of the rbdms
19:03:48  <pfraze>except sqlite
19:22:22  * timoxley_changed nick to timoxley
19:32:01  * peutetrequit (Quit: ...)
19:43:36  * shamajoined
19:54:56  * phatedjoined
20:10:41  * pfraze_joined
20:11:36  * pfrazequit (Ping timeout: 264 seconds)
20:41:07  * phatedquit (Remote host closed the connection)
21:02:03  * mrb_bk_quit (Quit: Connection closed for inactivity)
21:04:48  * pfrazejoined
21:07:53  * pfraze_quit (Ping timeout: 250 seconds)
21:09:06  * pfrazequit (Read error: Connection reset by peer)
21:10:39  * pfrazejoined
21:47:34  * phatedjoined
21:52:53  * phatedquit (Ping timeout: 276 seconds)
22:08:25  * ralphtheninjaquit (Quit: leaving)
22:10:17  * ralphtheninjajoined
22:24:50  * phatedjoined
23:18:55  * fotoverite_joined
23:20:38  * fotoveritequit (Ping timeout: 276 seconds)
23:20:38  * fotoverite_changed nick to fotoverite