00:00:35  <chapel>whiskey !== water what so ever
00:09:19  * xaqjoined
00:15:46  * xaqquit (Remote host closed the connection)
00:26:56  * saijanai_joined
00:28:11  <dominictarr>dools, have you seen the size of the curiosity though?
00:28:26  <dominictarr>(sorry guys I've been out drinking with mmalecki)
00:29:32  <dominictarr>http://i924.photobucket.com/albums/ad81/rdbrewer/PIA15279_3rovers-stand_D2011_1215_D-1.jpg
00:30:11  <dominictarr>Raynos, what module doesn't have a repo link?
00:31:22  <dominictarr>isaacs all my modules have readme.markdown. but npm doesn't notice if it's lowercase filename, and npm was collecting readmes for longer than it's been showing them, so I didn't realize.
00:31:48  <isaacs>dominictarr: there isn't even a readme in the tarball, though
00:32:09  <dools>dominictarr: yeah saw that photo. it's definitely bigger. i just sort of assumed that the general public would, you know, be bored of landing on mars. i understand it's an incredible achievement to do it engineering wise but, we've already "landed on mars" - where's the magic? where's the fascination?
00:32:59  <dominictarr>i'm kinda bored by it actually
00:33:05  <dominictarr>to be honest
00:33:29  <dominictarr>but I am pleased by everyone's enthusiasm, and you should be too.
00:33:55  <Raynos>dominictarr: through
00:34:13  <dools>yeah i suppose an electric atmosphere is an electric atmosphere. i remember when they had the olympics in sydney it was a massive party and i was out every night even though i don't give a shit about the olympics
00:34:18  <dominictarr>isaacs, what module are we talking about?
00:34:25  <isaacs>through
00:34:34  <dominictarr>oh, hang on.
00:34:49  <isaacs>dominictarr: what's the git repo url for it? if it actually does have a readme, i'll check out why it's not publishing it
00:34:56  <dominictarr>silly me for picking such a common name.
00:35:09  <dominictarr>next one I'll name 'what module?'
00:35:31  <isaacs>lol
00:36:20  <isaacs>"what module?" "yeah, that one." "which one?" "no, that's the dependency." "what module is the dependency?" "no, what module is the one i'm talking about." "what module?" "Exactly!"
00:36:43  <isaacs>"so it depends on which one?" "Yes."
00:36:55  <isaacs>too bad spaces and ? aren't allowed in module names
00:37:17  <dominictarr>damn, I wasn't fast enough.
00:37:42  <dominictarr>if only I could send that idea back in time.
00:38:31  <dominictarr>instead I will have to wait for the thing to make node look like ruby, and publish a module named "do you really think you should allow this name?"
00:39:42  <isaacs>lol
00:39:50  <isaacs>i wsn't fast enough with the case sensitivity, though
00:40:00  <isaacs>didn't realize what a pita that'd end up being
00:41:55  * stlsaintjoined
00:42:46  <dominictarr>raynos: see through@0.1.3
00:42:47  <Raynos>dominictarr: I keep getting does not have stream:2df9a99d60815d5e errors from mdm
00:42:55  <Raynos>are there any common bug patterns
00:43:00  <Raynos>I dont know what I'm doing wrong
00:43:12  <dominictarr>can you reproduce?
00:43:38  <Raynos>i need to decouple it from my production code
00:43:51  <dominictarr>it sounds like it's getting out of sync, like maybe this would happen if you reconneced a client to a server that has restarted.
00:44:13  <dominictarr>the error throws, right?
00:45:20  <Raynos>yes
00:45:23  <Raynos>thats exactly what im doing
00:45:29  <Raynos>reconnecting a client to a server that has restarted
00:45:36  <Raynos>yes the error throws
00:45:40  <Raynos>thus crashing the hell out of my computer
00:45:46  <Raynos>actually I dont do it just when I reconnect
00:45:58  <Raynos>its definitely an issue of the new event getting lost
00:47:19  <Raynos>dominictarr: https://github.com/Raynos/multi-channel-mdm/commit/b35c5a1d7eb201d65d0d4da029f80e020b13dd5f
00:47:24  <Raynos>reproduced
00:47:32  <Raynos>I should try reproducing it at a lower abstraction level :P
00:48:00  <dominictarr>you can't send an old stream to a new server, because it won't know what you are talking about.
00:48:15  <dominictarr>however, I've just realized that I can handle the error differently.
00:48:32  <dominictarr>I could actually send back a message that said that stream errored.
00:48:46  <dominictarr>and emit the error on the client side.
00:49:06  <dominictarr>the server shouldn't crash, because the client has done something wrong.
00:49:11  <Raynos>dominictarr: i know but thats not what Im doing
00:49:18  <Raynos>im not sending an old stream to a new server
00:49:23  <dominictarr>oh, what are you doing?
00:49:27  <Raynos>i think i found a race condition
00:50:37  <dominictarr>reading your code
00:50:42  <Raynos>dominictarr: https://github.com/Raynos/boot/commit/d262c055b568d5cb4b85c416a6c6d3511bd985da
00:50:45  <Raynos>ignore multi-channel
00:50:53  <Raynos>its reproducable in boot which is a lower abstraction level
00:51:03  <Raynos>ill see if I can reproduce it in mdm-shoe
00:51:23  <dominictarr>I need a tight script that I can add as a test case.
00:52:15  <Raynos>dominictarr: working on it
00:52:22  <dominictarr>like, in-process or just tcp
00:53:24  <Raynos>dominictar: https://github.com/Raynos/mux-demux-shoe/commit/bccb427447d964d169f02a20a0600e8607405ae4
00:53:43  <Raynos>it may only be reproducable over shoe, but at least it has nothing to do with boot or multi channel
00:55:10  <dominictarr>does that make the same error?
00:55:30  <dominictarr>can you paste the error trace?
00:57:26  <Raynos>dominictarr: https://gist.github.com/fd8bbfd46ca6be74bb7e
00:58:52  <rowbit>Hourly usage stats: []
01:02:36  <Raynos>dominictarr: I'll try to reproduce it without shoe later.
01:03:20  <dominictarr>mux-demux doesn't have a 'connect' event, that is shoe's thing.
01:05:08  <Raynos>dominictarr: I know, shoe has a connect event. but writing to a mux demux stream in that callback makes the race condition occur
01:05:17  <dominictarr>yes
01:05:58  <dominictarr>so when exactly does shoe emit the connect event?
01:06:08  <Raynos>when the connection to the websocket is open
01:06:31  <Raynos>https://github.com/substack/shoe/blob/master/browser.js#L49
01:06:33  <Raynos>Oh I see
01:06:40  <Raynos>I bet its because the buffer is send after the connect event
01:06:44  <Raynos>bingo!
01:06:55  <Raynos>i need to move that connect event lower
01:07:07  <Raynos>or use process.nextTick
01:07:39  <Raynos>dominictarr: this fixes my issue, its not an issue with mdm, its clearly an issue with shoe
01:09:20  <dominictarr>yeah, that is a bug in shoe.
01:09:37  <Raynos>dominictarr: thanks and sorry :(
01:09:53  <dominictarr>it souldn't emit the event until after it's cleared the buffer.
01:10:40  <dominictarr>or, even better, in write() it should push onto the buffer if it has a truthy length.
01:10:56  <Raynos>dominictarr: https://github.com/dominictarr/mux-demux/pull/4
01:11:20  <dominictarr>basically, it's writing the message before mux-demux has send the create stream message
01:13:39  <dominictarr>raynos, that pr isn't right, because it's the server still has to send the end message. it may still have more information to send.
01:14:09  <Raynos>dominictarr: Hmm, I agree, there is still the issue of the pipe memory leak and not knowing how to handle that memory leak
01:14:23  <Raynos>maybe it should emit another event like "ended" that I can catch and end the stream myself
01:14:39  <dominictarr>it's like this http://nodejs.org/api/net.html allowHalfOpen
01:14:59  <dominictarr>call destroy immediately after end.
01:15:33  <Raynos>dominictarr: I don't end it. the pipe ends it
01:15:58  <Raynos>I guess I can call close
01:16:42  <dominictarr>of course. you could listen to the previous stream, or wrap the function
01:17:12  <dominictarr>... or you could modify your pr so it used a allowHalfOpen option.
01:17:41  <dominictarr>I'd take that because it's consistent with net
01:18:13  <Raynos>dominictarr: how would I use allowHalfOpen?
01:18:20  <Raynos>default to false and call end for me and if true dont call end?
01:18:31  <Raynos>dominictarr: take this one instead :P https://github.com/dominictarr/mux-demux/pull/5
01:18:35  <dominictarr>yes
01:22:04  <isaacs>dominictarr: what version of npm did you just publish through with?
01:22:07  <isaacs>dominictarr: npm -v
01:22:58  <dominictarr>Raynos, yup, I'll pull that
01:23:19  <dominictarr>isaacs, npm@1.1.45
01:23:46  <isaacs>k, this is odd..
01:23:52  <isaacs>read-package-json finds the readme
01:23:54  <isaacs>but npm doesn't publish it
01:23:56  <isaacs>super weird
01:26:29  * tilgoviquit (Read error: Connection reset by peer)
01:29:40  * devaholicjoined
01:31:23  <dominictarr>raynos, are you gonna make a pull request on shoe?
01:31:32  <Raynos>i already did
01:31:46  <Raynos>https://github.com/substack/shoe/pull/3
01:31:52  <Raynos>SubStack: PONG
01:35:03  <Raynos>dominictarr: https://github.com/dominictarr/mux-demux/pull/4
01:35:55  <Raynos>add the halfOpen flag
01:36:26  <dominictarr>oh, can you also add && !this.ended ?
01:36:27  <SubStack>merged
01:36:34  <Raynos>SubStack: badass
01:36:42  <dominictarr>don't want to emit end twice
01:37:04  <dominictarr>SubStack, it's funny, in node you are more likely to get sync race conditions.
01:37:20  <Raynos>dominictarr: done
01:37:55  <SubStack>dominictarr: especially with events since those listeners fire right away
01:39:02  <dominictarr>Raynos, pr/4 breaks a test
01:39:14  <Raynos>dominictarr: I know, it says so in the PR
01:39:25  <Raynos>i dont know what the test should do correctly
01:39:28  <Raynos>because I cant read your tests
01:39:30  <Raynos>they are black magic
01:43:18  <isaacs>dominictarr: found the bug, and took the liberty of updating your through package
01:43:33  <isaacs>dominictarr: it'll be in the next release of npm, which will be in node 0.8.7
01:44:09  <dominictarr>it was my package.json ?
01:44:50  <dominictarr>there is a long error chain that caused this to not have a repo & readme.
01:45:30  <dominictarr>I refactored it out when offline, so I didn't have a git remote when I ran npm init.
01:45:53  <dominictarr>it didn't have a readme because I was in a rush (okay, that is no excuse)
01:46:15  <dominictarr>and then I miss wrote repo because I am drunk.
01:46:20  <dominictarr>excuses, excuses.
01:54:18  * thatguydanjoined
01:54:21  <SubStack>\o
01:55:19  <dominictarr>Raynos, does mdm/4 work for you?
01:55:24  <dominictarr>your using it right?
01:58:52  <rowbit>Hourly usage stats: []
02:01:41  * ITproquit (Ping timeout: 250 seconds)
02:02:27  * ITprojoined
02:03:40  <SubStack>thatguydan: looking through http://docs.ninja.is/ any chance that GET /user/stream could do streaming json?
02:03:55  <SubStack>not sure what you're using for the server part
02:04:07  <SubStack>but if you're using node, JSONStream rocks for that stuff
02:04:26  <thatguydan>definitely, v0 is pretty ghetto
02:04:35  <thatguydan>i'll throw it on the roadmap for v1
02:05:13  <thatguydan>right now stream is just rules that have executed, but in future will be new devices, and other important events
02:05:19  <thatguydan>like "Front door opened"
02:06:10  <SubStack>yeah it would just rock if I could have an http stream open over http that would just stay up and get populated with events as they happen
02:06:40  <chapel>thatguydan: you use pusher?
02:06:52  <SubStack>http outside of browsers is full duplex already, especially if you have https endpoints to prevent proxies from meddling
02:07:06  <thatguydan>chapel: yup, only to get events from our servers to browsers.
02:07:33  <chapel>how does that work with the api?
02:08:11  <thatguydan>basically we return a pusher-key with one of the endpoints and you can use one of their libraries to get data in
02:08:31  <thatguydan>we'll ultimately do it all ourselves, but for now pusher do their job pretty well and have tons of libs
02:08:45  <chapel>ah
02:09:00  <chapel>okay, so you just allow them to get the key to access the info
02:09:30  <thatguydan>yup, i think the endpoint is GET user/pusherchannel
02:09:41  <chapel>we use pusher at geeklist for our front end
02:10:02  <thatguydan>they're good at what they do
02:10:03  <chapel>there was talk of using pusher in the api
02:10:27  <dominictarr>thatguydan, your thing looks very cool!
02:10:53  <thatguydan>we haven't yet used it in the api, but we use it heaps for browser comms
02:11:03  <thatguydan>thanks, we're trying to do the whole internet of things but built for developers
02:11:46  <thatguydan>so devs can build apps ontop of user's devices
02:11:48  <dominictarr>that is what is gonna make the IoT happen.
02:11:49  * devaholicquit (Ping timeout: 256 seconds)
02:12:35  <thatguydan>we certainly think so :)
02:13:22  <thatguydan>IFTTT and Zapier are awesome, but if I want to build an app that monitors the energy output of solar panels and adjusts my air conditioner to match the PV array's output that's a bit more than a recipe/Zap
02:19:17  * devaholicjoined
02:19:18  <thatguydan>We're currently moving over from socket.io to dnode (dunno why anybody thought that was a good idea), I was wondering if anybody had any general advice for using up/dnode server side?
02:20:03  <thatguydan>It's all pretty straight forward, but after monkey patching socket.io-client and wading through endless issues on socket.io's github, this refactor has gone a little too…smoothly
02:20:12  <Raynos>dominictarr: the test that fails does not affect me negatively. I'm running it on some complex code and none of it breaks
02:20:21  <Raynos>dominictarr: I think that test case that breaks is a bad test case
02:20:34  <Raynos>dominictarr: I think that test case that breaks is a bad test case or I dont understand enough about the edgecases in streams
02:21:02  <dominictarr>the problem was the way readable and writable are set in createStream
02:21:28  <dominictarr>first, I had to modify the test case to have allowHalfOpen enabled,
02:21:32  <Raynos>dominictarr: Oh I see
02:21:47  <dominictarr>but that created a stream that was readable == writable == false
02:23:23  <Raynos>o.o
02:23:35  <Raynos>all the bugs
02:24:06  <dominictarr>raynos, mux-demux@2.0.0
02:24:12  <Raynos>oh sweet!
02:24:34  <isaacs>dominictarr: nvm, it'll be included in 0.8.6
02:25:21  <dominictarr>isaacs, what will be included?
02:25:30  <dominictarr>node version manager?
02:25:31  <SubStack>thatguydan: one thing that you need to be careful about is having too many requests buffer up that all fire right away when the connection comes back
02:25:34  <isaacs>dominictarr: the npm that does 'readme.markdown' properly
02:25:40  <isaacs>dominictarr: npm = 'nevermind'
02:25:42  <dominictarr>oh, good.
02:25:48  <isaacs>*nvm = 'nevermind'
02:25:56  <SubStack>like if the client has an internet outage for a few hours or something and then tries to resync
02:27:13  <dominictarr>the buffering model in upnode tends to assume that the outage will be short.
02:28:40  <dominictarr>SubStack, oh, you need to enable the shims?
02:29:06  <dominictarr>hmm. so ... these people must are mistaken.
02:30:04  <dominictarr>oh yeah, Raynos skates was intending to (maybe) be that framework, but I've since redone a bunch of stuff
02:30:12  <SubStack>well shims as in es5-shims? browserify hasn't had those for a year
02:30:16  <dominictarr>(that was before I wrote mux-demux)
02:30:45  <SubStack>and all the other modules you've got to require() explicitly for them to be included
02:30:59  <dominictarr>SubStack, oh... that is good. hmm. I'll definately correct people with that objection in future
02:32:11  <SubStack>it's only 4k compressed with uglify for an empty bundle
02:32:54  <SubStack>and yet the same people have no qualms about a 32k compressed jquery bundle
02:33:00  <SubStack>people are silly
02:33:21  <dominictarr>indeed they are
02:34:01  <devaholic>browserify is 4k compressed?
02:34:04  <dominictarr>Raynos, boot backs off with a fibonnaci series !
02:34:24  <SubStack>devaholic: the browser-side bundle with an empty entry file
02:34:28  <dominictarr>don't you know node isn't webscale for fibonnaci? it'll never work
02:34:55  <devaholic>nice, i always wondered how much it really was
02:35:23  <SubStack>$ browserify <(echo '') | uglifyjs | wc -c
02:35:24  <SubStack>4354
02:36:09  <dominictarr>(actually, using fib for that is totally classy)
02:36:12  <Raynos>dominictarr: yes it does
02:37:08  <dominictarr>does it transparently reconnect?
02:37:12  <Raynos>yes
02:37:18  <Raynos>boot gives you a new stream
02:37:26  <Raynos>that transparently pipes into fresh mux demux streams
02:37:34  <Raynos>and then you can use boot to drive upnode
02:37:40  <Raynos>and you can have two layers of reconnection!
02:38:01  <Raynos>The main thing thats nice about boot is having a persistance mdm stream that doesnt have to be recreated when ws reconnect
02:38:02  <dominictarr>so it gives you a fresh stream on each reconnect?
02:38:12  <Raynos>you keep one proxy stream
02:38:18  <Raynos>and it hooks a fresh mdm stream into it on reconnect
02:38:34  <Raynos>dominictarr: https://github.com/Raynos/boot/blob/master/browser.js#L64
02:39:00  <dominictarr>oh, can you add an option so that it breaks the stream each time, so that you have to start piping again?
02:39:22  <Raynos>dominictarr: I dont see the point of doing that. all it does is transparent reconnect
02:39:32  <Raynos>if you dont want transparent reconnect then just use mux-demux-shoe directly
02:40:09  <dominictarr>there are some applications that work better without transparent reconnect.
02:40:26  <Raynos>what advantage does boot give without the transparent reconnect?
02:40:34  <Raynos>why not use mux-demux-shoe directly?
02:40:52  <dominictarr>okay, I'll leave that for another module.
02:40:58  <devaholic>Raynos: can you write something that verifies that the chunks are all received in order and none are lost across redirects
02:41:09  <devaholic>reconnects rather
02:41:26  <Raynos>devaholic: shoe buffers when the connection isn't open
02:41:31  <dominictarr>devaholic, tcp does most of that...
02:41:31  <Raynos>I should add buffering back to boot
02:41:44  <dominictarr>hmm, unless the connection is dropped.
02:42:04  <devaholic>i just dont see anything that actually tests that will happen
02:42:28  <dominictarr>then you don't actually know if anything was still on th wire
02:42:33  <devaholic>and i mean as far as the chunks in the node streams, not tcp
02:43:45  <dominictarr>devaholic, tcp is a reliable stream, since each chunk is written to this they will be emitted in order.
02:44:13  <dominictarr>oh, you said "reconnects"
02:44:39  <dominictarr>then I explained why you had a good point.
02:44:51  <dominictarr>I need to sleep
02:45:20  <devaholic>probably my fault :D
02:48:22  <dominictarr>catch you guys later!
02:48:38  * dominictarrchanged nick to dominictarrsleep
02:48:40  <Raynos>devaholic: I can do that, putting the buffering in boot
02:48:46  <devaholic>dominictarrsleep: later
02:48:52  <dominictarrsleep>cheers
02:53:50  * st_lukequit (Remote host closed the connection)
02:58:52  <devaholic>Raynos: cool!
02:58:52  <rowbit>Hourly usage stats: []
02:59:06  <devaholic>Raynos: you should also consider using ``` js in your readmes
03:01:51  * thatguydanpart
03:01:56  * thatguydanjoined
03:01:56  * thatguydanpart
03:01:59  <Raynos>maybe
03:02:06  * thatguydanjoined
03:03:43  <devaholic>:p
03:04:23  <devaholic>oh and if you wanted to mke your chat stream thingy work distributed style you could consider making some streams from redis pub/subs with redis-stream
03:04:34  <devaholic>streams all the way up and down
03:10:47  <devaholic>just make a pub/sub channel based on the mux-demux id, even
03:21:38  <Raynos>devaholic: https://github.com/Raynos/boot
03:21:39  <Raynos>done !
03:22:06  <Raynos>devaholic: redis-stream, you are a badass
03:22:23  <Raynos>devaholic: https://github.com/Raynos/multi-channel-mdm/blob/master/index.js#L8
03:22:29  <Raynos>multi-channel takes a store
03:22:32  <Raynos>write a redis-stream-store
03:22:38  <Raynos>Actually I can write one
03:22:56  <Raynos>boot should handle reconnect and ensure no data gets lost or out of order
03:23:00  <Raynos>if it does its bugged
03:23:06  <devaholic>if you need to persist the chats you could fork the stream at some point before/after pubbing
03:23:45  <Raynos>explain what you mean fork?
03:24:30  <devaholic>well if you publish the stream coming from a browser to the chat channel, then have all the other people in the channel subscribe to that channel
03:25:34  <devaholic>its not going to be saved because its just pub/sub, so just also pipe the stream to a redis list possibly, or a function to save to mongodb maybe, then when someone refreshes they can get the persistent data
03:26:24  <devaholic>but with that even it would be a very small amount of code
03:27:39  <Raynos>hmm
03:27:48  <Raynos>getting data wont work cleanly with multi channel
03:29:01  <devaholic>you would need to stream down some persistent data at load time, i dunno how you are handling the chat data so its hard to say what the best data structure would be
03:29:25  <devaholic>are you saing a timestamp, username, just the text? etc
03:29:26  <Raynos>well
03:29:31  <Raynos>data structure is JSON
03:29:40  <Raynos>but first I need a redis store!
03:29:42  <Raynos>ALL THE REDIS
03:29:43  <LOUDBOT>OH JUST SUCK IT UP AND DEAL WITH WAP OVER GPRS LIKE I DO
03:29:47  <Raynos>i also need to install redis locally
03:29:55  <Raynos>i also need to go to best buy
03:30:13  <devaholic>redis-server you mean?
03:30:15  <Raynos>devaholic: btw did you see seaport-proxy ? it works
03:30:19  <Raynos>yeah redis server
03:30:27  <devaholic>oh
03:30:32  <thatguydan>how does redid-stream do its magic?
03:30:35  <devaholic>cuz yeah, you can store whatever with the redis client
03:30:42  <thatguydan>redis*
03:30:46  <thatguydan>is it polling?
03:30:50  <devaholic>thatguydan: https://github.com/tblobaum/redis-stream
03:31:25  <devaholic>its basically a net connection that processes the redis protocol
03:31:53  <devaholic>thatguydan: polling? never
03:32:29  <devaholic>Raynos: i need some code to test seaport-proxy and boot
03:33:02  <guybrush>devaholic: redis-stream looks sweet
03:33:10  <guybrush>wtf is a star?
03:33:21  <guybrush>now we can watch AND star a repo?
03:33:27  <devaholic>i dunno they gayed up the place abit
03:34:18  <guybrush>so watch is now notification-managment and star is star i guess
03:36:48  <guybrush>devaholic: did you have to scale redis yet? if yes did you use some sort of tls-proxy or just node?
03:36:48  <thatguydan>ah so it just streams responses for queries, I had an image of it streaming like an lrange whenever an element was pushed to it… it was about that time i realised i'm an idiot :p
03:37:18  <devaholic>thatguydan: you can actually do that!
03:37:39  <thatguydan>witchcraft!
03:37:40  <devaholic>thatguydan: there are blocking lpop and rpops in redis
03:38:10  <thatguydan>bah, there goes my lunch break :p
03:38:17  <devaholic>thatguydan: so you jsut blpop a list and node will sit there and wait and as soon as someone pushes to that list in another process your other stream will get the data
03:39:00  <devaholic>so its sort of polling in that case
03:39:36  <thatguydan>but it's not, say 1op/s (if you were polling 1/s)
03:39:53  <devaholic>no it would be a long poll, 1 poll per data item, never more
03:40:10  <thatguydan>sweet!
03:40:55  <devaholic>blpop is really useful, check it out on the redis site
03:41:19  <devaholic>guybrush: i havent had to scale redis beyond simple master-slave stuff
03:41:54  <thatguydan>redis cluster looks promising
03:42:23  <guybrush>i think it takes some more time to be finished
03:42:49  <guybrush>i mean antirez said in the mailinglist it will take some more time
03:43:11  <guybrush>so for now the recommended thing to do would be presharding i think
03:43:39  <devaholic>guybrush: and i usually just hide redis inside a private network so auth isnt much of an issue
03:44:00  <devaholic>yeah
03:44:04  <guybrush>i see, just wondered
03:44:57  <guybrush>if it would be ok to use node as tls-proxy (in consideration of performance?)
03:45:16  <devaholic>i have not run into the need to scale redis out like that, but i think just using some separate redis servers with data sharded by your applications concerns would take you a long way without needing any abstraction
03:45:19  <guybrush>like others use stud instead of node-https
03:45:40  <guybrush>right
03:45:50  <devaholic>put all your users on one redis server, put your chat rooms on another, and just connect to them separately in your code
03:45:55  <devaholic>ya
03:51:39  * timoxleyquit (Quit: Computer has gone to sleep.)
03:51:57  <devaholic>guybrush: i think node might become a bottleneck as a proxy in front of redis, but it could be done
03:52:00  <devaholic>and it would work
03:54:27  * thatguydanquit (Quit: thatguydan)
03:57:12  * thatguydanjoined
03:58:14  * ITproquit (Ping timeout: 250 seconds)
03:58:52  <rowbit>Hourly usage stats: []
03:59:36  * ITprojoined
04:11:45  * timoxleyjoined
04:16:25  <Raynos>devaholic: boot has a bunch of examples, multi-stream has a boot example, stream-chat uses both
04:16:43  <Raynos>devaholic: seaport-proxy isn't done yet, it just kind of works at the hackish implementation level, it needs a shiny API
04:17:01  <devaholic>Raynos: so are you basically solving the problem of how to provide streams over dnode?
04:17:12  <Raynos>devaholic: no, no streams over dnode
04:17:30  <Raynos>you can get streams directly by asking for them and piping them up
04:17:43  <devaholic>how about writing a test case for having multiple streams on the same connection between two processes
04:17:43  <Raynos>you cant pass streams over dnode and expect it to work
04:17:55  <Raynos>devaholic: boot handles that already it uses mdm
04:18:21  <Raynos>devaholic: https://github.com/Raynos/multi-channel-mdm/blob/master/examples/boot/client.js#L4
04:19:02  <devaholic>Raynos: ah, its not using dnode at all?
04:19:10  <Raynos>correct
04:19:17  <Raynos>but you can open a upnode connection using boot if you want
04:19:36  <devaholic>whats this upnode stuff then?
04:19:50  <Raynos>devaholic: https://github.com/Raynos/boot/tree/master/example/upnode
04:20:09  <Raynos>upnode does reconnects, dnode does protocol shit. you can pass your own streams to both
04:21:02  <devaholic>ah as i understood it upnode was just for dnode
04:21:28  <Raynos>yeah upnode is a thin wrapper around dnode
04:23:14  <Raynos>devaholic: I use seaport to not hardcode ports, how do I not hardcode my redis server?
04:25:22  <devaholic>what advantage is there to having redis on a dynamically allocated port
04:26:54  <Raynos>because I want just hard coded port in my code
04:26:56  <Raynos>which is seaport
04:28:07  <devaholic>i suppose it would be possible to provide different redis servers as named services in seaport
04:28:33  <devaholic>but with how scalable a single instance of redis is, writing the code might not be worth the effort
04:29:09  <devaholic>but if you wanted, you could have a node instance spin up a redis instance, and provide the port to listen on as a command line option, which is allocate dby seaport
04:29:09  <Raynos>how do you do redis's over multiple machines?
04:29:59  <devaholic>then you could have different redis instances named by different seaport services, and connect to them by connecting to seaport services without knowing the port
04:30:52  <devaholic>but that might introduce an extra layer of failability (i.e. the node process) to the redis server that wasnt there and doesnt have to be there, and the memory footprint of that process
04:31:20  <devaholic>i just store the redis port in a config.json
04:31:58  <devaholic>multiple machines? connect to the redis server with the private ip address
04:31:58  <Raynos>devaholic: what if I want to distribute it across multiple machines? I guess I can shard it
04:32:02  <Raynos>SHARD ALL THE THINGS
04:32:02  <LOUDBOT>POWER TO THE PEOPLE
04:32:05  <Raynos>SHARD ALL THE STREAMS
04:32:05  <LOUDBOT>THAT THING
04:32:18  <devaholic>i dont think you need to shard redis
04:32:24  <devaholic>or worry about that, at all
04:33:08  <devaholic>you will have thousands of concurrent users and dozens of node instances before redis would be an issue
04:33:48  <Raynos>i dont need to shard redis
04:33:54  <Raynos>I need to shared node
04:33:58  <Raynos>shard node*
04:34:28  <devaholic>you want to connect to redis from distributed node processes
04:34:35  <devaholic>?
04:35:17  <devaholic>put all your stuff on the same private network, and (if you needed) put redis on a dedicated machine, and connect to it with the private ip in your network
04:35:44  <guybrush>Raynos: you need to shard redis when size of data > ram
04:36:03  <Raynos>i seee
04:36:08  <Raynos>but waht about geolocated redis
04:36:13  <Raynos>GEO LOCATE ALL THE DATA
04:36:13  <LOUDBOT>BECAUSE IT COULD BE EITHER ONE
04:36:27  <Raynos>ugh
04:36:31  <Raynos>I need to finish seaport-proxy
04:36:34  <guybrush>that IS some sort of sharding isnt it?
04:36:55  <thatguydan>real men anycast
04:37:01  <Raynos>I need to write seaport stream
04:37:14  <Raynos>fuck getting things, give me streams! ALL THE STREAMS
04:37:16  <devaholic>Raynos: i dont think you need to worry about that stuff
04:39:26  <devaholic>youd have to be operating at pretty high scale to have a need to hvae networks in multiple zones
04:40:46  <guybrush>the thing is, you want to be able to do it later when its necessary without too much effort
04:41:10  <guybrush>and it is cool to think about such scenarios :D
04:41:24  <devaholic>you can solve that stuff with hackery on other layers
04:45:08  <Raynos>SOLVE ALL THE SCALING PROBLEMS UP FRONT. APPLY WATERFALL PROCESS TO YOUR SOFTWARE SCALING.
04:45:09  <LOUDBOT>I WILL GLADLY HLAGH UNDERAGE GIRLS
04:45:18  <Raynos>IF YOU CANT RUN AT GOOGLE SCALE FROM THE GET GO YOUR DOING IT WRONG
04:45:18  <LOUDBOT>HE'S DESCRIBING SO MANY OTHER LANGUAGES
04:45:39  <devaholic>oh loudbot
04:47:06  <Raynos>im installing a mounti
04:47:09  <Raynos>for stream chat :3
04:47:48  <devaholic>i am getting a pile of modules i need to release
04:47:50  <Raynos>devaholic: I dont actually know how to use redis at all
04:48:02  <Raynos>what redis feature do I use for my stream channels?
04:48:08  <devaholic>documenting is the suck
04:48:18  <Raynos>I just need a function get(name, cb) { cb(stream) }
04:49:10  <devaholic>Raynos: you could use publish/subscribe
04:49:32  <devaholic>there is only like 2-3 relevant commands for that
04:51:13  <devaholic>pub/sub isnt as feature rich as amqp and others but in your case it would work fine, and you can get streaming compatible stuff easy with redis-stream
04:51:33  <Raynos>whats amqp ?
04:52:04  <devaholic>message queue protocol
04:52:08  <devaholic>google is your friend
04:58:52  <rowbit>Hourly usage stats: []
05:21:35  * st_lukejoined
05:41:05  <Raynos>devaholic: https://gist.github.com/ba26282e39d876dd0954
05:41:30  <Raynos>why doesnt that work?
05:45:28  <devaholic>Raynos: what doesnt work?
05:45:31  <devaholic>about it
05:45:40  <Raynos>it outputs junk
05:46:01  <Raynos>subscribe
05:46:01  <Raynos>undefined:streamName
05:46:01  <Raynos>1
05:46:01  <Raynos>message
05:46:01  <Raynos>undefined:streamName
05:46:01  <Raynos>foo
05:46:15  <Raynos>i see command followed by success code
05:46:17  <Raynos>then some other junk
05:46:22  <Raynos>ugh. I dont want that junk
05:46:28  <devaholic>those are both readable/writable streams already, not sure what duplex might be doing
05:46:43  <Raynos>well subscribe outputs the data
05:46:46  <Raynos>and publish writes the data
05:46:53  <Raynos>I want a readable / writable stream
05:47:00  <Raynos>this redis thing was a horrible idea -.-
05:47:52  <Raynos>back to memory streams
05:47:54  <devaholic>i bet its pretty close to working
05:48:07  <Raynos>i dont want the redis protocol junk in my stream output
05:48:15  <devaholic>if you put up more code i can run i'd be happy to test it
05:48:25  <devaholic>yeah, that shouldnt happen
05:48:36  <devaholic>and it doesnt
05:49:07  <devaholic>so if thats the case its a bug and ill fix it
05:49:14  <Raynos>devaholic: https://github.com/Raynos/redis-stream-store
05:53:12  <devaholic>running it
05:55:03  <Raynos>the only data to be emitted that I would expect is "foo"
05:56:54  <devaholic>ok well, these are streams that go across the network so you cant expect that code execution has anything to do with the order of execution in redis
05:57:29  <devaholic>for instance the publish event would work except you arent subscribed to the channel when it is written
05:57:59  <devaholic>and second, the response from redis for a subscribe or a publish is a boolean, thats why you have a 1, it would be 0 if it failed
05:58:20  <devaholic>so you need to account for that, its how you handle an error connecting to a pub/sub channel not really junk
05:58:52  <rowbit>Hourly usage stats: []
05:59:05  <devaholic>so in this test you should assert that you get a `true` back before you test a publish
05:59:10  <Raynos>devaholic: oh ...
05:59:35  <Raynos>why does it return the name
05:59:39  <Raynos>of the thing im connecting to
05:59:47  <devaholic>thirdly, all publish events come with the channel as the first response, if you'd liek you can create a es.mapSync to filter those out fairly easily
06:00:02  <devaholic>so it'll be [ 'channel1
06:00:08  <devaholic>', 'message1' ]
06:00:16  <devaholic>[ 'channel1', 'message2' ]
06:01:10  <devaholic>and you could also use that to route so you didnt have to make a bunch of `publish` streams (which are net connections) to redis to handle a lot of publish channels
06:01:15  <Raynos>whats the message string?
06:02:00  <devaholic>i would expect the response to subscribe, after the initial boolean, to be \n undefined:streamName \n foo
06:02:35  <Raynos>oh
06:02:41  <Raynos>subscribe returns the channel name too
06:02:57  <devaholic>yeah that way you can subscribe to 10 different channels with one connection
06:03:33  <Raynos>i see
06:03:38  <devaholic>the api for redis-stream might not be perfect yet but even if you have to patch it a little bit the code is going to be very short
06:04:21  * st_lukequit (Remote host closed the connection)
06:04:39  <Raynos>how do I strip out the first few messages?
06:04:42  <Raynos>like the boolean
06:04:47  <devaholic>also you can use Redis.es instead of require('event-stream') so that you are sure you are using the same version of event-stream as is used internally in redis-stream, which may or may not be an issue at some point
06:05:35  <devaholic>you should probably use it to verify that you connected, but you could just create a little thing like...
06:06:09  <Raynos>devaholic: ...
06:06:32  <devaholic>var connected = false; Redis.es.mapSync(function (data) { if (!connected && data === 1) connected = true; return })
06:06:53  <devaholic>then pipe the stream to that and return that as your stream
06:07:30  <devaholic>`return undefined` will filter the 1 from making it to your browser stuff or wherever else
06:08:03  <Raynos>cool
06:08:19  <devaholic>then also make sure you are connected before publishing in your test
06:08:34  <devaholic>if that doesnt work let me know :D
06:12:24  <Raynos>devaholic: It doesnt close the redis connection :/
06:12:29  <Raynos>when I close the stream
06:12:30  <Raynos>balls
06:12:36  <devaholic>if you want to make an issue about how redis-stream should emit an error with a failed return value so those booleans can be ignored it might be a good addition
06:13:04  <devaholic>its a net connection, so stream.close() it
06:13:27  <devaholic>redis needs you to do that btw, or else it will only sometimes close its end
06:13:37  <devaholic>in any case
06:19:52  <Raynos>done!
06:20:02  <Raynos>https://github.com/Raynos/redis-stream-store
06:20:07  <Raynos>npm modules++
06:20:16  <Raynos>and dont stream.close it -.-
06:20:18  <Raynos>stream.end it
06:20:42  <devaholic>oh ok
06:21:03  <devaholic>its not junk haha, you should use that to possibly pause the writable stream until you get that return value
06:21:29  <Raynos>devaholic: I just callback it when its connected rather then pause it
06:21:59  <devaholic>why are you filtering "message" ?
06:23:36  <Raynos>devaholic: because it's an annoying junk word
06:23:51  <Raynos>Of course maybe you want to actually send the word message, well screw you :p
06:24:22  <Raynos>message streamName actualValue
06:24:35  <devaholic>i bet you could turn these subscribe streams into streaming json
06:24:41  <devaholic>with another line or two
06:26:36  <devaholic>ah it does return message first
06:26:41  <devaholic>redis stuff
06:28:58  <devaholic>this is pretty cool
06:29:05  <devaholic>its pretty much a stream broadcaster
06:29:24  <devaholic>heh
06:32:20  <Raynos>devaholic: yep :)
06:32:23  <devaholic>so to send down persistent data you could initially pause the readable stream and pipe your stored data to the destination, then when its done unpause the readable stream
06:32:29  <Raynos>its even cooler when used with MultiChannel
06:33:24  <devaholic>made me think of what other crazy stuff you could pipe to this, like broadcasting a file stream to an unlimited number of end streams
06:34:32  <devaholic>i should possibly add a pubsub parser along with the hmget parser to handle the message line properly
06:34:47  <devaholic>you should count the lines and strip every third line rather than testing the string value
06:35:12  <Raynos>i agree
06:35:15  <Raynos>feel free to PR :D
06:35:16  <devaholic>or pipe it to the Redis.hmget function to turn it into json
06:35:30  <devaholic>so that message becomes a key or element 0 in an array
06:38:48  <Raynos>devaholic: https://github.com/Raynos/seaport-proxy it is done cc SubStack
06:41:07  <devaholic>is the streamRouter stuff just some extra bit
06:41:34  <devaholic>why do you need a route
06:42:55  <Raynos>devaholic: it's kind of extra. I have yet to refactor it away
06:43:17  <Raynos>devaholic: you need some way to send named streams to specific destinations. The action service you are getting is encoded in the stream name
06:44:12  <Raynos>you can just call proxy.get(stream, { service: "magic@1.2.x" })
06:44:50  <Raynos>the browser.js code has a hard coded call to the uri :D
06:56:39  <Raynos>devaholic: https://github.com/Raynos/multi-channel-mdm/tree/master/examples/redis
06:56:48  <Raynos>run two servers on different ports then watch THE MAGIC
06:58:52  <rowbit>Hourly usage stats: []
07:11:07  <Raynos>devaholic: https://github.com/Raynos/stream-chat now web scale!
07:31:05  <Raynos>https://github.com/Raynos/stream-chat#the-interesting-bits
07:44:33  * chapelquit (*.net *.split)
07:44:34  * sorensenquit (*.net *.split)
07:58:52  <rowbit>Hourly usage stats: []
08:11:42  <guybrush>Raynos: awesome stuff
08:12:41  * chapeljoined
08:12:42  * sorensenjoined
08:13:11  <guybrush>i really like the mdm routing thing
08:21:23  <Raynos>guybrush: route all the streams!
08:22:26  <Raynos>i should pimp this way of coding to more people
08:22:37  <Raynos>maxogden: would appreciate the streamy goodness
08:58:52  <rowbit>Hourly usage stats: []
08:59:39  * thatguydanquit (Quit: thatguydan)
09:06:38  * thatguydanjoined
09:47:13  * ITproquit (Ping timeout: 260 seconds)
09:47:53  * ITprojoined
09:58:52  <rowbit>Hourly usage stats: []
10:39:15  * thatguydanquit (Ping timeout: 265 seconds)
10:58:52  <rowbit>Hourly usage stats: []
11:17:00  * Nexxypart
11:30:20  * timoxleyquit (Quit: Computer has gone to sleep.)
11:36:20  * thatguydanjoined
11:58:52  <rowbit>Hourly usage stats: []
12:15:13  * thatguydanquit (Quit: thatguydan)
12:58:52  <rowbit>Hourly usage stats: []
13:00:25  * stlsaintquit (Ping timeout: 252 seconds)
13:36:46  * stlsaintjoined
13:58:52  <rowbit>Hourly usage stats: []
14:58:52  <rowbit>Hourly usage stats: []
15:15:27  * _sorensenjoined
15:34:45  * dominictarrsleepchanged nick to dominictarr
15:58:52  <rowbit>Hourly usage stats: []
16:03:23  <rowbit>/!\ ATTENTION: (default-local) xb@... successfully signed up for developer browserling plan ($20). Cash money! /!\
16:03:23  <rowbit>/!\ ATTENTION: (default-local) paid account successfully upgraded /!\
16:08:44  * stlsaintquit (Quit: leaving)
16:35:14  * simcop2387quit (Excess Flood)
16:38:34  * simcop2387joined
16:40:09  * simcop2387quit (Excess Flood)
16:41:04  * simcop2387joined
16:58:52  <rowbit>Hourly usage stats: []
17:13:17  * blakmatrixjoined
17:35:48  * xaqjoined
17:58:17  <Raynos>SubStack: upnode and seaport-proxy work in the browser https://github.com/Raynos/boot/tree/master/example/upnode & https://github.com/Raynos/seaport-proxy/tree/master/examples/simple
17:58:31  <Raynos>SubStack: all I need is the hooks in airport to plug these guys in
17:58:52  <rowbit>Hourly usage stats: []
18:07:55  * st_lukejoined
18:12:18  * timoxleyjoined
18:38:57  * dominictarrquit (Remote host closed the connection)
18:58:52  <rowbit>Hourly usage stats: []
19:04:36  <rowbit>SubStack, pkrumins: Developers waiting in the queue for ie8 (Queue length: 2 on 1 servers. Total servers: 3)
19:32:59  <rowbit>/!\ ATTENTION: (default-local) antoni.wojcik@... successfully signed up for developer browserling plan ($20). Cash money! /!\
19:32:59  <rowbit>/!\ ATTENTION: (default-local) paid account successfully upgraded /!\
21:24:49  <SubStack>yay signups
21:30:06  * tilgovijoined
21:47:11  * devaholicquit (Ping timeout: 245 seconds)
22:18:35  <st_luke>the best one
22:18:36  <st_luke>http://24.media.tumblr.com/tumblr_m7g8tbZEzO1rzupqxo1_500.png
23:41:56  * tilgoviquit (Remote host closed the connection)
23:55:31  * xaqquit (Remote host closed the connection)
23:58:10  * tilgovijoined