00:21:09  * isaacs_awaychanged nick to isaacs
00:23:20  * brsonquit (Quit: leaving)
00:23:37  * brsonjoined
00:30:28  * benviejoined
00:44:22  * travis-cijoined
00:44:22  <travis-ci>[travis-ci] mmalecki/node#7 (travis-ci - 51b7c88 : Maciej Małecki): The build is still failing.
00:44:22  <travis-ci>[travis-ci] Change view : https://github.com/mmalecki/node/compare/3197d6f...51b7c88
00:44:22  <travis-ci>[travis-ci] Build details : http://travis-ci.org/mmalecki/node/builds/370910
00:44:22  * travis-cipart
00:44:50  <mmalecki>isaacs: ^
00:45:55  * travis-cijoined
00:45:55  <travis-ci>[travis-ci] mmalecki/node#9 (travis-ci - f00dbb5 : Maciej Małecki): The build is still failing.
00:45:55  <travis-ci>[travis-ci] Change view : https://github.com/mmalecki/node/compare/51b7c88...f00dbb5
00:45:55  <travis-ci>[travis-ci] Build details : http://travis-ci.org/mmalecki/node/builds/370920
00:45:55  * travis-cipart
00:46:56  <benvie>oh my
00:56:46  <indutny>isaacs: heya!
00:57:04  <indutny>isaacs: any luck with https://github.com/joyent/node/pull/2244 ?
01:09:17  * mralephquit (Quit: Leaving.)
01:16:46  * travis-cijoined
01:16:46  <travis-ci>[travis-ci] mmalecki/node#10 (travis-ci - 202ab16 : Maciej Małecki): The build is still failing.
01:16:46  <travis-ci>[travis-ci] Change view : https://github.com/mmalecki/node/compare/51b7c88...202ab16
01:16:46  <travis-ci>[travis-ci] Build details : http://travis-ci.org/mmalecki/node/builds/370956
01:16:46  * travis-cipart
01:29:28  * AvianFlu_joined
01:30:58  * AvianFluquit (Ping timeout: 260 seconds)
01:32:07  * AvianFlu_changed nick to AvianFlu
01:54:40  * brsonquit (Ping timeout: 255 seconds)
02:52:27  * sh1mmerquit (Quit: sh1mmer)
03:00:02  * ericktquit (Quit: erickt)
03:24:40  * AvianFluquit (Quit: Leaving)
03:56:40  * luxigoquit (Ping timeout: 252 seconds)
04:02:07  * mjr_joined
04:06:26  <mjr_>Hey guys, we rolled out node 0.6 to some pretty heavy production load yesterday, and there's some kind of bad memory leak.
04:07:09  <mjr_>Seems like the V8 heap is really small. I'm looking for suggestions on how to figure out where else in node it might be coming from.
06:29:06  * benviequit (Quit: Leaving.)
06:29:07  * bbenviejoined
07:20:10  <ryah>mjr_: presumably all that is buffers then
07:20:21  <ryah>mjr_: hm.. how can we approach this
07:20:22  <mjr_>hey
07:20:28  <ryah>hey
07:20:36  <mjr_>So, I think we might be getting confused with the v8-profiler module putting v8 into debug mode
07:21:14  <mjr_>After further analysis, it seems that the V8 heap DOES get very large, very quickly.
07:21:25  <ryah>hm
07:21:39  <mjr_>But debug mode uses up a lot of memory
07:21:43  <ryah>mjr_: process.memoryUsage() should confirm?
07:21:46  <mjr_>yes
07:22:08  <ryah>heap diffs.. hm
07:22:25  <mjr_>So we are now up to 170 router processes, because we have millions of users trying to come in a day now
07:22:28  <ryah>so - is danny's debugger helpful there?
07:22:40  <ryah>wow
07:22:54  <mjr_>Yeah, Danny is trying to grab heap profiles, but it seems that putting it into debug mode changes the characteristics.
07:23:22  <mjr_>So, we were on node 0.4, and running out of CPU, with a slight growth in memory.
07:23:37  <mjr_>We upgraded to 0.6, and CPU got cut in half, which is awesome, thanks, v8
07:24:01  <mjr_>BUT, now things run great for a while, and then something happens and we start growing the heap like crazy
07:24:06  * indexzerojoined
07:24:52  <mjr_>standby for image
07:25:00  <ryah>ok
07:25:45  <mjr_>https://skitch.com/mranney/gpt25/voxer-metrics
07:26:40  <mjr_>our code is the same at the switchover time, with just the minor changes to make it work with 0.6
07:26:59  <mjr_>So to be clear, I now no longer think this is a libuv / buffers thing.
07:27:39  <mjr_>oh, day stamps are off by one in that graph, thanks to a hilarious DST bug.
07:29:16  <ryah>looking through v8 list to see if there are any big bugs
07:29:21  <ryah>the only change is this http://codereview.chromium.org/8742020
07:29:32  <ryah>seems unrelated
07:29:39  <mjr_>So we have 300K HTTPS connections right now, and I think we'd have way more if we didn't have to keep restarting these processes every 10 minutes.
07:32:09  <CIA-95>node: Ryan Dahl v0.6 * r1cf26e2 / (4 files in 2 dirs): Upgrade V8 to 3.6.6.11 - http://git.io/q_yCAg
07:32:26  <mjr_>want me to try that patch?
07:33:18  <ryah>no
07:34:58  <ryah>did process.uvCounters() show anything interesting?
07:35:33  <ryah>unfortunately we're not counting handle closes yet
07:35:59  <mjr_>https://gist.github.com/cfddfc5ae9e89b96215a
07:36:38  <ryah>if you do that a few times - are they growing as expected?
07:37:09  <mjr_>yeah, we are burning through connections like crazy, as I would expect
07:37:14  <ryah>the http parser binding got changed in the v0.4 to v0.6 upgrade
07:37:28  <ryah>i wonder if you could be hitting something there
07:37:31  <mjr_>Danny thought he saw more http parser references than connections
07:37:35  <mjr_>like, lots more
07:37:53  <mjr_>Still trying to confirm. The heap snapshot doesn't cleanly fit into the viewer in 0.6 for some reason
07:39:22  <ryah>so we have a shortcut now for parsing http messages whose headers fit into one packet
07:39:52  <mjr_>going to write a program to parse this snapshot file
07:40:04  <mjr_>But yes, there are TONS of references like this:
07:40:09  <mjr_> "3338295056": {
07:40:09  <mjr_> "constructorName": "HTTPParser",
07:40:10  <mjr_> "count": 7519
07:40:10  <mjr_> },
07:40:49  <ryah>http://github.com/joyent/node/commit/84d0b1bcc5154f67d9f200f3c852bb8c5b64c6ef
07:43:18  <mjr_>That's a pretty big change.
07:44:15  * eddybjoined
07:44:30  <eddyb>if this isn't a ridiculous mistake, I don't know what it is
07:45:12  <eddyb>child_process.js (and net.js, I think): if(!Pipe) Pipe = new process.binding('pipe_wrap').Pipe; return new Pipe;
07:45:42  <eddyb>so it's constructing an instance of pipe_wrap.Pipe
07:45:57  <eddyb>then constructing an instance of an instance of Pipe
07:46:34  <eddyb>which unless Pipe (as a class) inherits Function, it's pretty dumb
07:56:19  * isaacsquit (Quit: isaacs)
07:56:59  <ryah>eddyb: where?
08:00:56  <eddyb>ryah: https://github.com/joyent/node/blob/master/lib/child_process.js#L29
08:01:05  * isaacsjoined
08:02:29  <ryah>isaacs: yt?
08:03:17  <ryah>eddyb: that seems broken
08:03:31  <eddyb>how on earth it doesn't break?
08:04:41  <ryah>eddyb: im not sure
08:05:12  <eddyb>I found it because I don't have process.binding and I have to replace all calls to it with equivalent calls to require
08:05:38  <eddyb>or if it's the same module's bindings, they're already in a local variable
08:08:12  <ryah>> x = new process.binding('pipe_wrap')
08:08:12  <ryah>{ Pipe: [Function: Pipe] }
08:08:12  <ryah>> x.Pipe
08:08:12  <ryah>[Function: Pipe]
08:09:03  <eddyb>wtf
08:09:47  <CIA-95>node: Ryan Dahl v0.6 * r60e2666 / lib/child_process.js : Remove superfluous 'new' - http://git.io/fHIRKw
08:10:54  <eddyb>ryah: somehow it works
08:11:21  <eddyb>but it's not sup... aaaaah
08:11:26  <eddyb>you know why it works?
08:12:32  <eddyb>new process.binding('pipe_wrap') calls process.binding with 'pipe_wrap' as an argument. because it explicitly returns, the return value is the one returned by new, not a new process.binding object
08:12:59  <ryah>> function foo() {
08:12:59  <ryah>... return { a: 5 }
08:12:59  <ryah>... }
08:12:59  <ryah>undefined
08:12:59  <ryah>> new foo()
08:13:01  <ryah>{ a: 5 }
08:13:16  <eddyb>yepp
08:30:26  <mjr_>ryah: I wrote my own stupid parser for the heap profiles since we can't get them into the webkit ui
08:30:27  <mjr_>https://gist.github.com/2ec5deab2c59fe16b306
08:32:56  <mjr_>There are a mere 910 connections open on that node at the moment.
08:34:10  <mjr_>Updated gist to include my program to parse the heap snapshot, in case I'm doing that stupidly
08:35:44  <ryah>i would expect tcp and http-parser to be roughly the same
08:37:08  <mjr_>I don't really know what any of that means. I'm just sort of guessing at the meaning of the data in those snapshots
08:43:09  <mjr_>I need to sleep. I was pretty much up all night last night wrestling with scaling issues.
08:43:39  <mjr_>We have a lot of room to try out patches on a small subset of the servers since we now have so many in operation
08:44:00  <ryah>ok
08:44:01  <mjr_>So if you come up with anything you'd like to try, I'm open to that.
08:44:09  <ryah>ill take a hard look at the http-parser stuff soon
08:44:16  <ryah>sounds like a likely canidate
08:44:26  <mjr_>yeah, seems like something in there
08:44:27  <ryah>i'll also notify the troops
08:44:33  * paddybyersjoined
08:44:39  <mjr_>Could be a bug in our code to that only seems to happen under certain circumstances
08:45:08  <mjr_>RSS on some nodes runs at 200MB for 20 mins or so before starting the clumb.
08:45:17  <mjr_>On other nodes, it starts climbing right away.
08:45:52  <mjr_>The number of Timer objects seems odd also
08:46:47  <mjr_>For tonight I'm just going to restart them all every 30 mins, whether they need it or not.
08:46:57  <mjr_>Kinda sad.
08:47:27  <mjr_>Anyway, talk to you later.
08:52:00  * indexzeroquit (Quit: indexzero)
08:59:47  <CIA-95>node: Ryan Dahl v0.6 * r6cc94db / (4 files in 3 dirs): Bump version to v0.6.5 - http://git.io/nm7mwQ
09:02:27  * paddybyersquit (Quit: paddybyers)
09:09:10  <eddyb>ryah: in stream_wrap.cc on line 172, global.slab is read
09:09:27  <CIA-95>node: Ryan Dahl v0.6 * rc50b0c9 / src/node_version.h : Now working on Node v0.6.6 - http://git.io/3gghGg
09:09:28  <eddyb>but it's never set anywhere, so it doesn't make too much sense
09:09:58  <eddyb>ryah: nevermind, it was close
09:10:05  <eddyb>no idea how I missed it :|
09:10:29  <eddyb>yet, a static handle could've been better
09:26:14  <eddyb>I'll be the annoying questions guy today
09:26:34  <eddyb>ryah: stream allocates Buffers for it's slab thing
09:26:43  <eddyb>yet it uses the SlowBuffer version
09:29:44  * isaacsquit (Quit: isaacs)
09:32:21  <eddyb>and I'm not sure if it's intended
10:09:23  * paddybyersjoined
10:17:51  * paddybyersquit (Quit: paddybyers)
10:30:52  <eddyb>if I were to rewrite libuv, btw, I would make everything C++ classes with virtual inheritance
10:30:58  <eddyb>or something like that
10:32:19  * paddybyersjoined
10:54:14  * mralephjoined
11:18:50  * bnoordhuisjoined
11:25:49  * luxigojoined
11:28:05  <CIA-95>libuv: Ben Noordhuis v0.6 * r248ca5d / src/unix/error.c : unix: translate ETIMEDOUT to UV_ETIMEDOUT - http://git.io/V0P7Zw
11:29:40  * _eddyb_joined
11:29:40  * _eddyb_quit (Changing host)
11:29:40  * _eddyb_joined
11:32:17  * eddybquit (Ping timeout: 240 seconds)
11:39:25  * _eddyb_changed nick to eddyb
11:52:51  * paddybyersquit (Quit: paddybyers)
12:13:19  <eddyb>child said: "hello world\r\n"
12:13:42  <eddyb>so apart from that stupid stdin thing, it
12:13:46  <eddyb>'s working out for me
12:16:36  <eddyb>ryah: why aren't you doing the unref thing for stdin, too?
12:16:51  <eddyb>it can be reffed only when it's reading
12:16:52  <eddyb>https://github.com/joyent/node/issues/1726
13:05:12  * piscisaureus_joined
13:07:37  <eddyb>O_O
13:07:43  <eddyb>only 8 tests failed
13:07:51  <eddyb>but 165 passed
13:08:10  <mmalecki>eddyb: node tests? last time I checked 2 failed
13:08:19  <eddyb>for me
13:08:27  <eddyb>it's weird, what are those tests doing?
13:08:52  <eddyb>try { do stuff } catch(e){/*don't care about errors*/}?
13:08:54  <mmalecki>one is pausing and resuming stdin
13:09:03  <eddyb>yeah, that fails for me, too
13:09:07  <mmalecki>there's some fork tests that failed, iirc
13:09:33  <mmalecki>http://travis-ci.org/#!/mmalecki/node/builds/370957
13:10:04  <eddyb>anyway, this is my rewrite, which has little stuff implemented
13:10:25  <eddyb>brb, food
13:11:50  * piscisaureus_quit (Quit: ~ Trillian Astra - www.trillian.im ~)
13:35:55  * paddybyersjoined
13:44:00  * paddybyersquit (Quit: paddybyers)
13:44:05  <eddyb>this is wrong... the only fails I'm seeing are timeouts, most likely from the referenced stdin thing
13:44:42  <eddyb>no "module not found"
13:44:56  <eddyb>no "method not found"
13:45:17  <eddyb>I don't even have a TCP implementation
13:46:26  <eddyb>yeah, I guessed so
13:46:57  * luxigoquit (*.net *.split)
13:46:58  * mralephquit (*.net *.split)
13:47:00  * pquernaquit (*.net *.split)
13:47:00  * mjr_quit (*.net *.split)
13:47:01  * CoverSlidequit (*.net *.split)
13:47:01  * mmaleckiquit (*.net *.split)
13:47:01  * creationixquit (*.net *.split)
13:47:02  * ljacksonquit (*.net *.split)
13:47:05  * bnoordhuisquit (*.net *.split)
13:47:05  * ryahquit (*.net *.split)
13:47:06  * raggi_quit (*.net *.split)
13:47:06  * eddybquit (*.net *.split)
13:47:07  * igorziquit (*.net *.split)
13:47:07  * DrPizzaquit (*.net *.split)
13:47:08  * DrPizza_joined
13:54:17  * igorzijoined
13:54:18  * eddybjoined
13:54:18  * raggi_joined
13:54:18  * ryahjoined
13:54:18  * bnoordhuisjoined
13:54:18  * pquernajoined
13:54:18  * mmaleckijoined
13:54:18  * CoverSlidejoined
13:54:18  * mjr_joined
13:54:18  * mralephjoined
13:54:18  * luxigojoined
13:54:18  * ljacksonjoined
13:54:18  * creationixjoined
13:57:15  <eddyb>the topic is still there
13:57:16  <eddyb>:)
14:07:15  * creationixquit (*.net *.split)
14:08:38  * creationixjoined
14:09:41  <eddyb>heh, you're late
14:09:51  <eddyb>someone can fix the topic now :)
14:56:18  * bnoordhuisquit (Ping timeout: 255 seconds)
15:26:44  <eddyb>ryah: seriously, someone fix the topic
15:27:15  <mmalecki>eddyb: what's wrong with it?
15:27:26  <eddyb>gimme a second
15:27:28  * eddybpart ("Konversation terminated!")
15:27:36  * eddybjoined
15:27:44  <eddyb>mmalecki: heh
15:27:54  <mmalecki>yeah, I'd guess so :)
15:27:57  <eddyb>I was seeing the topic I set during the netsplit
15:28:05  <mmalecki>what was it?
15:28:19  <eddyb>You set the channel topic to "EDDYB OWNS THIS CHANNEL. Just kiddin', waiting for the netsplit to be over :).".
15:28:28  <mmalecki>lol
15:56:53  <indutny>hm... is isaacs going to be online today?
15:57:11  <mmalecki>indutny: yes, for sure
15:57:27  <indutny>mmalecki: :)
15:57:34  <mmalecki>indutny: we're doing a podcast, so he should be here in ~3 hours or so?
15:57:44  <mmalecki>indutny: you know, silly timezones
15:57:48  <indutny>going to start refactoring of node-spdy, based on new zlib functionality
15:57:54  <indutny>mmalecki: yeah, I understand
15:58:05  <indutny>mmalecki: are you on nodeup with indexzero and isaacs?
15:58:14  <mmalecki>indutny: yeah
15:59:06  <mmalecki>I hope I'll be able to come up with some topic ;)
15:59:34  <indutny>mmalecki: hehe :) nice
16:08:50  * ericktjoined
16:09:37  <eddyb>https://github.com/eddyb/gearbox-node/commits/master
16:09:50  <eddyb>this is making me feel like I've done something
16:11:22  <mmalecki>eddyb: wtf
16:11:31  <mmalecki>eddyb: is it?
16:11:48  <mmalecki>I don't even
16:11:50  <eddyb>why the shock?
16:12:09  <eddyb>funny thing, last commit was exactly a month ago
16:12:43  <eddyb>mmalecki: all you're seeing is my local changes being selected into commits
16:14:09  <eddyb>hmm
16:14:26  <indutny>eddyb: oh man
16:14:28  <eddyb>I think I know why tests tend to fail silently
16:15:53  <eddyb>I wasn't doing exit(1) or anything like that
16:16:48  <eddyb>YES
16:16:49  <eddyb>now it works
16:17:40  <eddyb>indutny, mmalecki: I could make CIA spit my commits in here :)
16:17:51  <eddyb>right now #gearbox is pretty empty
16:19:50  <eddyb>ofc... anything tty/pipe-related tends to timeout
16:31:04  * piscisaureus_joined
16:43:22  * _eddyb_joined
16:43:23  * _eddyb_quit (Changing host)
16:43:23  * _eddyb_joined
16:43:59  * eddybquit (Ping timeout: 244 seconds)
16:48:08  * piscisaureus_quit (Quit: ~ Trillian Astra - www.trillian.im ~)
16:50:22  * isaacsjoined
16:56:49  <mmalecki>indutny: ^
16:57:32  * piscisaureus_joined
17:00:08  * piscisaureus_quit (Client Quit)
17:13:48  * luxigoquit (Remote host closed the connection)
17:43:56  * |eddyb|joined
17:44:00  * _eddyb_quit (Ping timeout: 252 seconds)
17:56:03  <indutny>mmalecki: :)
17:56:06  <indutny>isaacs: !!!
17:56:13  <indutny>heya!
17:56:17  <indutny>how are you?
18:06:01  * |eddyb|changed nick to eddyb
18:08:02  * paddybyersjoined
18:31:03  <eddyb>I think my rewrite is needed, node is really messy
18:37:10  <mmalecki>eddyb: or optimized
18:38:10  <eddyb>most stuff I've came across is not at all about optimization
18:38:43  <mmalecki>eddyb: pull request ;)
18:38:50  <eddyb>yeah, well
18:39:06  <eddyb>nobody's gonna look at a pull request which changes everything
18:39:53  <eddyb>this is how bindings should work: https://github.com/eddyb/gearbox-node/blob/master/src/modules/pipe.gear
18:40:31  <eddyb>and there is still space for improvement
18:52:43  <isaacs>indutny: hi
18:57:40  <indutny>isaacs: hey, any luck with my patch?
18:57:48  <isaacs>indutny: haven't looked at it.
18:57:49  <indutny>isaacs: looking forward to use it new node-spdy module
18:57:54  <indutny>isaacs: ah, ok
18:58:00  <isaacs>been working on tap stuff, and being lazy this weekend :)
18:58:04  <indutny>:)
18:58:06  <indutny>brb
18:58:09  <isaacs>mostly lazy
19:01:58  <eddyb>if you're lazy, I'm a sloth
20:05:07  <indutny>back
20:05:45  <indutny>isaacs: lets pull it! :)
20:06:09  <indutny>isaacs: test is passing and I mostly copied it from one of my modules
20:16:23  * ericktquit (Quit: erickt)
20:43:25  * mikealjoined
20:45:13  * brsonjoined
20:45:42  * indexzerojoined
20:49:47  * mraleph1joined
20:49:53  * mralephquit (Read error: Connection reset by peer)
21:00:28  * mraleph1quit (Quit: Leaving.)
21:00:51  * piscisaureus_joined
21:23:09  <indutny>isaacs: oh, looks like I'll probably need to add even more functionality to get zlib usable for spdy implementation
21:24:16  <indutny>or not... lets see
21:25:33  <indutny>isaacs: sorry, .flush() seems to fill my needs
21:27:37  * travis-cijoined
21:27:37  <travis-ci>[travis-ci] mmalecki/node#11 (travis-ci - 815fd83 : Maciej Małecki): The build is still failing.
21:27:37  <travis-ci>[travis-ci] Change view : https://github.com/mmalecki/node/compare/202ab16...815fd83
21:27:37  <travis-ci>[travis-ci] Build details : http://travis-ci.org/mmalecki/node/builds/373022
21:27:37  * travis-cipart
21:34:02  * indexzeroquit (Quit: indexzero)
21:37:41  * paddybyersquit (Quit: paddybyers)
21:46:14  * eddybquit (Ping timeout: 244 seconds)
21:49:40  * mikealquit (Quit: Leaving.)
21:55:16  * mralephjoined
21:57:56  * mikealjoined
22:01:14  * AvianFlujoined
22:10:41  * CIA-95quit (Read error: Operation timed out)
22:10:50  <piscisaureus_>I wonder if http://groups.google.com/group/nodejs/browse_thread/thread/b5eb060e6df3d1c9 could be related to mjr_'s problem
22:14:23  <mjr_>oh, interesting
22:14:45  <piscisaureus_>maybe - It's just speculation, this guy could just be doing something wrong
22:15:08  <mjr_>Yeah, and we could be doing something wrong
22:15:23  <piscisaureus_>I can't reproduce any leaks on windows w/ aborted connections on windows
22:15:25  <mjr_>That guy's on 0.4 though
22:15:28  <piscisaureus_>will try on mac shortly
22:15:30  <piscisaureus_>oh
22:15:31  <piscisaureus_>hmm
22:15:36  <mjr_>Otherwise node inspector wouldn't work
22:16:02  <piscisaureus_>inspector doesn't work in 0.4 - I didn't even know
22:16:09  <mjr_>it doesn't work on 0.6
22:16:13  <mjr_>The heap profiler part
22:16:14  <piscisaureus_>er, yeah
22:16:22  <mjr_>Danny is working on it.
22:16:45  <mjr_>Luckily for us, the one person in the world who maintains the one heap tool we have for V8 happens to work for us.
22:17:02  <mjr_>Sounds like it's another day or two off though.
22:17:23  <piscisaureus_>mjr_, are you running on joyent infra?
22:17:28  <mjr_>sadly, no
22:17:31  <piscisaureus_>hmm
22:17:37  <mjr_>I was about to migrate over, and then we got popular
22:17:41  <mjr_>Which makes it a lot harder
22:17:42  <piscisaureus_>they have this nice deep heap inspection stuff
22:17:50  <mjr_>I know, it would be great to have that right now.
22:17:53  <piscisaureus_>in the works, but I saw ith work
22:17:57  * CIA-111joined
22:17:58  <piscisaureus_>yeah ok
22:18:21  <mjr_>I'm about to roll back to 0.4 and start collecting data, so we'll see how it's different
22:18:46  <mjr_>I'm tempted to roll a custom 0.4 build that has the latest V8 and see how that fares.
22:19:14  <piscisaureus_>heh - that shouldn't be so hard
22:22:02  <mjr_>I wish there was an easy way to get just one of our processes running on Joyent, but we have all of these networking issues that get in the way.
22:28:12  * isaacsquit (Read error: Connection reset by peer)
22:28:12  * isaacs_joined
22:28:37  * mikealquit (Quit: Leaving.)
22:30:24  * paddybyersjoined
22:41:51  * mralephquit (Quit: Leaving.)
22:46:33  <piscisaureus_>well, I'm not saying you should, but it might have helped here
22:49:30  * paddybyersquit (Quit: paddybyers)
22:54:00  <piscisaureus_>mjr_: you're using https right?
22:54:07  <mjr_>oh yes
22:54:17  <piscisaureus_>that could matter
22:54:18  <mjr_>We are using the hell out of https
22:54:40  <mjr_>We have many long-lived connections on https, and a smaller number of short ones.
22:54:49  <mjr_>The bulk of the connection count and bytes goes over http.
22:55:12  <mjr_>But each process is holding over 1,000 https open for a long time and streaming updates down them.
22:57:02  <piscisaureus_>I can't find any leaks with normal http atm, but https is much trickier
22:59:47  <mjr_>I'll bet
23:00:25  <mjr_>I'm on Linux in production, but I could arrange you access to a machine if you thought that might help.
23:00:39  <mjr_>No dtrace, sadly.
23:00:49  <piscisaureus_>mjr_: if we can't figure it it out might help
23:01:16  <piscisaureus_>mjr_: but its midnight here, I am going to leave shortly
23:01:30  <mjr_>OK, well thanks for spending some time on it.
23:01:47  <piscisaureus_>mjr_: Oh I'll do much more, but not right now :-)
23:05:27  <piscisaureus_>mjr_: this makes me nervous: https://github.com/joyent/node/blob/0e8a55d2a22b88dc3b9b0165f344602b0fa8c977/lib/tls.js
23:06:00  <mjr_>the whole file, or some change in it?
23:06:17  <piscisaureus_>mjr_: oh, this commit: https://github.com/joyent/node/commit/0e8a55d2a22b88dc3b9b0165f344602b0fa8c977
23:06:18  <piscisaureus_>sorry
23:09:34  <mjr_>oh, interesting
23:09:37  <mjr_>so
23:09:48  <mjr_>We lean on flow control in https a lot
23:10:01  <mjr_>Since we are streaming updates out https to mobile clients, they often go away
23:10:13  <mjr_>But we don't get an end, because their phone just went out of range or wahtever
23:10:34  <piscisaureus_>mjr_: yeah - now you should notice that this only affects the encrypted text stream
23:10:36  <mjr_>So when we get TCP back pressure for a while, we destroy the response.
23:16:42  <mjr_>BTW, I had to patch node 0.4 to ignore a bunch of https errors that were uncatchable. I was sort of hoping that 0.6 would fix a lot of that.
23:16:44  <mjr_>But https is hard.
23:16:57  <piscisaureus_>yeah, it's very painful
23:17:04  <mjr_>To be fair, node 0.6 DID fix those uncatchable errors.
23:17:41  <piscisaureus_>mjr_: btw that event that this commit removes - I think it was new for 0.6 so your code won't rely on it
23:17:52  <piscisaureus_>iow - nvm
23:18:06  <mjr_>I see. It's not used in the node internals somewhere?
23:18:29  <piscisaureus_>I'm still looking for that
23:18:38  <piscisaureus_>but I don't think so
23:18:53  <mjr_>I think all we use on https is that write returns false. If we get too many writes in a row that return false, then we drop.
23:19:19  <piscisaureus_>there were changes to that
23:19:20  <piscisaureus_>https://github.com/joyent/node/commit/4cdf9d4
23:19:54  <mjr_>hmm
23:20:57  <mjr_>Our crude heap snapshot sort of points to http parser, but I wonder if that could be related to https.
23:21:29  <piscisaureus_>mjr_: it could also be the http header optimizations that bnoordhuis did
23:25:29  <piscisaureus_>anyway - gtg, will spend more time tomorrow
23:27:45  * piscisaureus_quit (Quit: ~ Trillian Astra - www.trillian.im ~)
23:36:08  * piscisaureus_joined
23:38:28  * piscisaureus_quit (Read error: Connection reset by peer)
23:38:49  * piscisaureus_joined
23:52:23  * mikealjoined
23:56:41  * bbenviequit