00:03:21  <piscisaureus>I wonder how one would only get "..."
00:03:59  <ryah>i bet they typed it in the repl
00:04:31  <piscisaureus>he says he used `node example.js`
00:12:14  <bnoordhuis>ryah piscisaureus: if you uv_close a handle that has pending writes
00:12:17  <bnoordhuis>what should happen?
00:12:17  <CIA-75>node: Ryan Dahl master * r05e6f31 / (129 files in 14 dirs): Upgrade V8 to 3.5.6 - http://bit.ly/p7ZXM7
00:12:24  <ryah>bnoordhuis: write callbacks fire
00:12:29  <ryah>then close callback
00:12:39  <bnoordhuis>okay, but with what status code?
00:12:53  <bnoordhuis>the writes don't actually happen do they?
00:13:12  <ryah>well we guarentee that uv_close() will close(2) synchronously
00:13:20  <ryah>so they can't
00:13:28  <ryah>not sure which status code - but they should error
00:13:53  <bnoordhuis>and callbacks fire
00:14:00  <bnoordhuis>but does that actually happen right now?
00:14:07  <bnoordhuis>i don't see code that does that
00:14:07  <ryah>yes
00:14:09  <ryah>i think
00:14:20  <ryah>i dont think we have a test for it though
00:14:23  <ryah>we should!
00:14:34  <bnoordhuis>hah, i'll write one
00:14:43  <bnoordhuis>but good to know that the callbacks should fire
00:14:53  <bnoordhuis>udp is ahead of the pack!
00:14:57  <DrPizza>I wonder if there would be value in some kind of a drain_and_close that allowed all outstanding activity to complete, then called close.
00:15:18  <bnoordhuis>yes, probably
00:15:20  <DrPizza>since the caller doesn't easily know how many outstanding actions there are
00:15:23  <DrPizza>(though uv itself does)
00:15:30  <DrPizza>I mean the caller could track it
00:15:36  <DrPizza>but it'd just be duplicating things that uv already pays attention to
00:15:42  <bnoordhuis>but the next feature request is going to be a drain_and_close that takes a timeout
00:16:03  <bnoordhuis>but yes, it's probably a good thing to have
00:16:52  <ryah>DrPizza: people can implement that with the write callbacks
00:16:57  <ryah>and we do in node
00:17:09  <piscisaureus>I am +1 for a drain-and-close
00:17:10  <ryah>socket.end()
00:17:20  <bnoordhuis>me too
00:17:24  <piscisaureus>right now we have shutdown()
00:17:26  <ryah>not in libuv
00:17:32  <DrPizza>ryah: right but wouldn't it mean that callers would have to track their own write_reqs value, which would just duplicate what uv already tracks?
00:17:34  <bnoordhuis>a fire-and-forget method is much easier to use
00:17:39  <piscisaureus>ryah: well...
00:17:44  <DrPizza>it just seems strange to track it in uv, but to require the caller to track it manually
00:17:51  <DrPizza>or does unix uv not track it?
00:18:06  <bnoordhuis>unix knows how many pending writes there are
00:18:24  <bnoordhuis>(of course)
00:18:42  <piscisaureus>ryah: node's closeSoon relies on socket linger etc to make sure no data is discarded but this is somewhat complicated on windows
00:18:44  <piscisaureus>also for pipes
00:18:50  <ryah>DrPizza: they just need to have a callback on the last write
00:19:01  <ryah>DrPizza: they dont have to track the earlier ones
00:19:07  <piscisaureus>I'd like to have something that does "graceful close after flushing the write buffer"
00:19:21  <DrPizza>ryah: but it means they hav eto know the last write is the last write at the time they submit it
00:19:32  <DrPizza>oh I suppose they could submit a zero write
00:19:40  <DrPizza>(I guess)
00:19:48  <piscisaureus>DrPizza: just shutdown() and close from the shutdown callback will do
00:19:55  <piscisaureus>right now
00:20:09  <DrPizza>I thought you were changing shutdown to immediately close
00:20:13  <DrPizza>or were you changing close to immediately close
00:20:41  <piscisaureus>DrPizza: uv-unix changed close to immediately close (windows has always done this)
00:20:48  <DrPizza>oh
00:20:57  <DrPizza>k.
00:21:21  <DrPizza>is that approach equally applicable to udp?
00:21:34  <DrPizza>it probably won't be for files
00:21:44  <piscisaureus>udp has no shutdown
00:22:11  <ryah>windows is clearly broken in this area
00:22:14  <DrPizza>so there's no way without manual tracking to ensure all outstanding requests are done
00:22:44  <piscisaureus>ryah: why is windows broken?
00:22:54  <ryah>write() close() should work as it does on unix
00:23:13  <piscisaureus>ryah: it almost works like that for sockets
00:23:26  <piscisaureus>ryah: pipes was a big problem
00:25:16  <piscisaureus>ryah: pipes are a bit fucked actually. right now to be sure that all data reaches the other side you have to call uv_shutdown on the pipe. just uv_close() will discard the kernel send buffer. But uv_shutdown on the pipe actually closes it (but not the uv handle).
00:26:25  <ryah>hm.. do we not have a shutdown test using pipes/
00:26:27  <ryah>we should..
00:26:40  <piscisaureus>shutdown can't work properly on pipes in windows
00:26:48  <ryah>:/
00:26:54  <ryah>you can't have a half-duplex pipe?
00:27:17  <piscisaureus>ryah: yeah but you can't switch from full to half duplex
00:27:32  <piscisaureus>ryah: but that's not the real problem
00:28:28  <piscisaureus>ryah: tcp semantics are that shutdown sends an EOF packet that will be noticed by the other side. We can't do this. So we could make shutdown a no-op that just ensures that all data in the kernel buffer is transmitted.
00:28:47  <piscisaureus>But the other side will never know that you want to close the connection so you will end up in a deadlock
00:29:06  <DrPizza>piscisaureus: it also marks the pipe as "shutdown" doesn't it, blocking further attempts to write to it?
00:29:23  <piscisaureus>DrPizza: yeah, this is implemented in libuv
00:29:30  <piscisaureus>it'll disallow further writes
00:29:31  <DrPizza>right
00:29:32  <DrPizza>yeah
00:29:43  <ryah>does the other side get marked?
00:29:48  <piscisaureus>DrPizza: but actually we CloseHandle on shutdown
00:29:56  <DrPizza>well, same thing
00:29:58  <DrPizza>it stops writes
00:30:10  <piscisaureus>DrPizza: unfortunately it also stops reads
00:30:15  <DrPizza>yes true
00:30:22  <DrPizza>well you could keep it open
00:30:25  <DrPizza>and just manually block writes
00:30:28  <piscisaureus>DrPizza: no
00:30:32  <DrPizza>no?
00:30:43  <ryah>the problem is the other side doesn't know the state of the pipe
00:30:51  <piscisaureus>exactly
00:31:02  <DrPizza>sure, but closing the handle doesn't change that does it?
00:31:16  <piscisaureus>DrPizza: if you close the pipe the other end will see ERROR_PIPE_CLOSED
00:31:21  <piscisaureus>so it knows :-)
00:31:22  <DrPizza>aah
00:31:29  <DrPizza>hmm
00:32:04  <piscisaureus>DrPizza: the bottom line is that you shouldn't expect tcp semantics from named pipes
00:32:26  <ryah>can you use GetNamedPipeHandleState() on the other side?
00:32:44  <piscisaureus>DrPizza: I'm pretty sure that protocols that are designed to work with named pipes have a protocol-layer graceful close.
00:33:17  <piscisaureus>ryah: what state are you interested in?
00:33:37  <piscisaureus>there is no "slot" for half-closed or something
00:34:13  <DrPizza>piscisaureus: yes I think they must
00:35:19  <piscisaureus>DrPizza: it could be a bit unfortunate for node users. They'll expect tcp-like semantics if they make pipes to other node processes. For connections to databases and such it probably won't matter
00:35:32  <DrPizza>the other option is you have two kinds of pipe
00:35:37  <DrPizza>a UV pipe which encapsulates its data
00:35:40  <DrPizza>and a raw pipe which doesn't
00:36:03  * graydonquit (Ping timeout: 258 seconds)
00:36:04  <DrPizza>so it can provided a built-in protocol for graceful closure
00:36:08  <DrPizza>maybe do it in node instead
00:36:09  <piscisaureus>DrPizza: yeah we may actually have this for ipc-type pipes
00:36:19  <piscisaureus>because then we could also send handles to the other side
00:36:22  <ryah>hmm sucks..
00:36:39  <ryah>so do you get UV_EOF immediately?
00:36:47  <ryah>when you call uv_shtudown on a pipe?
00:36:52  <piscisaureus>ryah: yeah
00:37:04  <piscisaureus>ryah: you might actually miss data that the other side is sending you
00:37:23  <DrPizza>you give the other side no chance to respond
00:37:59  <piscisaureus>DrPizza: I have actually tried to write 0 bytes to the other side
00:38:11  <piscisaureus>so the other end would see a read of 0 bytes
00:38:21  <DrPizza>as the shutdown protocol?
00:38:24  <piscisaureus>yes
00:38:35  <DrPizza>problem is that's still no good for connecting to 3rd party code
00:38:38  <piscisaureus>DrPizza: according to msdn this could work
00:38:46  <DrPizza>but for IPC sure
00:39:31  <piscisaureus>DrPizza: the issue is that if the kernel buffer is not empty at the time you write 0 bytes windows concatenates your 0 bytes to the rest of the send buffer
00:39:40  <DrPizza>ah yes
00:39:41  <piscisaureus>so the other end doesn't see it
00:39:45  <piscisaureus>DrPizza: although
00:39:51  <DrPizza>so that's why you do the buffer test
00:40:03  <piscisaureus>DrPizza: ?
00:40:08  <DrPizza>don't you?
00:40:10  <DrPizza>at the moment I mean
00:40:12  <piscisaureus>DrPizza: the buffer test?
00:40:24  <DrPizza>use some NTQIF function to test the kernel buffer?
00:41:04  <DrPizza>in the uv_pipe_endgame
00:41:12  <piscisaureus>DrPizza: no
00:41:18  <piscisaureus>DrPizza: oooh
00:41:30  <piscisaureus>DrPizza: that is just an optimization
00:41:53  <piscisaureus>DrPizza: to make sure the kernel buffer is flushed you have to call FlushFileBuffers on the pipe
00:42:00  <DrPizza>right
00:42:09  <piscisaureus>DrPizza: but there is no overlapped verion of that and it might block indefinitely
00:42:14  <DrPizza>so if you call FFB() zerowrite() that doesn't get coalesced?
00:42:34  <piscisaureus>DrPizza: not so fast...
00:43:31  <piscisaureus>DrPizza: so we run FlushFileBuffers in the thread pool. The NTQIF is just to check if the kernel buffer is already empty - because we can skip going to the thread pool then
00:43:39  <DrPizza>right I get that
00:43:47  <DrPizza>but once that's finished you know the buffer is empty
00:43:53  <piscisaureus>DrPizza: exactly
00:43:53  <DrPizza>and so a zero write won't get coalesced
00:43:55  <DrPizza>don't you?
00:44:05  <piscisaureus>I was just thinking that
00:44:07  <piscisaureus>that might work
00:44:12  <DrPizza>so you can always safely do a zero write
00:44:28  <piscisaureus>That would be pretty nice heh :-)
00:44:32  <DrPizza>piscisaureus: I think as a way of doing IPC pipes that should work
00:45:21  <DrPizza>piscisaureus: what intrigues me is that the NTQIF function can return a pipe state of "FILE_PIPE_CLOSING_STATE", almost as if pipes could exist in some half-way state, not quite closed, not quite open
00:45:40  <DrPizza>but I have no idea what could put the pipe into such a state
00:46:15  <piscisaureus>DrPizza: the only risk is that - say you're talking to mssql via a pipe and it inadvertedly sends you 0 bytes
00:46:24  <piscisaureus>DrPizza: libuv is going to mistake that for EOF
00:46:26  <DrPizza>piscisaureus: right, this is only safe for IPC pipes I think
00:46:33  <DrPizza>because it's still a custom protocol
00:46:35  <piscisaureus>for ipc pipes it's not really needed
00:46:49  <piscisaureus>Let me check FILE_PIPE_CLOSING_STATE
00:46:50  <DrPizza>well, ti's probably the simplest/lowest overhead protocol
00:47:20  <DrPizza>http://msdn.microsoft.com/en-us/library/dd414577%28v=PROT.10%29
00:47:25  <DrPizza>"The server end of the specified named pipe has been closed, but data is still available for the client to read."
00:47:32  <CIA-75>libuv: Igor Zinkovsky many_accepts * r8448ee4 / (include/uv-win.h src/win/tcp.c): Windows: Do simultaneous pending AcceptEx calls. - http://bit.ly/oqo0VE
00:47:41  <igorzi>piscisaureus: ---^
00:47:41  <piscisaureus>DrPizza: yeah isn't that weird
00:47:52  <DrPizza>piscisaureus: that feels very close to what you want
00:48:04  <DrPizza>maybe not exactly right, I dunno
00:48:49  <piscisaureus>DrPizza: I don't know how to put a pipe in that state. In my tests I found that CloseHandle would drop the kernel buffers.
00:49:04  <DrPizza>piscisaureus: no, nor me, and it gives no indication of how the pipe can get into that state
00:49:29  <piscisaureus>igorzi: can you still not email me the windows source code ;P
00:50:02  <piscisaureus>(btw Im reviewing ur patch)
00:55:43  * mjr__joined
00:56:33  * mjr__changed nick to mjr_
00:56:53  <mjr_>bnoordhuis: do you remember this patch 421b6e89aa58d1cdc3344d6fadb878f7b37eee1c
00:57:17  <mjr_>I'm wondering if it might help my mysterious https uncatchable exceptions.
00:57:24  <bnoordhuis>mjr_: yes
00:57:48  <mjr_>so what case did this one fix?
00:58:05  <bnoordhuis>that ReceivedShutdown always reported true
00:58:46  <igorzi>piscisaureus: if it was up to me - you'd already be looking at it :)
00:58:47  <mjr_>But how would you notice this?
00:59:00  <bnoordhuis>mjr_: can you rephrase that question?
00:59:30  <piscisaureus>igorzi: funny - somewhere in the libuv history we actually did post multiple accepts. But I removed that because it did not make a difference (on my local computer).
00:59:31  <mjr_>bnoordhuis: before that change went in, what would happen because that flag wasn't getting set properly?
00:59:40  <mjr_>bnoordhuis: connections would stay open after they should have been shutdown?
00:59:42  <piscisaureus>igorzi: btw, who *is* it up to then btw?
01:00:49  <DrPizza>steve sinofsky
01:01:16  <igorzi>piscisaureus: yep, it doesn't make any difference for me running locally either.. but it makes a pretty large difference (40-50% running over the network with a real workload)
01:01:23  <bnoordhuis>mjr_: nothing changes really, node core doesn't use that method
01:01:43  <mjr_>bnoordhuis: ahh, thanks.
01:02:13  <igorzi>(real workload = http_simple.js)
01:02:33  <piscisaureus>yes that is awesome
01:02:42  <piscisaureus>so 32 was the sweet spot
01:03:43  <igorzi>piscisaureus: yep, any higher or lower caused req/s to drop
01:04:43  <piscisaureus>http://www.microsoft.com/resources/sharedsource/mvp.mspx
01:04:43  <piscisaureus>^-- This is how one gets windows source code. I should go help grannies getting rid of adware...
01:04:53  <piscisaureus>igorzi: btw, patch looks good
01:04:59  <piscisaureus>+1 from me
01:05:10  <igorzi>piscisaureus: thanks
01:06:33  <piscisaureus>thanks yourself
01:08:35  <CIA-75>libuv: Igor Zinkovsky master * r8448ee4 / (include/uv-win.h src/win/tcp.c): Windows: Do simultaneous pending AcceptEx calls. - http://bit.ly/oqo0VE
01:09:16  <ryah>igorzi++
01:09:24  <ryah>not too often you get 100% perf improvements
01:10:25  <piscisaureus>ryah: you're seeing it too?
01:11:02  <ryah>no i havent tried
01:17:27  <piscisaureus>DrPizza: hey
01:17:52  <DrPizza>piscisaureus: hello
01:18:13  <piscisaureus>DrPizza: we may actually be able to switch a pipe from duplex to something else
01:18:35  <piscisaureus>Because this is mapped to share privileges under the hood
01:18:55  <piscisaureus>I just don't know if there is an api to do this. Do you?
01:19:07  <DrPizza>I think share privilege can only be specified at handle creation time
01:20:47  <DrPizza>I guess there may be some undocumented way of changing share state
01:20:49  <DrPizza>but I don't know it
01:20:55  <DrPizza>and I don't know how that woudl be communicated to the other end anyway
01:22:13  <piscisaureus>I'd think the other end would get a read error.
01:43:44  * isaacsquit (Quit: isaacs)
01:45:57  <piscisaureus>I'm going
01:46:06  <piscisaureus>I know it's early but
01:54:16  <bnoordhuis>sleep tight, piscisaureus
01:55:01  <piscisaureus>DrPizza: ReOpenfile. Sadly not on xp tho
01:55:31  <DrPizza>piscisaureus: oh interesting
01:56:03  <DrPizza>I did not know of that
01:56:09  <DrPizza>you can even force file handles to be overlapped
02:00:52  <piscisaureus>hmm. The sharing mode of the object. You cannot request a sharing mode that conflicts with the access mode specified in a previous open request whose handle is still open.
02:01:04  <piscisaureus>anyway. bye.
02:01:23  <DrPizza>bye
02:25:01  * isaacsjoined
02:30:18  * piscisaureusquit (Quit: ~ Trillian Astra - www.trillian.im ~)
02:33:18  * piscisaureusjoined
02:51:10  * piscisaureusquit (Ping timeout: 258 seconds)
02:56:44  * isaacsquit (Quit: isaacs)
03:02:11  * brsonquit (Ping timeout: 260 seconds)
03:36:29  * bnoordhuisquit (Read error: Operation timed out)
07:05:58  * mralephjoined
08:02:36  * mralephquit (Quit: Leaving.)
09:22:06  * mjr_quit (Quit: mjr_)
11:49:49  * piscisaureusjoined
12:08:50  * johnm1234quit (Ping timeout: 246 seconds)
12:15:56  * johnm1234joined
13:10:24  * bnoordhuisjoined
13:40:42  <piscisaureus>bnoordhuis: shouldn't uv_udp_send_s have a "handle" field?
13:40:56  <bnoordhuis>piscisaureus: maybe
13:41:05  <bnoordhuis>the other request types don't, i think
13:41:06  <piscisaureus>bnoordhuis: uv_write_t has it too
13:41:14  <bnoordhuis>yeah, but uv_connect_req hasn't
13:41:29  <piscisaureus>that may be dumb
13:41:46  <piscisaureus>I guess both unix and win define that in uv_connect_private_fields
13:41:47  <bnoordhuis>i'm okay with adding it, makes things easier for users
13:42:20  <piscisaureus>i'm adding it
13:42:43  <piscisaureus>also it needs a cb
13:42:44  <bnoordhuis>piscisaureus: want to hear a fun story?
13:42:48  <piscisaureus>sure
13:42:59  <piscisaureus>hear or "hear" ?
13:43:03  <bnoordhuis>so my ex co-workers at nrc have been battling with nrcnext.nl the whole morning
13:43:18  <bnoordhuis>the site was going down as often as your mom, so to speak
13:43:32  <bnoordhuis>they called me, asked if i wanted to step in
13:43:44  <bnoordhuis>so i tweaked the apache conf in five minutes
13:43:54  <bnoordhuis>problem solved and EUR 100 i can bill :)
13:44:02  <piscisaureus>ha!
13:44:04  <piscisaureus>way too little
13:44:15  <piscisaureus>you could have pulled 5 times as much :-)
13:44:25  <bnoordhuis>probably yes :)
13:44:38  <bnoordhuis>but the newspapers are falling on hard times already
13:45:39  <piscisaureus>bnoordhuis: well since nrc can still cover Peter Vandermeersch's travel expenses
13:45:53  <piscisaureus>(dwdd studio every other day)
13:46:00  <piscisaureus>they should be able to afford you
13:46:16  <bnoordhuis>true but peter v. lives in amsterdam nowadays
13:46:30  <bnoordhuis>amsterdam <-> hilversum is 30 minutes i think
13:46:34  <piscisaureus>guess who pays the rent?
13:46:41  <piscisaureus>dwdd is recorded in amsterdam btw
13:46:52  <bnoordhuis>yeah? didn't know that
13:47:07  <bnoordhuis>so he probably just pops across the street
13:47:31  <bnoordhuis>piscisaureus: 'also it needs a cb' <- why?
13:47:45  <piscisaureus>bnoordhuis: where are you going to stash the write_cb then?
13:47:51  <piscisaureus>oh send_cb
13:47:58  <bnoordhuis>in my, er, private parts
13:48:14  <piscisaureus>I like to share my private parts with you in public
13:49:04  <piscisaureus>btw: uv_write_s definition:
13:49:04  <piscisaureus>/* uv_write_t is a subclass of uv_req_t */
13:49:04  <piscisaureus>struct uv_write_s {
13:49:04  <piscisaureus> UV_REQ_FIELDS
13:49:04  <piscisaureus> uv_write_cb cb;
13:49:05  <piscisaureus> uv_stream_t* handle;
13:49:05  <piscisaureus> UV_WRITE_PRIVATE_FIELDS
13:49:06  <piscisaureus>};
13:49:22  <bnoordhuis>why is that public? it's an implementation detail
13:49:32  <piscisaureus>bnoordhuis: it is not public
13:49:35  <piscisaureus>it is just shared
13:49:45  <bnoordhuis>oh right
13:49:58  <bnoordhuis>well... i'm not the sharing type
13:58:17  <ryah>morning
13:58:28  <piscisaureus>mornin'
13:59:58  <piscisaureus>man. ryah is up at 7am
14:00:37  <piscisaureus>last time I was up at 7 it was just before I went to bed
14:01:18  <DrPizza>heh
14:01:33  <piscisaureus>bnoordhuis: what do we pass if recvfrom errs?
14:01:36  <piscisaureus>for the address
14:01:37  <piscisaureus>null?
14:02:06  <bnoordhuis>piscisaureus: yes
14:02:17  <piscisaureus>bnoordhuis: recv_cb doesn't even take a status or?
14:02:31  <bnoordhuis>piscisaureus: yes
14:02:36  <piscisaureus>oh nread
14:02:43  * piscisaureusfacepalm
14:02:59  <bnoordhuis>actually, i don't think recvfrom ever errors on unices unless you explicitly read out the error queue
14:03:15  <piscisaureus>I don't know
14:03:21  <piscisaureus>gotta handle it just in case
14:03:31  <piscisaureus>because we need to return the buffer
14:03:45  <bnoordhuis>that's what i do, handle it just in case
14:06:57  <piscisaureus>is it allowed to use the ? operator in c?
14:07:03  <piscisaureus>or is it bad style?
14:07:20  <bnoordhuis>the ternary operator?
14:07:22  <bnoordhuis>sure
14:07:32  <bnoordhuis>just don't go crazy
14:08:00  <piscisaureus>bnoordhuis: I don't like if, while, for etc
14:08:31  <piscisaureus>So I was thinking implementing this function using ?, setjmp / longjmp
14:08:33  <bnoordhuis>me neither, goto ftw
14:08:59  <piscisaureus>bnoordhuis: no I want to make it one big expression
14:09:10  <bnoordhuis>that's even better, that's job security
14:10:49  <piscisaureus>bnoordhuis: I guess nrc's httpd.conf looks like job security too :p
14:11:10  <bnoordhuis>piscisaureus: you should see the varnish default.vcl
14:11:25  <bnoordhuis>i wrote most of it :)
14:22:37  <ryah>we used to have something like sys.merge?
14:22:44  <ryah>do you guys remember what it was called?
14:22:51  <ryah>merged two objects
14:23:21  <piscisaureus>hmm not me
14:23:23  <ryah>mixin
14:23:31  <piscisaureus>I never actually use node
14:28:39  <ryah>me either
14:32:01  <piscisaureus>bnoordhuis: what was the trunc flag called again?
14:32:02  <piscisaureus>UV_TRUNC?
14:32:05  <piscisaureus>UV_UDP_TRUNC?
14:32:19  <bnoordhuis>piscisaureus: UV_UDP_PARTIAL
14:32:42  <piscisaureus>bnoordhuis: hmm. My headers are outdated
14:33:07  <bnoordhuis>piscisaureus: you know where to find them
14:33:14  <piscisaureus>bnoordhuis: sure
14:37:43  <bnoordhuis>udp on localhost works a little too well
14:37:53  <bnoordhuis>if you have a send_cb that sends off a new message
14:37:58  <bnoordhuis>you create an infinite loop :(
14:39:37  <piscisaureus>bnoordhuis: why?
14:39:47  <piscisaureus>don't you defer the send_cb until the next tick?
14:40:30  <bnoordhuis>piscisaureus: i do now
14:40:43  <bnoordhuis>but i tried to squeeze out a little extra performance
14:40:47  <bnoordhuis>well, it's complicated
14:41:04  <bnoordhuis>the bottom line is that sendmsg on localhost always succeeds, never returns EAGAIN
14:50:39  <piscisaureus>bnoordhuis: I think I am going to report some udp send
14:50:58  <piscisaureus>bnoordhuis: in case the user closes the socket before the send queue is flushed
14:51:22  <piscisaureus>"udp send" += "errors"
14:51:54  <bnoordhuis>piscisaureus: no need, the callbacks should fire
14:52:07  <bnoordhuis>that's apparently not fully implemented for tcp but it should
14:52:14  <piscisaureus>eh?
14:52:23  <bnoordhuis>i call the callback with status=-1 and error=UV_EINTR
14:52:30  <piscisaureus>ok yeah
14:52:38  <piscisaureus>that's implemented for tcp as well
14:52:50  <bnoordhuis>on windows? i don't think unix does it atm
14:53:01  <piscisaureus>windows always calls the write_cb for sockets
14:53:35  <bnoordhuis>needs a test!
14:53:46  <piscisaureus>bnoordhuis: it's tied deeply to how windows works. if you close a socket that has overlapped writes pending the kernel will notify you too. so we just pass the error on to the user
14:55:59  <ryah>more tests <3
15:04:39  <bnoordhuis>udp_packet_storm_1v1: 101632/s received, 101632/s sent
15:04:39  <bnoordhuis>udp_packet_storm_10v1: 102716/s received, 102716/s sent
15:04:39  <bnoordhuis>udp_packet_storm_100v1: 101864/s received, 101864/s sent
15:04:39  <bnoordhuis>udp_packet_storm_1000v1: 102136/s received, 102136/s sent
15:04:39  <bnoordhuis>udp_packet_storm_10v10: 172102/s received, 172104/s sent
15:04:40  <bnoordhuis>udp_packet_storm_100v10: 173084/s received, 173086/s sent
15:04:44  <bnoordhuis>udp_packet_storm_1000v10: 170744/s received, 170746/s sent
15:04:46  <bnoordhuis>udp_packet_storm_100v100: 174533/s received, 174553/s sent
15:05:23  <bnoordhuis>now to fix the 1000v1000 test :)
15:06:34  <bnoordhuis>EMFILE obviously...
15:07:41  <piscisaureus>wut?
15:07:53  <piscisaureus>100 000 / s
15:08:06  <ryah>bnoordhuis++
15:08:26  <ryah>udp = fast
15:08:45  <bnoordhuis>piscisaureus: give it a spin, it's in my repo
15:13:18  <piscisaureus>bnoordhuis: I just finished my implementation. Now I need to rebase on top of your crap
15:13:40  <piscisaureus>"crap" well crappy commits in a way
15:13:52  <piscisaureus>"WIP"
15:17:11  <ryah>given some library what's the best way to determine it's arch programatically?
15:17:28  <ryah>on unix
15:17:33  <bnoordhuis>objdump?
15:17:35  <ryah>parsing the output of file?
15:18:11  <ryah>objdump isnt on osx
15:18:57  <ryah>i need to determine the arch of the system for the gyp build
15:19:01  <ryah>which is super annoying
15:19:15  <ryah>waf does it by parsing gcc -E -dM -xc /dev/null
15:19:35  <bnoordhuis>so why don't you use that?
15:19:59  <ryah>i thought we could do it by checking what arch the libssl.so is
15:20:19  <bnoordhuis>that's kind of hacky
15:20:27  <bnoordhuis>won't work well on multi-arch systems either
15:20:44  <bnoordhuis>i have i386 and x64 libs installed, which one are you going to check?
15:21:02  <bnoordhuis>i'd probably look at `uname -r` and go from there
15:21:24  <bnoordhuis>eh, `uname -m`
15:23:17  <DrPizza>ryah: uname -a is standard
15:23:23  <DrPizza>checking that
15:23:38  <DrPizza>the v8 gyp uses uname's output
15:24:50  * isaacsjoined
15:27:39  <ryah>DrPizza: that doesn't work for computers with 32bit kernels and 64bit userland
15:27:51  <ryah>(most macs)
15:30:57  <DrPizza>I think most macs are now 64/64 aren't they?
15:31:10  <DrPizza>all the new ones are (should be)
15:31:13  <DrPizza>I think even my 2008 MBP is.
15:31:18  <ryah>http://news.cnet.com/8301-13579_3-10320314-37.html?tag=rtcol;pop
15:31:23  <DrPizza>though it may be the oldest model that defaults to 64/64
15:31:36  <ryah>probably in lion they have a 64bit kernel
15:32:06  <ryah>anyway - that throws out the possibility of using uname :)
15:32:12  <ryah>super annoying
15:32:25  <DrPizza>I think that article is wrong, or at least, insufficiently nuanced
15:32:39  <bnoordhuis>god, journalists :)
15:33:03  <ryah>ryan@mac1234:~/projects/node% file `which gcc`
15:33:04  <ryah>/Users/ryan/local/bin/gcc: Mach-O 64-bit executable x86_64
15:33:04  <ryah>ryan@mac1234:~/projects/node% uname -a
15:33:04  <ryah>Darwin mac1234.local 10.6.0 Darwin Kernel Version 10.6.0: Wed Nov 10 18:13:17 PST 2010; root:xnu-1504.9.26~3/RELEASE_I386 i386
15:33:40  <bnoordhuis>file `/bin/sh`?
15:33:45  <DrPizza>man if my dozy MBP would fucking wake up, I would take a look
15:34:07  <ryah>ryan@mac1234:~/projects/node% file /bin/sh
15:34:07  <ryah>/bin/sh: Mach-O universal binary with 2 architectures
15:34:07  <ryah>/bin/sh (for architecture x86_64): Mach-O 64-bit executable x86_64
15:34:07  <ryah>/bin/sh (for architecture i386): Mach-O executable i386
15:34:07  <ryah>ryan@mac1234:~/projects/node% file /opt/local/lib/libssl.dylib
15:34:10  <ryah>/opt/local/lib/libssl.dylib: Mach-O 64-bit dynamically linked shared library x86_64
15:34:23  <bnoordhuis>oh, that's sweet
15:34:27  <DrPizza>oh ffs
15:34:32  <DrPizza>I left a safari open looking at twitter
15:34:37  <DrPizza>twitter has now queued up 119714 messages
15:34:42  <DrPizza>and safari is very, very sad.
15:34:47  <bnoordhuis>haha
15:36:08  <DrPizza>what's the force quit shortcut
15:36:10  <DrPizza>I can never remember
15:36:40  <bnoordhuis>`pkill -9 safari.app`?
15:37:22  <DrPizza>bnoordhuis: yeah maybe if I could open a terminal...
15:37:46  <bnoordhuis>ssh into your machine :)
15:41:18  <DrPizza>Darwin macbookpro.local 11.0.0 Darwin Kernel Version 11.0.0: Sat Jun 18 12:56:35 PDT 2011; root:xnu-1699.22.73~1/RELEASE_X86_64 x86_64
15:41:24  <DrPizza>stupid mac os x
15:41:28  <DrPizza>it should not be macbookpro.local
15:43:06  <DrPizza>it surely should have picked up its domain name from DHCP
15:43:08  * DrPizzascowls
15:45:45  <ryah>https://github.com/joyent/node/blob/4cf931db1736a73717f4e6b9f11c3450de8606c6/tools/gyp/pylib/gyp/generator/make.py#L759
15:45:48  <ryah>:/
15:46:10  <DrPizza>heh
15:46:38  <bnoordhuis>i like uv_buf_init, less typing
15:50:20  <piscisaureus>bnoordhuis: it doesn't work yet but I have to go
15:50:24  <piscisaureus>I will put my branch up
15:50:44  <bnoordhuis>piscisaureus: right, say hi to your girlfriend from me
15:50:58  <piscisaureus>heh ok
15:51:43  <piscisaureus>bnoordhuis: https://github.com/piscisaureus/libuv/tree/udp
15:51:57  <bnoordhuis>piscisaureus: thanks, hf
15:52:07  <piscisaureus>thanks, yt
15:52:23  * piscisaureusquit (Quit: ~ Trillian Astra - www.trillian.im ~)
15:54:52  * piscisaureusjoined
15:59:00  <ryah>linking to ssl is such a pain in the ass
16:02:29  <DrPizza>pquerna will save us
16:04:40  * piscisaureusquit (Ping timeout: 258 seconds)
16:16:11  <ryah>http://www.google.com/codesearch#OAMlx_jo-ck/src/third_party/openssl/openssl.gyp
16:16:13  <ryah>:)
16:16:48  <DrPizza>ryah: doh
16:16:55  <DrPizza>ryah: I never even thought of checking chromium
16:18:14  <ryah>cant find it in subversion - must be in some experimental branch
16:18:16  <DrPizza>ryah: might be worth using those same patch sets too
16:23:40  <ryah>any idea how to download a tarball of http://src.chromium.org/viewvc/chrome/trunk/deps/third_party/openssl/ ?
16:23:50  * ryahis afraid of tyring to checkout chromium
16:23:55  <DrPizza>heh
16:25:18  <ryah>http://src.chromium.org/svn/trunk/src/build/linux/system.gyp
16:25:22  <ryah>^-- interesting how they use pkg-config
16:28:43  * ryahdownloads it..
16:28:50  <ryah>1.1 gig
16:29:03  <ryah>"The Chromium codebase consists of hundreds of thousands of files, which means that a checkout straight from the Subversion (SVN) repository can take a long time. To speed up the process, we have provided a tarball that you can use to bootstrap the download. "
16:29:42  <DrPizza>oh yeah, you grab the tarball and then run their funky script whose name I dont' remember to make it up-to-date
16:31:52  <bnoordhuis>firefox > chrome still
16:31:56  <bnoordhuis>code base-wise anyway
16:32:02  <bnoordhuis>slightly over 1.2 gb
16:33:02  <ryah>all this just to download openssl
16:33:06  <ryah>i hope its in this tarball
16:36:43  <bnoordhuis>download openssl? openssl.org?
16:36:53  <bnoordhuis>or is that whoosh sound i'm hearing a joke that's going over my head?
16:38:35  <DrPizza>bnoordhuis: chrome's openssl has gyp building and some possibly interesting patches
16:39:01  <bnoordhuis>oh right, thanks - missed that part
16:40:29  * rmustaccjoined
16:40:55  <bnoordhuis>hey rmustacc, congrats
16:45:21  <rmustacc>Thanks bnoordhuis
16:46:02  <bnoordhuis>i know the kvm source a little, it must've been a right bastard to port
17:04:54  <rmustacc>Mm, their code is pretty clean. Just a trick to try and understand it all.
17:05:03  <rmustacc>Debugging is really the harder problem.
17:05:17  <rmustacc>Or maybe I only say it's clean because I've been living in it for the last six months.
17:10:51  * graydonjoined
17:16:39  * brsonjoined
17:53:19  * mralephjoined
17:57:23  <igorzi>ryah: piscisaureus: https://gist.github.com/1157497
17:57:34  <igorzi>the api for eio stuff in libuv
18:32:37  * ryahquit (Ping timeout: 250 seconds)
18:32:37  * mralephquit (Ping timeout: 250 seconds)
18:36:40  * slurp1joined
18:38:19  <igorzi>ryah_: https://gist.github.com/1157497
18:38:29  * mralephquit (*.net *.split)
18:38:33  * slurpquit (*.net *.split)
18:38:34  * ryahquit (*.net *.split)
18:40:55  * mralephjoined
18:56:23  <bnoordhuis>call time?
19:26:58  <ryah_>hey
19:27:02  <ryah_>im very sorry
19:27:16  <ryah_>i had a meeting that ran late
19:27:30  <ryah_>can we do the call now?
19:28:02  <ryah_>well i'll just call you guys and see how it ges
19:28:58  <bnoordhuis>sure
19:38:02  <ryah_>my computer is still untaring chrome
19:38:08  <ryah_>its been like 3 hours
19:38:13  <ryah_>both cpus at 100%
20:04:02  * ryah_quit (Quit: leaving)
20:04:09  * ryahjoined
20:23:56  <DrPizza>ryah: lol
20:24:05  <DrPizza>can't svn do partial checkouts these days?
20:24:10  <DrPizza>if it can, I think that would have been quicker
20:26:08  <ryah>if dropping openssl.gyp into node just builds out of the box im going to be fucking shocked
20:26:19  <ryah>and will declare GYP the king of all build systems
20:26:30  <ryah>a problem that has broken great men for ages
20:26:50  <ryah>it seems to be working
20:26:52  <ryah>ryan@mac1234:~/projects/node% make -f Makefile-gyp
20:26:52  <ryah>tools/gyp_node -f make
20:26:52  <ryah>['-f', 'make', '/Users/ryan/projects/node/node.gyp', '-I', '/Users/ryan/projects/node/common.gypi', '-I', '/Users/ryan/projects/node/options.gypi', '--depth=.', '--generator-output', '/Users/ryan/projects/node/out', '-Goutput_dir=/Users/ryan/projects/node/out', '-Dtarget_arch=ia32', '-Dcomponent=static_library', '-Dlibrary=static_library']
20:26:57  <ryah>make -C out BUILDTYPE=Release CC(target) /Users/ryan/projects/node/out/Release/obj.target/openssl/deps/chrome-openssl/openssl/ssl/bio_ssl.o CC(target) /Users/ryan/projects/node/out/Release/obj.target/openssl/deps/chrome-openssl/openssl/ssl/d1_both.o CC(target) /Users/ryan/projects/node/out/Release/obj.target/openssl/deps/chrome-openssl/openssl/ssl/d1_clnt.o
20:27:03  <ryah> CC(target)/Users/ryan/projects/node/out/Reelease/obj.target/openssl/deps/chrome-openssl/openssl/ssl/d1_enc.o CC(target) /Users/ryan/projects/node/out/Release/obj.target/openssl/deps/chrome-openssl/openssl/ssl/d1_lib.o
20:27:09  <ryah>one line change to node.gyp
20:27:22  <DrPizza>:o
20:27:42  <ryah>there's no way this is going to work
20:27:47  * ryahwaits for the build fail
20:28:03  <ryah>doesn't openssl require perl or something to build?
20:28:08  <DrPizza>yup
20:28:17  <ryah>the gyp file doesn't ...
20:28:21  <DrPizza>it uses some perl script to generate recursive makefiles
20:28:22  <ryah>this can't be right
20:28:58  <DrPizza>ryah: the openssl build process is stupid, AFAICT it doesn't actually do anything very clever
20:29:10  <DrPizza>it's perfectly feasible to me that someone could create build scripts that don't use perl
20:29:28  <DrPizza>I don't think it's doing anything as complex as, say, js2c
20:29:48  <ryah>i thought it was like doing strange random number insertions into the object code
20:29:54  <ryah>build failed
20:30:19  <ryah> CC(target) /Users/ryan/projects/node/out/Release/obj.target/openssl/deps/chrome-openssl/openssl/crypto/ui/ui_openssl.o
20:30:22  <ryah>../deps/chrome-openssl/openssl/crypto/ui/ui_openssl.c:227:21: warning: termio.h: No such file or directory
20:30:25  <ryah>i knew it was too good to be true
20:35:05  <ryah>libtool: file: /Users/ryan/projects/node/out/Release/obj.target/openssl/deps/chrome-openssl/openssl/ssl/kssl.o has
20:35:13  <ryah> LIBTOOL-STATIC /Users/ryan/projects/node/out/Release/obj.target/deps/chrome-openssl/libopenssl.a
20:35:17  <ryah>:D
20:36:38  <DrPizza>is that good or bad?
20:37:30  <ryah>!!!
20:37:31  <ryah>% ./out/Release/node
20:37:31  <ryah>> process.versions
20:37:32  <ryah>{ node: '0.5.5-pre',
20:37:32  <ryah> v8: '3.5.6',
20:37:33  <ryah> ares: '1.7.4',
20:37:36  <ryah> uv: '0.1',
20:37:39  <ryah> openssl: '0.9.8o' }
20:37:41  <ryah>> process.features
20:37:43  <DrPizza>ooh
20:37:44  <ryah>{ uv: false,
20:37:46  <ryah> http1: false,
20:37:49  <ryah> ipv6: true,
20:37:51  <ryah> tls_npn: true,
20:37:54  <ryah> tls_sni: true,
20:37:56  <ryah> tls: true }
20:37:57  <DrPizza>ryah: so why 0.9.8 and not 1.whatever.it.is
20:37:59  <ryah>it fucking worked!
20:38:07  <DrPizza>lol
20:38:52  * ryahuploads a branch
20:39:27  <DrPizza>ryah: I'll check it out, see what needs fixing (if anything!)
20:51:56  <ryah> 18Mopenssl
20:51:57  <ryah>er
20:52:02  <ryah>it's 18 MB
20:52:03  <ryah>:/
20:54:52  <bentkus>process.features?
20:55:12  <bentkus>why should process contain that info :/
20:55:59  <bentkus>more like node.features
20:57:01  <ryah>bentkus: don't get all religous here - we have actual problems
20:57:18  <bentkus>:)
20:59:22  * felixgejoined
20:59:22  * felixgequit (Changing host)
20:59:22  * felixgejoined
21:05:12  <CIA-75>node: Ryan Dahl gyp-openssl * r13b853e / (1579 files in 122 dirs): import openssl from chrome - http://bit.ly/r2wUtG
21:05:14  <ryah>DrPizza: --^
21:05:18  <DrPizza>ryah: cool
21:05:25  <DrPizza>I will look in a couple of hours
21:05:29  <DrPizza>when my working day is done
21:17:42  <DrPizza>ryah: the biggest issue is that if we wanted to track chrome's upstream openssl then it would probably be a PITA
21:17:46  <DrPizza>I'm not saying we'd want to do that necessarily
21:17:56  <DrPizza>but if they're going to continue to maintain gyp files for openssl
21:18:04  <DrPizza>it would bemore convenient than having to maintain them ourselves
21:25:04  <ryah>yeah
21:25:18  <ryah>i figured i can just ignore that for nwo
21:30:47  <bnoordhuis>ryah: https://github.com/bnoordhuis/libuv/compare/udp <- if you're up for a code review
21:31:24  <ryah>bnoordhuis: ok
21:33:02  <bnoordhuis>oh, forgot to add more doc comments to uv.h
21:33:48  <ryah>bnoordhuis: start multiline comments with /*
21:33:58  <ryah>bnoordhuis: i gues that's the style we're going with
21:34:10  <bnoordhuis>ryah: eh?
21:34:26  <bnoordhuis>you mean /* comment 1 */\n/* comment 2 */ ?
21:35:07  <ryah> /*\n * blah blah\n * blah blah\n */
21:35:21  <ryah>instead of /* blah blah\n * blah blah\n */
21:35:25  <bnoordhuis>oh right
21:46:48  * isaacsquit (Quit: isaacs)
22:09:02  <bnoordhuis>... what should happen if someone calls uv_udp_send with bufcnt==0?
22:10:15  <ryah>bnoordhuis: *shrug*
22:10:19  <ryah>EINVAL
22:11:03  <bnoordhuis>cool
22:16:16  <igorzi>ryah: damn.. that patch to preallocate read buffers (instead of 0-size buffer) actually caused a pretty large regression with node (~20%)
22:16:42  <ryah>igorzi: oh no..
22:16:52  <igorzi>could that be because of node's slab allocator?
22:16:53  <ryah>how big were the buffers?
22:16:55  <ryah>yeah
22:17:04  <ryah>64k?
22:17:07  <igorzi>yep
22:17:20  <ryah>with what concurrency? what bench?
22:17:56  <igorzi>http_simple, 64 connections
22:18:19  <ryah>igorzi: would be interesting to see a profile of that
22:19:10  <ryah>i think normally node gets away with reusing most of the slab
22:19:24  <ryah>but with the prealloced read buffers it can't do that anymore
22:19:35  <ryah>we're just allocating massive numbers of these slabs...
22:20:07  <igorzi>ryah: ok, i'll get the profile; i was comparing all the perf improvements against node 0.5.4; and i noticed that we weren't actually that much better. i tried backing out some changes.. and this one caused a pretty large regression
22:20:51  <ryah>igorzi: maybe we need to invent a better way of doing the buffer allocs...
22:20:57  <bnoordhuis>can't we peek at the # of bytes actually available with ioctl(FIONREAD)?
22:21:43  <ryah>bnoordhuis: they need to give the kernel the buffer before data arrives
22:21:53  <bnoordhuis>oh, poor windows developers
22:21:56  <ryah>yes
22:22:14  <ryah>http://antirobotrobot.tumblr.com/post/5094720849/memory-is-the-price-you-pay-for-the-semantics-of
22:22:28  <bnoordhuis>hah
22:23:06  <bnoordhuis>is restricting the buffer size to the MTU size + a bit more an option?
22:23:39  <rmustacc>Which MTU size?
22:23:55  <rmustacc>Lots of places run with Jumbo frames for example.
22:23:58  <igorzi>well, we don't have to.. we were doing 0-reads before, but decided to preallocate if the number of active streams is below 50
22:24:23  <ryah>with large receive offloading limiting the buffer to the MTU is suboptimal
22:24:42  <ryah>even 64k is too small
22:25:02  <ryah>256kb is what modern ethernet controls can DMA
22:25:56  <ryah>(iirc)
22:27:04  <ryah>(my new hobby is reading datasheets for ethernet controllers)
22:32:00  * jonaslundquit (Quit: ChatZilla 0.9.87-rdmsoft [XULRunner 1.9.2.13/20101203074205])
22:34:46  <bnoordhuis>rmustacc: you'd ask the OS
22:34:53  <bnoordhuis>not sure if i follow ryah's argument
22:35:11  <bnoordhuis>but maybe i'm misunderstanding how those prealloc'd buffers work on windows
22:35:28  <ryah>bnoordhuis:
22:35:29  <ryah>[% 0|+ 20|- 0]: udp_bind_twice_v4
22:35:29  <ryah>`udp_bind_twice_v4` failed: exit code 6
22:35:29  <ryah>Output from process `udp_bind_twice_v4`:
22:35:29  <ryah>Assertion failed in test/test-udp-bind-twice.c on line 68: r == 0
22:35:31  <ryah>=============================================================
22:36:12  <ryah>bnoordhuis: expected?
22:36:15  <bnoordhuis>no
22:36:19  <bnoordhuis>can you strace `tests/run-tests udp_bind_twice_v4 udp_bind_twice_v4`?
22:36:26  <bnoordhuis>or dtruss
22:38:26  <ryah>bnoordhuis: https://gist.github.com/cf6cb69a54d5c394cfd7
22:38:40  <bnoordhuis>thanks
22:39:38  <bnoordhuis>what is errno 48 on darwin? EADDRINUSE?
22:40:19  <ryah>yes
22:41:17  <igorzi>ryah: i also noticed that i get this with that prealloc change
22:41:18  <igorzi>assert.js:104
22:41:25  <igorzi> throw new assert.AssertionError({
22:41:27  <igorzi><error: TypeError: Converting circular structure to JSON>
22:41:30  <igorzi> at TCP.onread (net_uv.js:277:10)
22:41:33  <igorzi>(sometimes)
22:41:59  <bnoordhuis>igorzi: ^ that's broken for some time now
22:42:03  <igorzi>(and only with that buffer prealloc patch)
22:42:21  <bnoordhuis>i sometimes see it to on linux
22:42:26  <ryah>on what test?
22:42:50  <igorzi>i see it very often with http_simple when the prealloc buffer patch is applied
22:42:54  <ryah> assert.equal(handle, self._handle);
22:43:02  <igorzi>and i haven't seen it at all without that patch
22:43:38  <ryah>hm...
22:47:39  <ryah>that seems like a real bug
22:47:54  <ryah>it would be good to root cause it
22:48:00  <ryah>but im taking off now.. bbl
23:00:59  <CIA-75>node: Maciej Małecki v0.4 * re150bc4 / doc/api/process.markdown : docs: process.memoryUsage returns memory usage measured in bytes - http://bit.ly/mZE3av
23:01:00  <CIA-75>node: Maciej Małecki master * r962a9e8 / doc/api/process.markdown : docs: process.memoryUsage returns memory usage measured in bytes - http://bit.ly/oA0Aql
23:10:36  * mralephquit (Quit: Leaving.)
23:13:56  * piscisaureusjoined
23:17:39  * felixgequit (Quit: felixge)
23:19:45  * piscisaureusquit (Ping timeout: 258 seconds)
23:54:43  * graydonquit (Quit: Leaving.)