00:01:31  <ryah>i've never used it
00:01:37  <ryah>looking forward to trying it out
00:01:51  <ryah>the guy who wrote it is a long time friend
00:01:59  <ryah>and a very very good programmer
00:02:10  <ryah>so i expect it to be awesome
00:02:32  <bnoordhuis>well, color me curious
00:04:06  <ryah>it's so awesome that GYP is very high-level and can support really good platform specific stuff
00:04:10  <ryah>liek this: http://code.google.com/p/gyp/source/detail?r=1002
00:04:19  <ryah>i have no idea what the hell that is
00:04:27  <ryah>but i'm glad node can somehow use it
00:07:22  <CIA-75>node: Ryan Dahl master * rc2ae39b / node.gyp : gyp: -ldl on linux - http://git.io/DZKRcg
00:10:41  <CIA-75>node: Ryan Dahl master * r80dd818 / (10 files in 3 dirs): Upgrade GYP to r1010 - http://git.io/A55c3w
00:11:13  <ryah>bnoordhuis: ./tools/gyp_node -f ninja
00:13:45  <bnoordhuis>ryah: NotImplementedError: --generator_output not implemented for ninja :(
00:14:07  <ryah>bnoordhuis: yeah..
00:14:25  <bnoordhuis>work in progress probably
00:15:41  <ryah>just a sec..
00:18:55  <ryah>yeah - i can't get it to work either
00:18:56  <ryah>oh well
00:19:02  <ryah>i guess we'll have to wait another week
00:19:05  <ryah>or so
00:20:14  * brson_joined
00:22:30  * brsonquit (Ping timeout: 276 seconds)
00:32:26  * graydonquit (Quit: Leaving.)
00:53:33  <bnoordhuis>piscisaureus: still around homie?
00:55:05  <piscisaureus>bnoordhuis: yes but not too long
00:55:09  <piscisaureus>what's up?
00:55:17  <bnoordhuis>this
00:55:34  <bnoordhuis>you mentioned that calling the recv_cb with nread==0 is an implementation detail?
00:55:49  <bnoordhuis>is that still true after tonight's conference call?
00:55:55  <piscisaureus>bnoordhuis: that per se is not an implementation detail but
00:56:07  <piscisaureus>bnoordhuis: the moment when that happens is
00:56:23  <bnoordhuis>piscisaureus: right - what do you propose?
00:56:51  <piscisaureus>bnoordhuis: applications should be ready to handle that case at any time, and not rely on it happening ever
00:57:19  <piscisaureus>bnoordhuis: what do I need to propose?
00:57:36  <bnoordhuis>piscisaureus: that doesn't sound acceptable, that's saying "i'm okay with memory leaks"
00:57:47  <piscisaureus>eh?
00:57:55  <bnoordhuis>'not rely on it happening ever'
00:58:18  <bnoordhuis>only hippies and commies are okay with memory leaks >:(
00:58:38  <piscisaureus>bnoordhuis: you are misrepresenting what I said :-(
00:58:58  <bnoordhuis>piscisaureus: no, i'm interpreting your words literally
00:59:16  <piscisaureus>bnoordhuis: applications should handle that case - by freeing the buffer
00:59:50  <bnoordhuis>piscisaureus: yeah, but they can't if libuv hasn't signalled that the buffer's released again
01:00:12  <piscisaureus>bnoordhuis: eh? did you drink too much?
01:00:25  <piscisaureus>bnoordhuis: I said, they should handle it by freeing the buffer
01:00:34  <bnoordhuis>piscisaureus: i suspect you didn't mean to add 'not rely on it happening ever' part
01:00:47  <piscisaureus>bnoordhuis: yes I am serious about that
01:00:48  <ryah>im getting a really crazy error
01:01:23  <bnoordhuis>piscisaureus: that's what i mean by mem leak - if libuv doesn't tell you that the buffer is cleared again, you can't free it
01:01:34  <bnoordhuis>ryah: what is it?
01:01:47  <piscisaureus>bnoordhuis: libuv will always give you back the buffer one way or another
01:02:05  <ryah>well some test is failing - but basiclly i can't im pretty sure im making a syscall - but i cant see it in dtrace
01:02:19  <bnoordhuis>piscisaureus: hence i figured you didn't really mean the 'not rely on it happening ever' part :)
01:02:34  <piscisaureus>bnoordhuis: the nread==0 case is the escape hatch for libuv, when it has requested a buffer that it doesn't need after all
01:02:39  <bnoordhuis>ryah: maybe it's not getting past the libc wrapper?
01:03:30  <ryah>bnoordhuis: that's what i was wondering - does apple have the libc open sourced?
01:04:01  <bnoordhuis>piscisaureus: so nread > 0 means 'here is the dgram, btw the buffer is yours again too kthxbye'?
01:04:05  <piscisaureus>bnoordhuis: about "not rely on it" - your udp tests (at least the send_and_recv test) only works if the makes the recv callback with nread==0 at some point. But there is no guarantee that libuv will ever do that, nread==0 is only an escape hatch, it has no further meaning. So your test should not rely on that ever happeing.
01:04:15  <piscisaureus>bnoordhuis: yes
01:04:19  <bnoordhuis>ryah: i think they have, i was digging through the dlopen parts recently
01:04:28  <piscisaureus>bnoordhuis: if libuv returns a buffer to you it's yours again
01:04:47  <bnoordhuis>piscisaureus: i should fix that in uv-unix then :)
01:04:59  <bnoordhuis>piscisaureus: sorry my bad
01:05:02  <ryah>http://www.opensource.apple.com/source/Libc/ <-- looks good
01:05:14  <piscisaureus>bnoordhuis: :-)
01:05:21  <ryah>if only i could grep it..
01:07:27  * ryahfound the download..
01:08:09  <DrPizza>ryah: did you see my question the other day, do you know where one might find NPN and other patches for openssl 1.0?
01:08:20  <DrPizza>if tehy do not exist then sticking with 0.9.8 may be better
01:08:34  <ryah>DrPizza: indutny is going to apply the patches
01:08:52  <ryah>i think he's asleep right now
01:08:59  <ryah>but he was in the middle of it
01:09:02  <DrPizza>oh he has suitable patches for 1.0?
01:09:18  <DrPizza>I looked briefly but did not see anything immediately obvious
01:09:21  <ryah>nos ure
01:09:23  <DrPizza>perhaps the 0.9.8 patches will apply
01:09:26  <ryah>not sure..
01:09:41  <DrPizza>but applying and working are two different things
01:11:51  <bnoordhuis>piscisaureus: makes me wonder if it'd be worthwhile to peek with ioctl(FIONREAD)
01:12:22  <bnoordhuis>piscisaureus: having the libuv consumer do a potentially expensive malloc when there's nothing to read possibly outweighs the cost of a second syscall...
01:12:41  <bnoordhuis>'benchmark it' you'll say and you'd be right of course
01:13:09  <DrPizza>malloc feasibly has to syscall anyway
01:13:28  <bnoordhuis>the dlmalloc implementation in glibc is pretty smart
01:13:35  <piscisaureus>bnoordhuis: about nread==0 again -- the "here is the dgram, btw the buffer is yours again too kthxbye" is what we had in mind, it seemed the most efficient thing we could do. But occiasonally (rarely!) we would ask for a buffer and not use it. That's why the nread==0 case was added (which is unfortunate because commonly nread==0 means EOF)
01:13:37  <bnoordhuis>small allocs come out of a special areane
01:13:40  <bnoordhuis>*arena
01:13:55  <piscisaureus>bnoordhuis: you could try that
01:14:22  <piscisaureus>bnoordhuis: but node's slab allocator is optimized for this
01:14:41  <bnoordhuis>piscisaureus: that's what happens now with every fd that becomes readable
01:15:07  <piscisaureus>bnoordhuis: that's wrong. It should not happen on windows
01:15:18  <piscisaureus>s/should/does/
01:15:38  <bnoordhuis>piscisaureus: let me link you to the recvmsg loop
01:15:44  <ryah>so the read is a pure syscall
01:15:50  <ryah>wtf
01:16:15  <bnoordhuis>piscisaureus: https://github.com/bnoordhuis/libuv/compare/udp#L3R500 <- this
01:16:25  <piscisaureus>bnoordhuis: yes, recvmsg is a special case. I though that you were referring to tcp.
01:16:33  <bnoordhuis>piscisaureus: what's tcp?
01:16:47  <bnoordhuis>of course i was talking about udp >:(
01:17:03  <piscisaureus>"that's what happens now with every fd that becomes readable"
01:17:16  <piscisaureus>"every"
01:17:20  <bnoordhuis>piscisaureus: oh sorry, right - every udp fd that becomes readable
01:17:23  <bnoordhuis>*udp
01:17:28  <bnoordhuis>damnit, it's getting late
01:17:31  <piscisaureus>bnoordhuis: yeah ok
01:17:47  <piscisaureus>bnoordhuis: I understand that recvmsg is a problem
01:18:11  <piscisaureus>Using fionread would be justifiable I think
01:18:25  <bnoordhuis>it does, doesn't it?
01:18:33  <bnoordhuis>needs a good benchmark though...
01:19:13  <piscisaureus>bnoordhuis: https://github.com/piscisaureus/libuv/blob/udp/src/win/udp.c#L491-526 :-)
01:19:32  <bnoordhuis>piscisaureus: haha
01:20:00  <piscisaureus>bnoordhuis: maybe this is going away btw since 0-reads don't work currently (MSG_PARTIAL support seems to be completely broken on my windows)
01:23:01  <bnoordhuis>piscisaureus: okay, think with me - what's a good approach to measuring the impact of FIONREAD?
01:23:22  <piscisaureus>bnoordhuis: hmm. do it a lot?
01:23:39  <piscisaureus>or run the packet storm benchmark
01:23:51  <bnoordhuis>with a matching recvmsg yes
01:24:11  <bnoordhuis>yeah, let's see what udp_packet_storm says
01:26:57  <piscisaureus>bnoordhuis: I bet fionread itself is pretty cheap, and it's cost is dominated by syscall overhead
01:27:52  <piscisaureus>bnoordhuis: if you can know number of syscalls that linux can do per second then you should be able to get an idea
01:28:03  <piscisaureus>damn, getting late yeah
01:28:22  <piscisaureus>s/can know/know the/
01:28:33  <bnoordhuis>piscisaureus: the fastest syscalll on my system takes about 0.75us
01:29:07  <bnoordhuis>piscisaureus: but a lot depends on the code path in the kernel and the amount of kernel-side locking involved
01:31:35  <piscisaureus>bnoordhuis: does FIONREAD return the size of the entire buffer or just the next message
01:31:56  <piscisaureus>(on windows it does either, depending on whether you call ioctlsocket or WSAIoclt :-/)
01:32:25  <bnoordhuis>piscisaureus: er... entire buffer i think
01:32:29  <bnoordhuis>i'd have to check though
01:34:31  <bnoordhuis>at a glance, the size of everything that's still in the queue
01:36:57  <piscisaureus>so you can just call FIONREAD once and then drain the kernel buffer
01:37:08  <piscisaureus>sounds like something you could do without hurting performance
01:38:10  <bnoordhuis>i'm wrong!
01:38:25  <bnoordhuis>FIONREAD is an alias for SIOCINQ
01:38:34  <piscisaureus>yes
01:38:34  <piscisaureus>FIONREAD (SIOCINQ)
01:38:34  <piscisaureus> Gets a pointer to an integer as argument. Returns the size of the next
01:38:34  <piscisaureus> pending datagram in the integer in bytes, or 0 when no datagram is
01:38:34  <piscisaureus> pending. Warning: Using FIONREAD, it is impossible to distinguish the
01:38:34  <piscisaureus> case where no datagram is pending from the case where the next pending
01:38:35  <piscisaureus> datagram contains zero bytes of data. It is safer to use select(2),
01:38:35  <piscisaureus> poll(2), or epoll(7) to distinguish these cases.
01:38:36  <bnoordhuis>and that returns the size of the first packet
01:39:04  <bnoordhuis>pretty confusing, it works completely different for file i/o :/
01:41:07  <piscisaureus>bnoordhuis: the other option you have is just read one dgram at a time
01:41:29  <piscisaureus>or is epoll overhead very big?
01:42:03  <bnoordhuis>piscisaureus: epoll overhead?
01:42:15  <piscisaureus>bnoordhuis: yes
01:42:28  <bnoordhuis>you mean do a quick epoll check after every dgram to see if there's more in the queue?
01:42:33  <piscisaureus>bnoordhuis: no
01:42:51  <bnoordhuis>^ because that'd be insanely expensive
01:43:15  <piscisaureus>bnoordhuis: I mean - if you read only one dgram then what libuv is going to do before the next read is epoll right?
01:43:34  <bnoordhuis>piscisaureus: no
01:43:46  <bnoordhuis>piscisaureus: it loops until recvmsg returns EAGAIN
01:43:57  <bnoordhuis>piscisaureus: and it should become there might be more than one packet pending
01:44:20  <piscisaureus>bnoordhuis: yes - but EAGAIN is what you are trying to avoid right?
01:44:25  <bnoordhuis>piscisaureus: yes
01:44:40  <piscisaureus>so - what happens if you read only once?
01:44:59  <bnoordhuis>piscisaureus: things will go bad
01:45:14  <piscisaureus>because?
01:45:28  <bnoordhuis>you really need to consume all data before you put a socket back in the epoll queue
01:45:45  <piscisaureus>bnoordhuis: that sounds like nonsense to me
01:45:56  <bnoordhuis>funny things happen if you don't, on older kernels anyway
01:46:11  <bnoordhuis>it depends on if your epoll is edge or level triggered
01:46:36  <piscisaureus>I bet libev uses EPOLLT
01:46:58  <bnoordhuis>you mean EPOLLET? i won't bet against you
01:46:59  <piscisaureus>since that is closest to what the poll and select backend also provide
01:47:22  <piscisaureus>no EPOLLET == edge triggered
01:47:32  <piscisaureus>I bet it uses level-triggered
01:47:43  <bnoordhuis>wait, here's the relevant part
01:47:44  <bnoordhuis>An application that employs the EPOLLET flag should use non-blocking file descriptors to avoid having a blocking read or write starve a task that is handling multiple file descriptors. The suggested way to use epoll as an edge-triggered (EPOLLET) interface is as follows:
01:47:44  <bnoordhuis> i with non-blocking file descriptors; and
01:47:45  <bnoordhuis> ii by waiting for an event only after read(2) or write(2) return EAGAIN.
01:48:01  <piscisaureus>bnoordhuis: that's for edge-triggered?
01:48:04  <bnoordhuis>yes
01:48:13  <piscisaureus>bnoordhuis: that's not what libev uses?
01:48:25  <piscisaureus>er. bnoordhuis: that's not what libev uses!
01:48:36  <bnoordhuis>it *is* what libev uses
01:49:17  <bnoordhuis>anyway, it's sound advice
01:49:24  <piscisaureus>bnoordhuis: grep EPOLLET -r *
01:49:33  <piscisaureus>(awkward silence)
01:49:38  <bnoordhuis>it may have been fixed but one issue with EPOLLET was that fds would never become readable again
01:49:53  <bnoordhuis>(if you didn't consume all data)
01:50:05  <piscisaureus>bnoordhuis: that makes sense. that's what edge-triggered means
01:50:20  <bnoordhuis>and even if libev doesn't use it now, i'm still going to take the defensive programmer approach here
01:50:20  <piscisaureus>bnoordhuis: libev uses level-triggered epoll
01:53:20  <DrPizza>flemish is dutch right?
01:53:24  <DrPizza>(+/-)
01:53:36  <piscisaureus>flemish is not a language
01:53:43  <piscisaureus>it's a condition
01:53:46  <DrPizza>lol
01:53:59  <DrPizza>what does "door de buffalo's" mean
01:54:04  <DrPizza>I know that buffalo's refers to the football team
01:54:12  <piscisaureus>it means "by the buffalos"
01:54:17  <DrPizza>ah
01:54:29  <DrPizza>oh shit yes
01:54:35  <DrPizza>now ryan paris
01:54:38  <DrPizza>ok this mix just got a lot better
01:55:23  * piscisaureusconfused
01:55:34  <DrPizza>radio soulwax/2manydj's
01:55:43  <bnoordhuis>piscisaureus: https://gist.github.com/8a133efad25706da2495 <- numbers
01:56:40  <bnoordhuis>so FIONREAD does have overhead :(
01:57:31  <DrPizza>over how many runs
01:57:54  <bnoordhuis>DrPizza: median of three
01:58:15  <bnoordhuis>the numbers were really close btw
01:58:26  <bnoordhuis>so it's pretty stable apparently
01:58:47  <DrPizza>hmm
01:59:09  <piscisaureus>not the kind of overhead I worry about btw
01:59:24  <bnoordhuis>i can gist or push the diffs if you want to check for yourself
01:59:44  <bnoordhuis>piscisaureus: well... >5% diff
01:59:59  <piscisaureus>more like 2.5%
02:00:13  <piscisaureus>with an exceptional outlier
02:00:47  <bnoordhuis>the difference gets smaller when senders x receivers gets larger
02:00:57  <bnoordhuis>still, death by a thousand cuts, right?
02:01:04  <bnoordhuis>don't want that
02:04:20  <ryah>it's so hard to test network software and get all the corner cases.
02:05:36  <bnoordhuis>and create a benchmark that represents a realistic workload
02:05:45  <bnoordhuis>https://gist.github.com/13e681faf7feca9ab402 <- the patches
02:12:01  <CIA-75>node: Ryan Dahl master * rc4454d2 / (Makefile lib/net_uv.js):
02:12:02  <CIA-75>node: net_uv: Don't error on ECONNRESET
02:12:02  <CIA-75>node: Fixes #1571. - http://git.io/0Kw-nw
02:17:13  <piscisaureus>http://code.google.com/p/memslab/
02:17:48  <piscisaureus>although what node needs is a really efficient realloc() (efficient when shrinking, that is)
02:18:52  <bnoordhuis>void* realloc(void* p, size_t new_size) { return p; } // <- like that?
02:19:14  <piscisaureus>sigh
02:25:54  <DrPizza>piscisaureus: any benchmarks for that?
02:26:23  <piscisaureus>like, when used in libuv? no
02:26:32  <piscisaureus>but there's http://code.google.com/p/memslab/source/browse/trunk/testbench.c
02:26:34  <DrPizza>no just in general
02:26:37  <DrPizza>compared to other mallocs
02:26:57  <piscisaureus>^-- benchmark comparing to normal malloc
02:27:23  <igorzi>node http_simple + non-zero reads will be a good benchmark on windows
02:27:24  <DrPizza>O_O
02:27:28  <DrPizza>factor 10 difference?
02:29:35  <CIA-75>libuv: Ryan Dahl master * rbdd880e / src/uv-unix.c : unix: add a few asserts, cosmetic touch ups - http://git.io/FGgR5g
02:29:35  <CIA-75>libuv: Ryan Dahl master * rce9171d / src/uv-unix.c : unix: uv_read_start should fail if UV_CLOSING set - http://git.io/n78S7g
02:31:31  <piscisaureus>that memslab may not be what we want
02:31:37  <piscisaureus>it's similar to using a freelist
02:31:41  <piscisaureus>what we need is efficient realloc
02:31:48  <piscisaureus>I will dream about this
02:31:50  <piscisaureus>goodbye all
02:38:12  * piscisaureusquit (Ping timeout: 258 seconds)
02:40:10  <bnoordhuis>yeah, i should be off to bed as well
02:40:19  <bnoordhuis>not going to dream about malloc impls though
02:40:39  <bnoordhuis>night, guys
02:46:05  * bnoordhuisquit (Ping timeout: 260 seconds)
02:47:35  * brson_quit (Ping timeout: 264 seconds)
02:53:46  * rmustaccjoined
04:51:35  * rmustaccpart
06:24:59  * isaacsquit (Quit: isaacs)
06:41:37  <ryah>damn it. npm does some annoying things during startup
06:41:50  <ryah>namely "new require('net').Socket(2)"
06:46:40  <ryah>DrPizza: you awake?
06:46:45  <ryah>igorzi: ?
06:46:57  <ryah>er.. igorzi: are you?
07:04:00  <igorzi>ryah: hey
07:06:06  * mralephjoined
07:07:11  <igorzi>ryah: that breaks on unix too? (with --use-uv)?
07:10:27  <ryah>yeah
07:10:47  <ryah>sorry i was going to ask you something - but now im trying to figure out something else.
07:11:02  <ryah>there are some really strange things happening with npm.
07:11:52  <ryah>i might actually be hitting a V8 bug
07:12:29  <ryah>mraleph: is it possible to ever hit a throw new Error("blah") but have the debugger step through it?
07:13:02  <ryah>mraleph: this *seems* to be happening to me in V8 3.5.6
07:13:31  <ryah>mraleph: (my test is a large code base right now - might take some time to extract)
07:21:01  <mraleph>ryah: you mean step through Error constructor?
07:22:50  <ryah>mraleph: step through the throw - but i think now that my debugger is throwing it into a special case
07:25:53  <mraleph>hmm. I am not sure stepping through throw is supported.
07:29:11  <mraleph>ryah: hmm. it seems it is actually supported. If I step over throw in DevTools i get into catch clause even if it is in the other function.
07:30:20  <mraleph>finally is also supported
07:32:30  <mraleph>ryah: oh I think I was misunerstanding you. did you mean that when in debugger throw-statement just does not throw?
07:38:55  <ryah>mraleph: yeah
07:39:25  <ryah>mraleph: but the more i look at it the more i think there's something else going on
07:39:42  <ryah>something deep inside npm - and npm seems to like doing tricky things
07:43:16  <ryah>mraleph: how fast is context spawning?
07:43:27  <ryah>mraleph: s/spawning/creation/
07:43:29  <mraleph>context as in Context?
07:43:33  <ryah>yes
07:43:38  <ryah>v8::Context
07:43:39  <mraleph>1.5 ms
07:43:54  <mraleph>with snapshots on good hardware
07:43:59  <ryah>is that ever going to be faster?
07:44:50  <mraleph>I doubt. Contexts can be made to share things that have no state — but it requires a lot of work
07:45:13  <mraleph>1.5 ms per iframe/tab is more than enough in the browser I think
07:47:22  <ryah>i want to start a context for each connection
07:47:46  <ryah>erlang style
07:48:01  <mraleph>oh
07:48:29  <ryah>i would need to do some work associating ev watchers and buffers to their owning contexts
07:49:59  <mraleph>erlang can do it because their "contexts" (= threads) are lightweight. v8 Context is the whole mutable JS world glued together — hardly lightweight.
07:50:23  <ryah>:(
07:51:40  <ryah>i wonder if i could do some tricks
07:51:44  <ryah>with a zygote
07:52:50  <mraleph>why do you want a separate context?
07:53:12  <ryah>so i can 'kill' them
08:01:52  <ryah>https://github.com/isaacs/npm/blob/c7824f412f0cb59d6f55cf0bc220253c39e6029f/lib/utils/output.js#L109-111
08:02:02  <ryah>^-- this just cost me an hour of debugging
08:02:20  <mraleph>omg
08:02:34  <mraleph>this is nasty formatting style
08:05:00  <mraleph>regarding Contexts: how are u going to deal with the state handlers create and leave in other contexts? e.g. modules they are loaded by require in the main context so all internal state will state in that context - even if you discard some context you will still have that state around.
08:05:16  <mraleph>*state will stay
08:06:19  <ryah>mraleph: i'm okay with them sharing those
08:07:01  <ryah>mraleph: i am of course scheduling these contexts in my single thread - that shouldn't be a problem
08:11:34  * mralephquit (Quit: Leaving.)
08:55:31  * piscisaureusjoined
09:32:15  <ryah>piscisaureus: we need a way to open net.Stream with fd
09:33:11  <CIA-75>node: Ryan Dahl master * re3413f0 / lib/net_uv.js :
09:33:11  <CIA-75>node: net_uv: throw when people construct net.Socket(fd)
09:33:11  <CIA-75>node: Easier to catch compatibility errors. - http://git.io/BoAnRw
09:33:49  <piscisaureus>ryah: why?
09:33:53  <piscisaureus>what's the use case?
09:34:26  <piscisaureus>ryah: in principle I am not agains this.
09:34:41  <piscisaureus>if we can find the right cross-platform abstraction, that is
09:43:02  <ryah>piscisaureus: https://github.com/isaacs/npm/blob/c7824f412f0cb59d6f55cf0bc220253c39e6029f/lib/utils/output.js#L110
09:43:55  <piscisaureus>hmm
09:44:01  <piscisaureus>ryah: what does it getStream for?
09:44:21  <ryah>stderr
09:44:31  <piscisaureus>hmm
09:44:52  <piscisaureus>net.Stream(2) won't work in windows
09:45:05  <piscisaureus>because you can't use the net stuff on a console handle
09:45:38  <ryah>can we guess the type of the fd?
09:45:43  <ryah>isatty?
09:45:50  <piscisaureus>libuv can
09:46:05  <ryah>okay - so that'd be okay
09:46:16  <piscisaureus>ryah: it would have to be more generic
09:46:22  <ryah>piscisaureus: void uv_stream_init(uv_stream_t*, int fd); ?
09:46:47  <ryah>make uv_stream_t non-abstract
09:47:12  <piscisaureus>maybe we should have uv_any_stream_t that is a union of all possible stream types
09:47:28  <ryah>hm
09:47:38  <ryah>i see your problem...
09:48:04  <piscisaureus>ryah: but fd==2 is possibly the only valid use case for this
09:48:35  <ryah>let's rename the current uv_stream_t to uv_abstract_stream_t
09:48:55  <ryah>and make uv_stream_t the union of all streams
09:48:59  <piscisaureus>uv_stream_base_t
09:49:01  <piscisaureus>maybe
09:49:09  <ryah>that will make it better for users
09:49:13  <piscisaureus>yes
09:49:40  <ryah>in fact uv_abstract_stream_t isn't so necessary
09:50:04  <piscisaureus>it's just an internal gimmick
09:50:04  <ryah>since we have those macros
09:51:00  <piscisaureus>ryah: it could be a bit risky though
09:51:09  <ryah>how so?
09:51:15  <ryah>encourage fd use?
09:51:48  <piscisaureus>oh hmm nvm
09:52:38  <piscisaureus>i was afraid that people could do
09:52:38  <piscisaureus>void close_cb(uv_stream_t* s) {
09:52:38  <piscisaureus> // Yay! our socket is close. Now let's make it a pipe
09:52:38  <piscisaureus> uv_pipe_init(s);
09:52:38  <piscisaureus>}
09:52:44  <piscisaureus>but close_cb takes a handle
09:55:14  <ryah>piscisaureus: off topic - i want to land multiplicity soon after the next release
09:55:27  <piscisaureus>yes, me too
09:55:41  <piscisaureus>let's make it a priority
10:10:35  <ryah>void uv_stream_init(uv_stream_t*, int fd, int readable, int writable);
10:10:46  <ryah>^-- determines if its readable or writable
10:11:14  <piscisaureus>ryah: you think libuv can't/shouldn't figure that out?
10:11:38  <ryah>not sure
10:11:53  <ryah>i have this function uv__stream_open() which is pretty much what we want
10:12:19  <ryah>except i need to specify readable and writable for some reason
10:13:10  <piscisaureus>hmm yeah
10:13:27  <piscisaureus>right uv-win doesn't support read-only / write-only at all
10:13:38  <piscisaureus>we'll need it for half-duplex pipes though
10:14:10  <ryah>supposedly uv_write on a process.stdout should fail
10:14:22  <piscisaureus>h,,
10:14:25  <piscisaureus>why?
10:14:28  <ryah>er - process->stdout uv_process_t
10:14:36  <ryah>process->stdout_stream with uv_write
10:14:46  <piscisaureus>oh ok
10:14:47  <piscisaureus>yeah
10:14:48  <ryah>because its only one directly
10:14:50  <ryah>direction
10:15:26  <ryah>i say we add the readable/writable initially
10:15:30  <ryah>and maybe we can remove it later?
10:15:47  <piscisaureus>ryah: I might just ingore it though
10:16:13  <ryah>piscisaureus: okay nevermind then
10:16:14  <piscisaureus>ryah: but seriously - what are the other uses cases besides uv_stream_init(s, 2)?
10:16:28  <ryah>simple first - i'll leave it out
10:16:33  <ryah>then we'll add it if needed :)
10:16:38  <piscisaureus>:-)
10:17:10  <piscisaureus>I wonder if on windows it should take an fd or a HANDLE
10:18:13  <piscisaureus>ryah: I can only think of using it for stdio. But we're going to have a separate api for that.
10:20:54  <ryah>im going to sleep. night
10:21:01  <piscisaureus>ok. sleep tight
10:25:26  * bnoordhuisjoined
12:56:35  <piscisaureus>https://github.com/joyent/node/commit/fcf9cfa1374bbbc0e77dd77908b55d45d7735450#comments <--let's troll some more
13:09:32  <DrPizza>piscisaureus: should it even be "uv_stream_init(..)"? I would think it would be better (less deceptive) to have some kind of uv_get_std_stream(STDOUT)
13:09:46  <DrPizza>i.e. don't even pretend that you can wrap arbitrary fds (which have no meaning on Windows anyway)
13:10:20  <piscisaureus>I think I agree
13:10:30  <piscisaureus>I was trying to tell ryah this
13:11:17  <piscisaureus>or, at least figure out what the use cases for stream_init would be other than opening stderr
13:12:13  <DrPizza>stdout and stdin, I guess
13:21:27  <DrPizza>but beyond that
13:26:50  <DrPizza>piscisaureus: how is the stdio stuff coming along?
13:27:08  <piscisaureus>not doing it
13:27:17  <DrPizza>oh how come?
13:27:25  <piscisaureus>got a little complicated
13:27:28  <DrPizza>heh
13:27:29  <piscisaureus>so I'm doing udp first
13:27:32  <DrPizza>ah
13:27:42  <piscisaureus>and ryah is taking a stab at stdio on linux when he has time
13:28:04  <DrPizza>where was your gist with the plan of things to do on stdio
13:29:00  <piscisaureus>https://gist.github.com/1149564
13:29:52  <DrPizza>man I wish we could ignore 2003
13:30:06  <DrPizza>then you could get rid of
13:30:07  <DrPizza> + (blocking) named pipe -> ReadFile / WriteFile in a thread.
13:30:08  <DrPizza> + (blocking) file handle -> ReadFile / WriteFile in a thread.
13:30:21  <DrPizza>since we could just reopen in overlapped mode
13:31:07  <piscisaureus>I am not sure that that would work well anyway
13:31:18  <DrPizza>no?
13:32:56  <DrPizza>I would defer the file handle stuff until we have a file API anyway
13:33:03  <DrPizza>since file handles are different anyway
13:50:25  <piscisaureus>DrPizza: fair enough
13:50:50  <piscisaureus>DrPizza: but since "blocking pipes" are common to have as stdin we need to have thread-pool WriteFile / ReadFile anyway
13:55:34  <DrPizza>unless we can find some way of reopening the pipes that works on 2003
14:00:59  <DrPizza>oh shit
14:01:01  <DrPizza>lol!
14:01:04  <DrPizza>it works in 2003 anyway
14:01:14  <DrPizza>I saw that it needed Vista on the client
14:01:19  <piscisaureus>and xp?
14:01:19  <DrPizza>so I just assumed it was 2008 on the server
14:01:24  <DrPizza>but the docs say 2003 server also works
14:01:32  <DrPizza>no, no XP
14:01:37  <DrPizza>but I thought 2003 was the minimum we cared about?
14:01:46  <piscisaureus>yeah well
14:01:52  <piscisaureus>I care about xp a bit
14:02:02  <piscisaureus>DrPizza: but does it actually work on a pipe?
14:02:08  <DrPizza>well that's the next question
14:02:10  <DrPizza>it may not
14:02:20  <DrPizza>but it's supposed to
14:02:28  <DrPizza>it even has specific notes about pipes
14:02:33  <DrPizza>and impersonation and other pipe stuff
14:08:58  <piscisaureus>DrPizza: this may provide a good solution for pipes
14:09:04  <DrPizza>yes
14:09:09  <DrPizza>real consoles still need thread pools
14:09:15  <DrPizza>since they can't do overlapped I/O anyway (ugh)
14:09:32  <piscisaureus>And so do files
14:09:37  <DrPizza>but I think pipes and files will be able to leverage the general pipe and file features
14:09:41  <DrPizza>files can overlap
14:09:53  <piscisaureus>files don't work well overlapped
14:10:03  <DrPizza>they work
14:10:07  <piscisaureus>we're going to have a tp implementation fist
14:10:20  <DrPizza>why?
14:10:27  <DrPizza>I know there is still apparenlty the compressed file thing
14:11:12  <piscisaureus>if possible we're going to make a conservative guess about whether overlapped operations will work
14:11:23  <piscisaureus>and use it when we're sure it's safe
14:11:51  <piscisaureus>I think
14:11:58  * igorziquit (Ping timeout: 252 seconds)
14:12:04  <piscisaureus>DrPizza: but so
14:12:14  <DrPizza>that seems kind of odd to me
14:12:21  <piscisaureus>why?
14:12:35  <DrPizza>well, we can always force the file into overlapped mode
14:12:45  <piscisaureus>yes
14:13:05  <piscisaureus>but there is no guarantee that ReadFile(handle, ..., &overlapped) will not block
14:13:16  <piscisaureus>if there is the risk that is blocks we should use the thread pool
14:13:38  <DrPizza>is the KB article still accurate for modern windows?
14:13:44  <piscisaureus>DrPizza: not entirely
14:13:54  <DrPizza>do compressed files still perform only blocking I/O?
14:14:31  <piscisaureus>DrPizza: but I did some tests and found that ReadFile on win7 would sometimes block even when none of the criteria in that kb article was met
14:14:43  <DrPizza>hmmm
14:15:01  <piscisaureus>The only way to make it work reliably is by setting FILE_FLAG_NO_BUFFERING
14:15:24  <DrPizza>well we don't want that
14:15:31  <piscisaureus>indeed
14:17:12  <DrPizza>I'm sure there was some APIthat would tell you what type a HANDLE actually was, but I'm damned if I can find it now
14:17:25  <piscisaureus>DrPizza: you can figure it out
14:17:32  <piscisaureus>DrPizza: what exactly do you need to know
14:17:39  <piscisaureus>?
14:17:55  <DrPizza>file versus named pipe versus console
14:19:02  <piscisaureus>DrPizza: call GetConsoleMode with the handle, if it fails it's not a console
14:19:09  <DrPizza>hmm
14:19:23  <piscisaureus>then call GetPipeInformationm, if it fails it's not a pipe :-)
14:19:24  <DrPizza>and then pipe versus file?
14:19:27  <DrPizza>haha
14:19:30  <DrPizza>how awful!
14:20:57  <piscisaureus>maybe there's another way
14:21:05  * piscisaureuschecking _isatty source
14:21:08  <DrPizza>yes I'm just looking through some old code of mine
14:21:43  <piscisaureus>GetFileType?
14:22:03  <piscisaureus>DrPizza: ^-- that does it
14:22:07  <DrPizza>doh yes of course
14:22:10  <DrPizza>I was looking for gethandletype
14:25:14  <DrPizza>NtQueryObject also, I think
14:25:21  <piscisaureus>hmm
14:25:47  <DrPizza>with ObjectTypeInformation
14:25:56  <DrPizza>I wonder if that's any more accurate
14:26:03  <DrPizza>since GetFileType doesn't distinguish between pipes and sockets
14:26:08  <DrPizza>maybe that doesn't matter
14:26:26  <piscisaureus>DrPizza: i think GetFileType returns UNKNOWN for sockets
14:26:44  <piscisaureus>hmm
14:26:51  <piscisaureus>allright the documentation says otherwise
14:26:55  <DrPizza>FILE_TYPE_PIPE
14:26:55  <DrPizza>0x0003
14:26:55  <DrPizza>
14:26:55  <DrPizza>The specified file is a socket, a named pipe, or an anonymous pipe.
14:26:57  <DrPizza>yes
14:29:11  <DrPizza>ugghghhghghg
14:29:18  <DrPizza>we'll have to call getfiletype from a thread
14:29:29  <DrPizza>"This is a well-known bug in NPFS. If the pipe has a pending read request which is not satisfied yet - GetFileType hangs waiting for read to be satisfied. "
14:29:33  <DrPizza>so you need to call it in a thread
14:29:40  <DrPizza>and then terminate the thread after a few seconds
14:29:52  <piscisaureus>I bet there are ways around that
14:30:01  <piscisaureus>maybe just call GetPipeInformation then?
14:30:03  <DrPizza>apparently NtQueryObject has the same issue
14:30:19  <DrPizza>yes maybe GetPipeInformation followed by GetFileType
14:30:26  <DrPizza>but who knows, that might hang to!
14:30:28  <DrPizza>too*
14:30:42  <piscisaureus>I bet PeekNamedPipe won't block :-)
14:32:07  <piscisaureus>DrPizza: the risky bit is that we'll probably have to call NtQueryInformationFile too to figure out whether the pipe is open in overlapped mode or not
14:32:18  <DrPizza>yes
14:32:34  <DrPizza>http://support.microsoft.com/kb/2009703
14:32:47  <DrPizza>"There were in-depth design changes for the GetFileType implemenation in Vista, so this kind of hang will not be seen in newer Operating Systems."
14:39:23  <DrPizza>so maybe getfiletype is now safe
14:39:25  <DrPizza>who can say
14:40:04  <piscisaureus>in other works
14:40:07  <piscisaureus>*words
14:40:14  <piscisaureus>GetFileType is not safe on xp and 3k
14:40:20  <piscisaureus>eh 2k3
14:40:36  <piscisaureus>that can be worked around
14:40:47  <DrPizza>so if you do a pipe call followed by getfiletype
14:40:53  <DrPizza>it should be safe anyway
14:41:01  <DrPizza>a non-blocking pipe call, I guess
14:41:02  <DrPizza>hmm
15:04:14  * isaacsjoined
16:06:22  <piscisaureus>igorzi: hey
16:07:20  <piscisaureus>oh hmm msft is not open yet?
16:23:58  <DrPizza>lol
16:52:15  * DrPizza_joined
16:53:32  <ryah>DrPizza_, piscisaureus: uv_stream_init() would be used on file fds too
16:53:39  <ryah>presumably
16:53:45  <ryah>we should ask isaac though
16:54:00  <isaacs>hello
16:54:04  * isaacsis summoned
16:54:23  <piscisaureus>ryah: when?
16:54:38  <piscisaureus>ryah: in windows open files mostly won't have an fd
16:55:23  <ryah>isaacs: what's the purpose of npm.config.get("outfd") ?
16:55:46  <isaacs>ryah: mostly so that it can be set programmatically to write somewhere else.
16:56:06  <isaacs>ryah: i'm ok with pulling that out, though, or just saying that you have to set it to a writable stream if you want to do it programmatically, and you set it up yourself.
16:56:29  <isaacs>i mean, you can already pipe stuff on the cli pretty easily
16:56:38  * DrPizzaquit (Ping timeout: 240 seconds)
16:57:23  <ryah>we don't want you to change stuff - we have to support existing API.
16:57:32  <isaacs>ryah: haha, too late.
16:57:35  <ryah>piscisaureus: im refing to https://github.com/isaacs/npm/blob/c7824f412f0cb59d6f55cf0bc220253c39e6029f/lib/utils/output.js#L34
16:57:48  <isaacs>ryah: i don't like any sort of completely silent failure in npm.
16:57:55  <isaacs>silent success is ok
16:57:58  <ryah>isaacs: i just mean that you changing it doesn't solve any problems - we still have to support the old npm
16:58:01  <ryah>:)
16:58:02  <isaacs>yeah
16:58:34  <isaacs>it's safe to say that --outfd will be a net stream, even if it's the shell faking one that gets piped to a file
16:58:46  <isaacs>--outfd 3 3>out.txt
16:58:55  <piscisaureus>hmm
16:59:04  <ryah>:/
16:59:04  <piscisaureus>that's not typically something that works on windows
16:59:12  <isaacs>piscisaureus: that's why i'm ok removing it
16:59:29  <isaacs>there's no reason why you can't just do 1>out.txt
16:59:29  <ryah>piscisaureus: so basically uv_stream_init should support any type of stream
16:59:34  <isaacs>and leave outfd as the default
16:59:38  <ryah>most likely tty or file
17:00:09  <piscisaureus>ryah: do we want to use HANDLEs or real fds on windows?
17:01:05  <ryah>piscisaureus: for these situations where people are using fds - i think we need to support fds
17:01:29  <piscisaureus>ryah: we can support fds for files that are managed by the crts
17:01:35  <piscisaureus>but the crt is so painful
17:01:41  <piscisaureus>libuv is not using that at all
17:01:48  <ryah>piscisaureus: is there a way to get a handle from fd?
17:01:53  <piscisaureus>yes
17:01:57  <piscisaureus>_get_osfhandle
17:01:59  <ryah>what's the call?
17:02:00  <ryah>thanks
17:02:11  <piscisaureus>ryah: it can also be done the other way around
17:02:16  <piscisaureus>_open_osfhandle
17:02:32  <ryah>so somehow we're going to figure out the type of the fd
17:02:57  <ryah>then if it's a file _get_osfhandle - then turn it into a uv_file_t
17:03:04  <ryah>(once that exists)
17:03:08  <piscisaureus>ryah: or - yeah whatever
17:03:12  <piscisaureus>a fd can be a console as well
17:03:33  <piscisaureus>ryah: supporting the fs api is a bit more painful
17:03:58  <piscisaureus>e.g. do you want to give the fd to the fs.open callback?
17:04:09  <piscisaureus>or something else
17:05:06  <piscisaureus>one of the annoying thins is that the windows crt has a hard limit of 2000-something fds
17:06:25  * piscisaureusneeds igorzi
17:07:05  <ryah>yes
17:07:14  <ryah>but this will just be used in one-off situations
17:07:18  <ryah>like isaacs has
17:07:29  <ryah>not for server connections
17:08:18  <isaacs>yeah, typically if you already have a file descriptor, and want to set up a stream around it, you're not going to have zillions of them
17:08:33  <piscisaureus>yes - ok we can do this
17:08:57  <piscisaureus>my main concern is whether we want FDs to come out of libuv
17:09:06  <piscisaureus>sure we can put fds in there
17:09:41  <ryah>as long as we're not using them for servers - i think it's fine
17:09:48  <ryah>fds are x-platform after all
17:09:54  <ryah>we're already depending heavily on CRT
17:10:41  <piscisaureus>not in windows
17:10:58  <ryah>i should say - we will be once igor finishes uv_async stuff
17:11:04  <ryah>er uv_tp_
17:11:08  <ryah>or whatever he's calling it
17:11:22  <ryah>or is he using native calls? :/
17:17:13  <ryah>piscisaureus: when we add this call i will add docs that say the user should beware of using it
17:17:31  * igorzijoined
17:17:33  <ryah>piscisaureus: that it should only be used for "compatibility" code
17:17:38  <piscisaureus>yeah
17:17:53  <piscisaureus>ryah: I was thinking also
17:18:46  <piscisaureus>maybe we should replace the "fd" that is used by the fs module by something opaque
17:18:50  <piscisaureus>a wrap or something
17:23:24  <DrPizza_>windows doesn't use fds, and I am not convinced that mixing fds and HANDLEs should even be mixed
17:23:28  <DrPizza_>I'm not sure how the buffering would work
17:23:31  <DrPizza_>the CRT has its own buffers
17:23:42  <DrPizza_>though I suppose you can turn it off
17:23:43  <DrPizza_>but still
17:23:45  <DrPizza_>nobody uses fds
17:23:50  <DrPizza_>except stdin, stdout, stderr
17:24:19  <DrPizza_>fds really aren't cross-platform in any useful sense
17:27:02  * johnm1234quit (Ping timeout: 240 seconds)
17:27:46  <DrPizza_>unix uses them, windows doesn't.
17:28:50  * brsonjoined
17:29:40  <bnoordhuis>piscisaureus: ping
17:35:02  <bnoordhuis>piscisaureus: https://gist.github.com/b0dfb564bb7c9d8af42f <- can you make that patch work on windows?
17:35:17  * johnm1234_joined
17:43:21  <ryah>well - we could probably support new net.Socket(1), net.Socket(2), net.Socket(0)
17:43:26  <ryah>and fail on everything else
17:43:39  <ryah>i bet that would solve 99% of the sisues
17:43:40  <ryah>issues
17:43:42  <piscisaureus>yes
17:43:46  <piscisaureus>although
17:43:50  <DrPizza_>I'm not really clear what that would even do, are they ever actually sockets?
17:43:53  <DrPizza_>can you do that on windows?
17:44:02  <piscisaureus>new net.Socket(1) should not actually create a socket
17:44:06  <piscisaureus>because that won't work
17:44:13  <ryah>DrPizza_: no - they're ttys or files
17:44:39  <ryah>right it should create a uv_tty_t wrap
17:44:42  <ryah>TTYWrap
17:44:47  <ryah>ConsoleWrap ?
17:44:59  <piscisaureus>either, whatever
17:45:10  <piscisaureus>bnoordhuis: I can make that work, but ping me later
17:45:18  <piscisaureus>It's a big mess in my repo right now
17:45:19  <ryah>DrPizza_: in node net.Socket is basically the fundemental stream class
17:45:28  <ryah>DrPizza_: because all types of fds are handled in the same way
17:45:38  <DrPizza_>hm
17:45:47  <DrPizza_>that seems rather unfortunate
17:45:50  <ryah>well - that's not totally true
17:46:08  <ryah>TTYs and sockets are handled by net.Stream. files are different
17:46:18  <DrPizza_>do files have their own random access thing?
17:46:25  <ryah>npm is actually broken in the case of using a file
17:46:47  <DrPizza_>ah
17:46:49  <DrPizza_>hm
17:47:10  <ryah>DrPizza_: we handle file streams and seekable files differntly
17:47:27  <DrPizza_>so you can't have one file object and treat it both ways?
17:47:43  <DrPizza_>(unlike e.g. c++ fstream)
17:47:43  <ryah>fs.ReadStream and fs.WriteStream are half-duplex streams - with the same interface as a socket has
17:47:54  <ryah>to seek around you have to use fs.read and fs.write
17:51:12  <DrPizza_>so really net.Socket should be named not-actually-net.Stream and net.Socket should be some kind of a subclass that (I assume) knows stuff like the names of its endpoints, and not-actually-net.Console would be a subclass that didn't know that kind of thing
17:56:03  <DrPizza_>hm
17:56:29  <igorzi>piscisaureus: ryah: new net.Socket(2) could also be a pipe, or it could even be a tcp socket (though that's probably rare)
17:58:00  <ryah>pipe, tty, or file
17:58:12  <ryah>but files should actually be constructed differently
17:58:40  <DrPizza_>files have a totally different response to I/O, since you can pull what you need, rather than just having some random person pushing whenever they feel like
18:00:10  <bnoordhuis>call time?
18:00:49  <piscisaureus>im in
18:01:00  <piscisaureus>but not the leader
18:02:03  <bnoordhuis>as am i
18:03:35  <CIA-75>node: koichik master * r485d5b5 / (4 files in 3 dirs):
18:03:35  <CIA-75>node: dns: Force the DNS module to invoke callbacks asynchronously.
18:03:35  <CIA-75>node: Fixes #1164. - http://git.io/d9ajFw
18:42:15  <CIA-75>libuv: Bert Belder winreqerr * r1edba40 / (6 files in 2 dirs): win: use uv_req_t.overlapped.Internal field to propagate request errors (+5 more commits...) - http://git.io/7sxBnA
18:44:01  <piscisaureus>igorzi: can you take a look at https://github.com/joyent/libuv/compare/master...winreqerr
18:44:14  <igorzi>piscisaureus: yep, looking
18:44:15  <piscisaureus>especially https://github.com/joyent/libuv/commit/1edba4006c5aca3ec583dfea7b0e059d83f1dab0
18:46:01  <ryah>igorzi, piscisaureus: anything against building the release this week with msvs?
18:46:15  <piscisaureus>i don't think so
18:46:33  <piscisaureus>if openssl works
18:46:38  <ryah>it seems to
18:47:07  <igorzi>i don't see any issues.. just need to make sure that CRT is statically linked, which I think DrPizza took care of
18:47:48  <DrPizza_>ryah: the only question is, do we want to add in zlib as well, so openssl can use it?
18:48:00  <DrPizza_>I would assume that unix distributions would include it
18:48:06  <DrPizza_>it should be easy enough to add
18:48:28  <ryah>DrPizza_: https://github.com/joyent/node/blob/e3413f08fdcd0476cd9e0c2f934a6f7c6816c322/src/node_crypto.cc#L3887-3898
18:48:38  * DrPizza_changed nick to DrPizza
18:48:58  <ryah>we notices good memory footprint improvements when disabling compression
18:49:08  <DrPizza>oh really
18:49:10  <DrPizza>huh
18:49:38  <ryah>openssl does not do compression well
18:49:51  <ryah>or at least not with high-concurrency servers in mind
18:50:07  <ryah>there's probably some tunables there - but we havent looked into it
18:50:16  <DrPizza>so they'd have to do it in a module or something?
18:50:27  <ryah>we want to have a core zlib module eventually
18:50:37  <ryah>which implements the stream interface
18:50:51  <DrPizza>can that integrate into the right places?
18:50:56  <DrPizza>I don't know how zlib with openssl works
18:51:02  <DrPizza>it doesn't need to set up zlib in the handshake or anything?
18:51:38  <ryah>we were thinking we'd do it in js
18:51:46  <ryah>socket.pipe(zlibInstace)
18:52:38  <DrPizza>hm
18:53:21  <DrPizza>is that the same thing though?
18:53:43  <DrPizza>like if I turn on compression on my ssh connection, both ends know abou tit
18:54:01  <DrPizza>it's not the same as just sending gzipped data through
18:54:08  <DrPizza>I don' tknow enough about how ssl works though
18:54:12  <DrPizza>so maybe it's the same
18:54:34  <ryah>no not the same- the compression is at a higher layer
18:55:15  <DrPizza>right, that's what Ia ssumed, I just wasn't sure if maybe openssl allows you to plug compression in or something
18:59:59  <DrPizza>ryah: would it not be the kind of thing perhaps better as a runtime/command-line tunable, so if it's not suitable for your application you can disable it, but if you'd rather have it, it's an option? because I have always assumed it's a fairly standard thing
19:00:18  <DrPizza>but maybe moreso for things like ssh/scp than https
19:00:50  <DrPizza>since with https you can always compress the bodies and it's just the headers that are not compressed
19:01:03  <igorzi>piscisaureus: looks good.. just this - https://github.com/joyent/libuv/commit/1edba4006c5aca3ec583dfea7b0e059d83f1dab0#L3R526
19:02:22  <piscisaureus>igorzi: rethinking, I don't like that SET_REQ_ERROR takes an error code but GET_REQ_ERROR returns uv_err_t
19:02:41  <piscisaureus>igorzi: should I rename GET_REQ_ERROR to GET_REQ_UV_ERROR?
19:02:56  <ryah>DrPizza: yeah probably
19:03:13  <igorzi>piscisaureus: yeah, that would make it cleaner
19:03:17  <ryah>DrPizza: we saw such a huge win with it disabled that we did it immediately
19:03:22  <ryah>DrPizza: and never looked back :)
19:03:26  <DrPizza>heh
19:04:01  <ryah>i forget exactly how much - but it might have been 10x
19:04:14  <ryah>pquerna: do you remember?
19:04:14  <DrPizza>that seems like a big difference
19:04:22  <DrPizza>like, alarmingly big
19:04:38  <ryah>DrPizza: pquerna did the change on his twisted system and also saw the same
19:04:46  <DrPizza>how strange
19:04:52  <piscisaureus>http://journal.paul.querna.org/ ?
19:05:31  <piscisaureus>^-- drpizza ryah
19:07:10  <DrPizza>interesting
19:07:31  <DrPizza>pquerna: did SSL_MODE_RELEASE_BUFFERS make zlib memory load tolerable again?
19:07:33  <DrPizza>or was it still high?
19:07:57  <DrPizza>ryah: how hard work is it to plumb in a command-line option, and would you accept a patch that did so?
19:08:17  <DrPizza>ryah: i.e. zero out the zlib stuff by default, but allow people to leave it enabeld
19:15:16  <ryah>DrPizza: relatively easy - yes i'd be donw with that
19:15:29  <DrPizza>ok I'll take a look
19:15:35  <ryah>just set a global node::use_ssl_compression variable
19:15:46  <ryah>src/node.cc is the command-line parsing
19:15:48  <DrPizza>ok
19:25:01  * mralephjoined
19:25:06  <CIA-75>libuv: Bert Belder master * r5cae6e4 / (6 files in 2 dirs): win: use uv_req_t.overlapped.Internal field to propagate request errors (+5 more commits...) - http://git.io/wSZ6Pg
19:28:20  <mraleph>ryah: so I was think about this whole Context thingy we talked about in the morning. I can't quite get what Context give you that you can't get without them (because you say that it's not isolation and you are fine with sharing state)
19:28:28  <mraleph>*I was thinking
19:30:04  <ryah>mraleph: hm - good point
19:30:16  <ryah>let me think about that...
19:47:25  <piscisaureus>udp_packet_storm_1v1: 118170/s received, 120190/s sent
19:47:25  <piscisaureus>udp_packet_storm_10v1: 119180/s received, 121200/s sent
19:47:25  <piscisaureus>udp_packet_storm_100v1: 118170/s received, 120190/s sent
19:47:25  <piscisaureus>udp_packet_storm_1000v1: 117160/s received, 119180/s sent
19:47:25  <piscisaureus>udp_packet_storm_10v10: 123222/s received, 125240/s sent
19:47:26  <piscisaureus>udp_packet_storm_100v10: 123222/s received, 125240/s sent
19:47:26  <piscisaureus>udp_packet_storm_1000v10: 122212/s received, 124230/s sent
19:47:27  <piscisaureus>udp_packet_storm_100v100: 107861/s received, 110319/s sent
19:47:27  <piscisaureus>udp_packet_storm_1000v100: 113921/s received, 116379/s sent
19:47:28  <piscisaureus>udp_packet_storm_1000v1000: 71261/s received, 84478/s sent
19:47:28  <piscisaureus>^-- not bad again
19:48:09  <ryah>mraleph: i was kind of thinking that only "core" modules would be shared
19:48:29  <ryah>mraleph: but perhaps we can do some trickery with require to make that the case
19:49:07  * mralephquit (Quit: Leaving.)
19:52:45  * mralephjoined
19:53:39  <mraleph>ryah: so you mean every module required by the "request handler" has to be reloaded for every request?
19:53:58  <mraleph>(sorry I accidentally closed laptop)
19:54:24  <mraleph>(so I did not get anything after "but perhaps we can do some trickery…" line
19:54:57  <ryah>mraleph: each request would re-execute its modules
19:55:25  <ryah>mraleph: there was nothing else said after that line
19:55:39  <mraleph>sounds pretty heavy.
19:57:06  <mraleph>if I understand you correctly you want to be able for a given connection be able to discard all activities that "grew" from that request
19:57:20  <ryah>yeah
19:58:52  <pquerna>DrPizza: no, mode_release_buffers is unrelated to zlib
19:59:01  <mraleph>hmm. that's pretty difficult especially because you have things like global "setInterval", global "nextTick" (or whatever it is called).
19:59:03  <DrPizza>pquerna: oh, too bad then.
19:59:05  <pquerna>DrPizza: zlib still keeps its own internal buffers at 512kb or whatnot
19:59:13  <DrPizza>is that not tunable at all?
19:59:20  <DrPizza>or is it not tunable when openssl is calling zlib
19:59:28  <mraleph>I think instead of going heavy style with Contexts you should just redesign the API
19:59:30  <pquerna>not tunable when openssl is calling zlib
19:59:45  <mraleph>ryah: just kill all the global ways to schedule
20:00:47  <mraleph>ryah: and give the connection/request/whatever some token/handle/gateway that allows to schedule things. then all activities will be "associated" with the connection explicitly
20:01:15  <mraleph>I am just throwing some ideas.
20:03:26  <ryah>mraleph: yeah
20:03:29  <ryah>good ideas
20:03:35  <ryah>we can actually do that implicitly
20:03:50  <ryah>since we always know who started the current "event"
20:04:21  <mraleph>oh, yeah that is also true.
20:05:40  <mraleph>in a sense there a two ways (Dark Side and Light SIde). Dark Side imposes limitations. Light Side gives opprotunities.
20:05:49  <mraleph>fuuuu
20:05:51  <mraleph>there are
20:05:57  <mraleph>not "there a"
20:06:03  <mraleph>this kills me
20:08:15  <DrPizza>pquerna: heh http://el-tramo.be/blog/more-squishy-data
20:09:32  <pquerna>wow, weird.
20:19:53  <CIA-75>node: Evan Martin master * ra916d88 / common.gypi :
20:19:53  <CIA-75>node: common.gypi: don't nest "target_defaults" within "target_defaults"
20:19:53  <CIA-75>node: The "conditions" block is already within a "target_defaults", so its
20:19:53  <CIA-75>node: children amend target default settings already. - http://git.io/Yiq0HA
20:19:53  <CIA-75>node: Evan Martin master * r8a29e5e / (common.gypi deps/openssl/openssl.gyp): gyp: Don't pass C++ flags to C compiler, and don't pass -ansi in OpenSSL builds. - http://git.io/ms8QTQ
20:19:53  <CIA-75>node: Ryan Dahl master * r85f7b78 / common.gypi : gyp: Further fixes to target_defaults for mac - http://git.io/-GZuxA
20:23:46  <piscisaureus>bnoordhuis: hey
20:46:18  <piscisaureus>bnoordhuis: test-udp-ipv6-only is broken
20:47:02  <piscisaureus>bnoordhuis: a ipv4 server gets bound to localhost first, then the ipv6 server
20:47:20  <piscisaureus>even if UDP_IPV6ONLY is a no-op then still the test would not fail
20:47:47  <piscisaureus>because the ipv6 socket can't listen on localhost since the ipv4 socket has already taken it
20:53:08  * graydonjoined
21:00:34  <ryah>using GYP for addons might be nice
21:01:45  <DrPizza>certainly it would make it easier to ensure that things like the C library etc. are specified correctly
21:04:43  <ryah>it would be nice if GYP understood real JSON
21:04:46  <ryah>and not python JSON
21:05:13  <ryah>but whatever
21:05:25  <DrPizza>I thought it did, just with the rule that you had to quote everything
21:05:37  <ryah>it uses # comments
21:05:41  <DrPizza>oh duh
21:05:42  <DrPizza>yes
21:05:49  <DrPizza>does it support // comments?
21:06:02  <ryah>not that i've seen ...
21:06:11  <piscisaureus>JSON also doesn't support // comments
21:06:20  <ryah>piscisaureus: true
21:06:30  <DrPizza>piscisaureus: no, but at least it'd be something that javascript could eval()
21:06:51  <DrPizza>at least I assume you can eval comments
21:06:58  <piscisaureus>sure
21:07:20  <ryah>i'm changing waf to build into out/Release/node and out/Debug/node_g
21:07:33  <ryah>which will allow us to unify the test runner at elast
21:07:35  <DrPizza>what does _g stand for?
21:07:40  <ryah>debug
21:07:45  <DrPizza>yeah but I mean
21:07:46  <DrPizza>why _g?
21:07:53  <ryah>that's what v8 does
21:08:07  <DrPizza>heh
21:08:09  <ryah>so i'll remove that too next :)
21:08:14  <ryah>since gyp doesn't add that
21:08:46  <DrPizza>how strange
21:08:53  <DrPizza>_g
21:08:58  <DrPizza>is it a gcc option or anything?
21:08:59  <DrPizza>-g?
21:11:13  <ryah>DrPizza: i think it comes from the gcc option - yah
21:12:41  <CIA-75>node: Ryan Dahl master * r879eb16 / (7 files in 5 dirs): Have WAF variants match GYP configuration names - http://git.io/iekkIQ
21:12:41  <CIA-75>node: Ryan Dahl master * ra7300c8 / (5 files in 3 dirs): WAF builds in out/ instead of build/ - http://git.io/Vyhv0A
21:13:04  <ryah>now if we could just get --generated-output in MSVS tow ork
21:13:17  <piscisaureus>--generated-output
21:13:20  <piscisaureus>??
21:13:46  <ryah>https://github.com/joyent/node/blob/a7300c8297693e26d2504da4a2e5813453306cd9/tools/gyp_node#L50
21:13:57  <DrPizza>hm, they have made some fixes
21:14:01  <DrPizza>so it's possible that it does now
21:14:11  <DrPizza>although I fear it may lead to projects with lots of ../../../ nonsense
21:14:56  <piscisaureus>yes these ../../ directories in the vs project suck
21:15:07  <ryah>http://code.google.com/p/gyp/issues/detail?id=40&q=output_dir
21:17:50  <piscisaureus>is this true? http://code.google.com/p/gyp/issues/detail?id=7
21:18:09  <ryah>yes
21:18:12  <piscisaureus>does gyp only work with cyg *cough* win installed?
21:18:16  <DrPizza>no
21:18:22  <ryah>we worked around it somehow
21:18:52  <DrPizza>I would have rewritten gyp if necessary
21:18:56  <DrPizza>not installing that shit on my PC
21:20:02  <piscisaureus>cygwin is a wonderful piece of engineering
21:20:27  <ryah>gyp is going to have a great future
21:20:37  <ryah>the cross compiling stuff is going to be great
21:20:38  <ryah>i think
21:29:14  <CIA-75>node: Ryan Dahl master * rea15635 / lib/net_legacy.js :
21:29:14  <CIA-75>node: net_legacy: Fix throw typo
21:29:14  <CIA-75>node: Thanks Tobi - http://git.io/JihLzw
21:30:46  <CIA-75>node: Ryan Dahl v0.4 * r827b9c5 / lib/net.js :
21:30:46  <CIA-75>node: net: fix throw error
21:30:46  <CIA-75>node: Thanks Tobi - http://git.io/MFIcHQ
21:31:10  <igorzi>ryah: piscisaureus: here's the eio API in libuv - https://gist.github.com/1157497
21:31:54  <igorzi>i'm trying to see how to make this API simple for sync operations.. maybe use the same functions, but run synchronously if cb is not specified?
21:32:19  <igorzi>)
21:32:20  <igorzi>(and read results out of uv_tp_req_t?
21:33:57  <ryah>igorzi: that would work for me
21:34:01  <ryah>NULL callback = sync
21:34:07  <piscisaureus>yes +1
21:35:02  <igorzi>ok, let's go with that
21:35:05  <piscisaureus>igorzi: what are you going to do with fd's
21:35:06  <piscisaureus>?
21:35:22  <piscisaureus>are you using _open_osfhandle ?
21:35:50  <piscisaureus>btw - if the functions can also be sync you should drop the tp_ suffix
21:36:13  <DrPizza>death to fds
21:36:19  <DrPizza>you shoul dhave a uv_native_file_type
21:36:25  <ryah>piscisaureus: what about uv_tp_write ?
21:36:26  <DrPizza>typedef.
21:36:28  <DrPizza>int on unix
21:36:31  <DrPizza>HANDLE on windows
21:37:10  <igorzi>hmm, yeah maybe we can can just drop _open_osfhandle, and do handles?
21:37:17  <piscisaureus>yes +1
21:37:20  <ryah>it's okay with me
21:37:35  <piscisaureus>igorzi: also uv_type_symlink should take a flags or a type argument
21:38:24  <igorzi>ok, now we need to agree on naming
21:38:33  <igorzi>(since it's no longer just thread-pool)
21:39:15  <piscisaureus>igorzi: on windows you need to distinguish between dir, file symlinks and junctions
21:40:08  <ryah>uv_fs_ ?
21:41:54  <igorzi>that would be fine with me.. and i guess we can leave uv_tp_custom as is
21:42:17  <piscisaureus>ryah: uv_object_t -- base class for uv_req_t and uv_handle_t?
21:42:22  <bnoordhuis>piscisaureus: re ipv6_only test: that's the point, ipv4 localhost != ipv6 localhost
21:42:34  <bnoordhuis>though i might have mistyped ::0 again :/
21:42:54  <piscisaureus>bnoordhuis: not exactly
21:43:03  <ryah>igorzi: sounds good
21:43:30  <ryah>piscisaureus: uv_base_t ?
21:43:51  <piscisaureus>ryah: yeah but uv_object_t sounds better
21:43:58  <piscisaureus>but base_t is also fine by me
21:43:59  <ryah>it's so long
21:44:07  <piscisaureus>you don;t use it a lot
21:44:14  <ryah>true..
21:44:29  <ryah>well whatver you want
21:45:27  <bnoordhuis>piscisaureus: hmm, looking at the test again...
21:45:41  <piscisaureus>bnoordhuis:
21:45:41  <piscisaureus>r = uv_udp_bind(&server1, $loopback:1234, 0);
21:45:41  <piscisaureus>r2 = uv_udp_bind6(&server2, $loopback:1234, 0); <-- IPV6ONLY implied because is already taken
21:46:03  <piscisaureus>bnoordhuis: so this test always passes even if IPV6ONLY does not work
21:46:37  <piscisaureus>bnoordhuis: I can't get it to fail on windows
21:46:43  <piscisaureus>unless I disable the ipv4 server
21:47:13  <piscisaureus>bnoordhuis: also I'd like a test for the opposite, e.g. test that when IPV6ONLY *not* set dualstack sockets work
21:55:00  <igorzi>latest benchmarks that i get with node http_simple (booting same 2-proc machine into linux and windows): https://gist.github.com/1166690
21:55:26  <ryah>nice!
21:55:27  <igorzi>ryah: piscisaureus: bnoordhuis: --^
21:55:40  <ryah>igorzi: may i share this?
21:56:43  <bnoordhuis>windows faster? i consider this a personal challenge, igorzi!
21:56:54  <bnoordhuis>good work :)
21:57:04  <igorzi>ryah: sure
21:57:10  <piscisaureus>still a regression on linux of about 5%
21:57:17  <piscisaureus>that is obviously not acceptable :-)
21:57:23  <igorzi>and we still have that 0-read patch in the queue :)
21:58:02  * piscisaureusawaits ryah's tweet
21:58:40  <igorzi>do we know why uv is slower than legacy? is it libuv? or node bindings?
21:59:23  <ryah>igorzi: we need to figure that out
22:00:23  <piscisaureus>igorzi: so this is because of the SetFileHandleNotificationMode patch?
22:01:32  <igorzi>piscisaureus: this is combination of: GetQueuedCompletionStatusEx + shortcut (skip IOCP for sync) + many accepts
22:01:41  <igorzi>many accepts was the most significant one
22:01:47  <piscisaureus>ah of course
22:06:08  <bentkus>nice, libuv on windows shines now
22:06:20  <bentkus>gj
22:06:30  <bentkus>keep that comming!
22:09:34  <mraleph> so can I actually build node with MS cl now?
22:10:50  <ryah>mraleph: MS cl?
22:11:19  <ryah>mraleph: you can build with msvs use the generate-projects.bat
22:11:21  <mraleph>msvc whatevar
22:11:32  <ryah>mraleph: via GYP :)
22:11:45  <mraleph>cl.exe is the name of the compiler
22:11:50  <ryah>oh
22:11:51  <mraleph>cool
22:12:06  <mraleph>I should try that
22:25:45  <ryah>it sure makes things easy when we compile ssl in the tree
22:26:09  <ryah>like - dont have to worry if we build ia32 instead of x64
22:27:47  <CIA-75>node: Ryan Dahl master * r42529dd / (Makefile wscript):
22:27:47  <CIA-75>node: waf: Don't build out/Debug/node_g - just out/Debug/node
22:27:47  <CIA-75>node: This is to match how GYP does it. - http://git.io/5sg4FA
22:27:47  <CIA-75>node: Ryan Dahl master * rb15ab5d / (35 files in 13 dirs): Upgrade V8 to 3.5.7 - http://git.io/vuPzAQ
22:37:38  <bnoordhuis>piscisaureus: sunos and linux have a system-wide ipv6 only setting - if it's enabled, the dual stack test would fail
22:37:44  <bnoordhuis>but no one ever does so...
22:38:10  <piscisaureus>bnoordhuis: yeah for windows too
22:39:07  <piscisaureus>bnoordhuis: there are issues - windows has ipv6only on by default
22:39:18  <piscisaureus>so to use dualstack you have to explicitly turn it off
22:39:22  <bnoordhuis>piscisaureus: seriously? gah
22:39:37  <piscisaureus>.... which I am doing if the user does not specify UV_IPV6ONLY
22:39:58  <bnoordhuis>yeah, i was toying with the idea of doing that on the unices
22:39:59  <piscisaureus>bnoordhuis: but this will fail on 3k and xp (no dualstack support) and on machines that have no ip4 stack
22:40:16  <bnoordhuis>windows 3000?
22:40:19  <piscisaureus>yeah
22:40:22  <bnoordhuis>rad!
22:40:38  <piscisaureus>it's nice
22:40:49  <piscisaureus>they have 9.394.093 new syscalls
22:40:52  <piscisaureus>compared to windows 7
22:41:10  <igorzi>ryah: piscisaureus: is the long-term goal to keep using these eio apis (that we're moving to libuv) for fs? or are we going to have uv_file_t at some point (which is uv_stream_t), and implement it through IOCP?
22:41:30  <piscisaureus>bnoordhuis: I don't know how to deal with it if turning IPV6ONLY off fails
22:41:39  <piscisaureus>right now I am just failing silently
22:41:44  <DrPizza>igorzi: the latter I hope!
22:41:59  <piscisaureus>ryah -> https://gist.github.com/1166783
22:42:08  <bnoordhuis>piscisaureus: can it plausibly fail? or are you talking about 2k3 and xp?
22:42:44  <piscisaureus>bnoordhuis: it always fails on 2k3 and xp, it also fails if the system has no ipv4 driver
22:42:54  <piscisaureus>which is not very plausible
22:43:07  <piscisaureus>(but removing ipv4 is possible - easy even)
22:43:35  <igorzi>i'm asking because if this eio work is really just temporary then do we really need to implement these APIs using win32? it'll be a lot faster to implement on top of crt
22:43:50  <igorzi>ryah: piscisaureus: DrPizza: ---^
22:44:08  <DrPizza>it will?
22:44:12  <DrPizza>crt sux
22:44:28  <igorzi>i know.. but if it's throw-away work anyway..
22:44:33  <piscisaureus>igorzi: read/write/open might be obsoleten (though I don't expect that)
22:44:33  <ryah>igorzi: i think we want both - the thread pool operations for manipulating the file system , and a file stream object
22:44:41  <DrPizza>well we don't know how throwaway it really is
22:44:55  <ryah>igorzi: i.e. not temporary
22:45:05  <piscisaureus>igorzi: but all the other stuff, mkdir stat etc are definitely there to stay
22:45:06  <DrPizza>and plenty o fthings can't use overlapped I/O anyway
22:45:20  <igorzi>ok, got it.. that's what i wanted to clear up
22:45:47  <piscisaureus>igorzi: if it's too much work stub it out with crt calls for now
22:46:19  <piscisaureus>igorzi: I can imagine that you don't want to spend a day implementing stat at this point
22:47:13  <piscisaureus>igorzi: actually I would be happy if you could get to a state that works as well/bad as libeio real soon
22:49:17  <igorzi>piscisaureus: ok.. let me do that first (so that we could get to the same state as eio, but without pthreads)
22:49:29  <igorzi>otherwise this has the potential to take very long time
22:50:28  <piscisaureus>igorzi: there's also a lot of platform-dependent nitty details that you would need to solve. I'd rather do that after we get the boilerplate in place
22:51:22  <piscisaureus>s/we/you/ :-)
22:51:31  <igorzi>piscisaureus: yep, agreed :)
22:52:07  <ryah>yes, fast > correct
22:52:32  <ryah>we need to get pthread-win32 out of there quickly
22:53:02  <igorzi>ok, for now I will get the infrastructure done, with using crt to implement eio functions. we can decide how to enhance that later.
22:56:28  <piscisaureus>ryah: https://gist.github.com/1166783 -- ok?
22:58:09  <ryah>piscisaureus: sure - but do the work you intend to do before landing that
22:58:42  <piscisaureus>ryah: ok
23:12:43  <piscisaureus>bnoorhuis++
23:12:46  <piscisaureus>bnoordhuis++
23:12:58  <piscisaureus>bnoordhuis: this udp-ipv6 test is good!
23:13:05  <bnoordhuis>piscisaureus: cool :)
23:13:20  <bnoordhuis>thanks for testing
23:17:35  <igorzi>piscisaureus: https://github.com/joyent/libuv/commit/1edba4006c5aca3ec583dfea7b0e059d83f1dab0#L2L145.. we no longer need to zero out overlapped?
23:17:59  <piscisaureus>igorzi: no, why would we
23:18:06  <igorzi>piscisaureus: nm.. because we now set Internal explicitely
23:18:51  <piscisaureus>igorzi: yes. and we only look at the Internal field if it has a meaning to uv_process_something_req, instead of always in uv_poll
23:54:33  <ryah>bnoordhuis: next week i want you to do perf work on the libuv backend - we need to get it up to legacy speeds...
23:55:03  <ryah>im not sure exactly what we should look at though :-/
23:55:07  <ryah>no low hanging fruit
23:55:25  <bnoordhuis>ryah: sounds like a plan
23:55:34  <ryah>https://twitter.com/#!/ronkorving/status/106147453719412736
23:55:49  <bnoordhuis>:/
23:57:01  <ryah>maybe we can convince brenden to take a look too