00:09:27  * LeftWingquit (Remote host closed the connection)
00:10:21  * LeftWingjoined
00:18:09  * fredk1quit (Quit: Leaving.)
00:24:58  * nfitchjoined
00:50:25  * bixujoined
00:54:44  * bixuquit (Ping timeout: 248 seconds)
01:20:00  * ed209quit (Remote host closed the connection)
01:20:07  * ed209joined
01:25:27  * ringzerojoined
01:27:10  * ringzeroquit (Client Quit)
01:30:53  * mindlacequit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
01:41:52  * nicholaswyoungjoined
01:48:05  * nicholaswyoungquit (Quit: Lingo - http://www.lingoirc.com)
02:05:46  * dap_quit (Quit: Leaving.)
02:05:46  * chorrelljoined
02:09:53  * mindlacejoined
02:12:18  * mindlacequit (Client Quit)
02:34:14  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
03:11:01  * AvianFluquit (Remote host closed the connection)
04:27:55  * bixujoined
04:32:20  * bixuquit (Ping timeout: 248 seconds)
04:39:47  * marsellquit (Quit: marsell)
06:16:46  * bsdgurujoined
06:53:06  * bsdguruquit (Ping timeout: 255 seconds)
06:53:12  * bsdguru_joined
07:07:08  * bixujoined
07:35:16  * bixuquit (Remote host closed the connection)
07:36:04  * bixujoined
07:45:56  * bsdguru_quit (Ping timeout: 248 seconds)
07:57:30  * bixuquit
08:04:36  * bixujoined
08:16:43  * bixuquit (Remote host closed the connection)
08:39:32  * bixujoined
08:40:45  * LeftWingquit (Remote host closed the connection)
08:40:54  * LeftWingjoined
08:42:33  * bixuquit (Remote host closed the connection)
08:42:43  * bixujoined
10:11:25  * marselljoined
10:20:01  * ed209quit (Remote host closed the connection)
10:20:18  * ed209joined
10:52:35  * jschmidtjoined
12:28:37  * nfitch1joined
12:29:23  * nfitchquit (Ping timeout: 264 seconds)
12:47:32  * nicholaswyoungjoined
13:31:08  <nahamu>good morning, nicholaswyoung
13:31:54  <nicholaswyoung>thanks, nahamu. Good morning to you, wherever you are. (I'm in Chicago.)
13:36:51  <nahamu>Boston
13:37:05  <nahamu>did you figure out a good solution to your problem?
13:39:07  <nahamu>after you left I was looking at that middleware a bit more, and it's annoying that it's unclear to me the relationship between when onParseEnd and onFileUploadComplete run
13:39:52  <nahamu>I wonder if in some sense it's a flaw of that middleware in a use-case like yours
13:39:56  <nahamu>but maybe I was misreading it.
13:48:13  * ed209quit (*.net *.split)
13:48:13  * christianboehlkequit (*.net *.split)
13:48:14  * e^ipiquit (*.net *.split)
13:55:05  * nfitch1quit (Quit: Leaving.)
14:09:25  * christianboehlkejoined
14:28:29  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
14:47:30  * e^ipijoined
14:50:37  * bixuquit (Remote host closed the connection)
14:51:06  * bixujoined
15:03:57  * ringzerojoined
15:06:24  * nfitchjoined
15:16:10  * dap_joined
15:18:24  * ryancnelsonjoined
15:36:16  * ed209joined
15:50:07  * fredk1joined
15:50:27  * dap_quit (Quit: Leaving.)
15:54:10  * ringzeroquit
15:57:11  * ringzerojoined
16:11:48  * ryancnelsonquit (Quit: Leaving.)
16:24:19  * dap_joined
16:25:31  * nicholaswyoungjoined
16:26:07  <nicholaswyoung>nahamu: Sorry, I just saw your messages.
16:26:11  <nicholaswyoung>It
16:26:29  <nicholaswyoung>It's highly possible that I'm misusing the middleware, but I don't think so.
16:26:47  <nicholaswyoung>I'm going to look into directly using busboy, rather than the middleware, and see if that gets me anywhere.
16:27:16  <nahamu>have you asked in a node or express channel?
16:27:31  <nicholaswyoung>Ooh, good idea. I haven't yet, but I will now.
16:27:40  <nahamu>it seems like a reasonable desire to not want to do things until the file is finished landing on disk.
16:28:11  <nahamu>and it currently seems like that's not the case when the middleware is in the seemingly obvious way.
16:28:20  <nahamu>*is used in the ...
16:30:13  <nahamu>but I'll be curious to hear back when you manage to figure out what's going on if you have the time. :)
16:30:15  <nicholaswyoung>Yeah, if I'm inserting the middleware as a part in my stack, rather than in a specific route or controller (which the documentation shows) that all files would be persisted before the next callback is fired.
16:30:28  <nicholaswyoung>Totally. I'm always glad to share.
16:31:53  <nicholaswyoung>It's been a wild and crazy journey through Node so far. In some ways, it's been awesome. In others, it makes me miss Ruby-land. But generally, I like it. And along the way, I've wrote a handful of useful modules: https://github.com/nicholaswyoung/node-chicago-june-2014
16:33:51  <nicholaswyoung>Okay, questions asked in the other channels. No response yet, though I'll just wait patiently...
16:35:43  * ringzeroquit
16:44:30  * ringzerojoined
17:07:49  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
17:13:39  * ringzeroquit
17:21:17  * ringzerojoined
17:22:39  * ringzeroquit (Client Quit)
17:25:57  * ringzerojoined
17:33:08  * ringzeroquit
17:46:58  * bixuquit (Ping timeout: 240 seconds)
18:08:49  * ringzerojoined
18:10:13  * mindlacejoined
18:12:22  * nicholaswyoungjoined
18:15:14  * ringzeroquit
18:21:18  * ringzerojoined
18:24:17  * ringzeroquit (Client Quit)
18:26:25  * chorrelljoined
18:29:37  * ringzerojoined
18:33:22  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
18:33:37  * saxbyjoined
18:40:13  * ringzeroquit
18:43:39  * bixujoined
18:48:18  * bixuquit (Ping timeout: 240 seconds)
18:58:33  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
19:48:43  * nicholaswyoungjoined
20:13:03  * ringzerojoined
20:20:00  * ed209quit (Remote host closed the connection)
20:20:07  * ed209joined
20:24:09  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
20:29:58  * ringzeroquit
20:36:45  * ringzerojoined
20:48:55  * ringzeroquit
20:50:24  * ringzerojoined
20:51:49  * ringzeroquit (Client Quit)
20:53:10  * ringzerojoined
20:59:36  * ringzeroquit
21:06:31  * nicholaswyoungjoined
21:13:46  * ringzerojoined
21:28:55  <nicholaswyoung>bahamas10: in manta#createWriteStream, is the md5 hash required?
21:40:31  <nicholaswyoung>rmustacc: Ping. Got a second to answer a question?
21:46:45  <nfitch>nicholaswyoung: The md5 isn't required, but without it there's no guarantee that what manta received is what you send. If you do send one, Manta will ensure that the the md5 of what you send matches what it puts down on disk.
21:47:18  <nfitch>s/send/sent/
21:50:04  <nfitch>That said, the probability that something is incorrect, is very low (since checksums happen in the lower networking layers), but there *is* still a chance.
21:52:00  <rmustacc>nicholaswyoung: It's always faster to just ask the question rather than asking me if you can ask a question. ;)
21:52:40  <nfitch>I guess to be pedantic, there is also a very, very, very low chance even with the md5sum that what you sent isn't what Manta put on disk. Anyways… :)
21:53:32  <nicholaswyoung>nfitch: Gotcha. How would I go about getting a MD5 of a read stream?
21:53:57  <nicholaswyoung>I'm having some socket reset errors and trying to debug now.
21:54:48  <nicholaswyoung>rmustacc: I generally just ask, but having put it in the channel beforehand, I didn't want to repost the exact same thing. I'm still feeling out IRC etiquette. ;)
21:55:30  <nfitch>nicholaswyoung: Something like this: https://github.com/joyent/node-manta/blob/master/bin/mput#L221
21:56:51  <nahamu>re etiquette: appending "(see above)" would probably have sufficed. :)
21:57:21  <nfitch>That's to compute it on a filestream as it goes by. What we do in manta is:
21:57:33  <nicholaswyoung>nfitch: Thanks. I wonder if not supplying the md5 could be causing issues in my case, so I'm going to give it a shot.
21:57:39  <nfitch>1) Let the frontend compute the md5 on the way through
21:57:58  <nfitch>2) Let the server streaming to disk also compute the md5
21:58:19  <nfitch>3) Return the md5 from #2 to the frontend
21:58:41  <nfitch>4) The frontend compares, and makes sure that the md5s match (if not, it fails the request)
21:58:57  <nicholaswyoung>Should I be watching for errors on my manta upload stream then?
21:59:01  <nfitch>If you send an md5 with the request, we make sure that one matches as well.
21:59:41  <nfitch>So if you have an application that is streaming to manta, you can't send an md5 (cause you're only seeing the bytes fly by).
21:59:52  <nicholaswyoung>Because my implementation basically matches the createWriteStream example in the SDK docs, except the md5.
22:00:14  <nicholaswyoung>In my case, the files should be landing to disk on my server, where it then grabs the stored file's path, and creates a stream that I pipe to Manta.
22:00:35  <nicholaswyoung>But that's what I'm looking at, because I'm getting some weird ECONNRESET errors, and I'm not sure where they're coming from.
22:01:17  <nfitch>Oh, good. So you have the file on disk- you can create the md5 just like the link I pasted.
22:01:58  <nfitch>ECONNRESETs are kinda the bane of any node app cause it's difficult to determine if they are because the client or the server is the one that initiates the reset.
22:01:59  <nicholaswyoung>I considered streaming the upload to Manta as it comes in, but the way I understand it, that could cause half-uploaded files to be stored, if the user's connection dies.
22:02:38  <nicholaswyoung>Ugh. Any advice for debugging this mess? It's a pretty simple app: express/mongoose, being developed on OSX.
22:03:05  <nfitch>You could do ^^ if the client can compute an md5 as well. Then you compare both the bytes transferred (content-length header) and the md5 (content-md5) header to make sure it all came through.
22:03:06  <nicholaswyoung>And it doesn't look like Manta is the culprit: because I don't see the upload stream emitting any errors.
22:03:49  <nahamu>oooh, if the client sent along the md5, that would be really cool... can browers do that easily?
22:03:54  <nicholaswyoung>(upload to Manta, that is.)
22:04:14  <nfitch>Typically what we do in situations like this is use wireshark to determine what initiates the close, then debug from there.
22:04:35  <nfitch>And typically in the past it's happened because the client goes idle for a period of time where the server times it out.
22:04:38  <nicholaswyoung>nahamu: You might be able to hack something out with the File API.
22:04:46  <nfitch>But in your case… dunno.
22:05:08  <nicholaswyoung>I need to setup Wireshark. I guess that's my next step.
22:05:32  <nicholaswyoung>It's highly possible that either the client or server is timing out, because I'm uploading a series of 50+mb files...
22:05:43  <nicholaswyoung>But I didn't think Node had default timeouts set.
22:06:40  <nicholaswyoung>The app doesn't crash, if that indicates anything.
22:08:13  <nfitch>Node servers do have a default timeout… I don't think client connections do.
22:08:47  <nicholaswyoung>Gotcha. Maybe that's the issue, because the error doesn't seem to be coming from my app. It looks like a generic, unwrapped error message that I'm not sending.
22:09:17  <nfitch>What node version are you using?
22:09:37  <nicholaswyoung>0.10.29
22:10:22  <nicholaswyoung>Basically, I've written a bootstrap script that uses request to upload data to the REST API. The API then passes data to Manta.
22:11:04  <nicholaswyoung>The problem begins when I try to do multiple file uploads, like I'm doing in this script: https://gist.github.com/nicholaswyoung/3a8ab6afaef0c25c092c
22:11:45  <nicholaswyoung>Lines 98 - 126 are basically one file upload after the other. The first file goes up fine, but after that, I see ECONNRESET and the client dies.
22:38:25  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
22:54:20  * nicholaswyoungjoined
23:06:59  * saxbyquit (Remote host closed the connection)
23:10:53  * ringzeroquit (Read error: Connection reset by peer)
23:10:58  * ringzero_joined
23:11:51  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
23:12:09  * saxbyjoined
23:13:18  * ringzero_quit (Read error: Connection reset by peer)
23:13:19  * nicholaswyoungjoined
23:13:24  * ringzerojoined
23:19:03  * ringzeroquit (Read error: Connection reset by peer)
23:19:09  * ringzero_joined
23:19:27  * ringzero_quit (Client Quit)
23:22:05  * nfitchquit (Quit: Leaving.)
23:23:30  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
23:29:24  * nicholaswyoungjoined
23:40:34  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
23:43:48  * nicholaswyoungjoined
23:43:57  * nicholaswyoungquit (Client Quit)