00:28:49  * jschmidtquit (Remote host closed the connection)
00:51:28  * fredkquit (Quit: Leaving.)
00:56:11  * ryancnelsonquit (Quit: Leaving.)
01:08:35  * yunongquit (Ping timeout: 256 seconds)
01:11:39  * dap_quit (Quit: Leaving.)
01:20:00  * ed209quit (Remote host closed the connection)
01:20:13  * ed209joined
01:37:23  * mindlacequit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
02:29:26  * mindlacejoined
02:37:28  * utlemmingquit (Remote host closed the connection)
03:07:03  * marsellquit (Ping timeout: 240 seconds)
03:09:51  * bixujoined
03:13:28  * mindlacequit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
03:15:34  * bixuquit (Ping timeout: 264 seconds)
03:33:48  * utlemming_mobilejoined
03:34:30  * utlemming_mobilequit (Client Quit)
03:34:48  * utlemming_mobilejoined
03:34:52  * utlemming_mobilequit (Client Quit)
03:45:43  * ringzerojoined
03:55:11  * ringzeroquit
03:55:35  * ringzerojoined
04:02:33  * yruss972joined
04:17:48  * ringzeroquit
05:00:22  * bixujoined
05:00:50  * ringzerojoined
05:04:46  * bixuquit (Ping timeout: 264 seconds)
05:05:11  * ringzeroquit (Ping timeout: 240 seconds)
05:05:49  * ringzerojoined
05:10:04  * yunongjoined
05:34:35  * yunongquit
05:58:06  * mindlacejoined
06:07:29  * mindlacequit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
06:48:54  * bixujoined
06:53:33  * bixuquit (Ping timeout: 255 seconds)
07:08:08  * ringzeroquit
07:25:54  * bixujoined
08:01:03  * marselljoined
08:35:01  * bixuquit (Ping timeout: 248 seconds)
08:53:33  * bixujoined
09:26:27  * bixuquit
09:27:34  * bixujoined
10:15:49  * bixuquit (Remote host closed the connection)
10:16:18  * bixujoined
10:20:01  * ed209quit (Remote host closed the connection)
10:20:08  * ed209joined
10:26:52  * bixuquit (Ping timeout: 245 seconds)
12:02:36  * bixujoined
12:07:58  * bixuquit (Ping timeout: 240 seconds)
12:12:54  * bsdgurujoined
13:53:03  * bixujoined
13:57:34  * bixuquit (Ping timeout: 264 seconds)
14:36:38  * bsdguruquit (Ping timeout: 240 seconds)
15:09:52  * chorrelljoined
15:10:41  * utlemmingjoined
15:13:24  * nfitchjoined
15:27:14  * ryancnelsonjoined
15:27:28  * mindlacejoined
15:37:27  * ryancnelsonquit (Quit: Leaving.)
15:41:48  * bixujoined
15:48:03  * bixuquit (Ping timeout: 240 seconds)
15:50:09  * fredk1joined
15:53:05  * yruss972quit
16:06:34  * yruss972joined
16:07:54  * chorrellquit (Quit: My Mac has gone to sleep. ZZZzzz…)
16:10:31  * yruss972quit (Remote host closed the connection)
16:12:44  * dap_joined
16:20:08  * ringzerojoined
17:22:00  * ryancnelsonjoined
17:26:47  * AvianFlujoined
17:27:41  * bsdgurujoined
17:33:08  * bixujoined
17:37:25  * bixuquit (Ping timeout: 256 seconds)
17:38:10  * chorrelljoined
17:44:10  * elijah-mbpjoined
17:50:50  * mindlacequit (Read error: Connection reset by peer)
17:55:35  * mindlacejoined
18:04:09  * ryancnelsonquit (Quit: Leaving.)
18:09:54  * bsdguruquit (Quit: bsdguru)
18:12:39  * nicholaswyoungjoined
18:20:13  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
18:34:34  <nicholaswyoung>Do you know of a situation when the node-manta client would emit ECONNRESET?
18:36:39  <nicholaswyoung>I'm tracking down a bug related to file uploads in my Express app, and would like some input.
18:53:33  * ringzeroquit
18:55:18  * ringzerojoined
18:56:30  * yruss972joined
19:15:00  * ringzeroquit
19:15:41  * bsdgurujoined
19:16:08  * yruss972quit
19:21:16  * ryancnelsonjoined
19:22:35  * bixujoined
19:27:42  * bixuquit (Ping timeout: 240 seconds)
19:38:09  * ryancnelsonpart
20:13:17  <nicholaswyoung> I think I might be experiencing this bug: https://github.com/joyent/node-manta/issues/53
20:13:23  <nicholaswyoung>Any insight?
20:15:48  <bahamas101>nicholaswyoung: are you using a private key explicitly, or ssh-agent?
20:16:04  <bahamas101>i've seen that issue with `ssh-agent` (before signatures were cached), so make sure you are running the latest node-manta
20:16:12  * bahamas101changed nick to bahamas10
20:17:22  <nicholaswyoung>private key, which I was told is the best for my scenario
20:17:45  <nahamu>nicholaswyoung: what would your expected behaviour be for attempting to upload a file that's still receiving writes?
20:17:59  <nicholaswyoung>So here's the storyline:
20:18:18  <nicholaswyoung>I've created a Mongoose plugin that grabs a specific attribute and uploads to manta.
20:18:29  <nicholaswyoung>I pass the file path in from Express, and try to save the model
20:19:05  <nicholaswyoung>manta grabs the path from the attribute, and creates a write stream. once the upload is done, it calls next() and continues with the mongoose middleware
20:19:28  <nicholaswyoung>So, in theory, the upload should be complete when the model instance is initialized, but I'm not 100% sure that's always the case
20:20:00  * ed209quit (Remote host closed the connection)
20:20:07  <nicholaswyoung>Sometimes I see this error, other times not. And since file sizes vary, it makes sense that some files could be finished going to disk while others aren't - hence the error
20:20:07  * ed209joined
20:20:18  <nicholaswyoung>I can show code if you need me to, but that's the gist
20:39:10  <nicholaswyoung>bahamas10: Does it sound like my workflow is even close to correct?
20:40:32  <nicholaswyoung>nahamu: Hopefully, the file is already finished being written by the time I kick off manta's stream, but I have no good way of knowing that to be always true
20:43:33  <nahamu>nicholaswyoung: well, I've never used Express or Mongoose so I don't really have much context to understand what you're talking about, but I find that my questions often elicit answers that make it easier for the smart people to help solve the problem...
20:44:11  <nicholaswyoung>nahamu: Do you know of a way to log when files are finished being written? Maybe that's a start.
20:44:19  <nahamu>so I'll just poke tjfontaine and dap_ and see what happens.
20:44:59  <nahamu>nicholaswyoung: I know nothing. My naive assumption is that a callback should be called when the file is finished being written and that that's the place where you want to do the next step
20:45:06  <nahamu>but like I said, I don't know much about this stuff.
20:47:12  <nicholaswyoung>Hey, it's more than I know, I'm sure.
20:49:46  <nicholaswyoung>Well, for anyone looking, here's the code in question: https://gist.github.com/nicholaswyoung/fcf45134c08f671f21e7
20:49:53  <nicholaswyoung>That's the mongoose plugin that handles file uploads.
20:50:43  <nicholaswyoung>Specifically, this bit handles moving files from local disk to Manta: https://gist.github.com/nicholaswyoung/fcf45134c08f671f21e7#file-media-js-L139-L181
21:05:09  <nahamu>which function lands the file to local disk?
21:09:37  <nahamu>(and does it take a callback? ;) )
21:09:51  <nicholaswyoung>nahamu: Oh, that's handled by Express' bodyparser. Let me grab a link for that...
21:10:34  <nicholaswyoung>nahamu: https://github.com/expressjs/multer/blob/master/index.js
21:11:31  <nicholaswyoung>There. That's the bodyparser middleware. It looks like busboy.on('finish') indicates that parsing is done, but the streams may not be finished writing to disk
21:13:12  <nahamu>https://github.com/expressjs/multer/blob/master/index.js#L107
21:14:05  <nicholaswyoung>Hmmm. I assumed I would only use that for logging, not doing any actual work.
21:14:48  <nahamu>If that's when the file has finished landing on disk, it might be a useful place to call the step of uploading the file...
21:15:04  <nahamu>but I know less about Express than you do.
21:15:32  <nicholaswyoung>That's not easily done, because this is designed to be included in the main application initializer as middleware. :-/
21:16:04  <nicholaswyoung>Plus, I don't think that guarantees that the file is done writing to disk, just that it's arrived at the server.
21:16:28  <nicholaswyoung>Times like this make me want to swear off Node, though.
21:16:59  <nicholaswyoung>Oh, I was wrong. Maybe that does guarantee that the file is finished writing...
21:17:11  <nahamu>Well, I have a couple of other crazy ideas...
21:17:20  <nicholaswyoung>Hit me.
21:17:36  <nahamu>One option is to sign a URL and have the client code upload directly to Manta and then just tell you that it's done.
21:18:32  <nicholaswyoung>That's definitely on the table, and something I've looked into implementing, but wouldn't it be hard to validate the file format? Because I only allow a handful of types.
21:18:52  <nicholaswyoung>And, I need to be absolutely sure that the type they upload is allowed.
21:19:10  <nahamu>Why?
21:19:43  <nicholaswyoung>Because if they don't upload a specific file type, ffmpeg (on Manta) will fail spectacularly.
21:19:47  <nahamu>IIRC you run a processing job in Manta after the file is uploaded, right?
21:20:02  <nahamu>Why not do the validation step just before you invoke ffmpeg?
21:20:16  <nahamu>And just delete the object if it fails the validation step
21:20:34  <nicholaswyoung>Right. But ffmpeg is incredibly picky about what it's fed. And I need to ensure that folks aren't re-encoding MP3's and outputting crappy audio.
21:20:47  <nahamu>.oO(what happens if you issue "mrm <object>" during a map step operating on object)
21:21:01  <nahamu>Where are you doing the validation currently?
21:21:27  <nicholaswyoung>It's in the initial gist: https://gist.github.com/nicholaswyoung/fcf45134c08f671f21e7#file-media-js-L118-L135
21:22:09  <nahamu>but all that does is check the mime type?
21:22:31  <nicholaswyoung>So far, that's been good enough.
21:22:48  <nicholaswyoung>But I'm unsure how to even handle that if we're uploading directly to Manta.
21:22:51  <nahamu>is that some node library doing the validation?
21:23:27  <nicholaswyoung>the invalidate() call basically prevents the document from ever being saved, and bypasses the remaining validation callbacks
21:24:02  <nahamu>okay, so let's go from another angle
21:24:47  <nahamu>is there any way that the callback that happens when the file is done landing on disk can tweak something in the process somewhere that your middleware can wait for?
21:25:49  <nicholaswyoung>I'm guessing that's the only way to really solve this. It'll be messy, that's for sure, but I think there's a way to handle it.
21:26:16  <nahamu>when you say that ffmpeg fails spectacularly, does it do so quickly or slowly?
21:26:21  <nicholaswyoung>But having manta just check the status of the file, as isaac suggested in his issue, would be a nice long-term solution
21:26:38  <nicholaswyoung>slowly. terribly slowly. It's not a very good feedback loop
21:26:44  <nahamu>blah
21:27:43  <nahamu>I don't know that you want to use Isaac's solution. He's uploading log files and doesn't seem to care if he gets all of it.
21:28:00  <nahamu>I take that back, it might work
21:28:34  <nahamu>rather than just print a warning, if it could let you call a particular function when that error case happens, you could reupload
21:28:47  <nicholaswyoung>I'm going to dig into this some more. But it'll probably be the end of my Node days if I can't work it out simply.
21:28:49  <nahamu>but Isaac seems to be expecting that he'd just manually run the script again
21:29:21  <nicholaswyoung>Because I don't want the user to need to care if my system has a crazy unexpected error. I should be able to rescue them from that, if at all possible.
21:29:42  <nahamu>agreed.
21:30:26  <nahamu>but currently it seems like you're trying to upload a file that isn't done being written which I think is an edge case that the Manta developers didn't forsee.
21:30:49  <nahamu>is there a way to get a stream of the file rather than land it locally on disk?
21:31:18  <nicholaswyoung>Possibly. But what happens if the upload stalls in-flight?
21:31:28  <nahamu>in theory could you stream the data into manta as it's streaming in to you?
21:31:37  <nahamu>uh, how does the current system handle that?
21:31:38  <nicholaswyoung>I've thought about this, and it might work, but I'm afraid of having lots of half-uploaded files in my manta bucket.
21:31:54  <nahamu>how do you know that the upload is incomplete?
21:31:58  <nicholaswyoung>Every so often, it purges tempfiles - so I don't worry about half-uploaded crap sitting around
21:32:13  <nicholaswyoung>Oh, that parsing middleware handles it.
21:32:36  <nahamu>hrm
21:32:38  <nicholaswyoung>If the 'finish' callback isn't fired, you know the multipart body was never completed.
21:32:48  <nicholaswyoung>Or event, rather.
21:33:07  <nahamu>I wish I knew more so that I could help you better...
21:33:20  <nicholaswyoung>I wish I knew more so I could just fix the damn problem. :-/
21:33:31  <nicholaswyoung>But you've done a lot, so don't worry.
21:33:56  <nicholaswyoung>But if mcavage was around, I'd certainly ask him!
21:33:58  <nahamu>All I do is ask simple questions... That got Socrates killed...
21:34:07  <nicholaswyoung>I think you're safe.
21:34:12  <nahamu>whew
21:34:40  <nicholaswyoung>Anyway, I'm going to take a break and try to get a fresh perspective on this. Food and beer will help also.
21:34:47  <nahamu>Enjoy!
21:34:54  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
21:39:59  * ringzerojoined
21:40:00  * ringzeroquit (Client Quit)
22:06:55  * ringzerojoined
22:11:09  * ringzeroquit (Client Quit)
22:34:57  * nicholaswyoungjoined
22:40:11  * ringzerojoined
22:40:45  * ringzeroquit (Client Quit)
22:47:53  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
23:01:00  * bixujoined
23:06:00  * bixuquit (Ping timeout: 255 seconds)
23:14:47  * bsdguruquit (Quit: bsdguru)
23:21:42  * nfitchquit (Quit: Leaving.)