00:44:59  * quijotejoined
00:49:14  * quijotequit (Ping timeout: 240 seconds)
00:55:03  * AvianFluquit (Remote host closed the connection)
01:20:00  * ed209quit (Remote host closed the connection)
01:20:08  * ed209joined
01:44:13  * marsellquit (Quit: marsell)
01:45:47  * quijotejoined
01:50:19  * quijotequit (Ping timeout: 240 seconds)
02:46:37  * quijotejoined
02:51:04  * quijotequit (Ping timeout: 240 seconds)
03:04:08  * utlemmingjoined
03:17:40  * utlemmingquit (Quit: Konversation terminated!)
03:18:29  * utlemmingjoined
03:28:22  * utlemmingquit (Quit: Konversation terminated!)
03:28:36  * utlemmingjoined
03:47:18  * quijotejoined
03:51:34  * quijotequit (Ping timeout: 240 seconds)
04:09:15  * utlemmingquit (Quit: Konversation terminated!)
04:13:09  * utlemming_mobilejoined
04:24:46  * utlemming_mobilequit (Ping timeout: 258 seconds)
04:48:12  * quijotejoined
04:51:30  * AvianFlujoined
04:52:55  * quijotequit (Ping timeout: 276 seconds)
04:57:36  * AvianFluquit (Remote host closed the connection)
04:57:51  * AvianFlujoined
05:31:16  * quijotejoined
06:59:36  * yruss972joined
07:38:57  * quijotequit (Ping timeout: 252 seconds)
07:57:10  * marselljoined
08:17:42  * quijotejoined
09:48:14  * quijotequit (Ping timeout: 255 seconds)
09:52:30  * irajoined
10:15:58  * quijotejoined
10:20:18  * quijotequit (Ping timeout: 240 seconds)
11:16:44  * quijotejoined
11:21:37  * quijotequit (Ping timeout: 276 seconds)
12:17:30  * quijotejoined
12:22:08  * quijotequit (Ping timeout: 255 seconds)
12:34:23  * iraquit (Read error: Connection reset by peer)
12:36:06  * nicholaswyoungjoined
13:18:16  * quijotejoined
13:18:37  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
13:22:53  * quijotequit (Ping timeout: 255 seconds)
13:24:37  * nicholaswyoungjoined
13:36:23  * quijotejoined
13:38:36  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
13:41:33  * nicholaswyoungjoined
13:45:12  * utlemming_mobilejoined
13:58:31  * irajoined
14:19:48  * utlemming_mobilequit (Quit: Konversation terminated!)
14:22:29  * quijotequit (Ping timeout: 265 seconds)
14:23:24  * quijotejoined
14:45:34  * nfitchjoined
15:13:30  * quijotequit (Ping timeout: 240 seconds)
15:41:08  * quijotejoined
15:45:31  * quijotequit (Ping timeout: 252 seconds)
15:53:18  * fredkjoined
15:53:18  * fredkquit (Client Quit)
15:53:41  * fredkjoined
16:19:04  <dap_>nicholaswyoung: are you still looking for help writing a poll function?
16:19:50  <nicholaswyoung>Yep. I'm still stuck. I know that mcavage had sent me one a while back, but I can't seem to find it.
16:20:01  <dap_>I'm digging it up now :)
16:20:22  <dap_>https://github.com/joyent/node-manta/blob/master/bin/mjob#L338
16:20:37  <dap_>I wouldn't actually use exponential backoff the way that thing does
16:21:23  * quijotejoined
16:22:15  * yunong1joined
16:22:21  <nicholaswyoung>Gotcha... I was just about to ask what that was doing. How would you recommend going about it then?
16:22:52  <nicholaswyoung>Basically, I'm pushing a audio encode job (with multiple phases) into Manta, and I would like to know when the jobs have completed (or failed).
16:23:26  <nicholaswyoung>If they succeed, call the callback without params (if they fail, pass the error). Regular node-style, I think.
16:24:22  <dap_>It depends on how clever you want to try to be to minimize latency, and how long you expect the job to take. Exponential backoff isn't actually a bad choice, but it looks like the max delay between polls that that code uses is 10s, which means it can take up to that long after the job completes before you notice
16:25:36  <dap_>The code is mostly the same either way: you call client.job(), which returns asynchronously. If it fails or the job's not done yet, you try again in a few seconds. If it succeeds and the job's done, you're done.
16:25:41  <nicholaswyoung>I would like to speed that up, if possible. I'm still digging around for the gist mcavage sent, but I think it was (a) private, and (b) sent via a direct message.
16:27:59  <nicholaswyoung>That should be easy enough to work around. So basically, I'll just watch the state outputted from client.job, and call until it's done.
16:29:22  <dap_>yeah
16:33:28  <nicholaswyoung>That makes sense. I don't know why I didn't realize that until now!
16:33:37  <nicholaswyoung>Thanks for your help, as always.
16:33:47  <dap_>No problem!
16:46:54  * quijotequit (Ping timeout: 240 seconds)
16:50:30  * yunong1quit (Quit: Leaving.)
17:01:25  * ryancnelsonjoined
17:43:21  * quijotejoined
17:47:37  * quijotequit (Ping timeout: 252 seconds)
17:51:59  <tjfontaine>I'll drop this here, over the weekend I wrote up a little thing to interact with manta in terms of node streams and something appearing "functional" https://github.com/tjfontaine/mlang
18:07:29  * yruss972_joined
18:10:43  * yruss972quit (Ping timeout: 252 seconds)
18:11:28  * yruss972_quit (Ping timeout: 240 seconds)
18:13:28  * marsellquit (Ping timeout: 240 seconds)
18:18:34  * marselljoined
18:29:27  * iraquit (Quit: Connection terminated.)
18:31:14  * quijotejoined
18:36:44  <nahamu>cool
18:38:37  * AvianFluquit
19:05:17  * nicholaswyoungquit (Quit: Computer has gone to sleep.)
19:05:47  * nicholaswyoungjoined
19:15:28  * quijotequit (Ping timeout: 276 seconds)
19:24:11  * AvianFlujoined
19:41:32  * quijotejoined
19:46:17  * quijotequit (Ping timeout: 255 seconds)
19:53:58  * isaacsquit (Ping timeout: 240 seconds)
19:54:32  * bahamas10quit (Ping timeout: 265 seconds)
19:55:56  * isaacsjoined
19:56:18  * yunongquit (Ping timeout: 240 seconds)
19:57:57  * yunongjoined
20:19:05  * seldojoined
20:27:18  <nicholaswyoung>tjfontaine: Would that module let me know when all of the phases of my manta job are done?
20:27:41  <nicholaswyoung>In other words, is it better than polling Manta and asking for periodic updates?
20:27:54  <tjfontaine>nicholaswyoung: an 'end' event is fired with the object contents, but it's achieved by polling
20:28:09  <tjfontaine>with the json parsed object of the job result
20:28:43  <nicholaswyoung>Ooh, I probably should look at that. Because I'm currently writing my own polling implementation, and so far, it's sorta a pain.
20:29:25  <nicholaswyoung>I guess my last question is, is the module production-ready, or is that unwise of me to do?
20:29:52  <tjfontaine>nicholaswyoung: https://gist.github.com/tjfontaine/730b7957d90134ba2e3a
20:30:16  <tjfontaine>define production ready? :)
20:30:29  <nicholaswyoung>As in, will it blow up in fun and unexpected ways.
20:30:48  <nicholaswyoung>Maybe it's best that I just take a look at how you executed polling...
20:31:14  <tjfontaine>probably, I just did a brain dump on it over the weekend, https://github.com/tjfontaine/mlang/blob/master/lib/mlang.js#L119-L134
20:34:50  <ryancnelson>per that "how do i know it's done" discussion in here last week, i was playing with https://pushover.net/ this weekend, and IFTTT … using curl as a reducer to make my phone buzz with jobs complete
20:35:13  <tjfontaine>ya, that's the other way that you could do it
20:35:27  <tjfontaine>with node that's reasonably trivial, presuming of course you don't have to deal with nat
20:36:40  <nicholaswyoung>I'll probably just poll. For my case, it'd be a pain to deal with triggering curl, etc.
20:36:54  <nicholaswyoung>Thanks for linking to the relevant code, tjfontaine. That clears things up a lot.
20:42:16  * quijotejoined
20:44:14  <nicholaswyoung>Crap. This is harder than I thought. I guess I don't quite understand setInterval yet. This gets stuck in a loop. https://gist.github.com/nicholaswyoung/701b75afafe5c5159a2d
20:45:49  <dap_>setInterval calls the function *every* 1000ms
20:45:51  <tjfontaine>nicholaswyoung: you're adding a timer for every key you add
20:45:51  <dap_>forever
20:46:14  <nicholaswyoung>Which should only be one, or so I thought.
20:46:17  <tjfontaine>you really only want to do it once you've successfully created a job, that is in createJob cb
20:46:21  <tjfontaine>no you're doing it in jobAddKey
20:46:36  <tjfontaine>addJobKey
20:46:42  <nicholaswyoung>Ah, I thought the array length [input] only added the key once.
20:46:55  * quijotequit (Ping timeout: 252 seconds)
20:47:24  <tjfontaine>well maybe, not clear on that -- it's a little bizare of a pattern to do it that way though
20:48:56  <nicholaswyoung>Do you see anything that I'm doing that seems off besides that?
20:49:17  <tjfontaine>well, clearInterval takes the result from setInterval
20:49:23  <tjfontaine>not the function
20:49:25  <nicholaswyoung>The job.process() basically accepts incoming jobs off the queue (with kue).
20:51:20  <nicholaswyoung>So I clear the interval like this, I guess: https://gist.github.com/nicholaswyoung/701b75afafe5c5159a2d
20:55:03  <tjfontaine>yup
21:00:46  <nicholaswyoung>I've updated the gist, and it works. Except that the job keeps running even after the outputs are done. https://gist.github.com/nicholaswyoung/701b75afafe5c5159a2d
21:01:07  <nicholaswyoung>Do I have to explicitly end the job?
21:02:45  <nahamu>yes, you have to close it to let it know that there are no more inputs
21:02:56  <nahamu>it will time out eventually, though.
21:04:17  <nicholaswyoung>nahamu: How do I close the job? client.close()?
21:09:04  <nahamu>http://apidocs.joyent.com/manta/nodesdk.html#clientendjobjobid-options-callback
21:09:10  <nahamu>client.endJob
21:11:12  <nahamu>nicholaswyoung: ^
21:11:36  <nicholaswyoung>Thanks. I'm working on debugging my worker now. I assume endJob takes the usual signature: jobid, (err)
21:11:46  <nahamu>I linked you to the docs
21:12:15  <nicholaswyoung>Ah, there's the link. Thanks.
21:12:20  <nicholaswyoung>It took a sec to show up.
21:16:07  * bahamas10joined
21:44:12  * quijotejoined
21:48:53  * quijotequit (Ping timeout: 264 seconds)
21:50:19  <nicholaswyoung>This polling thing is kicking my ass so far. If anyone would be willing to take a peek at my gist and help with a solution, I'd be very thankful.
21:50:48  <nicholaswyoung>It's the need for recursion, and calling watch() until the job is complete, that's flustering me. https://gist.github.com/nicholaswyoung/701b75afafe5c5159a2d
21:53:23  <tjfontaine>nicholaswyoung: is encode cb being fired a bunch of times?
21:53:42  <tjfontaine>that is to say, you're going to have multiple jobs in flight
21:53:56  <nicholaswyoung>It should only be called once for each job that needs to be processed.
21:54:30  <tjfontaine>need to take a step back for a moment, are you trying to encode multiple keys in manta?
21:54:30  <nicholaswyoung>i.e. it's called once for each raw media file (wav, etc) that needs to be encoded.
21:55:07  <nicholaswyoung>just one input key, but yes, there will be multiple output keys, if that's what you mean
21:55:10  <tjfontaine>so a.wav b.wav c.wav, encode will be called 3 times?
21:55:35  <nicholaswyoung>Just a.wav, that then outputs a.flac, .m4a, etc.
21:57:11  <tjfontaine>https://gist.github.com/nicholaswyoung/701b75afafe5c5159a2d#file-encode-js-L25 what's that for?
21:57:54  <nicholaswyoung>I thought I needed to recall watch() if the job wasn't done
21:57:58  <nicholaswyoung>Maybe that's wrong
21:58:23  <tjfontaine>the setInterval handles rescheduling the function, you are getting stuck in a recursive call right now?
21:58:48  <tjfontaine>you probalby just want if (job.state !== 'done') return;
21:59:11  <tjfontaine>and then if execution continues immediately clearInterval(interval)
22:00:13  <tjfontaine>otherwise while you're waiting on your output you could end up calling this code path multiple times
22:03:22  <nicholaswyoung>I cleaned it up a bit, and made the changes you suggested. It doesn't look like it's running any differently. :(
22:03:27  <nicholaswyoung>I'm at the tail end of engineering a rather large node app, and I think my brain is pretty well fried. Otherwise, this seems relatively simple.
22:04:16  <nicholaswyoung>If only I could find the gist mcavage sent me ages ago... it worked. Sadly, it's lost to a hard drive crash somewhere.
22:05:44  <tjfontaine>nicholaswyoung: oh, endJob needs to happen when you know there are no more keys to add to the job
22:08:54  <nicholaswyoung>well, once the encode callback is called, I know that I'm done. Maybe that should be called in the initial callback, rather than watch()
22:09:29  <tjfontaine>nicholaswyoung: if there's only ever one addJobKey then in its cb on success you should call endJob
22:10:24  <nicholaswyoung>Perfect! With that change, it works now. Thank you for your patience.
22:18:13  * christianboehlkequit (Read error: Operation timed out)
22:21:11  * LeftWingquit (Remote host closed the connection)
22:21:36  * christianboehlkejoined
22:21:37  * LeftWing__joined
22:22:26  * LeftWing__changed nick to LeftWing
22:33:56  * nfitchquit (Quit: Leaving.)
22:37:29  * LeftWingquit (Remote host closed the connection)
22:39:11  * LeftWingjoined
22:43:27  * LeftWing__joined
22:44:05  * LeftWing__quit (Remote host closed the connection)
22:44:07  * LeftWingquit (Ping timeout: 276 seconds)
22:44:50  * quijotejoined
22:47:58  * LeftWingjoined
22:48:58  * quijotequit (Ping timeout: 240 seconds)
23:10:43  <nicholaswyoung>Is there a way to get the size of a object stored in Manta?
23:12:49  <rmustacc>I generally get it through mls -lh
23:13:03  <rmustacc>Err, sorry, just mls -l, no h
23:14:04  <ryancnelson>there's also "minfo"
23:14:19  <nicholaswyoung>How about in the JS SDK?
23:14:56  <rmustacc>I'd just look at how those commands are implemented.
23:14:59  <nicholaswyoung>I see the client.info call. I assume that provides file size.
23:15:19  <ryancnelson>minfo *is* a node program, yeah
23:15:41  <tjfontaine>yes, client.info and it's just a HEAD on the key
23:15:51  <nicholaswyoung>Perfect. Thanks again!
23:16:22  * nicholaswyoungquit (Quit: Lingo - http://lingoirc.com)
23:36:08  * preillyjoined
23:45:35  * quijotejoined
23:50:24  * quijotequit (Ping timeout: 265 seconds)