00:04:44  * nfitchjoined
00:08:55  * nfitchquit (Ping timeout: 240 seconds)
00:14:14  * notmattquit (Remote host closed the connection)
01:01:08  * AvianFluquit (Remote host closed the connection)
01:01:38  * AvianFlujoined
01:33:27  * mattappersonquit (Remote host closed the connection)
01:34:04  * mattappersonjoined
01:36:49  * abraxasjoined
01:38:58  * mattappersonquit (Ping timeout: 265 seconds)
01:52:59  * nfitchjoined
01:57:37  * nfitchquit (Ping timeout: 272 seconds)
02:44:55  * mattappersonjoined
02:49:35  * mattappersonquit (Ping timeout: 245 seconds)
03:23:09  * abraxas_joined
03:24:53  * abraxasquit (Ping timeout: 248 seconds)
03:24:54  * wesolowsquit (Ping timeout: 248 seconds)
03:25:09  * wesolowsjoined
03:29:43  * notmattjoined
03:41:16  * nfitchjoined
03:45:21  * nfitchquit (Ping timeout: 240 seconds)
05:29:30  * nfitchjoined
05:33:45  * nfitchquit (Ping timeout: 245 seconds)
05:45:34  * AvianFluquit (Remote host closed the connection)
05:46:03  * AvianFlujoined
05:50:57  * AvianFluquit (Ping timeout: 272 seconds)
06:16:09  * mamashjoined
06:31:45  * yruss972joined
06:41:48  * yruss972quit (Remote host closed the connection)
06:42:23  * yruss972joined
06:46:17  * yruss972quit (Read error: No route to host)
07:34:39  * notmattquit (Remote host closed the connection)
09:06:18  * nfitchjoined
09:10:51  * nfitchquit (Ping timeout: 248 seconds)
10:16:13  * yruss972joined
10:33:42  * abraxas_quit (Remote host closed the connection)
10:34:16  * abraxasjoined
10:38:51  * abraxasquit (Ping timeout: 248 seconds)
10:54:41  * nfitchjoined
10:58:54  * nfitchquit (Ping timeout: 240 seconds)
11:35:20  * notmattjoined
11:39:39  * notmattquit (Ping timeout: 248 seconds)
12:42:57  * nfitchjoined
12:46:01  * AvianFlujoined
12:47:23  * nfitchquit (Ping timeout: 248 seconds)
13:25:15  * yruss972quit (Remote host closed the connection)
13:25:50  * yruss972joined
13:26:04  * AvianFluquit (Remote host closed the connection)
13:26:34  * AvianFlujoined
13:30:03  * yruss972quit (Ping timeout: 248 seconds)
13:31:07  * AvianFluquit (Ping timeout: 248 seconds)
13:56:48  * yruss972joined
13:57:11  * yruss972quit (Remote host closed the connection)
13:57:23  * yruss972joined
14:09:59  * mamashpart
14:31:12  * nfitchjoined
14:35:42  * nfitchquit (Ping timeout: 265 seconds)
14:54:59  * mamashjoined
15:03:02  * yruss972_joined
15:04:00  * yruss972quit (Read error: Connection reset by peer)
15:11:16  * ryancnelsonjoined
15:30:15  * chorrelljoined
15:30:47  * notmattjoined
15:31:04  * yruss972_quit (Remote host closed the connection)
15:46:20  * nfitchjoined
15:46:50  * mamashpart
15:49:44  * dap_joined
15:57:07  * mcavagejoined
15:57:47  * dap_quit (Ping timeout: 248 seconds)
16:00:56  * fredkjoined
16:01:26  <nahamu>mcavage: why doesn't mput do something like this:
16:02:01  <nahamu>(implemented in bash b/c I'm a sysadmin and I think in bash)
16:02:12  <nahamu>http://paste.ec/?d880a965a2ed0bdf#vnj/Xxa7sAquiuXk/ol/hiqTVL1dbipeC63dmCK9CFA=
16:02:50  * utlemmingjoined
16:02:50  <nahamu>stream the bits both into the upload, but also through an md5 calculation at the same time.
16:03:15  <mcavage>nahamu: mostly b/c that would require an extra round trip at the end.
16:03:31  * lloyddejoined
16:03:36  <mcavage>if you did the md5 calc, then sent -H content-md5:$md5, it gets enforced, so you don't have to check it.
16:04:13  <nahamu>right, and for small files that's totally the way to go.
16:04:21  <mcavage>yes, for small files.
16:04:43  <mcavage>well, really, it's going to depend on your throughput from your site what the "cutover point is"
16:04:55  <nahamu>right
16:04:55  <mcavage>where it's cheaper to do what your script does vs do it up front and save a wan trip to manta.
16:06:02  <mcavage>i actually would like to do this via trailers, except basically no http libraries correctly use trailers.
16:06:08  <nahamu>does mget do a streaming md5 checksum while it downloads?
16:06:12  <mcavage>yes
16:06:18  * lloyddequit (Remote host closed the connection)
16:06:18  <mcavage>well, let me check :)
16:06:29  <mcavage>I *think* so :)
16:07:00  * yunongjoined
16:07:18  <nahamu>just seems like if there's code for calculating the md5 while you're streaming the data, it could be pretty simple to add a feature to mput that allows you to request the extra round trip verification.
16:07:38  <nahamu>(which seemed to be what bixu was asking for)
16:07:47  <mcavage>yes it does; it's actually in the SDK, not the mget CLI
16:08:07  <mcavage>the crypto stream is easy -- that's just node built-in stuff.
16:08:11  <mcavage>it's really the extra RT at the end
16:08:24  <mcavage>but we could certainly add a flag for it or something
16:08:37  <mcavage>people that script large directory uploads on mput would get burned if this was always on
16:09:11  <nahamu>right, it should clearly default to the current behavio(u)r
16:09:29  <mcavage>can you open a GH issue? adding a thing at the end would be pretty easy.
16:10:00  <nahamu>I was hoping to mention this when bixu was around so that bixu could open the ticket... :-P
16:10:20  <mcavage>pro move ;)
16:10:48  <nahamu>but sure, I'll open one anyway. For me it's low priority. I just found the question interesting enough that I had to open a shell and poke around.
16:11:03  <mcavage>the other problem here though is that if it failed, you still have an object on the server you have to clean up.
16:11:20  <mcavage>like it's assuredly better to calculate the md5 and then send it so it doesnt' get saved.
16:11:26  <mcavage>that's probably what i'd actually make mput do
16:11:38  <nahamu>I don't follow.
16:11:48  <mcavage>is the equivalent of mput -H content-md5:$(cat file | openssl ...) -f file ...
16:12:01  * nfitchquit (Quit: Leaving.)
16:12:06  <mcavage>cat $1 | tee >(openssl md5 -binary | openssl enc -base64 > ${HASHFILE}) | mput ${MANTA_PATH}
16:12:07  <mcavage>diff <(mls -j ${MANTA_PATH} | json md5) <(cat ${HASHFILE}) || echo "MD5 Doesn't match"
16:12:07  <mcavage>rm $HASHFILE
16:12:14  <mcavage>that thing -- note that $MANTA_PATH
16:12:27  <mcavage>exists whether it's corrupt or not
16:12:52  <nahamu>right.
16:12:59  <mcavage>so really you need a if [] ; .. mrm ... fi
16:13:05  <nahamu>in theory it's preferable for there to be two different features:
16:13:26  <nahamu>1) precalculate md5 and set the header accordingly
16:13:43  <nahamu>2) calculate the md5 while streaming the data and do extra round-trips afterwards
16:14:18  <nahamu>because there's some cutoff somewhere where the size of the file being uploaded makes paying the price of reading it twice painful
16:14:26  <nahamu>(or at least I assume there is. I could be wrong)
16:14:51  <nahamu>also, for a streaming workload where we don't even have a complete file yet, it's nice to do verification on both ends.
16:14:52  <mcavage>btw: https://github.com/joyent/node-manta/issues/130
16:15:09  <mcavage>yes in theory both would be nice
16:15:19  <nahamu>nice.
16:15:22  <mcavage>and yes your way would be necessary for streaming uploads
16:15:53  * dap_joined
16:16:42  <nahamu>anyway, always nice chatting with you about such things. I should probably go do some real work, though. :)
16:16:57  <mcavage>haha -- i am a time sink ;)
16:26:33  * utlemmingquit (Ping timeout: 272 seconds)
16:26:38  * mamashjoined
16:44:08  * fennokin_joined
17:02:51  * nfitchjoined
17:10:59  * dap_quit (Quit: Leaving.)
17:40:24  * mcavagequit (Remote host closed the connection)
17:40:50  * mcavagejoined
17:41:08  * ryancnelsonquit (Quit: Leaving.)
17:45:31  * mcavagequit (Ping timeout: 248 seconds)
18:05:53  * mcavagejoined
18:06:05  * mcavagequit (Remote host closed the connection)
18:06:21  * mcavagejoined
18:16:55  * dap_joined
18:23:30  * utlemmingjoined
18:30:29  * utlemmingquit (Ping timeout: 248 seconds)
18:42:31  * notmattquit (Remote host closed the connection)
18:43:05  * bixujoined
18:43:40  * bixuquit (Client Quit)
18:44:37  * bixujoined
18:45:29  * bixuchanged nick to wanelo
19:02:48  * yruss972joined
19:59:21  <nahamu>wanelo: hashing while you upload: https://github.com/joyent/node-manta/issues/130#issuecomment-26750712
20:09:47  * ryancnelsonjoined
20:18:43  * yruss972quit (Remote host closed the connection)
20:19:17  * yruss972joined
20:19:37  * utlemmingjoined
20:23:26  * yruss972quit (Ping timeout: 240 seconds)
20:27:42  * daviddiasjoined
20:27:56  * mamashpart
20:36:10  * raboofquit (Ping timeout: 246 seconds)
20:38:00  * raboofjoined
20:43:00  * mamashjoined
20:54:14  * ins0mniajoined
21:14:11  * utlemmingquit (Ping timeout: 260 seconds)
21:19:28  * notmattjoined
21:29:56  * yruss972joined
21:34:13  * daviddiasquit (Remote host closed the connection)
21:34:15  * yruss972quit (Ping timeout: 260 seconds)
21:34:35  * mamashpart
21:34:42  * daviddiasjoined
21:39:02  * daviddiasquit (Ping timeout: 240 seconds)
22:11:51  * utlemmingjoined
22:17:28  * fennokin_quit (Remote host closed the connection)
22:17:31  * mattappersonjoined
22:58:47  * mattappersonquit (Remote host closed the connection)
22:58:54  * chorrellquit (Quit: My iMac has gone to sleep. ZZZzzz…)
22:59:12  * mattappersonjoined
23:03:50  * mattappersonquit (Ping timeout: 240 seconds)
23:08:34  * mcavagequit (Remote host closed the connection)
23:09:06  * mcavagejoined
23:11:01  * mcavagequit (Remote host closed the connection)
23:11:17  * mcavagejoined
23:23:20  * utlemmingquit (Read error: Connection reset by peer)
23:23:28  * utlemmingjoined
23:24:50  * utlemmingquit (Client Quit)
23:28:35  * fennokin_joined
23:32:38  * fennokin_quit (Ping timeout: 240 seconds)
23:34:12  * mattappersonjoined
23:37:14  * irajoined