00:00:12  * chorrellquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
00:02:53  * nfitchquit (Quit: Leaving.)
00:03:39  * yunongquit (Quit: Leaving.)
00:06:33  * trentmquit (Quit: Leaving.)
00:31:05  * dapquit (Quit: Leaving.)
00:42:09  * yunongjoined
01:01:40  * bixuquit (Remote host closed the connection)
01:06:03  * ryancnelsonquit (Quit: Leaving.)
01:19:46  * abraxasjoined
01:20:59  * chorrelljoined
01:42:10  * abraxasquit (Remote host closed the connection)
01:42:26  * abraxasjoined
01:51:38  * yunongquit (Ping timeout: 256 seconds)
02:55:19  * chorrellquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
03:33:01  * trentmjoined
05:12:17  * ghostbarquit (Remote host closed the connection)
05:12:47  * ghostbarjoined
05:17:20  * ghostbarquit (Ping timeout: 256 seconds)
05:42:18  * ghostbarjoined
06:01:09  * trentmquit (Quit: Leaving.)
06:16:15  * trentmjoined
06:19:23  * mamashjoined
06:23:04  * trentmquit (Quit: Leaving.)
06:48:19  * marsellquit (Quit: marsell)
07:02:59  * ghostbarquit (Remote host closed the connection)
07:03:29  * ghostbarjoined
07:08:19  * ghostbarquit (Ping timeout: 264 seconds)
09:21:03  * marselljoined
10:13:16  * abraxasquit (Remote host closed the connection)
12:31:30  * hkgranijoined
12:39:25  * hkgraniquit (Quit: Page closed)
13:04:24  * elijah-mbpquit (Read error: Connection reset by peer)
13:04:57  * elijah-mbpjoined
13:14:33  * elijah-mbpquit (Ping timeout: 264 seconds)
13:31:44  * chorrelljoined
13:46:15  * chorrellquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
14:06:48  * elijah-mbpjoined
14:21:34  * chorrelljoined
15:29:07  * ryancnelsonjoined
15:37:01  * mamashpart
15:44:20  * ryancnelsonquit (Quit: Leaving.)
15:57:02  * nfitchjoined
16:02:42  * yunongjoined
16:05:40  * fredkjoined
16:24:09  * yunongquit (Ping timeout: 264 seconds)
16:24:21  * dapjoined
16:32:18  * trentmjoined
16:40:29  * mcavagejoined
17:29:24  * mamashjoined
17:34:19  * aolsonjoined
17:51:31  * yunongjoined
17:52:38  * ghostbarjoined
18:01:10  * trevorojoined
18:04:17  <trevoro>Hey stupid manta question. It's possible to serve up manta "files" over HTTP publicly from the object store, right?
18:05:44  <e^ipi>yep, that's where smartos builds are being served from
18:05:48  <nfitch>Yes. mput /$MANTA_USER/public/...
18:05:56  <mcavage>trevoro! yes. just use the /:you/public space. "real" access control is top of the list on the feature list.
18:06:16  <nfitch>Then you can get with something like https://us-east.manta.joyent.com/$MANTA_USER/public/...
18:06:20  <trevoro>Ok
18:06:28  <trevoro>Its just... not really clear. Do you accept doc pull requests?
18:06:39  <mcavage>they're getting moved out to github "soon"
18:06:43  <mcavage>you're not the first to ask.
18:06:48  <trevoro>Awesomesauce.
18:06:50  <trevoro>TYVM!
18:24:47  * trevoroquit (Quit: leaving)
18:46:58  * yunong1joined
18:49:35  * yunongquit (Ping timeout: 245 seconds)
18:56:05  * bixujoined
18:58:26  <bixu>When I see "Server did not Upgrade" when running mlogin, what does that mean?
19:06:19  <mcavage>bixu: mlogin works by starting a websocket, and has a dedicated set of internal components to proxy "shell sessions back to you." We're doing some rolling upgrades this morning, so you probably just got unlucky, and one of those components was temporarily unavailable. Obviously the error message sucks.
19:10:16  <bixu>That's helpful.
19:10:33  <bixu>And the issue seems to be transient (and gone now) as you suggested.
19:11:06  <mcavage>yeah, makes sense - we're rolling out new OS upgrades to the fleet, which requires (at some point) everything upstack to restart. there are going to be brief "blips" when that happens.
19:11:12  <bixu>I'm still getting used to a cloud provider telling me what's really going on (cut my teeth on AWS, where 'abstraction' is valued over transparency).
19:11:50  <mcavage>ha! well, fwiw, 1/2 of us used to work at AWS, and I think we all very much believe in transparency ;)
19:11:51  <bixu>For a regular manta job, would manta just retry automatically?
19:12:17  <mcavage>yeah- if you do mjob get $jobid -- there's a "stats" section, where we tell you how many "tasks" got retried.
19:12:29  <bixu>Understood. I learned a lot from AWS (why the right abstractions are great, and why others suck).
19:12:29  <mcavage>certainly on "internal error" we just automatically retry for you.
19:12:40  <bixu>Nice.
19:14:03  <bixu>I've been using mlogin to explore the kinds of tools that are pre-bundled into manta jobs/zones. I'm guessing I can also store my own binary as an object and use it for jobs?
19:15:35  <mcavage>bixu: correct, that's "assets" in the API/CLI. Using that with "--init" you can bring in your own binaries/tarballs/etc and get them setup before "exec" is started.
19:17:08  <bixu>Maybe I should migrate my S3 images to Manta (we have thousands) using a distributed rake task...
19:17:54  <mcavage>is there such a thing?
19:18:02  <mcavage>or you mean in jobs?
19:18:07  <bixu>In jobs.
19:18:44  <mcavage>sure - the one caveat is that the manta compute zones are NAT'd, so the b/w is going to cap a lot faster than if you just ran a "pump" from a "normal" us-east machine.
19:18:56  <bixu>I see.
19:19:00  <mcavage>that said, it obviously depends on how much and how big your images are, so certainly try it ;)
19:22:07  <bixu>That reminds me...are there plans to extend Manta across more than one region? Or does that interfere with the strong consistency you are promising?
19:22:30  <mcavage>there are plans -- it would not as each region would be "standalone".
19:22:47  <mcavage>i.e., us-east manta would have a disjoint dataset from us-sw.
19:23:46  <bixu>I see.
19:24:10  <bixu>Would it be possible for me to specify that an object be multi-region at the expense of consistency?
19:24:54  <bixu>Not sure if this is something I need yet, but I can imagine wanting it for certain types of storage (especially stuff that might be needed for DR).
19:25:10  <mcavage>somebody else asked for that -- no such feature exists now (obviously we don't even have > 1 region right now). But if we did that it would be clearly built "on top" of manta.
19:25:29  <mcavage>i.e., the semantics would be a lot weaker (master/slave, high latency)
19:26:07  <bixu>That makes sense. Or leave it up to the client to mput to multiple regions...
19:27:39  <mcavage>pretty much
19:27:55  <mcavage>there's no magic when you start talking about x-region and the WAN :)
19:28:44  <bixu>mcavage: Stop dismantling the illusion!
19:29:40  <mcavage>sorry ;)
19:30:03  <bixu>I don't want to know how the internet works. I just want to write rails apps.
19:30:15  <mcavage>it's for the best. it barely works anyway.
19:31:29  <bixu>You mean Rails? Haha.
19:31:40  <mcavage>ha. actually both ;)
19:32:17  <nahamu>don't forget, IPv4 is going to run out any day now... ;)
19:32:40  <mcavage>and that
19:32:44  <mcavage>is why it's NAT'd ;)
19:33:58  <bixu>But.
19:34:03  <bixu>IPV6 Day
19:35:38  <nahamu>yeah... I'm still waiting for my ISP to give me IPv6
19:35:56  <nahamu>I suppose it's getting towards time to look into it for $work.
19:41:59  <nahamu>is it possible to run mjob on a signed URL belonging to someone else?
19:51:35  * mamashpart
20:08:14  <mcavage>nahamu: on a signed url? no.
20:08:19  <mcavage>you can access other people
20:08:22  <mcavage>s /public
20:16:41  <nahamu>right, so thoth sharing has to be with snaplinks.
20:29:53  <mcavage>yeah, for now.
21:30:45  * mikealjoined
21:31:56  <mikeal>so, i want to create an image magick job
21:31:59  <mikeal>convert image.png -resize 50000\> -strip -compress Zip -quality 40 output.png
21:32:00  <mikeal>basically
21:32:08  <mikeal>i see how to get the input filename
21:32:24  <mikeal>but how to construct an output filename that is available in /public
21:36:07  * chorrellquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
21:41:13  <mcavage>mikeal: you can name files with mpipe, or use mput.
21:41:35  <mcavage>do you have an example imagemagick command you want to run? (I can whip you up something real quick)
21:41:47  <mcavage>i just never remember the imagemagick syntax :)
21:45:07  <mikeal>$ convert image.png -resize 50000\> -strip -compress Zip -quality 40 output.png
21:45:08  <mikeal>:)
21:46:57  <mcavage>oh, so that was it ;)
21:47:10  <mcavage>so you just want output.png to go where?
21:47:26  <mcavage>like /:you/public/same/path/that/input/had.png?
21:47:33  <mcavage>or just /:you/public/foo.png?
21:52:36  <mikeal>basically /public/$OBJECT_NAME
21:52:39  <mikeal>well
21:52:40  <mikeal>now
21:53:13  <mikeal> /public/$OBJECT_NAME + '-500'
21:53:50  <mcavage>k
21:56:54  <mcavage>mikeal:
21:56:55  <mcavage>mfind -t o /$MANTA_USER/stor/images | mjob create -qo 'convert $MANTA_INPUT_FILE -resize 50000\> -strip -compress Zip -quality 40 /tmp/$(basename $MANTA_INPUT_OBJECT) && mmkdir -p $(dirname $(echo $MANTA_INPUT_OBJECT | sed "s|/stor/|/public/|")) && mput -f /tmp/$(basename $MANTA_INPUT_OBJECT) $(echo $MANTA_INPUT_OBJECT | sed "s|/stor/|/public/|") && echo $MANTA_INPUT_OBJECT | sed "s|/stor/|/public/|"'
21:57:04  <mcavage>that gives you /:you/public/%inputname
21:57:26  <mcavage>if you want the -500 1s let me tack an echo in the right place.
21:59:42  <mcavage>mikeal:
21:59:45  <mcavage>$ mfind -t o /$MANTA_USER/stor/images | mjob create -qo 'convert $MANTA_INPUT_FILE -resize 50000\> -strip -compress Zip -quality 40 /tmp/$(basename $MANTA_INPUT_OBJECT) && mmkdir -p $(dirname $(echo $MANTA_INPUT_OBJECT | sed "s|/stor/|/public/|")) && mput -f /tmp/$(basename $MANTA_INPUT_OBJECT) $(echo $MANTA_INPUT_OBJECT | sed "s|/stor/|/public/|" | sed "s|\.png$|-500\.png|") && echo $MANTA_INPUT_OBJECT | sed "s|/stor/|/p
21:59:45  <mcavage>ublic/|" | sed "s|\.png$|-500\.png|"' | xargs mls -l
21:59:45  <mcavage>-rwxr-xr-x 1 mark.cavage 99361 Jul 31 14:59 mushroom-500.png
21:59:46  <mcavage>Marks-MacBook-Pro:~ mcavage$
22:01:37  <mcavage>i don't think that "convert" command did the right thing, but in my case, i took this https://us-east.manta.joyent.com/mark.cavage/stor/images/nintendo/mushroom.png?algorithm=RSA-SHA1&expires=1375311604&keyId=%2Fmark.cavage%2Fkeys%2F7b%3Ac0%3A5c%3Ad6%3A9e%3A11%3A0c%3A76%3A04%3A4b%3A03%3Ac9%3A11%3Af2%3A72%3A7f&signature=dmJK2xoHGnaXW8GmWx6msF2dloIQ6ZaNIjZ%2FVy4q3CvFSO62k4nAwyh4M60zGOJts8wdpAoe1P8LEXgUdGkASV42uI1NFQQ4HtfujM
22:01:37  <mcavage>Rm9p3%2B3lTlOiDY6yW%2F8fNsAcQGJeR0RdqZVDkCsr9ce2mSR7PQwfI8%2FAWeBACPsEiA35unO%2FGAZnAt8mIqxPs9OB7jipSLsqGV9jnZ8g%2BV5r3CO%2BAF8Ecs3Hkbb8POrCNE7HgjRAnaxMQVzUi%2FW76%2F0RTWOduqSBBiAVkvYbRiXrDWICS112yaKA95JSkKn1ezsRwqmKYm98MFdm319RuolKBVXFrgmzgD4GRkWAFFqITLUw%3D%3D and made this https://us-east.manta.joyent.com/mark.cavage/public/images/nintendo/mushroom-500.png
22:02:22  <trentm>signed URL IRC paste #fail
22:02:22  <mcavage>anyway - the gist is, you want to just run mmkdir -p and then mput by hand. the "echo" so it shows up in job outputs is optional, but i find that helpful, so you can figure out what wrote what down the road.
22:02:36  <mcavage>trentm: ugh
22:02:53  <mcavage>https://us-east.manta.joyent.com/mark.cavage/stor/images/nintendo/mushroom.png?algorithm=RSA-SHA1&expires=1375311604&keyId=%2Fmark.cavage%2Fkeys%2F7b%3Ac0%3A5c%3Ad6%3A9e%3A11%3A0c%3A76%3A04%3A4b%3A03%3Ac9%3A11%3Af2%3A72%3A7f&signature=dmJK2xoHGnaXW8GmWx6msF2dloIQ6ZaNIjZ%2FVy4q3CvFSO62k4nAwyh4M60zGOJts8wdpAoe1P8LEXgUdGkASV42uI1NFQQ4HtfujMRm9p3%2B3lTlOiDY6yW%2F8fNsAcQGJeR0RdqZVDkCsr9ce2mSR7PQwfI8%2FAWeBACPsEiA35unO%2FGAZnAt
22:02:53  <mcavage>8mIqxPs9OB7jipSLsqGV9jnZ8g%2BV5r3CO%2BAF8Ecs3Hkbb8POrCNE7HgjRAnaxMQVzUi%2FW76%2F0RTWOduqSBBiAVkvYbRiXrDWICS112yaKA95JSkKn1ezsRwqmKYm98MFdm319RuolKBVXFrgmzgD4GRkWAFFqITLUw%3D%3D
22:02:55  <mcavage>take 2?
22:03:04  <mcavage>oh, no my IRC client is fucking it up.
22:03:06  <mcavage>anyway.
22:03:09  <mcavage>you get the idea, i hope
22:05:25  <mikeal>yeah
22:05:25  <trentm>msign needs a "shorten the url" option
22:05:27  <mikeal>i think
22:07:39  <mikeal>oh
22:07:46  <mikeal>hrm
22:07:53  <mikeal>i need to rethink how i did something
22:07:56  <mcavage>trentm: ok, well, when you build a joyent URL service ;)
22:08:09  <trentm>meh, pluggable for existing ones
22:08:10  <mikeal>all of my images are in public
22:08:26  <trentm>bryan, e.g. wants to get hit counts on us clicking his disaster pr0n
22:08:46  <mikeal>how can i quickly copy all the objects in public to another directory in stor?
22:08:52  <mcavage>so build a bit.ly CLI. msign | xargs bitly
22:08:56  <mcavage>mikeal: mln
22:09:13  <mcavage>mln will let you make new names without moving the data.
22:09:34  <mcavage>you'll have to do a little goober of mfind | parallel/xargs mln sed.
22:11:44  * elijah-mbpquit (Remote host closed the connection)
22:16:31  * mikealquit (Quit: Leaving.)
22:16:53  * mikealjoined
22:23:38  * elijah-mbpjoined
22:26:24  <trentm>mcavage:
22:26:25  <trentm>$ msign /trentm/stor/tmp/man-of-blue-steel.jpg | googl
22:26:25  <trentm>http://goo.gl/9sXjTt
22:26:39  <mcavage>super ;)
22:26:48  <mcavage>googl is a new trentism?
22:27:35  <trentm>https://gist.github.com/trentm/6126755
22:27:49  <trentm>no... I just found a curl for goo.gl before I did for bit.ly
22:31:39  <mikeal>mcavage: how do i avoid running this for objects it has already run on when i add new objects and want the task to run against the new ones?
22:32:10  <mcavage>mikeal: you'd have to do bookkeeping yourself for that.
22:32:17  <mikeal>ok
22:33:53  <fredk>could you mls and check exit code?
22:36:21  <mcavage>fred: not really - that would only tell you things that aren't there. it seems to me the only sane thing is just to maintain a queue or something of "new" files.
22:36:40  <mcavage>mikeal: the other "hot feature" is "triggers" which would do what you want. they just don't exist yet.
22:36:49  <mikeal>i see
23:13:16  <nahamu>within manta, is mln kind of like a hard link; adding one increases a refcount and then after you mrm all the links the underlying object is clear for deletion?
23:13:33  <nahamu>(or after you mln a name to a new object)
23:28:04  <mcavage>nahamu: yeah, kind of. It's like a hard link if hard links were copy on write.
23:41:36  * mikealquit (Quit: Leaving.)
23:55:49  <nahamu>in theory, could joyent run a service for requesting a signed URL for PUT for thoth? e.g. customer runs SDC, something dumps core, SDC sends a request to SDC-thoth-service for a signed URL that dumps into /thoth/stor/thoth/whatever, the service returns the URL, customer SDC uploads the dump, analyzers analyze it, etc.
23:57:01  <nahamu>holy crap, when I was kicking the tires on SDC6.5 support used to have me run a command that would collect a bunch of stuff and upload it somewhere... that should totally be rewritten to use manta...
23:59:23  <nahamu>also, logrotate plugins that upload to manta.
23:59:30  <nahamu>I think I'm done for now.