00:04:44  <dap>mjn: Sorry for disappearing there. mpipe always suppresses output, not just when it's last.
00:05:38  <mjn>dap: but i thought if you do an mpipe in phase 1 of a 2-phase job it gets passed on tot he next phase? or am i misunderstanding?
00:06:00  <mjn>or maybe i'm reading what 'output' means wrong
00:06:11  <nahamu>mjn: I think mpipe writes to a manta object, not the next phase.
00:06:12  <dap>mpipe always causes that output to be passed to the next phase, and it always causes the stdout of whatever task it's running in to be suppressed.
00:06:22  <nahamu>nevermind, I was totally wrong.
00:06:34  <dap>nahamu: Yeah, it's explicitly not a Manta object unless you name it :)
00:06:34  * bixuquit (Remote host closed the connection)
00:06:46  <mjn>ah ok, that makes sense that, i was thinking of 'output' as in the default of stdout of a phase 1 goes to a phase 2, if you don't mpipe it and it's a 1-to-1 relationship
00:06:59  <mjn>the comment in the mpipe manpage that it suppresses output made me worry it suppressed that part too
00:07:42  <dap>It does seem like some elements may be overloaded here, but I'm not sure what should change. We could separate "pipe this stream to the next phase" from "name this output", but of course, you might want to do both.
00:08:04  <dap>But we could also provide special-case aliases that are clearer
00:08:44  <mjn>yeah, i'm not even sure the current behavior is wrong, just wrapping my head around the difference between piping and piping-between-phases and naming-an-object
00:09:20  <dap>That's fair. If you think of something that could be clearer, do let us know :)
00:09:52  <mjn>the default if you do nothing is that phases get piped and *not* named if they're not last, i believe; while mpipe in a non-final phase stops the local (within-phase) pipe *but* does pipe it to the next phase while also giving an opportunity to name it (mtee-like, but only at phase boundaries)
00:09:58  <mjn>and mpipe in the final phase is more like >
00:10:50  <dap>That sounds right.
00:11:42  <dap>The two main use cases, really, are emitting multiple outputs from a given task (e.g., chunking up an input in some custom way for parallel processing in the next phase), and naming an output object (so you don't have to do the mln/mrm crap)
00:12:05  <mjn>yeah that makes sense, those are my two use-cases also
00:12:10  <mjn>but i was surprised that mpipe is both of them ;-)
00:12:30  <mjn>i don't see anything in that that i can't work with, just took a bit to arrive it... mainly the slightly intertwined semantics of when things get piped on (within a phase and between phases) and when they get saved
00:13:49  <mjn>i think i'd feel more comfortable with something like msave (kills any piping on in any case, saves the result to an object) vs. mpipe (pipes this onwards transiently, never saves to an object)
00:14:10  <mjn>but it's not a strong preference
00:15:45  <mjn>oh and mtee as the 3rd in that set, "both pipes this on and saves it"
00:23:57  <dap>Yeah.
00:28:57  * dapquit (Quit: Leaving.)
00:58:10  * asongequit (Read error: Connection reset by peer)
01:48:35  * notmattquit (Read error: Connection reset by peer)
01:49:05  * notmattjoined
01:53:20  * yunongquit (Quit: Leaving.)
01:54:10  * fredkquit (Quit: Leaving.)
02:00:26  * ghostbarjoined
02:11:13  * ghostbarquit (Remote host closed the connection)
02:11:41  * ghostbarjoined
02:15:50  * ghostbarquit (Ping timeout: 240 seconds)
02:32:57  * mcavagequit (Remote host closed the connection)
02:36:22  * bcantrillquit (Quit: Leaving.)
03:07:44  * yunongjoined
03:09:11  * _Tenchi_quit (Quit: Leaving)
03:10:59  * yunongquit (Client Quit)
03:43:26  * mcavagejoined
03:51:16  * mcavagequit (Ping timeout: 268 seconds)
04:04:06  * ghostbar_joined
05:03:59  * ghostbar_quit (Remote host closed the connection)
05:04:31  * ghostbarjoined
05:09:35  * ghostbarquit (Ping timeout: 268 seconds)
06:18:02  * marsellquit (Quit: marsell)
07:35:50  * mamashjoined
08:01:43  * mamashpart
08:06:49  * mamashjoined
08:06:57  * mamashpart
08:44:19  * notmattquit (Remote host closed the connection)
09:03:33  * almostobsoletejoined
09:14:53  * notmattjoined
09:23:17  * notmattquit (Ping timeout: 248 seconds)
11:40:12  * bcantrilljoined
11:57:28  * cburroughsjoined
12:36:27  * mamashjoined
13:08:23  <nahamu>I think mlogin in current master of node-manta is busted.
13:08:58  <nahamu>I'd poke mcavage but he's probably asleep, so I'll just poke bcantrill instead.
13:11:36  <nahamu>http://paste.ec/?e564e50d9870a11f#kWoC9tsbFqE1kh3DxF2tKo6irJxlGgpGKTMaOU6biLE=
13:19:20  * AvianFluquit (Remote host closed the connection)
13:21:00  * AvianFlujoined
13:49:14  * AvianFluquit (Remote host closed the connection)
13:49:42  * AvianFlujoined
14:51:13  * mamashpart
14:54:38  * CarlosCjoined
14:56:03  * mcavagejoined
15:01:46  <nahamu>mcavage: is it just me or is mlogin broken in the current master of node-manta?
15:01:49  <nahamu>http://paste.ec/?e564e50d9870a11f#kWoC9tsbFqE1kh3DxF2tKo6irJxlGgpGKTMaOU6biLE=
15:03:56  * AvianFluquit (Remote host closed the connection)
15:05:12  * AvianFlujoined
15:19:41  <mcavage>nahamu: let me check :)
15:20:34  <nahamu>(I was so excited for ~~ syntax that I pulled down the latest, but then mlogin seemed broken and I was sad.)
15:21:09  <mcavage>hahahaha
15:21:35  <mcavage>1m - i can see this too. let me check the npm one first to make sure this isn't server side.
15:22:13  <mcavage>side point: I'm glad somebody else wanted ~~ ;)
15:22:31  <mcavage>that was my selfish feature (I was so tired of typing /$MANTA_LOGIN
15:22:43  <mcavage>ok, works from npm so i must have broken this yesterday.
15:23:48  * dapjoined
15:32:21  <nahamu>yeah, my working version is the one from npm.
15:33:04  <mcavage>nahamu: i'm tracing and figuring out what broke.
15:33:16  <mcavage>(it's not obvious from the diffs what's making this grumpy)
15:34:00  <nahamu>I was writing a small helper function for using mlogin to launch mdb because there's obvious duplication in the example in the docs and I hate typing something twice when I could only type it once.
15:36:07  <nahamu>interesting random thought. for all objects where copies>1, Ops can probably do a rolling upgrade of the platform one machine at a time in a cluster without availability problems.
15:38:42  <mcavage>we do this without telling you, actually ;)
15:38:50  <mcavage>it's already happened at least 3x since GA.
15:39:28  <nahamu>I figured.
15:39:41  <mcavage>i think i see what broke in mlogin, give me a few more minutes; cooking a fix.
15:39:52  <nahamu>no rush for me, but I figured you'd want to know.
15:40:32  <mcavage>i do :)
15:51:15  <mcavage>nahamu: https://github.com/joyent/node-manta/issues/94
15:51:29  <mcavage>nahmu: git pull, it should work now.
16:00:31  <nahamu>mcavage: confirmed
16:01:41  <mcavage>great!
16:03:51  * almostobsoletequit (Ping timeout: 260 seconds)
16:04:22  * nfitch1joined
16:08:04  * AvianFluquit (Remote host closed the connection)
16:09:31  * fredkjoined
16:09:50  * fredkquit (Read error: Connection reset by peer)
16:09:56  * fredk1joined
16:16:37  * yunongjoined
16:24:23  * bcantrillquit (Ping timeout: 260 seconds)
16:25:34  <nahamu>are there example tools for using manta to go through manta reporting?
16:25:54  <mcavage>that's on the short list (like an estimate my bill)
16:28:55  <nahamu>in theory an estimate my bill tool should be similar to the tool you folks end up using for actual billing, no?
16:29:09  * bcantrilljoined
16:30:08  <nahamu>hell, there's probably a business to be built around building a billing tool for companies for whom billing takes too long...
16:32:01  <mcavage>nahumu: yeah, we've already got a thing -- just one of us needs to make it polished and "runnable as you". It's on the super short list :)
16:32:11  <nahamu>fair enough. :)
16:32:32  <nahamu>I suppose if I stop bugging you on IRC you'll have more time for such things. ;)
16:37:53  <mcavage>haha - no worries ;) -- it's just scrum time on our internal IRC ;)
16:38:36  * notmattjoined
16:47:48  <mjn>if you don't mind pulling the files locally (they're pretty tiny) and have maggr or some workalike installed locally, this works for me for compute time: mfind -type o /:login/reports/usage/summary/2013/07|xargs mget|json -g -a computeGBSeconds|maggr sum|awk '{n=$1*0.00004; print "$"n}'
16:50:43  <mcavage>yeah, I was going to ship a sample script that runs a job for you to do that and price out the storage|bw|compute.
16:51:38  <mjn>it could be convenient to ship maggr with the package that installs mjob/mget/mput btw, it's sort of useful to have locally to test out scripts
16:51:50  <mjn>it looks like it's in the same package as mpipe/etc. but unlike them it makes sense to run on e.g. my osx laptop
16:56:56  <mcavage>and here you convinced me we should just push people to crush tools ;)
16:57:30  <mjn>hah well, they do have more stuff
16:57:32  <nahamu>is it just me or do the access logs only show remoteAddress that's an IPv6 address that's probably something internal.
16:57:42  <mcavage>btw: if you want to try them out, use this as an asset: /mark.cavage/public/crush-tools-2013-04.tar.gz
16:57:46  <mcavage>I built them the other day.
16:57:49  <mjn>excellent
16:57:49  <rmustacc>nahamu: That's an ipv4 address encoded as an IPv6 address.
16:57:57  <mcavage>nahamu: there's a bug for that.
16:58:23  <rmustacc>Basically there's a syntax in ipv6 that says this is really an old ipv4 addr.
16:58:35  <mcavage>yeah there's two things:
16:58:35  <nahamu>well, at least I'm poking around in the interesting bits.
16:58:46  <mjn>is the right way to use tools from an asset like that to untar them to /var/tmp?
16:59:09  <mcavage>(1) i think we want to just ship this in ipv6 to be future proof, and as rmustacc says, they're using the tunneled form
16:59:40  <mcavage>(2) we probably should just translate all internal addresses to the same address (like 169.254 or something) so you can reliably mark that "internal"
17:00:13  <mcavage>mjn: in this case, b/c I built it to be this way, just untar in / - they'll end up in /opt/local/crushtools and the libpath crap is all set for that.
17:00:37  <mjn>ah ok cool
17:01:06  <mcavage>we're going to get them in pkgsrc and then it will just be in the manta image at some point in $PATH (really /opt/local/bin)
17:01:14  <mcavage>i just wanted a sharable tarball
17:01:18  * yunongquit (Quit: Leaving.)
17:04:16  <fredk1>nahamu: if we don't have a remote address for a request, we stick in 169.254 just so the field is always there
17:04:31  <fredk1>and yeah, all addresses are converted to ipv6
17:05:12  <nahamu>I didn't recognize the ::ffff:<stuff> format as converted ipv6.
17:05:24  <nahamu>*converted ipv4
17:06:04  <fredk1>yeah, the library always uses a compact form
17:41:18  * bcantrill1joined
17:41:48  * bcantrillquit (Read error: Connection reset by peer)
17:46:42  * bcantrilljoined
17:48:47  * bcantrill1quit (Read error: Connection reset by peer)
17:53:49  * yunongjoined
18:20:50  * bixu_joined
18:28:55  * bixu_changed nick to bixu
18:35:49  * notmattquit (Remote host closed the connection)
18:42:21  * cburroughsquit (Quit: Leaving)
18:42:34  * cburroughsjoined
18:49:16  * bcantrillquit (Ping timeout: 240 seconds)
18:52:04  * mamashjoined
19:22:40  * notmattjoined
19:33:20  * mamashpart
19:43:11  * bcantrilljoined
19:44:11  * bcantrill1joined
19:48:25  * bcantrillquit (Ping timeout: 276 seconds)
20:10:04  * bcantrilljoined
20:10:39  * bcantrill1quit (Read error: Connection reset by peer)
20:36:44  * ghostbarjoined
20:47:37  * notmattquit (Remote host closed the connection)
20:48:43  * ghostbar_joined
20:50:22  * ghostbar_quit (Read error: Connection reset by peer)
20:51:37  * ghostba__joined
20:52:15  * ghostba__changed nick to ghostbar__
20:52:28  * ghostbarquit (Ping timeout: 268 seconds)
20:52:36  * ghostbar__changed nick to ghostbar
21:04:53  * ghostbarquit (Remote host closed the connection)
21:05:22  * ghostbarjoined
21:05:44  * bcantrillquit (Ping timeout: 256 seconds)
21:09:43  * ghostbarquit (Ping timeout: 246 seconds)
21:10:09  * bcantrilljoined
21:29:44  * notmattjoined
22:08:31  * bcantrillquit (Read error: Connection reset by peer)
22:08:36  * bcantrill1joined
22:12:53  * bcantrill1quit (Ping timeout: 240 seconds)
22:44:10  * AvianFlujoined
23:16:09  * ghostbarjoined
23:27:31  * cburroughsquit (Ping timeout: 240 seconds)
23:29:22  * nfitch1quit (Quit: Leaving.)