00:01:46  * sintaxiquit (Quit: sintaxi)
00:04:59  * ghostbarquit (Ping timeout: 252 seconds)
00:16:24  * AvianFlujoined
00:16:58  * sintaxijoined
00:22:09  * mikealquit (Quit: Leaving.)
00:44:11  * paulfryzeljoined
01:05:04  * mikealjoined
01:05:19  <isaacs>nfitch: Manta's responding, at least for my cursory curl tess.
01:05:26  <isaacs>nfitch: shall i switch the npm registry back over to it
01:05:27  <isaacs>?
01:05:33  <isaacs>nfitch: or do you recommend waiting further?
01:06:02  <nfitch>isaacs: dap_ is writing up details. Could you hold off for a few minutes?
01:08:39  <isaacs>nfitch: will do
01:10:08  * mikealpart
01:11:58  <dap_>isaacs: I think it's probably fine, actually
01:15:17  * fredkquit (Quit: Leaving.)
01:16:38  <isaacs>dap_: k, i'll throw some traffic at it, and let you knwo what i see
01:19:54  <isaacs>not seeing any errors
01:23:54  <isaacs>well, no errors from GET requests anyway
01:23:57  <isaacs>PUTs still failing\
01:23:59  <isaacs>dap_: https://gist.github.com/isaacs/8448150
01:28:47  <isaacs>dap_: on the plus side, it's failing slow enough to not put the service into maintance mode, so it'll come back whenever things are going well
01:29:25  <nfitch>isaacs: What's a rought rate of failure for you? Also, are the puts to many different directories or to the same directory?
01:29:29  <nfitch>*rough
01:32:03  <dap_>isaacs: when was the last one of those you saw?
01:33:45  <isaacs>dap_: about 30 seconds ago?
01:33:59  <isaacs>dap_: wait a second... why's it parsing a JSON response from manta? that stack looks weird.
01:34:04  <isaacs>hold on, this might be a false alarm.
01:38:34  <isaacs>ok, I just need to use soething other than nginx, it looks like.
01:38:39  <isaacs>yeah, this is not failing on manta's side.
01:38:53  <dap_>phew!
01:41:57  <isaacs>nginx: Terrible TLS Terminator.
01:41:59  <isaacs>just FYI
01:42:05  <isaacs>don't use nginx as a HTTP proxy. it sucks for that.
01:42:28  <isaacs>everytime i use nginx for anything other than a file server, i'm reminded of what a bad idea that is.
01:42:35  <isaacs>likewise anytime i use something other thna nginx to serve files.
01:44:16  <isaacs>so, tomorrow I'll swap that piece out
01:46:03  <dap_>Sorry everyone for the downtime. The details: we had a legitimate user job that hit a pathological case in the API, which in turn hit a pathological case in the indexing tier. Since it was done massively in parallel, it ended up effectively DOS'ing our metadata tier. The fixes are well understood. Sorry again for the issue.
01:58:42  * chorrelljoined
01:59:38  * chorrellquit (Client Quit)
02:04:47  * nfitchquit (Quit: Leaving.)
02:07:19  * paulfryzelquit (Read error: Connection reset by peer)
02:07:57  * paulfryzeljoined
02:22:44  * therealkoopaquit (Remote host closed the connection)
02:28:16  * paulfryzelquit (Remote host closed the connection)
02:28:52  * paulfryzeljoined
02:29:40  * sintaxiquit (Quit: sintaxi)
02:29:48  * therealkoopajoined
02:33:30  * paulfryzelquit (Ping timeout: 276 seconds)
02:58:38  * ghostbarjoined
03:02:56  * sintaxijoined
03:25:04  * sintaxiquit (Quit: sintaxi)
03:29:30  * paulfryzeljoined
03:33:24  * therealkoopaquit (Remote host closed the connection)
03:34:12  * paulfryzelquit (Ping timeout: 252 seconds)
03:35:55  * therealkoopajoined
03:40:23  * therealkoopaquit (Ping timeout: 260 seconds)
04:00:49  * sintaxijoined
04:19:54  * paulfryzeljoined
04:24:15  * paulfryzelquit (Ping timeout: 252 seconds)
04:34:38  * ghostbar_joined
04:36:05  * ghostbarquit (Ping timeout: 252 seconds)
04:38:41  * ghostbarjoined
04:38:50  * ghostbar_quit (Ping timeout: 252 seconds)
05:20:42  * paulfryzeljoined
05:22:09  * paulfryzelquit (Read error: Connection reset by peer)
05:22:22  * paulfryzeljoined
05:26:57  * paulfryzelquit (Ping timeout: 252 seconds)
05:51:59  * sintaxiquit (Quit: sintaxi)
05:55:25  * AvianFluquit (Remote host closed the connection)
06:23:59  * Azbruhquit (Remote host closed the connection)
06:24:35  * Azbruhjoined
06:34:43  * marsellquit (Quit: marsell)
06:35:03  * ghostbarquit
06:36:23  * ghostbarjoined
06:41:39  * ghostbarquit (Ping timeout: 272 seconds)
06:45:54  * ghostbarjoined
07:02:11  * sintaxijoined
07:04:37  * ghostbarquit (Remote host closed the connection)
07:48:09  * therealkoopajoined
07:50:06  * therealkoopaquit (Remote host closed the connection)
08:24:53  * paulfryzeljoined
08:29:33  * paulfryzelquit (Ping timeout: 252 seconds)
08:30:39  * mamashjoined
09:24:37  * mamashpart
10:02:20  * daviddiasjoined
10:15:46  * mamashjoined
10:26:14  * paulfryzeljoined
10:30:33  * paulfryzelquit (Ping timeout: 252 seconds)
11:02:02  * marselljoined
11:11:27  * irajoined
11:27:04  * paulfryzeljoined
11:31:51  * paulfryzelquit (Ping timeout: 265 seconds)
12:39:32  * sintaxiquit (Quit: sintaxi)
12:41:34  * sintaxijoined
12:44:01  * therealkoopajoined
12:48:37  * therealkoopaquit (Ping timeout: 248 seconds)
13:18:22  * iraquit (Quit: Computer has gone to sleep.)
13:24:31  * therealkoopajoined
13:28:32  * paulfryzeljoined
13:33:09  * paulfryzelquit (Ping timeout: 252 seconds)
13:45:42  * AvianFlujoined
13:46:27  * therealkoopaquit (Remote host closed the connection)
13:46:43  * mamashpart
13:50:46  * therealkoopajoined
14:29:15  * paulfryzeljoined
14:33:39  * paulfryzelquit (Ping timeout: 252 seconds)
14:58:31  * AvianFluquit (Remote host closed the connection)
15:30:00  * paulfryzeljoined
15:34:42  * paulfryzelquit (Ping timeout: 252 seconds)
15:51:03  * daviddiasquit (Ping timeout: 276 seconds)
15:56:53  * daviddiasjoined
15:57:26  * daviddia_joined
15:58:45  * AvianFlujoined
16:01:15  * daviddiasquit (Ping timeout: 245 seconds)
16:05:53  * daviddia_changed nick to daviddias
16:07:02  * sintaxiquit (Quit: sintaxi)
16:08:00  * ryancnelsonjoined
16:30:52  * paulfryzeljoined
16:35:54  * paulfryzelquit (Ping timeout: 276 seconds)
16:37:42  * paulfryzeljoined
16:39:36  * nfitchjoined
16:41:38  * ghostbarjoined
16:45:32  * ghostbar_joined
16:46:15  * ghostbarquit (Ping timeout: 245 seconds)
16:57:23  * ghostbar_changed nick to ghostbar
17:02:50  * fredkjoined
17:02:55  * fredkquit (Client Quit)
17:03:07  * fredkjoined
17:14:28  * notmattjoined
17:35:21  * daviddiasquit (Remote host closed the connection)
17:35:58  * daviddiasjoined
17:40:54  * daviddiasquit (Ping timeout: 276 seconds)
17:46:11  * ryancnelsonquit (Quit: Leaving.)
17:51:10  * therealkoopaquit (Remote host closed the connection)
17:59:00  * paulfryz_joined
18:01:00  * paulfryzelquit (Ping timeout: 252 seconds)
18:10:11  * therealkoopajoined
18:14:57  * ghostbarquit (Ping timeout: 265 seconds)
18:31:43  * daviddiasjoined
18:36:32  * daviddiasquit (Client Quit)
18:56:46  * daviddiasjoined
19:18:51  * mamashjoined
19:46:24  * ira_joined
19:49:16  * ghostbarjoined
19:53:12  * ghostbar_joined
19:53:54  * ghostbarquit (Ping timeout: 252 seconds)
19:56:32  * tna1quit (Quit: Leaving.)
19:56:51  * tnajoined
19:58:44  * ira_quit (Quit: Computer has gone to sleep.)
20:25:15  * ghostbar_quit (Ping timeout: 272 seconds)
20:38:13  * papajuansjoined
20:38:39  <papajuans>Is there anyway to see current statistics about manta usage through the web console?
20:38:49  <papajuans>eg, bytes used, traffic, etc.
20:44:25  * konobijoined
20:44:45  * konobipart
20:51:16  <dap_>papajuans: Not today, but it's something we'd like to do
21:00:17  * daviddiasquit (Remote host closed the connection)
21:07:59  * chorrelljoined
21:48:38  * ghostbarjoined
21:49:33  * AvianFlu_joined
21:49:50  * AvianFluquit (Disconnected by services)
21:49:52  * AvianFlu_changed nick to AvianFlu
21:52:12  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
21:55:27  * ryancnelsonjoined
21:59:41  * mamashpart
22:00:15  * bahamas10quit (Ping timeout: 252 seconds)
22:02:29  * bahamas10joined
22:02:38  * therealkoopaquit (Remote host closed the connection)
22:04:42  * ghostbarquit
22:06:22  * notmattquit (Remote host closed the connection)
22:14:19  * therealkoopajoined
22:20:25  * AvianFluquit (Remote host closed the connection)
22:33:53  <papajuans>for manta, if you store a file with a %20 in the filename, i assume manta does a decodeUri to look for a file with an actual space in the filename?
22:34:16  * ryancnelsonpart
22:35:26  <papajuans>I have an image in my public directory where the filename has a %20 in it. When i try to reach it via https://us-east.manta.joyent.com, I get a 404.
22:37:31  * AvianFlujoined
22:38:03  <dap_>papajuans: The name is encoded only when it goes over the wire. So if you "mput '/foo/stor/my file'", the object name in Manta (which you will see in "mls", and "mjob inputs", and so on) will be "/foo/stor/my file", even though it gets encoded with a %20 in the actual request. So similarly, if you "mput '/foo/stor/my%20file'", it's re-encoded over the wire, and the *actual* name is '/foo/stor/my%20file', not '/foo/stor/my file'
22:39:23  <dap_>So if you stored it with "%20", the actual name has %20, so if you request it with, say, Chrome, I wouldn't be surprised if you have to encode the %
22:41:03  <papajuans>right, so I need to double encode it
22:41:12  <papajuans>which fixes my problem. OK, thx
22:41:25  <dap_>no problem
22:41:58  * daviddiasjoined
22:46:13  <isaacs>Can someone tell me what i'm doing wrong here?
22:46:14  <isaacs>mfind -n '2014-01-\.*' /npm/stor/logs/ | mjob create -m 'gzcat | grep DELETE'
22:46:23  <isaacs>this results in 170 errors from 170 files
22:46:45  <isaacs>the errors all look like this:
22:46:46  <isaacs>{"phase":"0","what":"phase 0: map input \"/npm/stor/logs/2014-01-05-1388880094.site-npm-lylog_default.log.gz\"","code":"ResourceNotFoundError","message":"failed to load object","input":"/npm/stor/logs/2014-01-05-1388880094.site-npm-lylog_default.log.gz","p0input":"/npm/stor/logs/2014-01-05-1388880094.site-npm-lylog_default.log.gz"}
22:47:29  <isaacs>dap_: am i mjobing wrong?
22:48:18  <dap_>isaacs: I'll take a look
22:49:13  <isaacs>those objects are there, afaict. i can mget them, and gzcat|grep locally, but obvs that's silly
22:57:37  * therealkoopaquit (Remote host closed the connection)
22:59:36  <dap_>isaacs: This looks like a bug in the way we do cross-account snaplinks. There's no workaround except to "mget | mput" to create a new file. Sorry about that.
23:02:40  * daviddiasquit (Remote host closed the connection)
23:07:41  * daviddiasjoined
23:09:27  <isaacs>dap_: ugh
23:09:33  * daviddiasquit (Remote host closed the connection)
23:10:03  <isaacs>dap_: what's the scope of fixing the bug?
23:10:31  <dap_>I'm not sure how to answer that :) It shouldn't be that bad, if that's what you're asking, but realistically it would be tough to deploy before Monday
23:10:42  <dap_>How badly are you hosed?
23:22:06  <isaacs>dap_: i mean, days/weeks/months/probably never/patch welcome/?
23:22:21  <isaacs>dap_: i'm not so hosed, really. just annoying that i can only grep through those files one by one.
23:22:26  <dap_>isaacs: days
23:22:28  <isaacs>dap_: good to know that i'm not crazy, though :)
23:22:43  <isaacs>dap_: it does work if i do mjob create -r '...'
23:22:48  <isaacs>dap_: but takes a hecka long time
23:23:14  <dap_>Definitely not crazy. It's a nasty oversight — sorry about it.
23:23:30  <dap_>Yeah, "-r" will work, but by fetching the file instead of operating locally
23:24:04  * paulfryz_quit (Remote host closed the connection)
23:25:10  <isaacs>dap_: ok, thanks. i'll wait.
23:28:17  * therealkoopajoined
23:32:43  * therealkoopaquit (Ping timeout: 260 seconds)
23:42:33  * tnaquit (Ping timeout: 252 seconds)
23:55:05  * paulfryzeljoined
23:59:35  * therealkoopajoined