00:14:14  <bsdguru>you guys having some issues again? :)
00:14:46  <bsdguru>we are seeing an elevated error rate on collecting tarballs from manta
00:17:38  <dap_1>bsdguru: I'm not seeing any issues
00:18:37  * nfitchquit (Ping timeout: 244 seconds)
00:18:39  <bsdguru>all three backends go out of rotation and we see large error counts
00:18:48  <dap_1>right now?
00:19:03  <bsdguru>let me refresh the stats view
00:19:46  <rmustacc>What exactly is the nature of the error?
00:20:05  <bsdguru>we were spiking around 400 r/s that were not going to the manta backends
00:20:14  <bsdguru>for 503's
00:20:26  <bsdguru>we 503 when we have three unhealthy manta backends
00:22:10  <dap_1>We haven't made any changes in hours, and I don't see any failures in the last half an hour (that's the view I'm looking at). Unfortunately I've got to run at the moment.
00:23:02  <bsdguru>k.
00:23:21  <mcavage>bsdguru: on top of that, a bunch of "sniff tests" seem to be just fine.
00:23:32  <mcavage>also - our client-side alerts have not tripped, from *anywhere*
00:23:39  <mcavage>(there are about 30 edge locations)
00:23:42  <bsdguru>I'm going to grep through our syslog messages
00:25:41  * nfitchjoined
00:25:41  * nfitchquit (Client Quit)
00:28:19  <mcavage>bsdguru: ok - i do see some 500s starting to pop up in client monitors -- 1m we are looking.
00:29:40  <bsdguru>thanks
00:33:29  <mcavage>bsdguru: ok it seems that we had exactly 1 of 64 web api servers go into a state where it was pitching 500s without restarting (as best I can see right now -- checking others). It's restarted.
00:34:38  * ghostbarjoined
00:36:32  * notmatt_quit (Remote host closed the connection)
00:37:12  * seldoquit (Remote host closed the connection)
00:37:56  * ghostbar_quit (Ping timeout: 252 seconds)
00:41:18  * seldojoined
00:48:24  * AvianFlujoined
00:56:08  * nfitchjoined
01:00:49  * nfitchquit (Ping timeout: 264 seconds)
01:10:43  * seldoquit (Remote host closed the connection)
01:11:18  * seldojoined
01:11:51  * mcavagequit (Remote host closed the connection)
01:12:39  * fredk1quit (Quit: Leaving.)
01:15:36  * seldoquit (Ping timeout: 240 seconds)
01:18:56  * paulfryz_quit (Remote host closed the connection)
01:19:33  * paulfryzeljoined
01:19:54  * dap_1quit (Quit: Leaving.)
01:20:07  * ed209joined
01:24:08  * paulfryzelquit (Ping timeout: 252 seconds)
01:26:30  * yunongquit (Quit: Leaving.)
01:31:54  * abraxasjoined
01:36:24  * abraxasquit (Ping timeout: 240 seconds)
01:42:59  * seldojoined
01:43:54  * seldoquit (Remote host closed the connection)
01:44:08  * seldojoined
01:49:53  * paulfryzeljoined
01:54:17  * bixu_joined
01:54:23  * paulfryzelquit (Ping timeout: 252 seconds)
01:57:18  * bixuquit (Ping timeout: 244 seconds)
02:23:29  * seldoquit (Remote host closed the connection)
02:23:57  * bsdguruquit (Quit: bsdguru)
02:24:03  * seldojoined
02:28:00  * seldoquit (Ping timeout: 240 seconds)
02:35:36  * irajoined
02:35:56  * iraquit (Client Quit)
02:44:58  * nfitchjoined
02:49:25  * nfitchquit (Ping timeout: 264 seconds)
02:50:53  * paulfryzeljoined
02:55:26  * paulfryzelquit (Ping timeout: 252 seconds)
03:02:33  * bsdgurujoined
03:32:53  * abraxasjoined
03:37:12  * abraxasquit (Ping timeout: 240 seconds)
03:38:24  * seldojoined
03:51:33  * paulfryzeljoined
03:55:56  * paulfryzelquit (Ping timeout: 252 seconds)
04:33:41  * nfitchjoined
04:38:17  * nfitchquit (Ping timeout: 252 seconds)
04:52:23  * paulfryzeljoined
04:56:59  * paulfryzelquit (Ping timeout: 252 seconds)
05:09:59  * marsellquit (Quit: marsell)
05:15:49  * nfitchjoined
05:20:37  * nfitchquit (Ping timeout: 264 seconds)
05:27:33  * ghostbar_joined
05:28:53  * ghostbarquit (Ping timeout: 252 seconds)
05:33:47  * abraxasjoined
05:34:08  * ghostbar_quit (Remote host closed the connection)
05:38:14  * abraxasquit (Ping timeout: 252 seconds)
05:58:30  * notmattjoined
05:58:41  * AvianFluquit (Remote host closed the connection)
06:50:09  * elijah-mbpchanged nick to eli-away
07:04:36  * nfitchjoined
07:08:59  * nfitchquit (Ping timeout: 252 seconds)
07:34:22  * abraxasjoined
07:38:48  * abraxasquit (Ping timeout: 240 seconds)
08:32:02  * ins0mniajoined
08:53:19  * nfitchjoined
08:55:10  * nfitch1joined
08:55:55  * paulfryzeljoined
08:57:49  * nfitchquit (Ping timeout: 264 seconds)
08:59:37  * nfitch1quit (Ping timeout: 264 seconds)
09:00:05  * paulfryzelquit (Ping timeout: 252 seconds)
09:35:32  * abraxasjoined
09:40:00  * abraxasquit (Ping timeout: 240 seconds)
10:01:12  * ins0mniaquit (Ping timeout: 240 seconds)
10:04:25  * notmattquit (Remote host closed the connection)
10:04:58  * notmattjoined
10:09:23  * notmattquit (Ping timeout: 252 seconds)
10:43:51  * nfitchjoined
10:48:00  * nfitchquit (Ping timeout: 240 seconds)
10:57:34  * paulfryzeljoined
11:02:11  * paulfryzelquit (Ping timeout: 252 seconds)
11:05:10  * irajoined
11:07:56  * iraquit (Client Quit)
11:15:22  * bsdguruquit (Quit: bsdguru)
11:18:20  * bsdgurujoined
11:27:00  * bsdguruquit (Quit: bsdguru)
11:28:00  * bsdgurujoined
11:29:56  * marselljoined
11:36:22  * abraxasjoined
11:38:42  * bsdguruquit (Quit: bsdguru)
11:39:36  * bsdgurujoined
11:40:48  * abraxasquit (Ping timeout: 240 seconds)
11:45:47  * ins0mniajoined
11:57:48  * bsdguruquit (Quit: bsdguru)
11:58:19  * paulfryzeljoined
12:02:41  * paulfryzelquit (Ping timeout: 252 seconds)
12:32:49  * nfitchjoined
12:37:25  * nfitchquit (Ping timeout: 264 seconds)
12:59:07  * paulfryzeljoined
13:03:49  * paulfryzelquit (Ping timeout: 264 seconds)
13:37:13  * abraxasjoined
13:41:36  * abraxasquit (Ping timeout: 240 seconds)
14:00:02  * paulfryzeljoined
14:04:25  * paulfryzelquit (Ping timeout: 264 seconds)
14:16:10  * AvianFlujoined
14:21:34  * nfitchjoined
14:26:01  * nfitchquit (Ping timeout: 264 seconds)
14:28:36  * ghostbarjoined
14:34:12  * ghostbarquit (Ping timeout: 252 seconds)
14:38:43  * ghostbarjoined
15:00:47  * paulfryzeljoined
15:05:37  * paulfryzelquit (Ping timeout: 264 seconds)
15:18:02  * daviddiasjoined
15:28:35  * daviddiasquit (Remote host closed the connection)
15:29:31  * daviddiasjoined
15:38:15  * abraxasjoined
15:42:49  * abraxasquit (Ping timeout: 264 seconds)
15:44:04  * daviddiasquit (Ping timeout: 252 seconds)
15:45:28  * daviddiasjoined
15:59:21  * daviddiasquit (Remote host closed the connection)
16:01:37  * paulfryzeljoined
16:06:13  * paulfryzelquit (Ping timeout: 264 seconds)
16:07:51  * daviddiasjoined
16:10:19  * nfitchjoined
16:15:13  * nfitchquit (Ping timeout: 264 seconds)
16:19:25  * daviddiasquit (Ping timeout: 264 seconds)
16:28:37  * ghostbarquit
16:41:15  * ghostbarjoined
16:43:42  * ghostbar_joined
16:45:12  * ghostbarquit (Ping timeout: 240 seconds)
16:59:16  * daviddiasjoined
17:02:22  * paulfryzeljoined
17:04:52  * chorrelljoined
17:06:49  * paulfryzelquit (Ping timeout: 264 seconds)
17:18:49  * marsellquit (Ping timeout: 264 seconds)
17:26:00  * chorrellquit (Ping timeout: 240 seconds)
17:28:18  * chorrelljoined
17:32:59  * daviddiasquit (Remote host closed the connection)
17:33:36  * daviddiasjoined
17:36:13  * daviddiasquit (Remote host closed the connection)
17:38:12  * chorrellquit (Quit: My Mac has gone to sleep. ZZZzzz…)
17:39:01  * abraxasjoined
17:43:27  * abraxasquit (Ping timeout: 252 seconds)
17:59:07  * nfitchjoined
18:03:22  * paulfryzeljoined
18:03:49  * nfitchquit (Ping timeout: 252 seconds)
18:08:01  * paulfryzelquit (Ping timeout: 264 seconds)
18:30:43  * daviddiasjoined
19:02:59  * daviddia_joined
19:04:05  * paulfryzeljoined
19:05:09  * daviddiasquit (Ping timeout: 245 seconds)
19:08:23  * daviddia_quit (Read error: Connection reset by peer)
19:08:24  * daviddiasjoined
19:08:37  * paulfryzelquit (Ping timeout: 264 seconds)
19:10:13  * daviddiasquit (Remote host closed the connection)
19:16:04  * daviddiasjoined
19:31:02  * bsdgurujoined
19:39:56  * abraxasjoined
19:44:37  * abraxasquit (Ping timeout: 264 seconds)
19:47:50  * nfitchjoined
19:52:12  * nfitchquit (Ping timeout: 252 seconds)
20:05:03  * paulfryzeljoined
20:09:49  * paulfryzelquit (Ping timeout: 264 seconds)
20:10:23  * daviddiasquit (Remote host closed the connection)
20:12:50  * daviddiasjoined
20:12:52  * daviddiasquit (Remote host closed the connection)
20:13:29  * daviddiasjoined
20:15:37  * daviddiasquit (Client Quit)
20:44:05  * daviddiasjoined
21:05:53  * paulfryzeljoined
21:10:25  * paulfryzelquit (Ping timeout: 264 seconds)
21:36:35  * nfitchjoined
21:40:50  * abraxasjoined
21:41:01  * nfitchquit (Ping timeout: 264 seconds)
21:45:32  * abraxasquit (Ping timeout: 252 seconds)
22:06:38  * paulfryzeljoined
22:09:51  * isaacs_mobilejoined
22:11:37  * paulfryzelquit (Ping timeout: 264 seconds)
22:16:24  * isaacs_mobilequit (Ping timeout: 240 seconds)
22:29:12  * ins0mniaquit (Ping timeout: 240 seconds)
22:51:13  * bsdguruquit (Ping timeout: 240 seconds)
22:54:13  * bsdgurujoined
22:58:25  * bsdguruquit (Ping timeout: 240 seconds)
22:58:25  * AvianFluquit (Ping timeout: 240 seconds)
22:58:39  * AvianFlujoined
22:58:47  * bsdgurujoined
23:07:23  * paulfryzeljoined
23:10:28  * AvianFluquit (Remote host closed the connection)
23:10:58  * AvianFlujoined
23:12:13  * paulfryzelquit (Ping timeout: 264 seconds)
23:25:25  * nfitchjoined
23:30:13  * nfitchquit (Ping timeout: 264 seconds)
23:41:24  * abraxasjoined
23:46:00  * abraxasquit (Ping timeout: 240 seconds)
23:57:01  * nfitchjoined
23:58:51  * nfitch1joined
23:58:51  * nfitchquit (Read error: Connection reset by peer)