00:00:38  * yunongjoined
00:17:20  * lloyddequit (Remote host closed the connection)
00:17:59  * lloyddejoined
00:22:09  * lloyddequit (Ping timeout: 240 seconds)
00:27:48  * paulfryzeljoined
00:34:08  * lloyddejoined
00:34:09  * lloyddequit (Remote host closed the connection)
00:34:43  * lloyddejoined
00:39:51  * papajuansquit (Remote host closed the connection)
00:54:32  * chorrell-awaychanged nick to chorrell
00:55:26  * lloyddequit (Ping timeout: 264 seconds)
01:02:38  * nfitchquit (Quit: Leaving.)
01:04:44  * fredkquit (Quit: Leaving.)
01:11:21  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
01:20:08  * ed209joined
01:31:24  * paulfryzelquit (Remote host closed the connection)
01:42:10  * abraxasjoined
01:44:47  * abraxasquit (Read error: Connection reset by peer)
01:45:06  * abraxasjoined
01:46:41  * abraxasquit (Remote host closed the connection)
01:52:59  * lloyddejoined
01:55:45  * abraxasjoined
02:10:40  * lloyddequit (Remote host closed the connection)
02:29:28  * bixujoined
02:32:32  * bixu_quit (Ping timeout: 244 seconds)
02:35:31  * seldojoined
02:35:47  * seldoquit (Remote host closed the connection)
02:36:12  * paulfryzeljoined
02:40:37  * bsdguruquit (Quit: bsdguru)
02:59:22  * yunongquit (Quit: Leaving.)
03:06:43  * abraxasquit (Read error: Connection reset by peer)
03:07:15  * abraxasjoined
03:09:02  * therealkoopaquit (Remote host closed the connection)
03:39:59  * therealkoopajoined
03:47:12  * therealkoopaquit (Ping timeout: 240 seconds)
04:01:31  * dap_quit (Quit: Leaving.)
04:21:05  * bsdgurujoined
04:23:51  * AvianFlujoined
04:36:24  * abraxasquit (Remote host closed the connection)
04:37:00  * abraxasjoined
04:41:38  * abraxasquit (Ping timeout: 264 seconds)
04:42:21  * AvianFluquit (Remote host closed the connection)
04:49:13  * bsdguruquit (Quit: bsdguru)
04:52:17  * ryancnelson1quit (Quit: Leaving.)
04:54:53  * lloyddejoined
05:06:28  * abraxasjoined
05:09:58  * bsdgurujoined
05:16:37  * abraxasquit (Remote host closed the connection)
05:17:10  * abraxasjoined
05:21:38  * abraxasquit (Ping timeout: 240 seconds)
05:22:39  * paulfryzelquit (Remote host closed the connection)
05:35:40  * abraxasjoined
05:47:24  * bsdguruquit (Remote host closed the connection)
05:47:45  * bsdgurujoined
05:52:04  * ghostbarquit (Remote host closed the connection)
06:23:38  * paulfryzeljoined
06:27:58  * paulfryzelquit (Ping timeout: 240 seconds)
06:39:11  * notmatt_quit (Remote host closed the connection)
06:44:04  * |woody_|changed nick to |woody|
07:01:28  * lloyddequit (Remote host closed the connection)
07:01:55  * lloyddejoined
07:06:12  * lloyddequit (Ping timeout: 245 seconds)
07:07:51  * marsellquit (Quit: marsell)
07:14:30  * daviddiasjoined
07:24:31  * paulfryzeljoined
07:28:38  * paulfryzelquit (Ping timeout: 240 seconds)
07:34:21  * abraxasquit (Remote host closed the connection)
07:48:11  * notmattjoined
07:52:18  * notmattquit (Ping timeout: 240 seconds)
07:58:46  * abraxasjoined
08:32:51  * paulfryzeljoined
08:37:20  * paulfryzelquit (Ping timeout: 265 seconds)
09:33:34  * paulfryzeljoined
09:34:47  * paulfryzelquit (Read error: No route to host)
09:35:20  * paulfryzeljoined
09:36:01  * notmattjoined
09:39:36  * paulfryzelquit (Ping timeout: 240 seconds)
09:40:05  * daviddiasquit (Read error: Connection reset by peer)
09:40:39  * notmattquit (Ping timeout: 265 seconds)
09:41:08  * daviddiasjoined
09:42:39  * therealkoopajoined
09:44:20  * bsdguruquit (Quit: bsdguru)
09:44:46  * daviddiasquit (Read error: Connection reset by peer)
09:47:26  * therealkoopaquit (Ping timeout: 252 seconds)
09:52:51  * ghostbarjoined
09:57:50  * ghostbarquit (Ping timeout: 264 seconds)
10:16:44  * abraxasquit (Remote host closed the connection)
10:29:00  * abraxasjoined
10:36:11  * paulfryzeljoined
10:39:48  * notmattjoined
10:40:24  * paulfryzelquit (Ping timeout: 240 seconds)
10:44:00  * notmattquit (Ping timeout: 240 seconds)
11:18:32  * daviddiasjoined
11:27:40  * bixuquit (Remote host closed the connection)
11:28:07  * bixujoined
11:33:03  * bixuquit (Ping timeout: 272 seconds)
11:36:48  * paulfryzeljoined
11:41:29  * paulfryzelquit (Ping timeout: 265 seconds)
11:51:48  * abraxasquit (Remote host closed the connection)
12:37:41  * paulfryzeljoined
12:40:52  * ghostbarjoined
12:41:38  * paulfryzelquit (Ping timeout: 240 seconds)
12:46:52  * therealkoopajoined
13:19:26  * bixujoined
13:32:27  * bixu_joined
13:33:26  * bixuquit (Read error: Operation timed out)
13:34:05  * daviddiasquit (Remote host closed the connection)
13:34:19  * daviddiasjoined
13:35:55  * bixu_quit (Read error: Connection reset by peer)
13:36:03  * bixujoined
13:39:27  * bixu_joined
13:42:41  * bixu__joined
13:42:49  * bixuquit (Ping timeout: 265 seconds)
13:46:12  * bixu_quit (Ping timeout: 265 seconds)
13:48:47  * therealkoopaquit (Remote host closed the connection)
13:52:12  * bixujoined
13:53:59  * bixu__quit (Read error: Operation timed out)
13:54:44  * therealkoopajoined
13:57:18  * ghostbarquit (Remote host closed the connection)
14:01:40  * AvianFlujoined
14:26:08  * paulfryzeljoined
14:55:28  * paulfryzelquit (Remote host closed the connection)
15:13:34  * bixu_joined
15:15:20  * therealk_joined
15:21:44  * therealkoopaquit (*.net *.split)
15:21:44  * bixuquit (*.net *.split)
15:21:45  * ed209quit (*.net *.split)
15:23:38  * doug__joined
15:27:42  * doug_quit (Ping timeout: 265 seconds)
15:44:48  * AvianFluquit (Remote host closed the connection)
15:48:09  * chorrelljoined
15:53:31  * paulfryzeljoined
15:53:35  * abraxasjoined
15:57:59  * abraxasquit (Ping timeout: 240 seconds)
16:36:36  * dap_joined
16:36:43  <bixu_>anybody here using manta to do distributed unit tests?
16:36:48  * bixu_changed nick to bixu
16:37:13  * AvianFlujoined
16:38:42  * daviddiasquit (Ping timeout: 245 seconds)
16:40:04  * notmattjoined
16:41:15  * doug__quit (Quit: Ex-Chat)
16:41:30  * doug__joined
16:42:20  * lloyddejoined
16:42:23  * lloyddequit (Remote host closed the connection)
16:43:00  * lloyddejoined
16:55:03  * marselljoined
16:59:31  * nfitchjoined
17:04:51  * lloydde_joined
17:06:23  * fredkjoined
17:06:50  * fredkquit (Client Quit)
17:07:11  * fredkjoined
17:07:58  * lloyddequit (Ping timeout: 240 seconds)
17:10:18  * notmattquit (Ping timeout: 240 seconds)
17:13:07  * paulfryzelquit (Read error: Connection reset by peer)
17:13:58  * paulfryzeljoined
17:14:47  * notmattjoined
17:27:28  * lloydde_quit (Remote host closed the connection)
17:27:31  * yunongjoined
17:28:06  * lloyddejoined
17:28:47  * lloyddequit (Read error: Connection reset by peer)
17:35:32  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
17:39:13  * yunongquit (Quit: Leaving.)
17:45:05  * lloyddejoined
17:54:22  * abraxasjoined
17:58:40  * ryancnelsonjoined
17:58:59  * abraxasquit (Ping timeout: 240 seconds)
18:15:12  * notmattquit (Remote host closed the connection)
18:15:46  * notmattjoined
18:19:58  * notmattquit (Ping timeout: 240 seconds)
18:21:03  * seldojoined
18:35:38  * yunongjoined
18:42:09  * daviddiasjoined
18:42:39  * daviddiasquit (Remote host closed the connection)
18:43:10  * daviddiasjoined
18:46:47  * yunong1joined
18:47:29  * daviddiasquit (Ping timeout: 240 seconds)
18:49:59  * yunongquit (Ping timeout: 241 seconds)
18:55:42  * bsdgurujoined
18:59:28  * notmattjoined
19:05:17  * therealk_quit (Remote host closed the connection)
19:36:50  * therealkoopajoined
19:51:33  * bsdguruquit (Quit: bsdguru)
19:55:18  * abraxasjoined
19:59:38  * abraxasquit (Ping timeout: 240 seconds)
20:05:00  * lloyddequit (Remote host closed the connection)
20:15:50  * seldoquit (Remote host closed the connection)
20:16:40  * seldojoined
20:18:51  * bsdgurujoined
20:23:49  * bsdguru_joined
20:25:03  * bsdguruquit (Ping timeout: 272 seconds)
20:25:04  * bsdguru_changed nick to bsdguru
20:25:43  * paulfryzelquit (Remote host closed the connection)
20:28:28  * paulfryzeljoined
20:33:32  * ghostbarjoined
20:36:24  * paulfryzelquit (Remote host closed the connection)
20:44:21  * paulfryzeljoined
20:57:10  * ghostbarquit (Remote host closed the connection)
21:10:05  * paulfryzelquit (Remote host closed the connection)
21:11:25  * seldoquit (Remote host closed the connection)
21:11:41  * seldojoined
21:13:00  * mamashjoined
21:25:05  * seldoquit (Quit: Leaving...)
21:45:07  * mamashpart
22:04:21  * paulfryzeljoined
22:22:37  * paulfryzelquit (Remote host closed the connection)
22:46:41  <bsdguru>hi guys
22:46:53  <bsdguru>one of the mantas in as burn is not responding to http requests
22:47:19  <bsdguru>165.225.172.11
22:53:01  * mcavagejoined
22:53:38  * paulfryzeljoined
22:54:07  <mcavage>bsdguru: thanks - your alarms beat ours -- it's the same FD leak again; restarted. We rolled out the first "real fix" today - the others are going tomorrow. Sorry.
22:54:58  * paulfryzelquit (Read error: Connection reset by peer)
22:55:34  * paulfryzeljoined
22:58:21  <nfitch>bsdguru: We just talked about it and we're going to go ahead and push the fix out to all of the LBs this afternoon. You should expect ~5 min blip for each of the external ips.
22:59:36  * paulfryzelquit (Ping timeout: 240 seconds)
23:05:19  * mcavagequit (Remote host closed the connection)
23:10:02  <bsdguru>many is down again *sigh*
23:10:05  <bsdguru>manta even
23:10:14  <bsdguru>mcavaga: ping :)
23:10:42  <nfitch>bsdguru: I told you there would be 3 ~5 minute blips. This is #1.
23:10:53  <nfitch>Happy to tell you when the other two are coming :)
23:14:01  <nfitch>Side question: Manta was still up while that lb was rebooting. Was manta totally down for you or just one of the lbs?
23:15:42  <bsdguru>seems manta was down for us
23:15:49  <bsdguru>we reverse proxy via varnish
23:16:01  <bsdguru>on the us-east.manta.joyent.com DNS record
23:16:15  <bsdguru>I should prod you guys to use stingray's ;)
23:19:04  <nfitch>The other two lbs should have been fine…. does your reverse proxy reconnect to one of the other ips when something downstream disconnects?
23:23:36  <nfitch>I mean, any of our LBs could fail at any time (that's why we have multiple).
23:25:39  * therealkoopaquit (Remote host closed the connection)
23:29:54  <nfitch>bsdguru: I hear that npm is having issues- does that trace back to manta? Should we hold off on the next 2 reboots?
23:30:59  * lloyddejoined
23:33:29  * therealkoopajoined
23:34:22  <bsdguru>pinging fastly to see how we can bypass the issues
23:35:05  * AvianFluquit (Remote host closed the connection)
23:35:49  <nfitch>bsdguru: This is why we have 3 lbs. If (fastly?) isn't reconnecting correctly on failure, ya, that's a problem.
23:36:16  <nfitch>Obviously our LBs *all* being down is a problem, but we (and you) should be able to tolerate the failure of one.
23:38:01  <nfitch>bsdguru: So… we should hold off on the other reboots then?
23:38:46  <nfitch>We were planning on holding off until tomorrow anyways, but we figured it was better to get the fix out sooner. If it'd help, I think we're happy to wait.
23:45:11  * ryancnelsonquit (Ping timeout: 272 seconds)
23:45:12  * marsellquit (Ping timeout: 272 seconds)
23:54:46  * ryancnelsonjoined
23:56:13  * paulfryzeljoined
23:57:07  * lloyddequit (Remote host closed the connection)
23:57:57  * ryancnelson1joined
23:58:04  * ryancnelson1quit (Read error: Connection reset by peer)
23:58:16  * ryancnelson1joined
23:58:34  * ryancnelsonquit (Disconnected by services)
23:58:40  * ryancnelson1changed nick to ryancnelson