00:11:31  * paulfryzelquit (Remote host closed the connection)
00:12:06  * paulfryzeljoined
00:16:36  * paulfryzelquit (Ping timeout: 252 seconds)
00:17:37  * fredkquit (Quit: Leaving.)
00:19:43  * mattappersonjoined
00:24:08  * mattappersonquit (Ping timeout: 240 seconds)
00:32:52  * nfitchquit (Quit: Leaving.)
00:39:40  * dapquit (Quit: Leaving.)
00:56:48  * yunongquit (Quit: Leaving.)
01:09:24  * mattappersonjoined
01:18:41  * ryancnelsonquit (Read error: Operation timed out)
02:16:19  * abraxasjoined
02:20:21  * mattappersonquit (Remote host closed the connection)
02:21:00  * mattappersonjoined
02:26:01  * mattappersonquit (Ping timeout: 272 seconds)
03:31:50  * mattappersonjoined
03:36:21  * AvianFluquit (Remote host closed the connection)
03:36:50  * AvianFlujoined
03:36:58  * mattappersonquit (Ping timeout: 256 seconds)
03:41:15  * AvianFluquit (Ping timeout: 248 seconds)
03:47:37  * mcavagequit (Remote host closed the connection)
03:48:05  * mcavagejoined
03:48:43  * mcavage_joined
03:48:43  * mcavagequit (Read error: Connection reset by peer)
03:48:52  * ins0mniaquit (Ping timeout: 256 seconds)
04:25:42  * mcavage_quit (Remote host closed the connection)
04:26:09  * mcavagejoined
04:30:48  * mcavagequit (Ping timeout: 256 seconds)
04:55:55  * bixu__quit (Remote host closed the connection)
04:56:30  * bixujoined
04:56:41  * mcavagejoined
05:00:57  * bixuquit (Ping timeout: 252 seconds)
05:05:10  * mcavagequit (Ping timeout: 260 seconds)
06:01:13  * mcavagejoined
06:05:47  * mcavagequit (Ping timeout: 260 seconds)
06:18:55  * marsellquit (Quit: marsell)
07:08:29  * mamashjoined
07:09:10  * yruss972joined
07:31:38  * mcavagejoined
07:35:50  * mcavagequit (Ping timeout: 245 seconds)
07:40:06  * yruss972quit (Remote host closed the connection)
07:40:42  * yruss972joined
07:44:48  * yruss972quit (Ping timeout: 240 seconds)
08:23:22  * notmattquit (Remote host closed the connection)
08:39:20  * yruss972joined
09:30:10  * marselljoined
09:32:20  * mcavagejoined
09:37:19  * mcavagequit (Ping timeout: 272 seconds)
09:57:20  * ins0mniajoined
10:17:24  * abraxasquit (Remote host closed the connection)
10:18:01  * abraxasjoined
10:22:28  * abraxasquit (Ping timeout: 260 seconds)
10:32:42  * mcavagejoined
10:36:37  * mcavagequit (Ping timeout: 240 seconds)
11:02:32  * AvianFlujoined
11:33:03  * mcavagejoined
11:37:30  * mcavagequit (Ping timeout: 252 seconds)
11:51:21  * mattappersonjoined
11:52:48  * yruss972quit (Remote host closed the connection)
11:53:23  * yruss972joined
11:57:30  * yruss972quit (Ping timeout: 245 seconds)
12:23:57  * notmattjoined
12:24:20  * yruss972joined
12:29:47  * yruss972quit (Ping timeout: 248 seconds)
12:31:37  * notmattquit (Ping timeout: 265 seconds)
12:33:23  * mcavagejoined
12:38:05  * mcavagequit (Ping timeout: 272 seconds)
13:01:02  * yruss972joined
13:06:32  * yruss972_joined
13:07:35  * yruss972quit (Ping timeout: 272 seconds)
13:33:45  * mcavagejoined
13:37:42  * mcavagequit (Ping timeout: 241 seconds)
13:52:33  * yruss972_quit (Ping timeout: 272 seconds)
14:10:05  * AvianFluquit (Remote host closed the connection)
14:10:34  * AvianFlujoined
14:14:49  * AvianFluquit (Ping timeout: 246 seconds)
14:34:05  * mcavagejoined
14:35:01  * mattappersonquit (Remote host closed the connection)
14:38:08  * mcavagequit (Ping timeout: 240 seconds)
15:07:40  * mcavagejoined
15:08:17  * mcavage_joined
15:08:17  * mcavagequit (Read error: Connection reset by peer)
15:13:09  * mcavage_quit (Remote host closed the connection)
15:13:36  * mcavagejoined
15:14:30  * mamashpart
15:17:55  * mcavagequit (Ping timeout: 245 seconds)
15:30:43  * ryancnelsonjoined
15:32:48  * yruss972joined
15:36:02  * ryancnelsonquit (Quit: Leaving.)
15:40:25  * mcavagejoined
15:42:59  * mcavagequit (Remote host closed the connection)
15:43:26  * mcavagejoined
15:48:11  * mcavagequit (Ping timeout: 272 seconds)
15:50:23  * nfitchjoined
15:51:37  * nfitchquit (Client Quit)
15:52:35  * nfitchjoined
15:56:18  * nfitchquit (Client Quit)
15:58:51  * nfitchjoined
16:06:06  * fredkjoined
16:12:52  * chorrelljoined
16:20:43  * dapjoined
16:22:57  * yunongjoined
16:32:59  * notmattjoined
16:37:10  * ins0mniaquit (Read error: Operation timed out)
16:47:27  * notmattquit (Remote host closed the connection)
16:48:42  * yunongquit (Quit: Leaving.)
17:08:11  * yunongjoined
17:10:40  * notmattjoined
17:11:52  * bixujoined
17:15:51  * yunongquit (Quit: Leaving.)
17:22:32  * mamashjoined
17:45:21  * yunongjoined
17:49:03  * yruss972quit (Ping timeout: 260 seconds)
17:54:05  * paulfryzeljoined
18:02:36  * mcavagejoined
18:18:33  * bixuquit (Ping timeout: 272 seconds)
18:31:04  * mamashpart
18:33:25  <nahamu>If manta had only 7 servers named "server1" -> "server7" and I had an object with copies=2 stored on server1 and server2
18:33:54  <nahamu>and server1 suffered simultaneous failure of all the drives (unlikely thanks to wesolows' engineering...)
18:34:20  <nahamu>would Manta make a copy of my object from the one stored on server2 to one of the remaining servers?
18:34:50  <nahamu>or does the system generally assume that a server that goes away will generally come back at some point?
18:35:19  <tjfontaine>I think your question more generally is: "does manta rebalance objects"
18:35:23  <tjfontaine>not that I know the answer to that
18:35:44  <nahamu>I suppose that would be a more succint way to ask, yes.
18:36:40  <nfitch>nahamu: We assume that either server2 will come back, or a replica of server2 will come back. We don't automagically rebalance objects.
18:37:28  <nahamu>"replica of server2" meaning "zfs pools are damn resilient, we'll get that pool back online whether in that chassis or a different one"?
18:38:07  * mariuspjoined
18:38:17  <nfitch>No, that would be a case of server2 "coming back".
18:38:29  <nahamu>oh
18:38:46  <nahamu>so what would a "replica of server2" be?
18:38:49  <nfitch>We could rebuild a replica of server2 from the objects stored everywhere else, which is what I meant by a "replica".
18:39:07  <nahamu>ah, got it.
18:40:07  <nahamu>So if that pool is unrecoverable you'd recreate that same collection of objects again based on the knowledge elsewhere in the system of what was on that box.
18:40:17  <nfitch>Given our architecture, the tooling around that is actually fairly simple- it would just take a really long time (days).
18:40:19  <nfitch>Correct.
18:40:26  <nahamu>Cool. Thanks for explaining.
18:40:54  <dap>nahamu: For details: http://dtrace.org/blogs/dap/2013/07/03/fault-tolerance-in-manta/
18:41:29  * notmattquit (Remote host closed the connection)
18:43:47  <nahamu>I'd read that. did I miss the answer to this specific question buried in there?
18:44:14  <nahamu>(but a good read, so reason not to point anyone asking questions about Manta at it...)
18:44:26  <nahamu>*so *no* reason not to...
18:44:51  <nahamu>there it is.
18:45:09  <nahamu>" If we lose a ZFS storage pool, we would have to replicate objects to other pools, but that's not something that's triggered automatically in response to failure since it's not appropriate for most failures."
18:45:46  <dap>Yeah. I guess that's it, and you already got that :)
18:46:05  <nahamu>having Keith build the hardware does seem like a good route to avoiding this sort of problem in general, though.
18:46:15  <nahamu>:)
18:56:49  * yruss972joined
19:00:19  * ins0mniajoined
19:01:23  * yruss972quit (Ping timeout: 260 seconds)
19:08:38  * paulfryzelquit (Remote host closed the connection)
19:09:13  * paulfryzeljoined
19:13:47  * paulfryzelquit (Ping timeout: 268 seconds)
19:16:17  * utlemmingquit (Remote host closed the connection)
19:18:43  * utlemmingjoined
19:21:15  * mariuspquit (Remote host closed the connection)
19:30:59  * mamashjoined
19:39:10  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
19:49:53  * paulfryzeljoined
19:59:01  * dapchanged nick to Guest84659
20:04:18  * ryancnelsonjoined
20:04:24  * ryancnelsonquit (Quit: Leaving.)
20:17:29  * ryancnelsonjoined
20:18:14  * mariuspjoined
20:20:04  * notmattjoined
20:21:49  * mariuspquit (Remote host closed the connection)
20:27:45  * utlemmingquit (Remote host closed the connection)
20:29:12  * mamashpart
20:29:44  * paulfryzelquit (Remote host closed the connection)
20:41:01  * paulfryzeljoined
20:42:07  * paulfryzelquit (Read error: Connection reset by peer)
20:42:26  * paulfryzeljoined
20:47:27  * Guest84659quit (Quit: Leaving.)
20:52:13  * paulfryz_joined
20:55:26  * paulfryzelquit (Ping timeout: 240 seconds)
21:02:41  * nfitchquit (Ping timeout: 245 seconds)
21:03:35  * mariuspjoined
21:05:34  * nfitchjoined
21:16:11  * yruss972joined
21:29:53  * mariuspquit (Remote host closed the connection)
21:31:54  * yunongquit (Quit: Leaving.)
21:59:23  * nfitchquit (Quit: Leaving.)
22:23:47  * paulfryz_quit (Remote host closed the connection)
22:24:24  * paulfryzeljoined
22:28:38  * paulfryzelquit (Ping timeout: 240 seconds)
22:30:49  * utlemmingjoined
22:52:34  * mcavagequit (Remote host closed the connection)
22:53:07  * mcavagejoined
22:57:02  * mcavagequit (Ping timeout: 240 seconds)
23:22:20  * yruss972quit (Remote host closed the connection)
23:22:52  * yruss972joined
23:25:43  * paulfryzeljoined
23:25:47  * paulfryzelquit (Remote host closed the connection)
23:27:03  * yruss972quit (Ping timeout: 240 seconds)
23:35:55  * mcavagejoined
23:36:15  * mcavagequit (Read error: Connection reset by peer)
23:36:30  * mcavagejoined
23:46:16  * ins0mniaquit (Ping timeout: 245 seconds)
23:50:27  * yunongjoined