00:03:18  * mellocjoined
00:03:18  * melloc1quit (Read error: Connection reset by peer)
00:08:38  * mellocquit (Quit: Leaving.)
00:09:30  * mhicksquit (Quit: Leaving.)
00:14:08  * mellocjoined
00:15:36  * mellocquit (Client Quit)
00:44:48  * bahamatjoined
01:02:57  * jayschmidtquit (Quit: Leaving.)
01:03:25  * jayschmidtjoined
01:04:15  * jayschmidtquit (Client Quit)
01:06:51  * pwhackquit (Quit: Textual IRC Client: www.textualapp.com)
01:20:01  * ed209quit (Remote host closed the connection)
01:20:08  * ed209joined
01:20:53  * bahamatquit (Quit: Leaving.)
01:21:19  * bahamatjoined
02:00:10  * yruss972quit (Remote host closed the connection)
02:04:50  * jayschmidtjoined
02:04:51  * jayschmidtquit (Client Quit)
02:14:14  * dap_quit (Quit: Leaving.)
02:57:43  * bahamatquit (Quit: Leaving.)
03:21:31  * wills64quit (Quit: ZNC - http://znc.in)
03:22:29  * wills64joined
03:31:26  * jhendricksquit (Quit: Leaving.)
03:51:31  * trentmquit (Quit: Leaving.)
03:55:19  * pmooneyquit (Ping timeout: 258 seconds)
04:41:26  * mellocjoined
04:49:18  * jhendricksjoined
04:49:33  * mellocquit (Quit: Leaving.)
05:20:07  * pmooneyjoined
05:45:25  * pmooneyquit (Ping timeout: 260 seconds)
05:50:11  * trentmjoined
06:24:02  * trentmquit (Quit: Leaving.)
06:50:54  * yruss972joined
07:02:17  * yruss972quit (Remote host closed the connection)
08:02:04  * yruss972_joined
08:02:20  * yruss972_quit (Remote host closed the connection)
08:09:41  * yruss972joined
10:20:01  * ed209quit (Remote host closed the connection)
10:20:07  * ed209joined
11:22:21  * yruss972quit (Remote host closed the connection)
11:28:41  * yruss972joined
12:22:37  * jayschmidtjoined
12:31:31  * yruss972quit (Remote host closed the connection)
12:33:45  * yruss972joined
13:06:08  * yruss972quit (Remote host closed the connection)
13:09:44  * yruss972joined
13:42:32  * yruss972_joined
13:46:51  * yruss972quit (Ping timeout: 268 seconds)
13:49:59  * yruss972joined
13:50:38  * yruss972_quit (Read error: Connection reset by peer)
14:10:35  * jhendricksquit (Quit: Leaving.)
14:15:25  * chorrelljoined
14:39:24  * mhicksjoined
14:40:05  * pmooneyjoined
15:11:20  * yruss972_joined
15:12:33  * yruss972_quit (Remote host closed the connection)
15:13:09  * yruss972_joined
15:14:43  * yruss972quit (Ping timeout: 260 seconds)
15:17:11  * yruss972_quit (Ping timeout: 246 seconds)
15:22:48  * dap_joined
15:23:57  * mellocjoined
15:28:10  * jhendricksjoined
15:33:21  * bahamatjoined
15:38:20  * elijahZ241joined
15:38:20  * elijahZ24quit (Read error: Connection reset by peer)
15:52:26  * trentmjoined
15:54:07  * mellocquit (Read error: Connection reset by peer)
15:54:13  * bahamatquit (Quit: Leaving.)
15:54:21  * mellocjoined
16:24:37  * melloc1joined
16:24:38  * mellocquit (Read error: Connection reset by peer)
16:40:13  * yruss972joined
16:43:28  * chorrellquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
16:44:37  * chorrelljoined
16:49:29  * chorrellquit (Ping timeout: 260 seconds)
16:54:52  * mellocjoined
16:54:52  * melloc1quit (Read error: Connection reset by peer)
17:01:49  * yruss972quit (Remote host closed the connection)
17:11:10  * elijahZ241quit (Quit: Leaving.)
17:11:15  * elijahZ24joined
17:17:10  * chorrelljoined
17:25:05  * melloc1joined
17:25:05  * mellocquit (Read error: Connection reset by peer)
17:42:52  * chorrellquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
17:46:28  * chorrelljoined
17:55:22  * mellocjoined
17:55:22  * melloc1quit (Read error: Connection reset by peer)
18:00:07  * jhendricksquit (Quit: Leaving.)
18:03:02  * dap_1joined
18:03:13  * wizard113_joined
18:05:38  * dap_quit (Ping timeout: 260 seconds)
18:07:57  * jhendricksjoined
18:25:39  * melloc1joined
18:25:39  * mellocquit (Read error: Connection reset by peer)
18:32:06  <wizard113_>howdy, I am getting an 'InternalError: an unexpected error occurred' from manatee, at non-deterministic intervals, which makes it hard to catch with dtrace/bunyan.
18:32:43  <wizard113_>Is there a way to turn on debugging for the manta stack, so that I can grab the stored logs once I see this happen?
18:43:42  <wizard113_>gah, muskie, not manatee.
18:54:13  <melloc1>wizard113_: You can modify the config template inside the zone, and restart config-agent.
18:54:17  * melloc1changed nick to melloc
18:54:36  <dap_1>Also, all of the logs are uploaded to Manta.
18:54:42  <melloc>Specifically, the "level": "info" line.
18:54:54  <dap_1>If you need more verbose logs, that itself is likely a bug.
18:56:42  <wizard113_>The logs I am looking at for muskie on my manta instance don't indicate the cause of the "InternalError". I am not yet good at debugging through the manta stack to find the correlations.
18:57:03  <wizard113_>would this be the right tmie for me to be running an mjob to dig through all those logs?
18:57:08  <dap_1>There should at least be a stack trace in them
18:57:28  <dap_1>Yes, maybe with mlogin.
19:11:58  <wizard113_>OK, forgot I had done this monday. The stack trace is there, it leads to something going wrong in moray.js. Was then trying to figure out how to drill down through other logs and got distracted.
19:12:13  <wizard113_>Here is the stack trace from the latest err: https://gist.github.com/pccowboy/a5d9648bd1a46095226ac70af69db395
19:14:49  <dap_1>There may be a log entry slightly earlier than that one that has more details. You might try to grep for the request_id from the entry you pasted.
19:15:25  * jhendricksquit (Quit: Leaving.)
19:18:54  <wizard113_>cool, I should have just tried that, thank you. Kinda awesome, I could just kickoff a job to grab all these msgs when I catch an error, rather than stress out my ELK instance.
19:29:17  * chorrellquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
19:29:18  * jayschmidtquit (Quit: Leaving.)
19:29:21  * jayschmidt1joined
19:29:22  * jayschmidt1quit (Client Quit)
19:29:44  * jayschmidtjoined
19:49:51  <wizard113_>OK, got it, I think. There is a query timeout on the commit in the moray client, probably an artifact of my lab setup.
19:50:05  <wizard113_>I updated https://gist.github.com/pccowboy/a5d9648bd1a46095226ac70af69db395 with two more traces
19:51:40  <wizard113_>I probably need to bump up cpu shares/cap my primary postgres instance to get past this, I'd expect
19:52:35  <wizard113_>a totally different question then comes up - I have an SSD array that I could delegate a dataset on, for use by postgres.
19:53:03  <wizard113_>I imagine that would play havoc with manta upgrades, or some other portion of manta.
19:54:07  <wizard113_>dap_1: Any thoughts about putting a manta's postgres dataset on an SSD dataset that was added after triton installation?
19:56:24  * melloc1joined
19:56:24  * mellocquit (Read error: Connection reset by peer)
20:03:37  <dap_1>The timeout is 30s, so that's pretty long. You might want to check the health of the Manatee shards by using "manatee-adm show -v" on one of the shards
20:04:06  <dap_1>As for the SSD question: Triton really assumes there's only one storage pool and it's called "zones"
20:04:48  * yruss972joined
20:09:27  * jhendricksjoined
20:09:35  * yruss972quit (Ping timeout: 240 seconds)
20:18:00  * bahamatjoined
20:19:27  <wizard113_>primary and sync show ok, my current lab incarnation does not have an async.
20:19:29  <wizard113_>From prior experience with my rig, I'm gonna bump cpu_shares to 128 on these two shards and see if I get past the timeout. My lab rig is a bit out-of-spec, in that I run everything on one host (6core xeon, 128gb ecc ram), which has given me some fits in the past.
20:20:00  * ed209quit (Remote host closed the connection)
20:20:07  * ed209joined
20:20:13  <wizard113_>Thank you very much for the help and advice today, what I learned about logging on manta is going to allow me to drop a bunch of code from my app
20:24:10  <dap_1>wizard113_: Cool. FWIW, we do run test deployments on single systems as well, so that works fine if you've got enough resources. I've done it on 48GB DRAM and 16 cores.
20:24:14  <dap_1>Glad to help!
20:25:08  <wizard113_>hmm, actually - In a Joyent DC, is there a method that a customer can access muskie logs that are generated by their own traffic, or are the muskie logs in production not accessible to a customer?
20:25:52  <melloc1>wizard113_: They're not accessible to customers, only operators.
20:25:56  * melloc1changed nick to melloc
20:26:30  <wizard113_>ok, cool. so I can switch my logging based on whether I am running in my lab, or in prod, then.
20:26:55  <wizard113_>good while I am developing that I don't have to clutter my logs wit so much debug when it is all stored for me
20:27:30  <dap_1>wizard113_, melloc: Your logs should be available in /$MANTA_USER/reports
20:27:56  <dap_1>It's just the access logs, though, not all the debug logging.
20:28:19  <wizard113_>ok, cool. That makes sense.
20:29:18  <wizard113_>By the time I am running in prod, I better have my debugging done to some degree, at least on how I am using Triton-provided services :-)
20:33:02  * bahamatquit (Quit: Leaving.)
20:43:58  * chorrelljoined
20:52:08  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
21:25:48  * elijahZ24quit (Quit: Leaving.)
21:27:33  * elijahZ24joined
21:30:59  * jayschmidtquit (Quit: Leaving.)
21:31:01  * jayschmidt1joined
21:31:05  * jayschmidt1quit (Client Quit)
21:31:21  * jayschmidtjoined
21:38:37  * elijahZ24quit (Quit: Leaving.)
21:38:44  * elijahZ24joined
22:11:20  * elijahZ24quit (Quit: Leaving.)
22:11:25  * elijahZ241joined
22:27:39  * melloc1joined
22:27:39  * mellocquit (Read error: Connection reset by peer)
22:52:06  * jhendricksquit (Quit: Leaving.)
22:57:58  * mellocjoined
22:57:58  * melloc1quit (Read error: Connection reset by peer)
23:17:14  * jhendricksjoined