00:18:27  * fredkquit (Quit: Leaving.)
01:04:05  * eVikingquit (Remote host closed the connection)
01:17:39  * bahamatquit (Quit: Leaving.)
01:20:00  * ed209quit (Remote host closed the connection)
01:20:07  * ed209joined
01:24:32  * bahamatjoined
02:04:48  * eVikingjoined
02:13:25  * eVikingquit (Ping timeout: 256 seconds)
02:29:02  * bahamatquit (Quit: Leaving.)
03:09:34  * eVikingjoined
03:14:03  * eVikingquit (Ping timeout: 256 seconds)
03:16:12  * bahamatjoined
03:48:42  * bahamatquit (Quit: Leaving.)
04:11:10  * eVikingjoined
04:17:33  * eVikingquit (Ping timeout: 256 seconds)
04:26:03  * bahamatjoined
04:39:47  * bahamatquit (Quit: Leaving.)
04:47:24  * marsellquit (Ping timeout: 256 seconds)
05:10:23  * pmooneyjoined
05:10:42  * eVikingjoined
05:15:36  * eVikingquit (Ping timeout: 264 seconds)
06:03:00  * cxe1joined
06:09:04  * cxe1quit (Read error: Connection reset by peer)
06:09:21  * cxe1joined
06:11:16  * eVikingjoined
06:13:48  * bahamatjoined
06:15:49  * eVikingquit (Ping timeout: 248 seconds)
06:16:56  * pmooneyquit (Ping timeout: 256 seconds)
06:25:17  * pmooneyjoined
06:25:58  * bahamatquit (Quit: Leaving.)
06:40:56  * pmooneyquit (Ping timeout: 265 seconds)
06:50:39  * pmooneyjoined
07:11:48  * eVikingjoined
07:16:26  * eVikingquit (Ping timeout: 256 seconds)
07:58:53  * _Tenchi_quit (Ping timeout: 250 seconds)
08:12:22  * eVikingjoined
08:16:39  * eVikingquit (Ping timeout: 256 seconds)
09:12:58  * eVikingjoined
09:17:33  * eVikingquit (Ping timeout: 244 seconds)
10:13:32  * eVikingjoined
10:18:36  * eVikingquit (Ping timeout: 272 seconds)
10:20:00  * ed209quit (Remote host closed the connection)
10:20:08  * ed209joined
10:37:18  * pmooneyquit (Ping timeout: 250 seconds)
10:55:09  * marselljoined
11:14:05  * eVikingjoined
11:18:43  * eVikingquit (Ping timeout: 255 seconds)
11:23:21  * pmooneyjoined
12:14:40  * eVikingjoined
12:19:08  * eVikingquit (Ping timeout: 250 seconds)
13:01:34  * chorrelljoined
13:05:37  * _Tenchi_joined
13:06:44  * _Tenchi_quit (Client Quit)
13:07:17  * _Tenchi_joined
13:12:31  * happy-dudejoined
13:15:18  * eVikingjoined
13:19:49  * eVikingquit (Ping timeout: 256 seconds)
13:21:56  * pmooneyquit (Ping timeout: 244 seconds)
13:29:38  * pmooneyjoined
14:15:51  * eVikingjoined
14:20:19  * eVikingquit (Ping timeout: 244 seconds)
14:37:47  <cxe1>minfo return "503 Service Unavaliable". All services are running and loadbalancer is listening on ports 443 and 80
14:38:01  <cxe1>what can I check?
14:41:54  <jayschmidt>do you have a madtom instance deployed? You could check that
14:42:21  <jayschmidt>https://github.com/joyent/manta/blob/master/docs/manta-ops.md#madtom-dashboard-service-health
14:48:30  <cxe1>yes I do.
14:48:41  <jayschmidt>Is it showing all green?
14:49:33  <cxe1>it is on the internal network. Need to build an ssh tunnel from the headnode
14:49:39  <jayschmidt>ah, ok.
14:50:01  <cxe1>I have two. Which one of them active?
14:50:23  <jayschmidt>you should only have one madtom.
14:50:46  <jayschmidt>But I think if you have two they will both probably do the same checks - I've just not run in that configuration.
14:51:19  <jayschmidt>fwiw, I use this stanza to connect to mine in .ssh/config
14:51:22  <jayschmidt>Host madtom-dashboard.test
14:51:22  <jayschmidt> User root
14:51:22  <jayschmidt> HostName 172.26.5.4
14:51:22  <jayschmidt> LocalForward 6681 172.25.5.53:80
14:51:22  <jayschmidt> ServerAliveInterval 300
14:53:05  <cxe1>I don't see madtom listening on port 80
14:53:47  <jayschmidt>if you manta-login to madtom can you curl against localhost:80?
14:54:38  <jayschmidt>Also, I would check svcs inside that zone (madtom) to make sure everything is running.
14:56:09  * bahamatjoined
15:15:31  * bahamatquit (Quit: Leaving.)
15:16:24  * eVikingjoined
15:19:12  <cxe1><jayschmidt> how are you using this .ssh/config?
15:20:48  * eVikingquit (Ping timeout: 252 seconds)
15:43:45  * pmooneyquit (Ping timeout: 256 seconds)
15:58:48  * pmooneyjoined
15:59:25  * bahamatjoined
16:08:08  * fredkjoined
16:16:59  * eVikingjoined
16:21:16  * eVikingquit (Ping timeout: 240 seconds)
16:23:32  <jayschmidt><cxe1> - I have stanzas for all the hosts I connect to, including those I do port forwarding. This way I just ssh to madtom-dashboard.test and it sets local 6681 to point to 80 on my madtom. Then I can just https://localhost:6681. 172.26.5.4 is my headnode, the other my madtom zone. I use the same bit to get to the marlin-dash.
16:24:28  <cxe1>why not to run madtom-dashboard on the HN?
16:26:59  <jayschmidt>Doesn't matter where you run it - you just need to be able to connect to it from your browser.
16:29:21  * eVikingjoined
16:30:53  <cxe1><jayschmidt> can I show you my config for sanity check?
16:31:07  <jayschmidt>sure
16:31:53  <cxe1><jayschmidt> what is the "acceptable" way to post it?
16:32:47  <jayschmidt>throw it in a gist and link it here.
16:36:33  <cxe1>https://gist.github.com/ceizner
16:36:46  <cxe1>Oops
16:36:47  <cxe1>https://gist.github.com/ceizner/d57bd5b1e03519601dcb#file-gistfile1-txt
16:38:29  <cxe1>here are the servers: https://gist.github.com/ceizner/2b5a12603f68fcb7b36a
16:41:58  <jayschmidt>How did you come up with the # of components that you used?
16:42:18  <cxe1>two commute nodes are identical
16:42:25  <cxe1>compute
16:42:42  <cxe1>shall I remove name service from one or make it an odd number in total?
16:44:10  <jayschmidt>What I normally do when setting up a deployment is to start with one of the pre-generated configs, or follow along in the deployment guide.
16:44:41  <jayschmidt>But yes, I would go with an odd number of nameservices.
16:45:40  <cxe1>you mean you start with the lab config and then move services?
16:46:24  <jayschmidt>That's one way to do it. When I was learning manta, I went through a number of deployments - started with everything on one node, clearing everything off, adding in new nodes.
16:46:59  <jayschmidt>So, start with the bare minimum config, validate that it works.
16:47:09  <jayschmidt>Then reset your gear and add on to that config.
16:47:42  <cxe1>good idea. So basically everything would run on the HN in the begining
16:47:53  <jayschmidt>yes
16:48:05  <cxe1>with "lab" config
16:48:15  <jayschmidt>yes.
16:48:19  <cxe1>gotcha
17:00:34  <cxe1>how would you distribute components in the 2-CN,1-SN and 1-HN setup?
17:13:34  * chorrellquit (Quit: My Mac has gone to sleep. ZZZzzz…)
17:16:16  <cxe1>looks like madtom has no ssh service
17:16:30  <cxe1>how one would build a tunnel?
17:49:48  * pmooneyquit (Ping timeout: 264 seconds)
17:51:04  * jayschmidtquit (Quit: Leaving.)
17:54:33  * chorrelljoined
17:54:52  * jayschmidtjoined
18:23:19  <cxe1><jayschmidt> Do you have an example of the single node config?
18:28:50  <jayschmidt><cxe1> - sorry, fixing a few problems right now. What I do is just generate a config for a lab machine (with manta-adm genconfig lab) then I edit the resulting file as required.
18:31:23  <cxe1>I am a bit confused about manhattan. My external network is already noted. Can I just use another range of IP on the external network and use external NIC for mantanat?
18:31:51  <cxe1>sorry "not noted rather natted"
18:36:15  <jayschmidt>manta is rather picky about it's networks. You need an internal manta network and an external mantanat network that is NAT'd. That's in addition to the admin and external networks that SDC requires. As far as manta is concerned, external and mantanat have to be two separate networks within SDC.
18:37:41  <cxe1>understood
18:41:43  <cxe1>so if I need access to manta from nodejs , nodes should have a mantanat NIC. right?
18:42:09  <jayschmidt>no…all your traffic to/from manate will go through the loadbalancer(s) which will have an external IP.
18:42:30  <jayschmidt>mantanat is so the marlins can get to the internet to do thigns like install software, curl against remote hosts, etc.
18:42:44  <cxe1>ah.
18:43:16  <cxe1>for whatever reason my LB were created without an external NIC
18:44:16  <cxe1>will try your way now. Would you be able to review my netconfig.json before I sure up another 4 hours?
19:04:10  <cxe1><jayschmidt> do you use manta-deploy-lab?
19:29:30  * pmooneyjoined
19:29:41  * eVikingquit (Remote host closed the connection)
19:36:33  * chorrellquit (Quit: My Mac has gone to sleep. ZZZzzz…)
19:52:37  * chorrelljoined
20:00:10  * eVikingjoined
20:04:29  <cxe1>very last step of deployment gives an error: Error: findServerUuid failed, invalid number of servers in cnapi
20:04:36  * eVikingquit (Ping timeout: 252 seconds)
20:04:42  <cxe1>I cnapi runnig for manta
20:04:47  <cxe1>I see cnapi runnig for manta
20:15:55  * bsdgurujoined
20:20:01  * ed209quit (Remote host closed the connection)
20:20:08  * ed209joined
20:24:11  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
20:29:43  * eVikingjoined
20:31:02  * pmooneyquit (Ping timeout: 272 seconds)
20:39:52  * eVikingquit (Read error: No route to host)
20:40:24  * eVikingjoined
20:42:46  * chorrelljoined
20:43:52  <cxe1><jayschmidt> very last step of deployment gives an error: Error: findServerUuid failed, invalid number of servers in cnapi
20:46:57  <jayschmidt><cxe1> - on a call so can't really chat, but please check your config - if your first line shows <any> you are going to want to change that to the uuid of the server that will host the components. "any" only works if you have 1 and only 1 node (ie, 1 HN and no CN's).
20:49:19  <cxe1>I do not have any. Used lab config. everything got installed but manta-marlin failing
20:53:17  * chorrell_joined
20:54:52  <jayschmidt>Check the marlin image via sdc-imgapi - make sure that it's got a file and is activated
20:56:32  * chorrellquit (Ping timeout: 250 seconds)
21:00:12  * chorrell_quit (Quit: My Mac has gone to sleep. ZZZzzz…)
21:05:09  <cxe1>yeah. that was done. As I said marlin is working
21:05:34  <cxe1>I can mls, mput mget etc
21:05:44  * bsdguruquit (Quit: bsdguru)
21:07:47  <cxe1>but any compute on the objects failing
21:09:09  <cxe1>storage is showing red for mako and minnow
21:10:48  * bsdgurujoined
21:15:49  <jayschmidt><cxe1> you need to deploy the marlin agents.
21:16:43  <cxe1>tell me more please
21:17:31  <jayschmidt>in your manta deployment zone.
21:17:52  <jayschmidt>(mxpa:manta0) /opt/smartdc/manta-deployment/bin]# ./manta-marlin
21:19:20  <cxe1>ah. found it. this is another image
21:19:53  <cxe1><jayschmidt> this is exactly what is failing
21:21:05  <cxe1>looks like it is require a server UUID
21:21:06  <cxe1>manta-marlin -s 44454c4c-3100-1058-804a-b7c04f4b5331
21:21:35  <jayschmidt>yes, that's the server you want to deploy the agents to.
21:21:55  <cxe1>that work
21:22:06  <cxe1>storage is still showing red
21:22:41  <jayschmidt>if you log into the storage zone, are all the services running correctly?
21:22:51  <cxe1>mput, mfind, mget mls all work
21:22:52  <jayschmidt>"svcs -xvZ" on your node.
21:23:06  <cxe1>from GZ right?
21:23:08  <jayschmidt>yes.
21:23:31  <cxe1>empty
21:24:15  <cxe1>all online inside the storage zones
21:27:31  <jayschmidt>Try restarting one of the storage zones - let's see if that clears your issue.
21:28:37  <cxe1>with " vmadm reboot >
21:28:39  <cxe1>?
21:30:50  <cxe1>didn't help
21:31:18  <cxe1>going to do cleanup again
21:32:12  * bsdguruquit (Ping timeout: 265 seconds)
21:32:17  <cxe1>BTW in your ssh tunnel how do you pass -L and -f and -N?
21:32:25  <cxe1>aka "ssh -L 8080:10.77.77.37:80 -f root@192.168.11.20 -N"
21:36:22  <jayschmidt>If you put it in the .ssh/config file you just need to use the hostname.
21:36:52  <jayschmidt>Host marlin-mxpa
21:36:52  <jayschmidt> User root
21:36:52  <jayschmidt> Hostname mxpa
21:36:52  <jayschmidt> LocalForward 5589 10.0.1.106:80
21:36:52  <jayschmidt> ServerAliveInterval 300
21:36:53  <jayschmidt> StrictHostKeyChecking no
21:36:53  <jayschmidt> UserKnownHostsFile=/dev/null
21:37:03  <jayschmidt>That stanza allows me to
21:37:06  <jayschmidt>ssh marlin-mxpa
21:37:29  <jayschmidt>which forwards port 80 on 10.0.1.106 to my local 5589 through the host called mxpa.
21:40:08  * bsdgurujoined
22:01:59  <cxe1>yes but it is not persistent tunnel. It exists only of you keep terminal open
22:03:13  <cxe1>ssh -L 5589:10.0.1.106:80 -f root@mxpa -N
22:03:21  <cxe1>will do better job in this case
22:06:15  * eVikingquit (Remote host closed the connection)
22:24:39  * eVikingjoined
23:02:57  * eVikingquit (Remote host closed the connection)
23:09:48  <cxe1><jayschmidt> ALL WORKS!!!
23:09:57  <cxe1>EVERYTHING
23:10:43  * marsellquit (Quit: marsell)