00:04:12  <bahamas10>PSA: any manta-sync users, if you upgrade to v0.3.0 you can do "reverse" sync with `-r`... synchronize from manta to local
00:18:29  * trentmjoined
01:00:47  * ryancnelsonquit (Ping timeout: 250 seconds)
01:19:58  * dap_quit (Quit: Leaving.)
01:20:00  * ed209quit (Remote host closed the connection)
01:20:07  * ed209joined
01:47:51  * chorrelljoined
01:48:55  <nahamu>nice!
02:12:25  * pmooneyquit (Ping timeout: 255 seconds)
02:14:39  * marsellquit (Ping timeout: 265 seconds)
02:35:56  * therealkoopaquit (Remote host closed the connection)
02:37:22  * chorrellquit (Quit: My Mac has gone to sleep. ZZZzzz…)
02:57:28  * happy-dudejoined
03:06:42  * trentmquit (Quit: Leaving.)
03:10:40  * therealkoopajoined
03:13:16  * pmooneyjoined
03:17:51  * pmooneyquit (Ping timeout: 250 seconds)
03:21:07  * therealkoopaquit (Ping timeout: 245 seconds)
04:03:54  * therealkoopajoined
04:08:31  * therealkoopaquit (Ping timeout: 255 seconds)
04:11:39  * therealkoopajoined
04:14:40  * trentmjoined
04:26:27  * therealkoopaquit (Ping timeout: 264 seconds)
04:50:44  * trentmquit (Quit: Leaving.)
04:56:18  * trentmjoined
05:04:35  * therealkoopajoined
05:09:13  * therealkoopaquit (Ping timeout: 250 seconds)
05:11:49  * therealkoopajoined
05:19:52  * therealkoopaquit (Ping timeout: 240 seconds)
05:25:53  * therealkoopajoined
05:30:25  * therealkoopaquit (Ping timeout: 255 seconds)
05:36:45  * therealkoopajoined
05:41:27  * therealkoopaquit (Ping timeout: 264 seconds)
05:54:40  * happy-dudequit (Quit: Connection closed for inactivity)
05:57:01  * therealkoopajoined
06:04:32  * therealkoopaquit (Ping timeout: 252 seconds)
06:11:56  * therealkoopajoined
06:16:13  * therealkoopaquit (Ping timeout: 245 seconds)
06:33:47  * trentmquit (Quit: Leaving.)
06:50:19  * therealkoopajoined
06:54:59  * therealkoopaquit (Ping timeout: 265 seconds)
07:00:21  * therealkoopajoined
07:05:08  * therealkoopaquit (Ping timeout: 265 seconds)
07:11:38  * therealkoopajoined
07:18:45  * therealkoopaquit (Ping timeout: 256 seconds)
07:34:10  * therealkoopajoined
07:40:28  * therealkoopaquit (Ping timeout: 255 seconds)
07:42:13  * therealkoopajoined
07:51:44  * therealkoopaquit (Ping timeout: 264 seconds)
08:00:13  * therealkoopajoined
08:07:00  * therealkoopaquit (Ping timeout: 252 seconds)
08:42:46  * therealkoopajoined
08:47:44  * therealkoopaquit (Ping timeout: 245 seconds)
09:23:42  * therealkoopajoined
09:29:19  * therealkoopaquit (Ping timeout: 244 seconds)
10:20:01  * ed209quit (Remote host closed the connection)
10:20:08  * ed209joined
10:20:27  * pgalejoined
10:31:02  * therealkoopajoined
10:35:31  * therealkoopaquit (Ping timeout: 250 seconds)
10:36:26  * therealkoopajoined
10:41:13  * therealkoopaquit (Ping timeout: 264 seconds)
10:54:40  * therealkoopajoined
11:00:18  * marselljoined
11:02:27  * therealkoopaquit (Ping timeout: 265 seconds)
11:04:10  * pgalequit (Quit: Leaving.)
11:11:38  * therealkoopajoined
11:17:09  * therealkoopaquit (Ping timeout: 252 seconds)
11:30:57  * therealkoopajoined
11:35:23  * therealkoopaquit (Ping timeout: 244 seconds)
11:49:19  * therealkoopajoined
11:50:38  * xmerlinjoined
11:53:58  * therealkoopaquit (Ping timeout: 252 seconds)
11:59:28  * therealkoopajoined
12:04:15  * therealkoopaquit (Ping timeout: 264 seconds)
12:05:56  * therealkoopajoined
12:13:19  * {slurp}1joined
12:13:19  * rmustacctopic: Manta: Big Data Unix | Now Open Source! -- https://github.com/joyent/manta | http://apidocs.joyent.com/manta/ | http://logs.libuv.org/manta/latest
12:16:10  * nahamu_joined
12:17:22  * therealkoopaquit (Ping timeout: 240 seconds)
12:18:19  * goberle_joined
12:20:55  * {slurp}quit (*.net *.split)
12:20:56  * nahamuquit (*.net *.split)
12:20:56  * _Tenchi_quit (*.net *.split)
12:20:57  * bahamas10quit (*.net *.split)
12:20:57  * goberlequit (*.net *.split)
12:27:10  * rmustacctopic: Manta: Big Data Unix | Now Open Source! -- https://github.com/joyent/manta | http://apidocs.joyent.com/manta/ | http://logs.libuv.org/manta/latest
12:30:22  * therealkoopajoined
12:32:07  * bahamas10joined
13:08:44  * happy-dudejoined
14:02:46  * chorrelljoined
14:26:32  * nahamu_changed nick to nahamu
14:38:48  * ffahimijoined
14:55:04  * pgalejoined
14:58:13  * pmooneyjoined
15:16:42  * pgalequit (Quit: Leaving.)
15:26:15  * _Tenchi_joined
15:40:41  * ffahimiquit (Remote host closed the connection)
15:41:08  * ffahimijoined
15:41:16  * pgalejoined
15:48:46  * pgalequit (Quit: Leaving.)
16:06:58  * pgalejoined
16:08:32  * ffahimiquit (Remote host closed the connection)
16:17:47  * pgalequit (Quit: Leaving.)
16:42:25  * trentmjoined
16:43:54  * trentmquit (Client Quit)
16:45:18  * trentmjoined
17:07:57  * marsellquit (Quit: marsell)
17:22:25  * dap_joined
17:54:11  * marselljoined
18:12:25  * pgalejoined
18:21:13  * lloyddejoined
18:21:18  * lloyddequit (Remote host closed the connection)
18:21:31  * lloyddejoined
18:31:48  * ryancnelsonjoined
18:55:50  * pgalequit (Quit: Leaving.)
19:09:27  * ffahimijoined
19:13:43  * ffahimiquit (Ping timeout: 245 seconds)
19:47:57  * pmooneyquit (Quit: coffee)
20:13:09  * pmooneyjoined
20:20:00  * ed209quit (Remote host closed the connection)
20:20:07  * ed209joined
20:45:42  * ffahimijoined
21:04:27  * pmooneyquit (Quit: WeeChat 1.1.1)
21:07:43  * pmooneyjoined
21:31:17  * ffahimiquit (Remote host closed the connection)
21:37:28  * namtziglajoined
21:42:25  * ffahimijoined
21:44:32  <namtzigla>hi everyone
21:45:06  <namtzigla>I have a strange error when I'm attempting deploying manta on two nodes
21:45:06  <namtzigla>manta-adm: first of 2 errors: job provision-7.1.6 (1352ba21-10c0-43e1-b8af-0d6b310c734b) failed: cnapi.check_manual_server_nics: Manual server does not meet NIC Tag requirements
21:52:39  <dap_>namtzigla: Are you using manta-deploy-lab?
21:53:33  <namtzigla>no, I'm trying to production path, I've created a config for two nodes
21:54:17  <dap_>I think the networking step may have to be completed on each of the nodes you intend to deploy Manta components on. Sorry, that's not clear in the docs, and I'm not sure you want exactly the same script.
21:55:06  <dap_>For background: It sounds like the problem is that there's no manta0 nic tag on your second node. For single-node deployments, that gets set up on the single node by running that networking script.
21:55:15  <namtzigla>oh
21:55:15  <namtzigla>ok
21:55:17  <namtzigla>thanks
21:55:33  <dap_>Actually, the script's documentation claims that it sets up all the nodes in the DC...
21:55:39  <dap_>https://github.com/joyent/sdc-manta/blob/master/networking/manta-net.sh#L13-L15
21:55:43  <dap_>Did you run it before setting up the second CN?
21:55:49  <namtzigla>no
21:56:03  <namtzigla>I had the nodes done before
21:56:46  <dap_>Okay. It looks like the script figures out which nodes to set up based on the config file that you give it.
21:56:52  <dap_>Did that file contain mac_mappings for both nodes?
21:57:13  <namtzigla>yes
21:57:29  <dap_>Oh, that's surprising.
21:57:41  <dap_>Is this a two-node SDC install?
21:57:44  <namtzigla>I've executed the script only on headnode
21:57:54  <namtzigla>except the node head yes
21:57:57  <dap_>That should be fine, actually. I was wrong about thinking it needed to be run on each node.
21:58:02  <dap_>can you try this:
21:58:13  <dap_>sdc-oneachnode 'ifconfig -a' ?
21:58:22  <namtzigla> I have a manta0 interface on both servers
21:58:29  <dap_>Okay, that's what I was trying to figure out.
21:59:16  <namtzigla>=== Output from ac3c0f53-2cac-e411-8694-e1ff4e82aa92 (a0-36-9f-1f-69-4a): igb0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 1 inet 10.31.5.46 netmask ffffff00 broadcast 10.31.5.255 ether a0:36:9f:1f:69:4a lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 2 inet 127.0.0.1 netmask ff000000 manta0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 3
21:59:32  <namtzigla>oh
21:59:33  <namtzigla>sorry
21:59:53  <namtzigla>I don't have a mantanat ointerface on either
22:00:15  <dap_>I think that's fine
22:00:41  <dap_>We should start from the top. You should be able to get details about the failed provision with something like this on the HN:
22:00:46  <dap_>sdc-workflow /jobs?limit=2
22:00:58  <dap_>That will show info about the last two workflows, which are probably your last two provisions
22:01:25  <namtzigla>the output is a bit long
22:01:36  <namtzigla>do you want it all ?
22:01:37  <dap_>Yeah. If you're okay with posting that somewhere, you could gist it up. Otherwise, you want to see if you can figure out which CN it tried to provision to
22:01:43  <namtzigla>k
22:01:44  <namtzigla>1 sec
22:01:47  <dap_>And which nic tags it wanted to be present.
22:02:20  <namtzigla>http://pastebin.com/zhsz75yr
22:03:01  <namtzigla>on both nodes same error
22:03:44  <dap_>Can you also paste "sdc-network list"?
22:04:04  <dap_>The list of networks it wanted for that zone is at lines 75-86 in your gist, but I want to map those uuids to names.
22:04:44  <namtzigla>sure
22:04:45  <namtzigla>https://gist.github.com/namtzigla/9fbb81ec7f705dfde544
22:05:22  <dap_>So it looks like it's external (primary), then manta, and then admin.
22:06:01  <namtzigla>yes
22:06:21  <dap_>and we tried to provision it to server ac3c0f53-2cac-e411-8694-e1ff4e82aa92
22:07:06  <dap_>how about this: sdc-oneachnode -n ac3c0f53-2cac-e411-8694-e1ff4e82aa92 'nictagadm list'
22:07:53  <namtzigla>https://gist.github.com/namtzigla/42adee42540c9dfa0946
22:08:13  <dap_>Hmm…no external.
22:08:13  * pmooneyquit (Read error: Connection reset by peer)
22:08:25  <dap_>I suspect that's the problem, but I'm not sure why it's not there.
22:08:30  <namtzigla>I did not provision any vm's there
22:08:38  <dap_>Well, manta-adm tried to
22:08:49  <namtzigla>until now the external showed up every time we provision 1st vm
22:09:05  * pmooneyjoined
22:09:50  <dap_>Interesting. We're now beyond my knowledge of SDC networking config, but let me see if I can pull in someone who knows more.
22:10:04  <namtzigla>sure, thanks
22:12:21  <namtzigla>ok ... that was it
22:12:24  <namtzigla>@dap_
22:12:45  <namtzigla>we created the external0 and it seems that it's provisioning now
22:12:50  <dap_>how'd you do that?
22:13:22  <namtzigla>I've added at nic tags in the CN
22:13:33  * sigxcpujoined
22:14:34  <dap_>Cool.
22:14:44  <dap_>This looks like a missing piece of documentation — sorry about that!
22:15:39  <namtzigla>no problem
22:15:45  <namtzigla>thanks a lot for the debug
22:16:18  <dap_>No problem!
22:16:32  <dap_>We don't hit it very often because we don't set up new, multi-CN install that often. Our test systems are largely either single-CN or not reinstalled very often
22:17:55  <namtzigla>do you guys offer manta training ?
22:18:44  * chorrellquit (Quit: My Mac has gone to sleep. ZZZzzz…)
22:19:41  <dap_>That's a good question, but I'm not the person to ask about that :)
22:19:46  <namtzigla>ok
22:19:48  <namtzigla>np
22:27:39  <dap_>If you care, I've filed https://smartos.org/bugview/MANTA-2575 for the documentation fix. I can probably fix that by the end of today.
22:29:34  <dap_>namtzigla: I'm told that for training, you should contact Jens at mailto:jens@joyent.com.
22:30:14  <namtzigla>great, Thanks a lot
22:31:55  <dap_>no problem
22:42:52  <sigxcpu>dap_: https://github.com/joyent/sdc-manta/blob/master/networking/manta-net.sh#L463-464
22:43:06  <sigxcpu>dap_: i was looking over that script
22:43:19  <sigxcpu>dap_: shouldn't be another | before json -a routes?
22:53:21  <dap_>sigxcpu: Either that, or the second "json" should be removed. I believe the current code works a little by accident because the first "json" parses the "-a" and (correctly) selects the "routes" property of each entry in the array. Basically, the only problem with the current code is that if there were a property called "json" in the output for a network, we'd erroneously grab that one in addition to the "routes" property.
22:54:03  <sigxcpu>dap_: i've tested it myself and it works with or without pipe. I've thought that there is some clever Joyent hack :)
22:54:51  <dap_>It's basically just the fact that jsontool keeps parsing arguments even after properties, and the fact that "json ARGS1 | json -a ARGS2" === "json ARGS1 -a ARGS2"
22:54:57  <dap_>Well, for these ARGS1 and ARGS2
22:55:05  <rmustacc>Sometimes I get lucky!
22:55:15  <rmustacc>I try not to be too clever in scripts like that.
22:57:00  <namtzigla>I just found another thins, when I try to add madtom service to deployed manta I got this message in the log {"name":"manta-adm","hostname":"headnode","pid":40127,"level":20,"cnid":"ac3c0f53-2cac-e411-8694-e1ff4e82aa92","service":"madtom","config":["c7444f0c-a8c3-11e4-ae84-dbdb9b4d7b53"],"wanted":1,"have":0,"delta":1,"msg":"match count in new config","time":"2015-02-04T22:55:41.580Z","v":0}
22:57:33  <namtzigla>"wanted":1,"have":0," with msg":"match count in new config" don't make sens
22:58:00  <namtzigla>and no other error
22:58:39  <dap_>That's not an error. That's just a (somewhat sloppy) message reporting the number of madtom instances desired compared to what's present.
22:58:47  <dap_>https://github.com/joyent/sdc-manta/blob/master/lib/adm.js#L1042-L1067
22:58:58  * sigxcpuquit (Quit: sigxcpu)
22:59:02  <dap_>It always logs that, but it only takes action if the count is non-zero.
22:59:13  <namtzigla>yes but I scify that I need 1 and I have 0
22:59:25  <namtzigla>*specify
22:59:32  <dap_>Did it not try to provision one?
22:59:45  <namtzigla> "madtom": { "c7444f0c-a8c3-11e4-ae84-dbdb9b4d7b53":1 }, "marlin-dashboard":{ "fe52c744-a8c3-11e4-9e8e-835368695f92":1 }
22:59:51  <namtzigla>no
22:59:57  <dap_>What's the next message in the log?
23:00:34  <namtzigla>https://gist.github.com/namtzigla/c4f8b7d189e9ddc48d50
23:00:38  <namtzigla>the whole log
23:01:22  <dap_>That doesn't seem to have the message you pasted above
23:01:28  <namtzigla>so what I did after the manta install was done I've generated the config.json with manta-adm show -s -j and I added there madtom and marlin-dashboard services
23:01:43  <namtzigla>and for both of them I'm getting that message
23:01:54  <dap_>That's the entire log file?
23:02:02  <dap_>Oh, I bet I know what this is.
23:02:33  <dap_>I'll bet your manta deployment zone is missing https://github.com/joyent/sdc-manta/commit/8eaab2149dc1e88b1d7ddc6a03c13bfa0c5c831f
23:02:58  <dap_>It was just an oversight that manta-adm wouldn't deploy those two zones
23:02:59  * ffahimiquit (Remote host closed the connection)
23:03:07  <namtzigla>ok ... so should I update the images
23:03:09  <namtzigla>thanks
23:03:36  <dap_>Yeah, the official path would be to update the manta-deployment zone and do it again. (You'll lose any data stored in the manta-deployment zone)
23:04:20  <namtzigla>ok, thanks for the warning
23:04:22  <dap_>If you want to hack it to test it out, you can just apply the change, which is pretty small… but I don't recommend that on anything other than a dev system because it becomes hard to keep track of what version you actually have.
23:04:41  <dap_>Apparently, updating this zone is simple:
23:04:41  <dap_>http://joyent.github.io/manta/#manta-deployment-zone-upgrades
23:04:45  <dap_>I've never actually used that procedure :)
23:04:54  <dap_>Sorry for the bug.
23:12:38  <namtzigla>yeah, that I was going to say
23:12:39  <namtzigla>:)
23:44:32  * namtziglaquit (Ping timeout: 264 seconds)