11:47:02  * rmustacctopic: Manta: Big Data Unix | Now Open Source! -- https://github.com/joyent/manta | http://apidocs.joyent.com/manta/ | http://logs.libuv.org/manta/latest
12:53:26  * bahamatquit (Quit: Leaving.)
14:01:11  * pmooneyjoined
14:04:31  * kkantorjoined
14:28:52  * neuroservequit (Ping timeout: 260 seconds)
14:45:04  * jhendricksjoined
15:20:47  * jhendricksquit (Quit: Leaving.)
15:56:58  * jhendricksjoined
15:57:00  * jhendrickspart
16:05:21  * pmooneyquit (Read error: Connection reset by peer)
16:06:02  * wizard113joined
16:07:35  <wizard113>today's theater of mistakes entry: I used 'vmadm delete' to remove a marlin zone, instead of 'manta-undeploy'.
16:07:59  <wizard113>Now I have what appears to be a permanent disabled marlin zone in the marlin dashboard
16:08:15  <wizard113>jobs run on the new zones just fine, but I'd like to remove that entry
16:08:35  <wizard113>Do I need to poke around somewhere besides moray to remove it?
16:09:45  <wizard113>I walked through the manta-undeploy code, and manually executed all the steps after it threw an error, so I think it should be gone
16:09:46  * pmooneyjoined
16:17:53  * neuroservejoined
16:27:39  * bahamatjoined
16:31:11  <melloc>wizard113: You'll need to remove the instance from SAPI, I believe
16:31:24  <melloc>For example: sdc-sapi /instances/<uuid> -X DELETE
16:31:59  <melloc>Although there used to be an issue with SAPI trying to delete instances that had been removed via `vmadm delete`
16:32:13  <melloc>Hopefully that's been fixed now, though.
16:34:21  <wizard113>melloc: It appears to be fixed? I did remove it from sapi (that was where it errored out earlier today), and I just double-checked, sapi reports 'no such instance'.
16:36:04  <melloc>Oh, was it already out of SAPI when the Marlin dashboard was still showing?
16:36:59  <melloc>* showing it
16:36:59  <wizard113>Yes
16:37:28  <wizard113>I rebooted the marlin-dashboard zone, and the job supervisor zone, just in case it was a ghost
16:37:48  <wizard113>but the dashboard still shows it, about 5 hours later
16:45:13  <wizard113>melloc: sorry, train went under the bay. Yes, it was still showing in the dashboard after I manually removed it from sapi
16:45:41  * wizard113quit (Quit: Leaving)
17:31:26  <neuroserve>can somebody help with a multi-dc manta deployment (in my lab)?
17:33:50  <neuroserve>i have three AZs - one master, two slaves - i have multi-dc dns setup and i have one master-sapi zone
17:34:32  <neuroserve>i didn't know, that i should create the poseidon user in the master-dc first
17:35:05  <neuroserve>now i have two different poseidon users (because i started the deployment in one of the slave-dcs)
17:36:20  <neuroserve>should i do sdc-ufds-m2s in the respective slave-dc to get rid of the "wrong" poseidon user?
17:50:01  <neuroserve>just realized, that sdc-ufds-m2s is not possible: Error: already converted :-(
18:25:05  * jhendricksjoined
19:17:45  * kkantorquit (Quit: kkantor)
19:52:43  <bahamat>neuroserve: Can you check the status of the ufds-replicator service in the ufds zone for the broken DC?
19:53:22  * kkantorjoined
19:57:42  <neuroserve>bahamat: in maintenance - restarting too quickly - the logs show the problem, that there are two poseidons now
19:58:57  <bahamat>neuroserve: Ok, that's what I would expect. You'll need to use sdc-ldap on the headnode of the problem datacenter to delete the poseidon user then clear ufds-replicator.
19:59:49  <neuroserve>bahamat: tried that - but sdc-ldap del login=poseidon says: not implemented
20:01:28  <bahamat>You'll need to do it by the dn
20:01:34  <neuroserve>ah ok
20:01:48  <bahamat>neuroserve: FWIW, this is MANTA-3246.
20:01:50  <jinni8>https://smartos.org/bugview/MANTA-3246
20:02:09  * jinni8quit (Remote host closed the connection)
20:02:18  * jinnijoined
20:04:00  <bahamat>actually, this is MANTA-2116
20:04:00  <jinni>Sorry, MANTA-2116 is not public.
20:04:33  <bahamat>But they're very similar. In both cases, the poseidon user is being created/updated only in the local DC, which is wrong.
20:09:07  <neuroserve>bahamat: hm - seems not so easy to get rid of that
20:11:45  <neuroserve> sdc-ldap delete dn="uuid=fc10f8cd-6335-66fe-9aff-98754d90ab8a, ou=users, o=smartdc" - no such object
20:20:01  * ed209quit (Remote host closed the connection)
20:20:08  * ed209joined
20:24:44  * kkantorquit (Quit: kkantor)
20:25:53  * kkantorjoined
20:27:51  <neuroserve>worked - replication has replicated the right poseidon now
20:32:47  <bahamat>neuroserve: You should be able to continue with Manta setup now.
20:34:54  <neuroserve>bahamat: yep - i'm running manta-create-topology.sh now in the master-dc - but how do i proceed in the slave-dcs?
20:47:09  <bahamat>You only need to create the topology once
20:49:25  <neuroserve>bahamat: i learned, that manta-sharadm is used only once, manta-create-topology once - what about manta-adm genconfig and manta-adm update?
20:49:48  <neuroserve>i guess, these have to be executed in each dc, right?
20:50:08  <bahamat>genconfig is only used during setup, but update is used in all DCs any time you want to update manta.
20:51:45  <neuroserve>hm - the generated json-file for importing the hash-ring still has the 2nd poseidon uuid in it - manifest data is not valid JSON - wonder where this still comes from
21:00:20  <neuroserve>hm - sapi still has the two uuids for poseidon
21:03:51  <bahamat>neuroserve: You should only have the manta application in the primary datacenter.
21:09:08  <neuroserve>bahamat: true - but i first started the installation in the slave-dc and then in the master-dc
21:09:33  <bahamat>Do you have a significant amount of usage on triton itself in those datacenters?
21:10:01  <neuroserve>bahamat: in the master-dc - slave-dc is disposable
21:10:21  <neuroserve>but its only lab usage
21:12:25  <neuroserve>can't i just delete the wrong sapi entry in the master-dc?
21:22:40  <bahamat>neuroserve: Possibly. I've never done that though.
21:24:33  <neuroserve>hm - it says sdc-sapi /applications?uuid=1a829001-b4b3-4b56-be48-2c2103be266d -X DELETE - HTTP/1.1 405 Method Not Allowed
21:33:56  <neuroserve>don't want so setup the dcs again ...
21:49:39  * neuroservequit (Ping timeout: 252 seconds)
21:57:28  * pmooneyquit (Quit: WeeChat 1.7.1)
23:48:54  * johngrastyquit (Ping timeout: 240 seconds)
23:48:55  * johngrastyjoined
23:51:58  * ed209quit (Ping timeout: 240 seconds)