00:23:36  <Trott>Something wrong with CI? Can't seem to start any stress jobs.
00:33:39  * Fishrock123quit (Remote host closed the connection)
00:39:59  * Fishrock123joined
00:40:00  * Fishrock123quit (Remote host closed the connection)
01:01:38  * Fishrock123joined
01:01:41  * Fishrock123quit (Remote host closed the connection)
01:16:32  <maclover7>looks like queue is all messed up
01:16:34  <maclover7>128 jobs waiting
01:16:35  <maclover7>http://node-build-monitor.herokuapp.com/
01:16:43  <maclover7>rvagg: something's up with the pis
01:24:46  <maclover7>now up to 147 jobs in queue
01:25:00  <maclover7>killed a bunch of phantom stress test jobs, hopefully helping
01:43:12  * Fishrock123joined
01:43:21  * Fishrock123quit (Remote host closed the connection)
07:37:34  * seishunjoined
08:34:40  * seishunquit (Ping timeout: 268 seconds)
08:40:10  * seishunjoined
09:03:46  * seishunquit (Disconnected by services)
09:03:52  * seishunjoined
10:53:02  <rvagg>I've cleaned up a bunch of jobs as well, I guess they were "phantom", it says they are running but they are either completely stalled or there is no build page for them when you click through.
10:53:57  <rvagg>here's a tip for anyone with jenkins-admin perms if things look stalled: go through the list of running builds on the left side of the main page and click through to any build that has a progress bar far to the right, if it's been running for more than a few hours or the page doesn't exist, kill it
10:56:57  <rvagg>I'd really like to figure out the cause of some of these phantoms, I tried to dig into it today but have run out of time
10:57:35  <rvagg>the post-build-update job, or whatever its called, seemed to be causing problems last week when we were having similar issues, heaps of them banked up, killing a few seemed to ease the strain
10:57:52  <rvagg>I don't see evidence of that now, however, so it might be more of a symptom than a cause
11:25:11  * mylesborinsquit (Quit: farewell for now)
11:25:42  * mylesborinsjoined
11:40:39  <rvagg>queue has mostly drained, looking pretty healthy
11:40:42  <rvagg>💤
14:19:52  * jbergstroemquit (Remote host closed the connection)
14:19:55  * evanlucasjoined
14:33:35  * sgimenoquit (Ping timeout: 248 seconds)
14:47:08  * sgimenojoined
16:15:11  * node-ghjoined
16:15:11  * node-ghpart
16:17:52  * node-ghjoined
16:17:52  * node-ghpart
16:21:08  * node-ghjoined
16:21:08  * node-ghpart
17:03:13  <Trott>I'm still unable to start a Pi 1 stress job. rvagg maclover7
17:05:34  <Trott>Sometimes, it appears briefly on the left and then vanishes.
17:06:09  * apapirovskijoined
17:06:14  <Trott>Pi 2, same result.
17:29:15  * apapirovskiquit (Remote host closed the connection)
17:29:34  * apapirovskijoined
18:41:36  * seishunquit (Disconnected by services)
18:41:43  * seishunjoined
19:30:34  <Trott>Can't seem to start *any* tasks on node-stress-single-test regardless of OS.
19:32:59  <Trott>Is it OK for me to use ""Wipe Out Current Workspace" to see if that gets things unstuck?
19:33:44  <Trott>I went ahead and did it, figuring it's extremely unlikely to make anything *worse*....
19:35:58  <Trott>Alas, that didn't fix it, though...
19:36:44  <Trott>Build queue is empty.
20:24:12  * node-ghjoined
20:24:12  * node-ghpart
20:35:04  <joaocgreis>Trott: same thing here.. never seen this before, it just ignores the build command. The only thing I can think of is to restart Jenkins, but there are citgm and benchmark jobs running and I don't know how long they will take. I don't know if this still works and is up to date and all, but why not try https://ci.nodejs.org/view/All/job/node-stress-single-test-pi1-fanned/ ?
21:05:23  <Trott>I was looking for that job joaocgreis! Not sure why I couldn't find it! Anyway, that's working, so cool! Thank you!
21:21:17  * evanlucasquit (Remote host closed the connection)
21:43:24  * apapirovskiquit (Remote host closed the connection)
21:45:36  * apapirovskijoined
21:46:36  * node-ghjoined
21:46:36  * node-ghpart
21:47:01  * node-ghjoined
21:47:01  * node-ghpart
22:15:42  * apapirovskiquit (Remote host closed the connection)
22:29:03  <rvagg>I think CI is due for a restart, I'll do that in a few hours during quiet time
22:29:21  * apapirovskijoined
22:29:25  <rvagg>some workers are taking longer than they should do, aix and arn-wheezy are two I'm seeing that are concerning
22:44:44  * apapirovskiquit (Remote host closed the connection)
22:50:47  * apapirovskijoined
22:51:57  * apapirovskiquit (Remote host closed the connection)
22:52:42  * apapirovskijoined
22:55:19  * apapirovskiquit (Remote host closed the connection)
23:00:18  * seishunquit (Ping timeout: 248 seconds)
23:26:22  <rvagg>I think I've fixed the arm-wheezy worker that was causing problems
23:26:26  <rvagg>aix still backed up