00:14:49  <mylesborins>ummm
00:14:50  <mylesborins>everything seems broken in private job now
00:14:51  <mylesborins>😅
00:16:49  <mylesborins>rvagg CI is completely down now
00:16:50  <mylesborins>for private at least
00:16:52  <mylesborins>:(
00:16:53  <mylesborins>ERROR: Error fetching remote repo 'origin'
00:16:54  <mylesborins>hudson.plugins.git.GitException: Failed to fetch from https://github.com/nodejs-private/node-private.git
00:16:57  <mylesborins>I've got to step away but please lmk asap if you have any luck fixing this
00:17:04  <rvagg>wait WHAT?
00:17:54  <mylesborins>https://ci.nodejs.org/job/git-rebase/25130/console is a failure of the rebase job
00:18:50  <mylesborins>AHHHH because you testing nodejs/node-private
00:18:51  <mylesborins>not nodejs-private/node-private
00:18:52  <mylesborins>🙃
00:21:02  <mylesborins>rvagg it was moved and I don't think the forwarded is working the way you handled permissions
00:21:12  <mylesborins>AFK for a minute while I head home from work... it is too damn late rn :D
00:22:04  <rvagg>hm
00:22:30  <rvagg>might just need to update node-private with the latest on node, branches and all
00:26:31  <rvagg>mylesborins: so in this instance, it's purely because v4.8.5-proposal doesn't exist on node-private and you're submitting that as the build head. Either duplicate that onto node-private or use node-test-commit instead of node-test-commit-private
00:27:36  <rvagg>mylesborins: looking at the branches i think maybe you meant to submit v4.8.5-sec-proposal instead? that's on the node-private repo
00:28:35  <rvagg>mylesborins: done that here for you: https://ci.nodejs.org/job/node-test-commit-private/166/
00:32:14  <rvagg>osx failure I don't understand, we normally get DefaultMatrixExecutionStrategyImpl failures when all machines for a label are out of action but in this case it should just have gone into the queue
00:32:21  <rvagg>that stupid DefaultMatrixExecutionStrategyImpl failure seems a bit random
00:34:44  <rvagg>maybe it's related to queue depth
00:34:57  <rvagg>the jenkins queue hasn't been working properly of late, it doesn't show what's in there
00:35:06  <rvagg>maybe a reboot later today when everyone's offline will help
00:37:47  <rvagg>this kind of thing: https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx1010/ - see the build numbers on the left, they're sequential except for after #13412 they jump for the queue where #13416 is waiting and it seems that nothing else can be queued
00:39:11  <rvagg>and https://ci.nodejs.org/computer/test-requireio-osx1010-x64-1/ says #19 is running but #19 also doesn't exist
00:40:12  <rvagg>but it's running
00:40:17  <rvagg>on the server at least
00:40:59  <rvagg>:facepalm: heading out to find a cafe to work from, back in 30 mins or so
00:50:08  <mylesborins>rvagg thanks... sorry for the false alarm regarding the job not running
00:50:12  * mylesborinsadds to the facepalm
01:08:10  * jasnellquit (Ping timeout: 264 seconds)
01:21:17  * joyeejoined
01:25:21  * joyeequit (Ping timeout: 248 seconds)
01:41:42  <mylesborins>rvagg CI looks like
01:41:47  <mylesborins>windows citgm is still being weird
01:41:48  <mylesborins>https://ci.nodejs.org/view/All/job/citgm-smoker-private/27/nodes=win-vs2015/console
01:41:54  <mylesborins>but that isn't a huge deal...
01:45:34  <rvagg>mylesborins: the "ERROR: Step ‘Publish JUnit test result report’ failed" failure? I can't find anything different in the config between citgm-smoker and citgm-smoker-private that has anything to do with this. Is it possible that there is actually no JUnit output file here?
01:46:22  <mylesborins>it fails earlier
01:46:23  <mylesborins>true was unexpected at this time.
01:46:24  <mylesborins>c:\workspace\citgm-smoker-private\nodes\win-vs2015\node\Release>if equ true set "READABLE_STREAM=disable"
01:46:25  <mylesborins>that happens after the citgm install
01:47:08  <mylesborins>sucessful run using the other job https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1026/nodes=win-vs2015/
01:47:11  * Fishrock123quit (Quit: Leaving...)
01:47:28  <rvagg>ah, ok them, that's missing in the config
01:48:51  <mylesborins>heh
01:49:02  <rvagg>mylesborins: added in DISABLE_READABLE_STREAM as a parameter, do you normally tick that?
01:49:02  <mylesborins>btw... did you notice that leveldb 2.0 does not support 4.x
01:49:09  <mylesborins>nope
01:49:15  <mylesborins>generally don't disable it
01:49:59  <rvagg>huh, https://github.com/Level/levelup/blob/master/.travis.yml#L18-L20, only 8 and 6, I missed that memo
01:51:27  <mylesborins>just noticed that it was broken in CITGM for 4.x
01:51:40  <mylesborins>suprised such a large project would stop support for 4.x before EOL but I get it
01:51:41  <mylesborins>¯\_(ツ)_/¯
01:51:44  <mylesborins>did it in a major
01:52:33  <mylesborins>leveldown seems to still work fine
01:54:36  <rvagg>mm, https://github.com/Level/levelup/pull/455 promisification and some ES6 stuff
01:56:22  <mylesborins>neat
02:02:42  * joyeejoined
02:05:43  * joyeequit (Client Quit)
02:35:32  * joyeejoined
03:14:49  * jasnelljoined
05:34:41  * node-ghjoined
05:34:41  * node-ghpart
05:36:17  * node-ghjoined
05:36:17  * node-ghpart
05:37:06  * node-ghjoined
05:37:06  * node-ghpart
06:07:26  <rvagg>ok, I screwed up iptables on ci.nodejs.org while trying to clean it up (swapping out arm64 machines). Sorry, mylesborins I know stuff was running, I hope I haven't set things back too far.
06:07:36  <rvagg>i.e. I had to hard-reboot the machine
06:07:39  <mylesborins>heh
06:07:40  <mylesborins>its ok
06:13:16  * joyeequit (Remote host closed the connection)
06:13:48  * joyeejoined
06:13:53  * jasnellquit (Ping timeout: 252 seconds)
06:41:42  * seishunjoined
06:51:55  * joyeequit (Remote host closed the connection)
06:53:35  * joyeejoined
06:57:49  * joyeequit (Ping timeout: 255 seconds)
07:17:49  * node-ghjoined
07:17:49  * node-ghpart
07:19:19  * node-ghjoined
07:19:19  * node-ghpart
07:36:05  * seishunquit (Ping timeout: 240 seconds)
07:43:23  * joyeejoined
07:43:43  * seishunjoined
07:44:34  * joyeequit (Remote host closed the connection)
07:47:32  * joyeejoined
08:02:39  * joyeequit (Remote host closed the connection)
08:04:59  * joyeejoined
08:09:08  * seishunquit (Ping timeout: 240 seconds)
08:09:58  * joyeequit (Remote host closed the connection)
08:17:36  * joyeejoined
08:21:57  * joyeequit (Ping timeout: 240 seconds)
08:26:59  * node-ghjoined
08:26:59  * node-ghpart
08:27:51  * joyeejoined
10:25:08  * mylesborinsquit (Quit: farewell for now)
10:25:39  * mylesborinsjoined
10:58:58  * node-ghjoined
10:58:58  * node-ghpart
11:07:17  * node-ghjoined
11:07:17  * node-ghpart
11:27:38  * node-ghjoined
11:27:38  * node-ghpart
11:29:33  * joyeequit (Remote host closed the connection)
12:01:34  * joyeejoined
12:03:33  * node-ghjoined
12:03:33  * node-ghpart
12:05:51  * joyeequit (Ping timeout: 248 seconds)
12:26:45  * node-ghjoined
12:26:45  * node-ghpart
12:29:19  * node-ghjoined
12:29:19  * node-ghpart
12:35:55  * node-ghjoined
12:35:55  * node-ghpart
12:46:18  * node-ghjoined
12:46:18  * node-ghpart
12:47:22  * node-ghjoined
12:47:22  * node-ghpart
13:53:02  * jasnelljoined
13:54:36  * joyeejoined
13:55:10  * joyeequit (Client Quit)
14:04:08  * chorrelljoined
15:03:53  <joaocgreis>mylesborins, rvagg: I've been away, still have have a lot of reading to do. Thanks for taking care of it yesterday and let me know if there's still something causing issues
15:11:18  * node-ghjoined
15:11:18  * node-ghpart
15:12:05  * node-ghjoined
15:12:05  * node-ghpart
15:26:21  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
15:30:35  <mylesborins>hey anyone who can help
15:30:36  <mylesborins>ci-release is not working with nodejs-private 🎉
15:30:39  <mylesborins>rvagg joaocgreis mhdawson_
15:36:11  <joaocgreis>mylesborins: let me take a look
15:40:40  <mylesborins>danke shen
15:54:04  <joaocgreis>mylesborins: keys are all good. You have to use git@github.com: . Let me know if you want me to relaunch
15:54:32  <joaocgreis>there were also some jobs in the queue, I checked and cleared
16:08:27  * seishunjoined
17:14:32  * node-ghjoined
17:14:32  * node-ghpart
17:38:27  * joyeejoined
17:41:11  <mylesborins>joaocgreis https://ci-release.nodejs.org/job/iojs+release/2153/console
17:41:15  <mylesborins>git@github.com not working either
17:42:08  <mylesborins>weird... another one did work
17:42:09  <mylesborins>ok digging in
17:42:55  <mylesborins>and now it worked
17:42:57  <mylesborins>no idea why
17:43:42  <mylesborins>is centos7 a new platform?
17:43:44  <mylesborins>it failed on 6.x
17:45:58  <mylesborins>in fact centos7 is failing on all builds
17:45:58  <mylesborins>https://ci-release.nodejs.org/job/iojs+release/2155/nodes=centos7-arm64/console
17:47:40  <mylesborins>it looks like the build job is not supplying an argument to destcpu
17:55:17  <joaocgreis>mylesborins: it's strange, no variables are getting passed. let me see..
17:55:33  <mylesborins>it looks like you made some changes to stuff involving variables today
17:55:38  <mylesborins>can we revert that?
18:05:28  * node-ghjoined
18:05:28  * node-ghpart
18:12:52  <mylesborins>joaocgreis any word?
18:12:53  <mylesborins>sorry for being so impatient but I'm worried about the timing on this
18:13:41  <joaocgreis>mylesborins: my changes are unrelated, that host is failing since saturday
18:13:49  <mylesborins>ah ok
18:13:59  <mylesborins>so what is weird is that the env vars are being passed for all other build
18:14:21  <joaocgreis>(my changes were only automatic plugin updated, i clicked config and then save without changing anything, so it would not be in the diff if I changed anything later)
18:14:54  <joaocgreis>the env vars should be set in the machine script to start jenkins, aparently got removed on saturday
18:15:13  <joaocgreis>I'm trying to access the machine but can't
18:15:27  <joaocgreis>investigating
18:19:53  <mylesborins>joaocgreis in ansible?
18:19:57  <mylesborins>or in jenkins
18:20:08  <mylesborins>would this affect any of the other builds or is it an isolated issue?
18:20:38  <joaocgreis>seems to be isolated, all other machines should be ok
18:21:13  <joaocgreis>should be part of ansible, I'll check later after I figure out the problem with the machine now
18:24:52  <mylesborins>thanks!
18:26:51  <joaocgreis>mylesborins: found the issue, it will take me a while to solve (1h perhaps)
18:27:16  <mylesborins>is there a hot fix that can be done? This is blocking the sec release from coming out, and we'll have to re run the builds after this is fixed
18:28:38  <refack>joaocgreis: if there's anything I can do to help in a staging environment, ping me
18:29:12  <mylesborins>refack these are all on the build machines
18:29:52  <refack>I'm here if you need non-infra help
18:31:23  <mylesborins>:P
18:40:49  <joaocgreis>mylesborins: give it a try, let's see if I got it right
18:40:53  * node-ghjoined
18:40:55  <mylesborins>fingers corssed
18:40:59  * node-ghpart
18:41:47  <mylesborins>running job rn
18:41:58  <mylesborins>https://ci-release.nodejs.org/job/iojs+release/2160/nodes=centos7-arm64/console if you want to watch live
18:42:07  <mylesborins>oh doodle I didn't set the right repo
18:42:08  <mylesborins>LOL
18:42:55  <mylesborins>well that just scared the doodle out of me
18:43:04  <mylesborins>it still worked because it had already pull those changes from private
18:43:10  <mylesborins>but I was like HOW TF DID IT JUST BUILD
18:43:20  <mylesborins>and scrambled to the repo to see if I accidentally leaked the build
18:43:50  <joaocgreis>so, ARCH and DESTCPU are usually set in the scripts that launch jenkins in the workers, but that is only for release machines. Some machines already set it in the jenkins worker configuration, so I did that for now. We can change ansible to set those vars for release machines if there is a good reason to, for me setting in Jenkins makes more sense because it's closer to the job that requires them (cc rvagg
18:43:50  <joaocgreis>jbergstroem )
18:44:55  <mylesborins>so that worked for now
18:44:57  <mylesborins>thank you so much
18:46:22  <joaocgreis>np! lmk if something else happens
18:46:54  <mylesborins>I think I unintentionally found a security hole though, and a good reason as to why we need to embargo for sec
18:46:55  <mylesborins>I was able to build off of the same reference without pull down private
18:46:56  <mylesborins>because it existed in the git tree
18:47:14  <mylesborins>so in theory, on the main CI, this could leak security fixes early if people wanted to try and guess the name of the head of our release branch (not that hard)
18:48:25  <joaocgreis>mylesborins: Jenkins must always be locked for security releases. There's plenty of ways to get info and even extract code and metadata if it is not locked.
18:49:00  <mylesborins>yup.
18:49:02  <mylesborins>rvagg and I were discussing if the private job could be sufficient and we concluded it wasn't
18:49:03  <mylesborins>just another vector
18:49:41  <mylesborins>I would really like to see us make it easier to spin up new instances of the CI infra... so we can spin up a fresh infra when we do sec releases and not block the main project
18:49:50  <joaocgreis>subjobs are not private... all the parameters are there
18:51:16  <joaocgreis>the private jobs should only spin the normal jobs passing parameters, but have more useful defaults on them
18:52:46  <joaocgreis>spining up new instances of jenkins would be useful, but I'm not sure it's doable because they would need a similar set of slaves
18:53:16  <joaocgreis>so we'd need to duplicate all the infrastructure, perhaps without so much redundance but it's still plenty of platforms
18:55:24  * node-ghjoined
18:55:24  * node-ghpart
19:00:31  * node-ghjoined
19:00:31  * node-ghpart
19:01:03  * node-ghjoined
19:01:03  * node-ghpart
19:01:54  * node-ghjoined
19:01:54  * node-ghpart
19:06:49  * node-ghjoined
19:06:49  * node-ghpart
19:09:39  * node-ghjoined
19:09:39  * node-ghpart
19:11:41  * node-ghjoined
19:11:41  * node-ghpart
19:22:17  * joyeequit (Remote host closed the connection)
19:22:49  * joyeejoined
19:27:22  * joyeequit (Ping timeout: 264 seconds)
19:34:37  * node-ghjoined
19:34:37  * node-ghpart
19:37:56  * node-ghjoined
19:37:56  * node-ghpart
20:07:52  * node-ghjoined
20:07:52  * node-ghpart
20:10:10  * node-ghjoined
20:10:10  * node-ghpart
20:24:05  * seishunquit (Ping timeout: 240 seconds)
21:04:26  * joyeejoined
21:08:57  * joyeequit (Ping timeout: 258 seconds)
21:10:05  * joyeejoined
21:14:32  * joyeequit (Ping timeout: 255 seconds)
21:14:37  * node-ghjoined
21:14:37  * node-ghpart
21:22:37  * node-ghjoined
21:22:37  * node-ghpart
21:46:50  * node-ghjoined
21:46:50  * node-ghpart
21:53:27  * node-ghjoined
21:53:27  * node-ghpart
21:58:49  * node-ghjoined
21:58:49  * node-ghpart
21:59:41  * node-ghjoined
21:59:41  * node-ghpart
22:03:24  * node-ghjoined
22:03:24  * node-ghpart
22:03:49  * node-ghjoined
22:03:49  * node-ghpart
22:05:04  * node-ghjoined
22:05:04  * node-ghpart
22:07:37  * node-ghjoined
22:07:37  * node-ghpart
22:07:57  * node-ghjoined
22:07:57  * node-ghpart
22:10:47  * joyeejoined
22:13:46  * node-ghjoined
22:13:46  * node-ghpart
22:15:21  * joyeequit (Ping timeout: 240 seconds)
22:16:44  <ofrobots>mylesborins: I still see: ofrobots is missing the Overall/Read permission
22:22:17  * node-ghjoined
22:22:17  * node-ghpart
22:37:37  * node-ghjoined
22:37:37  * node-ghpart
22:38:22  * node-ghjoined
22:38:22  * node-ghpart
22:51:38  * joyeejoined
22:56:41  * joyeequit (Ping timeout: 255 seconds)
23:07:47  * node-ghjoined
23:07:47  * node-ghpart
23:16:30  * gibfahnjoined
23:40:56  * node-ghjoined
23:40:56  * node-ghpart
23:47:07  * node-ghjoined
23:47:07  * node-ghpart