01:27:34  <rvagg>refack: what did you do with Jenkins? I was fiddling with it on the weekend too with similar unresponsiveness.
01:31:11  <rvagg>refack: `-Xmx16g -Xms8g` - you've just reduced what I already reduced. It was, and has been, at 32G for ages. That machine has 32G of RAM and the JVM does a lot of caching so it's not terrible that Jenkins uses a lot of memory. I reduced it to 24G but the fact that we're still at "unresponsive" suggests the problem is elsewhere. It'll be interesting if 16G makes a difference but it's potentially wasting resources.
01:32:34  <rvagg>I also trimmed old builds, having a large amount of those hanging around has been a problem for us in the past, but since that hasn't helped, it may not be the problem
02:12:17  <refack>rvagg: I didn't (intentionally) change the memory limits. ( Not sure I even can )
02:13:07  <refack>I was reading tread dumps trying to figure out what components are eating CPU
02:15:15  <refack>I found two plugins that collect stats, that were new and seemed to be doing a huge amount of work 6000 seconds of CPU time in total.
02:17:20  <refack>Then after a restart the main config got slightly curropted and something called GlobalNodeEnv couldn't load or be saved, so until I figured that out, all jobs failed right after start
02:18:02  <refack>I restored a config from a week ago, and not things seem fine
02:19:03  <refack>Memory changes might be mhdawson__ ?
02:31:16  <rvagg>refack: oh, sorry, figured it was you
02:31:37  <rvagg>well let's see what it does with this new limit
02:33:56  <refack>Yeah, what I was thinking. If it works 🤷
06:23:05  * node-ghjoined
06:23:06  * node-ghpart
09:31:58  * sxajoined
11:01:11  * Guest90641changed nick to addaleax
14:56:10  * tniessenjoined
14:56:52  <tniessen>LinuxOne seems to be failing a lot: can't open 'test.tap': [Errno 2] No such file or directory: 'test.tap'
14:56:58  <tniessen>And Jenkins is generally extremely slow
14:57:04  <tniessen>Even gateway timeouts
14:57:18  <refack>checking
15:11:44  <tniessen>Also doesn't seem to report to GH anymore.
15:11:47  <tniessen>Thanks refack
15:12:35  <refack>tniessen: there was a problem with a specific linuxONE host, it is solved
15:13:04  <refack>as for speed, it's a complicated problem, but we're trying to figure it out
15:13:32  <tniessen>okay, thanks for letting me know. is it a known problem that it doesnt update the GH checks?
15:13:45  <refack>no, that's new
15:14:22  <tniessen>see e.g. https://github.com/nodejs/node/pull/22747, https://github.com/nodejs/node/pull/22766
18:01:54  * sxaquit (Ping timeout: 252 seconds)
18:57:52  <refack>tniessen: github status fixed
20:06:24  * node-ghjoined
20:06:24  * node-ghpart
20:52:20  * node-ghjoined
20:52:20  * node-ghpart
21:25:50  * node-ghjoined
21:25:50  * node-ghpart
21:27:24  * node-ghjoined
21:27:24  * node-ghpart
21:37:48  <node-slack-bot>[george.adams] http://ci.nodejs.org still seems to be terribly slow for me today?
21:39:13  <node-slack-bot>[refack] Yes.
21:39:55  <node-slack-bot>[refack] I works well after a restart for a few hours then becomes very slow
21:41:10  <node-slack-bot>[refack] https://blog.github.com/2018-09-10-azure-pipelines-now-available-in-github-marketplace/
22:24:40  * node-ghjoined
22:24:40  * node-ghpart
22:48:37  <joaocgreis>refack: I see the windows backlog is huge. Are you working on it or can I take a look?
22:49:03  <refack>I'll look
22:53:30  <refack>joaocgreis: what I see is not extraordinary... I've seen that much binary-test jobs often
22:53:41  <refack>there are the git-clean jobs
22:53:53  <joaocgreis>I started the git-clean
22:54:28  <joaocgreis>there was a job failing because of git lock
22:54:33  <refack>I saw. OTher then that there are about 1.5*matrix-of-binary test worth
22:55:01  <joaocgreis>strange that it's mostly VS2017 and win2016
22:55:19  <joaocgreis>might be related to the VS2017 issue
22:56:24  <joaocgreis>refack: let me know when you're done, I'll take a look
22:56:32  <refack>the VS2017 is "compiled by 2017" since for v10 and v11 we only build vs2017
22:56:39  <refack>I'm done.
22:57:06  <joaocgreis>thanks
23:02:26  <refack>joaocgreis: P.S. since my change didn't seem to help, feel free to revert it
23:03:04  <joaocgreis>refack: ok, I'll do it
23:13:40  * node-ghjoined
23:13:40  * node-ghpart