00:08:40  * apapirovskijoined
03:07:45  * Ya_ALLAH_Ya_Muhmjoined
03:07:46  * Ya_ALLAH_Ya_Muhmpart
04:05:16  * apapirovskiquit (Remote host closed the connection)
04:29:58  <Trott>Anybody know what causes this? https://ci.nodejs.org/job/node-test-commit-linux/16010/nodes=debian8-64/console
04:30:48  <Trott>https://www.irccloud.com/pastebin/xK8jdaWr/
04:32:38  <Trott>Related: Really super-frustrated right now that CI is permanently red and nobody seems to mind, TBQH. There seems to be a disconnect where neither collaborators nor build feel responsible for making sure it stays green. Not a ding on build or collaborators. Just a culture and systemic problem that I don't know how to address.
04:35:43  <Trott>Sorry, I'm grumpy right now.
04:40:43  * apapirovskijoined
04:50:41  * apapirovskiquit (Remote host closed the connection)
04:50:59  * apapirovskijoined
04:53:00  * apapirovskiquit (Remote host closed the connection)
04:53:17  * apapirovskijoined
04:53:46  * apapirovskiquit (Remote host closed the connection)
09:54:58  * apapirovskijoined
09:56:43  * seishunjoined
09:59:45  * apapirovskiquit (Ping timeout: 268 seconds)
11:11:17  * thefourtheyequit (Ping timeout: 276 seconds)
11:11:36  * thefourtheyejoined
11:25:08  * mylesborinsquit (Remote host closed the connection)
11:25:44  * mylesborinsjoined
12:46:45  * node-ghjoined
12:46:45  * node-ghpart
13:02:46  * apapirovskijoined
14:15:12  * apapirovskiquit (Remote host closed the connection)
14:28:52  * apapirovskijoined
14:31:18  * apapirovskiquit (Remote host closed the connection)
14:34:48  <joyee>I think we are probably surpressing too much info from Make so when the build fails there is not much useful from the logs
14:36:40  * apapirovskijoined
14:44:18  <joyee>Hmm, also I think it might make more sense to remove the doctool tests from test-ci and somehow run them with lint-ci...
14:50:25  <joyee>There are a lot of ENOSPC: no space left on device on the arm cluster right now, e.g. https://ci.nodejs.org/job/node-test-binary-arm/RUN_SUBSET=3,label=pi3-raspbian-jessie/13297/
14:50:49  * apapirovskiquit (Remote host closed the connection)
14:52:46  * node-ghjoined
14:52:46  * node-ghpart
15:00:10  * Fishrock123joined
15:26:31  * apapirovskijoined
15:50:51  * evanlucasjoined
15:57:15  * apapirovskiquit (Remote host closed the connection)
16:25:46  * apapirovskijoined
16:40:56  * chorrelljoined
17:00:50  * apapirovskiquit (Remote host closed the connection)
17:08:29  * apapirovskijoined
17:30:07  <Trott>test-mininodes-ubuntu1604-arm64_odroid_c2-3 seems to have 18+ hours of nothing but failures. Haven't gone through them all, but it looks like build failures and not test failures.
17:30:35  <Trott>Ah, it's a stale .git/lock file. I'll see if I can log on and remove it.
17:32:46  <Trott>Rats! I can't log into ubuntu1604-arm64_odroid_c2 even after running `ansible-playbook playbooks/write-ssh-config.yml `. Who can log into that machine and remove the stale `.git/index` file?
17:33:06  <Trott>I feel like that's an IBM machine, but I don't know? mhdawson ^^^^^
17:33:42  <Trott>Oh, wait, it's not a stale `.git/index` file, it's a read-only file system. Sheesh. OK, I'm going to just mark that host temporarily offline and hope someone else fixes it....
17:36:11  <Trott>Next one: Anyone understand what happened here? I don't see why this failed. https://ci.nodejs.org/job/node-test-commit-plinux/14997/nodes=ppcle-ubuntu1404/console mhdawson
17:36:31  <Trott>Last several lines:
17:36:33  <Trott>https://www.irccloud.com/pastebin/mq0Rx0mx/
17:37:03  <Trott>(Oops, ignore everything after FAILURE, that's just stuff I accidentally copy/pasted from the web interface.)
17:37:57  * node-ghjoined
17:37:57  * node-ghpart
17:39:53  * node-ghjoined
17:39:53  * node-ghpart
17:40:42  * node-ghjoined
17:40:42  * node-ghpart
17:41:09  * node-ghjoined
17:41:09  * node-ghpart
17:48:22  * chorrellquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
18:12:05  <Trott>Manually kicking off a node-daily-master to see what's working and what's not at this point. https://ci.nodejs.org/job/node-daily-master/1021/
18:17:10  * chorrelljoined
18:24:30  <Trott>test-requireio_svincent-debian7-arm_pi2-2 failing for hours and hours. Can't `mkdir`? Anyway, going to take it offline and hope some other device picks up for the git-nodesource-update-reference task. rvagg
18:26:56  <Trott>Rebuilding that arm failure to see if taking the node offline fixed it: https://ci.nodejs.org/job/node-test-commit-arm-fanned/14154/
18:30:23  <Trott>Nope, still broken, and in the same way....
18:30:42  <Trott>Bringing the other node back online.
18:31:54  <Trott>Anybody know what to do about this (below)? rvagg
18:31:56  <Trott>https://www.irccloud.com/pastebin/M0QBkVWv/
18:39:19  * srl295joined
18:43:25  <addaleax>Trott: I guess the easiest way would be trying to create that directory yourself and seeing what error you get?
18:45:02  <Trott>addaleax: Ah, that gave some useful info! :-D "mkdir: cannot create directory `b6a0a350-e415-4540-9eb1-3c60dd7159e2': No space left on device"
18:45:27  <addaleax>Yeah, it's not great that Java doesn't relay that :( I guess you have more experience with *fixing* that anyway?
18:45:29  <Trott>It's an NFS mount, so it will affect anything probably.
18:49:53  <Trott>Hmmm...I deleted pretty much everything but it's still 99% full.
18:50:17  <Trott>My next trick would be to reboot whatever machine is hosting the NFS. Because I'm pretty limited. :-D
18:55:50  <addaleax>Trott: idk, is it worth figuring out *what* is using up all that memory?
18:56:04  <addaleax>happy to help if you're not sure how do to that, even if i don't know how the whole nfs setup works
18:56:18  <Trott>Sure would be, but I suspect I'd need access to the NFS server.
18:56:43  <Trott>I'm at the top of the mount point and doing a `du -sh .` returns 12K and no permissions issues.
18:57:02  <addaleax>`df -h`? Maybe it isn't the right mount that's causing trouble?
18:57:49  <Trott>Oh, yeah, I was doing `df -h .` but that's a better idea...
18:58:25  <Trott>Yeah, that reveals two other paths to check. Thanks!
18:59:07  <addaleax>ok, cool if that helped :)
18:59:10  <Trott>The culprit is the `.ccache` directory.
18:59:14  <Trott>That should be pretty safe to delete.
18:59:44  <addaleax>yes; but
19:00:04  <addaleax>there should be a config file in that directory that tells ccache how much data to cache
19:00:20  <addaleax>you could keep that file and edit it in order to make this not happen in the future, i guess?
19:00:44  <addaleax>(again, this might be totally naive thinking, not knowing how everything is set up on these machines)
19:02:14  <Trott>It has the max size set to just 10 Gb or so. I wonder if it somehow got ignored? Guess I'll see how much space there is once I clear out everything but the config file.
19:12:00  <srl295>addaleax: there's a ccache option to adjust its config
19:12:22  <srl295>`ccache -M 5G` etc
19:14:41  <Trott>Removing .ccache frees up 6.7 Gb. No idea what is taking up the other 95 Gb on the NFS share. rvagg
19:16:27  * chorrellquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
19:16:37  <Trott>Anyway, set it down to 5G so that it doesn't blow up again, hopefully. Thanks, addaleax!
19:17:13  * chorrelljoined
19:17:19  <Trott>Trying arm fanned again: https://ci.nodejs.org/job/node-test-commit-arm-fanned/14155/
19:30:10  * node-ghjoined
19:30:10  * node-ghpart
19:35:05  * chorrellquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
19:36:47  * chorrelljoined
19:43:05  <addaleax>Trott: Looks like there are similar issues happening when trying to run the tests? :/
19:43:44  <Trott>I think those might be weird leftovers. LIke, an out-of-date NFS cache locally or something like that. Once that run finishes, I'll run another one and see if it's clean.
19:50:08  <Trott>Argh, rebuilding still shows some mkdir failures.
19:50:30  <Trott>Guess I'll try logging into those machines and running the mkdir manually to see if it's space or something else.
19:51:59  <Trott>Ooh, interesting. This time, it's permission denied.
19:53:10  <Trott>Will have to compare permissions against a working machine after I get back from lunch. If someone beats me to it and fixes it, awesome!
20:06:40  * Fishrock123quit (Remote host closed the connection)
20:13:05  * mmarchinijoined
20:14:26  <srl295>the 'Nodes' view of jenkins ought to show how much disk space is free on each system
20:24:18  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
20:28:29  * Fishrock123joined
20:46:59  * Fishrock123quit (Remote host closed the connection)
20:49:14  * Fishrock123joined
21:16:31  <Trott>600 vs 755 perms. Pretty significant. :-D
21:19:18  <Trott>Also a lot of 600 vs 700 fixes too.
21:22:05  <Trott>OK, hopefully fixed them all but let's see. https://ci.nodejs.org/job/node-test-binary-arm/13326/
21:25:10  <Trott>Hmmm...any idea what might be causing this failure?
21:25:12  <Trott>https://www.irccloud.com/pastebin/KIxIF6xD/
21:25:36  <Trott>Oh, guess it's this: 16:22:00 fatal: Couldn't find remote ref refs/heads/v7.x-staging
21:26:12  <Trott>Not gonna stop until I get green though. :-D https://ci.nodejs.org/job/node-test-binary-arm/13327/
21:29:29  <Trott>Next https://ci.nodejs.org/job/node-test-binary-arm/13328/
21:32:00  <Trott>https://ci.nodejs.org/job/node-test-binary-arm/13329/
21:51:23  * seishunquit (Ping timeout: 268 seconds)
21:56:16  * seishunjoined
22:00:05  <maclover7>Trott: would it help to create a different location to discuss these types of failures?
22:00:10  <maclover7>maybe a trello board or something like that?
22:00:23  <Trott>Is that your way of asking me not to flood the channel with this stuff? :-D
22:00:41  <maclover7>I feel like part of the issue is that we're bad at logging exactly what the problem is, and things are a little bit all over the place (nodejs/build nodejs/node trackers)
22:00:47  <maclover7>no the chat is ok I think too
22:00:58  <maclover7>just a place to log an incident, and give it a name and number
22:01:11  <maclover7>could also track issues that are recurring that way, too
22:01:39  <maclover7>addaleax: would also +1 a build wg access request from you
22:04:04  <maclover7>Trott: also, personally, I have been trying to cleanup the ansible playbooks (it's a little bit of a hot mess), and it's a little bit tough to figure out what the state of play is re: build failures unless you are monitoring closely
22:04:59  * evanlucasquit (Remote host closed the connection)
22:05:54  <Trott>Would be happy to use a Trello board or something else if it was helpful to others and as long as I'm not the only one using it. :-D
22:07:23  <Trott>https://ci.nodejs.org/job/node-test-binary-arm/13330/
22:07:26  <maclover7>Are you able to make the next WG meeting?
22:07:32  <maclover7>Maybe we can discuss it then with everybody else
22:12:52  <Trott>I can probably be there for the first half (a little less, actually) if there's no messed up domino effect in my schedule that day. (There usually isn't, but there sometimes is.)
22:14:48  * seishunquit (Ping timeout: 240 seconds)
22:15:06  * node-ghjoined
22:15:06  * node-ghpart
22:15:51  <maclover7>Ok, cool. Do you want to open up an issue, or just comment on the meeting issue so don't forget
23:34:36  * apapirovskiquit (Remote host closed the connection)
23:49:51  * node-ghjoined
23:49:51  * node-ghpart
23:50:34  <Trott>Kicking off another node-daily-master to see if we can get green or yellow this time: https://ci.nodejs.org/job/node-daily-master/1022/
23:53:27  * apapirovskijoined