00:09:31  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
00:13:10  * ofrobotsjoined
00:26:25  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
00:27:36  * ofrobotsjoined
00:28:20  * chorrelljoined
00:28:46  <Trott>I've got a serious case of the 504 blues right now.
00:34:25  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
00:39:51  <jbergstroem>yeah, i'd like to proceed with deleting old stuff but would like some review.
00:40:59  <thealphanerd>jbergstroem I'm getting time-outs on ci
00:40:59  <thealphanerd>:(
00:41:05  <thealphanerd>https://ci.nodejs.org/job/node-accept-pull-request/
00:41:10  <thealphanerd>gateway timeout 504
00:41:56  <jbergstroem>thealphanerd: yeah; i did testing with the job purging plugin this morning (few hours ago) but it just doesn't work as I want it to. I have a suggested bash script in place but need review before i slash 80G from disk
00:42:08  <jbergstroem>i'd link you to the issue if gh wasn't down :D
00:42:11  <thealphanerd>womp
00:43:08  <jbergstroem>github going down is starting to be the equivalent of "internet is down"
00:44:48  <jbergstroem>come to think about it; its likely the ci issues are related to github not giving access tokens
00:45:19  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
00:46:43  <jbergstroem>its in: We're investigating a significant network disruption effecting all github.com services.
00:46:50  <jbergstroem>orly :/
00:48:39  <thealphanerd>womp
00:51:31  * Fishrock123quit (Quit: Leaving...)
01:03:32  * jgiquit (Quit: jgi)
02:08:32  * chorrelljoined
02:16:50  * chorrellquit (Quit: My Mac has gone to sleep. ZZZzzz…)
02:17:56  * chorrelljoined
02:30:28  * chorrellquit (Quit: My Mac has gone to sleep. ZZZzzz…)
02:35:11  * chorrelljoined
02:35:35  * chorrellquit (Client Quit)
02:42:58  <jbergstroem>coming back online now it seems
02:46:41  * node-ghjoined
02:46:41  * node-ghpart
03:00:17  <jbergstroem>I wonder
03:00:54  <jbergstroem>should i just remove all folders regarding the backup job? I pretty much said it would happen today regardless. We have a full backup if we need to revert.
03:02:59  <jbergstroem>i'd almost prefer that over making the rsync snapshot stale.
03:05:08  <jbergstroem>making -> letting the rsync snapshot grow stale
03:07:16  <jbergstroem>as my mom always says; reasoning with yourself always leads to the best answers.
03:10:37  * node-ghjoined
03:10:38  * node-ghpart
03:25:53  <thealphanerd>jbergstroem is CI about to restart?
03:26:10  <jbergstroem>thealphanerd: i was doing a few things and didn't want new jobs in. give me a few more minutes
03:26:19  <thealphanerd>ok cool beans
03:31:49  <jbergstroem>i wonder if I can kill that stress test job
03:31:54  <jbergstroem>3525 ok, 15 not ok
03:33:09  <jbergstroem>Trott: ^
03:33:25  <Trott>Yes, that's fine.
03:34:11  <Trott>(If it's the stress test job I started, of course.)
03:34:30  <jbergstroem>'twas
03:35:11  <jbergstroem>ok, back in action now. I'd still like to execute above but i'll wait diligently for feedback
03:36:30  <Trott>I have a few jobs I want to run. Are you saying I should go ahead and run them so we can suss out problems now? Or I should wait until you do some things you'd like to do first?
03:36:53  <Trott>I can totally wait, or I can totally start jobs. I'm all totally something totally something totally.
03:37:34  <jbergstroem>nah feel free -- i just need more people to look at my `find` stuff so i can proceed and shave off some 100G logs
04:50:48  * ofrobotsjoined
04:58:58  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
05:45:20  * node-ghjoined
05:45:20  * node-ghpart
05:56:08  * jgijoined
05:56:50  * node-ghjoined
05:56:51  * node-ghpart
05:58:27  * node-ghjoined
05:58:28  * node-ghpart
06:04:19  * jgiquit (Quit: jgi)
06:05:25  * jgijoined
06:14:16  <Trott>No 504 errors or super sluggish interface issues so far tonight. Small sample size, but still, that's good.
06:35:34  * jgiquit (Quit: jgi)
06:36:16  * ofrobotsjoined
06:42:36  * jgijoined
06:52:39  * jgiquit (Quit: jgi)
06:57:13  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
07:22:27  <jbergstroem>Trott: i haven't really cleaned just yet.
07:22:39  <jbergstroem>rvagg: when will we cut releases (hours from now)?
07:23:01  <Trott>Maybe it's the reboot, or it's a light usage time right now (evening in the U.S.).
07:24:19  <thealphanerd>jbergstroem last I hear the plan was to wait until Monday
07:24:26  <jbergstroem>oh
07:24:32  <jbergstroem>yeah its friday after all
07:24:33  <thealphanerd>but that was depending on severity and there was the potential to do it friday
07:24:48  <jbergstroem>i'd prefer doing some house cleaning ahead
07:25:04  <thealphanerd>I'm pretty sure you are clear for the next couple hours if that is your concern
07:25:21  <thealphanerd>I don't even know if the security vuln is being released until friday
07:25:21  <jbergstroem>i will do an a rsync in ~5 hours then another one in ~14h. was planning to do the delete after that
07:25:29  <jbergstroem>openssl?
07:25:32  <thealphanerd>yeah
07:25:40  <jbergstroem>openssl is set for jan 28 probs 12gmt
07:25:46  <thealphanerd>ok so thursday
07:25:51  <jbergstroem>well :D
07:25:53  <jbergstroem>semantics
07:26:16  <thealphanerd>rvagg was talking about wanting to make the CI private for the two days to do testing of the fixes
07:27:17  <jbergstroem>ok, np
07:27:41  <jbergstroem>i'll be around; just give me some heads up and i'll try to have everything done and quick for cutting
07:42:03  * rmgquit (Remote host closed the connection)
07:42:37  * rmgjoined
07:47:06  * rmgquit (Ping timeout: 250 seconds)
09:11:50  <joaocgreis>jbergstroem: did you understand what the ppc problem was?
09:37:20  * imjacobclarkjoined
09:42:23  * imjacobclarkquit (Ping timeout: 276 seconds)
09:44:07  * node-ghjoined
09:44:07  * node-ghpart
10:16:10  * imjacobclarkjoined
10:17:19  * node-ghjoined
10:17:19  * node-ghpart
11:10:30  * imjacobclarkquit (Remote host closed the connection)
11:11:02  * imjacobclarkjoined
11:15:31  * imjacobclarkquit (Ping timeout: 245 seconds)
11:26:33  * node-ghjoined
11:26:34  * node-ghpart
11:37:07  * imjacobclarkjoined
12:00:01  * jenkins-monitorquit (Remote host closed the connection)
12:00:08  * jenkins-monitorjoined
12:33:42  * imjacobclarkquit (Remote host closed the connection)
12:34:36  * imjacobclarkjoined
12:34:40  * imjacobclarkquit (Remote host closed the connection)
12:34:47  * imjacobclarkjoined
12:40:19  * imjacobc_joined
12:40:19  * imjacobclarkquit (Remote host closed the connection)
12:45:02  * imjacobc_quit (Ping timeout: 272 seconds)
12:47:18  * node-ghjoined
12:47:19  * node-ghpart
13:00:02  * jenkins-monitorquit (Remote host closed the connection)
13:00:09  * jenkins-monitorjoined
13:04:19  <jbergstroem>So, in ~8 hours i'll be cleaning up the job history unless people think otherwise. Backups are in place.
13:04:47  <jbergstroem>(I've confirmed backups working by deleting old jobs, reloading jenkins config; restoring them and reloading again)
14:00:01  * jenkins-monitorquit (Remote host closed the connection)
14:00:08  * jenkins-monitorjoined
14:00:15  * node-ghjoined
14:00:15  * node-ghpart
15:00:37  * jenkins-monitorquit (Remote host closed the connection)
15:00:44  * jenkins-monitorjoined
15:23:45  * ofrobotsjoined
15:42:34  * Fishrock123joined
15:42:44  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
16:00:37  * jenkins-monitorquit (Remote host closed the connection)
16:00:44  * jenkins-monitorjoined
16:12:32  * ofrobotsjoined
16:14:27  * rmgjoined
16:19:34  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
17:00:01  * jenkins-monitorquit (Remote host closed the connection)
17:00:08  * jenkins-monitorjoined
17:06:51  * ofrobotsjoined
17:09:06  * Fishrock123quit (Quit: Leaving...)
17:21:39  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
17:35:39  * ofrobotsjoined
17:44:29  * jgijoined
18:00:02  * jenkins-monitorquit (Remote host closed the connection)
18:00:09  * jenkins-monitorjoined
18:19:14  * jgiquit (Quit: jgi)
18:21:44  * jgijoined
18:23:34  <Trott>Not sure who will be using the build infrastructure for the OpenSSL-fix release coming soon... rvagg probably? Anyway, I'm currently running 5 win10 stress tests on CI to troubleshoot a particularly vexingly unpredictable flaky test. If this causes a bottleneck for the release, please abort all my stress test jobs (or ping me here to do it if you want, for
18:23:34  <Trott>whatever reason).
18:38:42  <Trott>Looks like test-azure_msft-win10-x64-4 keeps going offline during builds (which is a shame because that's specifically the host I want to run the stress test on as the problem I'm looking at has manifested on that host and that host alone, oddly).
18:38:50  <Trott>jbergstroem: joaocgreis ^
18:39:17  <Trott>Examples: https://ci.nodejs.org/job/node-stress-single-test/392/nodes=win10/console https://ci.nodejs.org/job/node-stress-single-test/391/nodes=win10/console
18:40:24  <joaocgreis>Trott: can I try to reboot it right now?
18:41:00  <Trott>Sure, please do! I just started another build on it, so feel free to cancel it (or I can).
18:41:10  <Trott>It's this one: https://ci.nodejs.org/job/node-stress-single-test/393/nodes=win10/
18:42:12  <joaocgreis>rebooting, the job will abort on its own
18:43:17  <joaocgreis>Trott: it's back
18:49:08  <Trott>great thanks!
19:00:40  * jenkins-monitorquit (Remote host closed the connection)
19:00:46  * jenkins-monitorjoined
19:19:18  <Trott>Wow, four green CI runs in a row on node-test-pull-request. We should all get commemorative coffee mugs or something for this day.
19:23:27  <jbergstroem>hehe
19:48:29  * jgiquit (Quit: jgi)
19:50:42  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
19:59:42  <jbergstroem>ok, getting ready to delete.
20:00:01  * jenkins-monitorquit (Remote host closed the connection)
20:00:08  * jenkins-monitorjoined
20:00:34  * jgijoined
20:02:25  * Fishrock123joined
20:39:41  <jbergstroem>does anyone know why the jenkins master checks code out locally for each job?
20:49:00  <jbergstroem>running now.
20:52:56  * node-ghjoined
20:52:57  * node-ghpart
20:58:12  <jbergstroem>84G cleaned.
21:00:48  * jenkins-monitorquit (Remote host closed the connection)
21:00:56  * jenkins-monitorjoined
21:09:13  <jbergstroem>Ok; reloading done. Took about 10 minutes. I'm restarting master; hopefully we should be ready to go from here.
21:12:05  <thealphanerd>yyay!
21:12:12  <thealphanerd>awesome job jbergstroem
21:12:24  <jbergstroem>it's still reindexing jobs
21:12:32  <jbergstroem>but hopefully will be much quicker
21:16:54  <jbergstroem>orangemocha, joaocgreis: got any idea why each job checks out the git source on master? (also, once per workspace)
21:20:02  <jbergstroem>61G total <- all workspaces
21:37:33  <joaocgreis>jbergstroem: at least it is needed for the changelog that appears on the job page
21:37:56  <jbergstroem>joaocgreis: seems to be some kind of assumption in jenkins-land
21:38:12  <jbergstroem>joaocgreis: its possible to share the workspace but i'm not sure what that could lead to
21:38:26  <joaocgreis>Would be for the "build only if source changes" too
21:40:17  <joaocgreis>I tried sharing the workspaces on Windows with links and messed the realpath test. Not sure what would happen on master.
21:41:37  <joaocgreis>Let's see how much it improves with just this.. Great job!
21:41:47  <jbergstroem>the workspaces seems very old
21:42:07  <jbergstroem>we're talking november
21:42:33  <jbergstroem>ah; just some of them
21:43:44  * ofrobotsjoined
21:44:15  <jbergstroem>joaocgreis: i'll probalby just exclude them from backups or something
21:56:45  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
22:00:02  * jenkins-monitorquit (Remote host closed the connection)
22:00:09  * jenkins-monitorjoined
22:00:32  * ofrobotsjoined
22:06:59  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
22:10:19  * ofrobotsjoined
22:17:51  * node-ghjoined
22:17:52  * node-ghpart
22:19:47  * node-ghjoined
22:19:47  * node-ghpart
22:24:59  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
22:26:48  * ofrobotsjoined
22:31:10  * ofrobotsquit (Client Quit)
22:33:30  * ofrobotsjoined
22:35:22  * ofrobotsquit (Client Quit)
22:38:44  * ofrobotsjoined
22:54:23  * ofrobotsquit (Quit: My Mac has gone to sleep. ZZZzzz…)
22:59:52  * ofrobotsjoined
23:00:01  * jenkins-monitorquit (Remote host closed the connection)
23:00:09  * jenkins-monitorjoined
23:40:56  * chorrelljoined
23:56:57  * chorrellquit (Quit: My Mac has gone to sleep. ZZZzzz…)