v10.x CI failures

[gabrielschulhof]

https://ci.nodejs.org/job/node-test-binary-windows/25706/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=0/consoleFull

There seem to be some async_hooks failures.

[BethGriggs]

They seem to be intermittently flaky - I did manage to get a "green" v10.x run earlier
(still seeing the Alpine failures, but they're covered by #29977)

[BethGriggs]

I reran the tests on some old v10.x releases and those are showing failures now too (v10.15.0 run). I'm fairly certain I've not seen these failures, at least to this extent, before.

@nodejs/build have there been any changes to our Windows infrastructure recently?
Also ping @nodejs/async_hooks in case you have any ideas as to what the cause is 🤔

[mhdawson]

@joaocgreis can you comment on any changes to the Windows infa?

[BethGriggs]

Seeing the same failures across historic releases:

v10.10.0 - https://ci.nodejs.org/job/node-test-binary-windows/25724/
v10.15.0 - https://ci.nodejs.org/job/node-test-binary-windows/25725/
v10.17.0 - https://ci.nodejs.org/job/node-test-binary-windows/25726/

Failures always seem to be on - test-azure_msft-win10-x64-3 and test-azure_msft-win2016-x64-3. Note that I got a green run for Win10/VS2017 on https://ci.nodejs.org/job/node-test-binary-windows/25725/ - where none of the test runs were scheduled on -3.

[BethGriggs]

My run of v10.17.0 once those two machines were taken offline passed 🎉
https://ci.nodejs.org/job/node-test-commit-windows-fanned/31775/console

[joaocgreis]

This was caused by an environment variable NODE_V8_COVERAGE=cov being set at user level. The failing machines are back online, testing: https://ci.nodejs.org/job/node-test-binary-windows/25730/ ✔️

@nodejs/async_hooks is this interaction a bug? Or perhaps https://github.com/nodejs/node/blob/v10.x-staging/test/async-hooks/init-hooks.js should check if the variable is defined and fail with a clear message?

@bcoe @Trott I see this variable defined in the Windows coverage job. I cleared the user variable from these two machines and a few Windows 2012 ones. I don't know if either of you set it, but if you did and need it let me know and we'll find a way.

[Trott]

@bcoe @Trott I see this variable defined in the Windows coverage job. I cleared the user variable from these two machines and a few Windows 2012 ones. I don't know if either of you set it, but if you did and need it let me know and we'll find a way.

I set it in node-test-binary-windows-coverage. If it was set in node-test-binary-windows, that was either a mistake on my part or something done by someone else. Probably my mistake, so I'm really sorry for the trouble!

[joaocgreis]

@Trott that was not it, the variable was not set in the Jenkins job. It was actually a persistent variable (similar to adding an export to .bashrc in Linux). Possibly a left-over from manual testing. Or an unrelated mistake. Or a Jenkins bug. Anyway, it's fixed now!

[BethGriggs]

ubuntu1804_sharedlibs_debug_x64 is seeing similar failures (plus a few extras). I think something may have changed for this platform too as I don't recognise the failures and they're showing up on some historic releases.

v10.x-staging: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/16373/
v10.17.0: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/16366/
v10.15.0: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/16375/

[jasnell]

@nodejs/build ... does this need to remain open?

[richardlau]

@nodejs/build ... does this need to remain open?

I think the specific issues here have been resolved. I've volunteered to prepare the next Node.js 10.x release so I'll guess I'll find out firsthand otherwise 😅.

(nodejs/build#2072 remains open because to resolve this issue we reverted the change to run flaky tests in the debug builds (currently we skip them).)