Skip to content

Almost all the windows machines are DOWN #3435

Closed

Description

@UlisesGascon

Overview

This incident started the last Friday, seems like all the testing windows machines are currently down, in the release ci some windows machines are running.

Slack discussion

Release machines

Captura de pantalla 2023-07-30 a las 19 33 15

Test machines

Captura de pantalla 2023-07-30 a las 19 33 52
Captura de pantalla 2023-07-30 a las 19 34 54

Next steps

  • Reboot all the machines

Activity

  1. UlisesGascon commented on Jul 30, 2023

    @UlisesGascon
    MemberAuthor

    Seems like the root cause is a connectivity issue, I will check Azure and Rackspace

    Captura de pantalla 2023-07-30 a las 19 50 38

  2. self-assigned this
    on Jul 30, 2023
  3. UlisesGascon commented on Jul 30, 2023

    @UlisesGascon
    MemberAuthor

    I rebooted the machines in Azure and Rackspace. Overall, around 20 machines are back. Most of the down machuines currently are from Rackspace

    Relese

    Captura de pantalla 2023-07-30 a las 20 31 38

    Testing

    Captura de pantalla 2023-07-30 a las 20 30 09
    Captura de pantalla 2023-07-30 a las 20 30 34

    @nodejs/build: I will suggest to check more in detail the root cause and the logs from the machine to check by the connection with Jenkins is not properly stablished.

  4. removed their assignment
    on Jul 30, 2023
  5. rluvaton commented on Jul 30, 2023

    @rluvaton
    Member

    This may be related to recent Jenkins upgrade (Cc @targos )

  6. StefanStojanovic commented on Jul 30, 2023

    @StefanStojanovic
    Contributor

    A quick update: I continued what @UlisesGascon started and went through machines and got all of them except a few back online. Looks like the worst has passed, but this should be investigated so we know the root cause.

  7. UlisesGascon commented on Jul 31, 2023

    @UlisesGascon
    MemberAuthor

    Thanks a lot @StefanStojanovic!

  8. mhdawson commented on Nov 29, 2023

    @mhdawson
    Member

    Discussed in the Build WG meeting, agreed to close. Will re-open if we get another similar outage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions