/dist/ returns 500 or is very slow

[bnoordhuis]

URL:

https://nodejs.org/dist/latest-v20.x/node-v20.0.0.tar.xz

Browser Name:

wget

Browser Version:

n/a

Operating System:

n/a

How to reproduce the issue:

$ wget https://nodejs.org/dist/latest-v19.x/node-v19.9.0.tar.xz
--2023-04-20 13:08:51--  https://nodejs.org/dist/latest-v19.x/node-v19.9.0.tar.xz
Resolving nodejs.org (nodejs.org)... 104.20.23.46, 104.20.22.46, 2606:4700:10::6814:172e, ...
Connecting to nodejs.org (nodejs.org)|104.20.23.46|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2023-04-20 13:08:52 ERROR 500: Internal Server Error.

$ wget https://nodejs.org/dist/latest-v20.x/node-v20.0.0.tar.xz
--2023-04-20 13:09:00--  https://nodejs.org/dist/latest-v20.x/node-v20.0.0.tar.xz
Resolving nodejs.org (nodejs.org)... 104.20.23.46, 104.20.22.46, 2606:4700:10::6814:162e, ...
Connecting to nodejs.org (nodejs.org)|104.20.23.46|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2023-04-20 13:09:04 ERROR 500: Internal Server Error

Sometimes it starts downloading but at a veeerrrryyyy sloooow raaatttteeee, like 8k/s, or stalls completely.

Interestingly, https://status.nodejs.org/ is green but yeah, but no.

[ovflowd]

Hey there 👋 this is a recurring issue. Here are some details: #4495 and details about the latest incident here.

We're working on numerous efforts to alleviate the current infrastructure, such as moving the website to a third-party provider (Vercel).

There are some details about why this happens on these links.

We apologise for any inconvenience 😅

Closing this for now as duplicated and as there's no action besides of what we're working already on.

[bnoordhuis]

@ovflowd I hit this again today. If you want something actionable, https://status.nodejs.org/ should reflect the current flakiness; it's all green now.

[ovflowd]

Sorry, we won't update the status page because one or two persons are hitting an occasional error that happens rarely. We're working behind the doors on actionable, but the flakiness level is irrelevant enough to be an actual concern.

[bnoordhuis]

@ovflowd it's late June now and I'm still hitting this regularly.

[ovflowd]

Yes, and? Sorry, what is your comment supposed to mean? We're aware of the issue, but things take time. The issue will only get solved once certain planned infrastructural things get done.

Hitting on a closed issue and mentioning that you're still facing this doesn't bring anything meaningful for us; we're aware of the issue. And once a permanent solution comes in hand, we will update the community.

Thanks!

[bnoordhuis]

Then why don't you have a pinned issue with an explanation and occasional status updates? I'm a man of many talents but a mind reader I am not.

And to clarify: it's been two months now and I still get either 500 errors or insufferably slow downloads. It's not incidental either, it's structural, and it's unclear to me why it happens. In 2023 file serving ought to be a solved problem, IMO.

[ovflowd]

Then why don't you have a pinned issue with an explanation and occasional status updates?

Because these issues are related to occasional flakiness, the content is explained across different issues. You can search if you're curious.

And to clarify: it's been two months now and I still get either 500 errors or insufferably slow downloads.

Slow downloads have nothing to do with this, as Cloudflare serves these files… And no changes were made to how Cloudflare serves our binaries (afaik)

, it's structural, and it's unclear to me why it happens. In 2023 file serving ought to be a solved problem, IMO.

There are many issues explaining why this happens if you care to search. I'm not obliged to give a 1:1 full explanation of things already explained elsewhere.

Most of the information is available here (https://nodejs.org/en/blog/announcements/node-js-march-17-incident) and links to related issues.

[ovflowd]

And we monitor the 5XX errors; they are insignificant to the point that if you're hitting this regularly, there's something wrong on your side (which I doubt).

To iterate on this, this was explained in the related issues.

To save you time, the issue at hand is that every time we deploy the Website, we do a complete Cloudflare cache purge; this means that a gigantic load of CDN and Download requests go for a few seconds to our server, which exhausts the server. Hence these "occasional" 5XX errors; should only happen briefly when there's a new website deployment.

And yes, there are long-term plans to address these and there's a lot of work being diligently put into this, if you also read the open pinned discussions of this repository.

Now, all of this is explained on all these issues. It becomes really painful on my side to explain this to every person that asks this. I could curate a centralised issue, but that would remain the fact that people would still open more issues and ask the same thing repeatedly. (We did this before)

[ovflowd]

Also, @bnoordhuis, I would expect a bit of a more understanding behaviour from another fellow Node.js collaborator (I just noticed now you're also a collaborator).

If you're really really curious about the specifics, #nodejs-build is there for that ;)

[bnoordhuis]

Now, all of this is explained on all these issues.

As a bit of user feedback, I don't think the average visitor can be expected to understand that those pinned issues are about the 5xx errors he or she is seeing. Their titles are way too oblique; they read like developer todo lists, not operational status reports.

My suggestion: pin an issue with an explicit title, like "5xx errors / download problems".

I could curate a centralised issue, but that would remain the fact that people would still open more issues and ask the same thing repeatedly.

Sure, but then you just close those as duplicates.

(I just noticed now you're also a collaborator)

Oh, only for the last 13 years or so.

[bmuenzenmeyer]

it costs us nearly nothing to pin an issue. I am going to do so on #4495

will also edit that issue body with https://nodejs.org/en/blog/announcements/node-js-march-17-incident

[ovflowd]

Thanks @bmuenzenmeyer appreciate that!

[ovflowd]

FYI @bmuenzenmeyer we pinned the issue 🙇 thanks for your patience on this matter 🙏