[eliottness]

What does this PR do?:

Stops this repository's CI images from trusting emSign ECC Root CA - G3 and AAA Certificate Services, the roots behind the compromised eMudhra intermediates (incident IR-62343). The APPSEC-71273 ledger lists 16 impacted image references here; images CI never builds or pulls are left alone.

• .gitlab/base/Dockerfile (Debian base images): installs current ca-certificates, points awscli's botocore bundle at the system bundle, replaces Debian's packaged Node (which embeds the old roots) with the checksum-verified upstream Node 22.23.4 archive, and deletes both roots from the SDKMAN JDK cacerts.
• .gitlab/base/Dockerfile.musl: upgrades ca-certificates, points awscli's botocore bundle at the system bundle, and deletes both roots from the JDK cacerts.
• .gitlab/base/el7/Dockerfile: Oracle Linux 7 has no fixed ca-certificates package, so both roots are removed from the p11-kit trust source (every other object and its purpose restrictions are kept) and the trust store is re-extracted; both roots are also deleted from the JDK cacerts.
• .gitlab/base/el8/Dockerfile: dnf update ca-certificates + update-ca-trust extract, and both roots deleted from the JDK cacerts.

The base-image Dockerfile changes take effect when REBUILD_IMAGES republishes the async-profiler-build images and the pinned digests are updated (7 references are fixed this way).

Motivation:

Incident IR-62343: CI images that trust these two roots keep the exposure from the compromised eMudhra intermediates. Jira APPSEC-71294 (parent APPSEC-71273).

Additional Notes:

• Datadog's GitLab runners only admit images from approved registries. The merge-queue pipelines for this PR were rejected for that reason: Docker Hub alpine:3.24.3 was denied by the third-party-registry admission policy (job 2174477532), and the reliability child pipeline failed because its skip job had also moved to Docker Hub alpine. .gitlab-ci.yml, .gitlab/build-deploy/.gitlab-ci.yml and the benchmark files are therefore back to main's version; this PR now only changes the base-image Dockerfiles.
• Follow-ups (not in this PR):
  • Default CI image alpine:3.22.1 (.gitlab-ci.yml, .gitlab/build-deploy/.gitlab-ci.yml): moving to Alpine 3.24 needs a mirrored alpine 3.24 image in registry.ddbuild.io first, because the runners reject Docker Hub images.
  • Needs a rebuild at source: benchmarking-platform-tools-ubuntu:newest (DataDog/images; newest and latest still resolve to the bad digest 1e27d877); ci/async-profiler-build:v137281788-datadog-ci (rebuild .gitlab/Dockerfile.datadog-ci via .gitlab/scripts/rebuild-images.sh, then update the pin); ci/async-profiler-build-amd64 / -arm64 benchmark images (rebuild .gitlab/benchmarks/docker/Dockerfile, then update the pins); slack-notifier:v91289620-4ec922a-latest (DataDog/slack-notifier; latest is the same bad digest 06b24f39).
  • Upstream bases with no clean version in line: eclipse-temurin:21-jdk-alpine@sha256:c98f0d2e171c (JDK 21 is pinned on purpose; the current digest still trusts the roots, patched via Dockerfile.musl above) and almalinux:8@sha256:9f355ae942d6 (pins the shipped binaries' glibc 2.28 floor; AlmaLinux 9 and 10 also still trust the roots; patched via el8/Dockerfile above).
  • Unscannable: 486234852809.dkr.ecr.us-east-1.amazonaws.com/images/docker:24.0.4-gbi-focal (pull access denied: no basic auth credentials for that ECR registry).

How to test the change?:

Every new image reference and every changed Dockerfile was pulled or built and scanned with the eMudhra scanner, which checks every file in the image for the two roots' public keys; all returned empty hits. GitHub CI is not fully green on 44cbb26: test-linux-glibc-amd64 (8-ibm, debug, regular) was cancelled at its 180-minute timeout after hanging in CTimerSamplerTest (cstack=dwarf), run 40387577386. The same commit passed that job on earlier runs, and the GitHub workflows do not read the Dockerfiles this PR changes. The GitLab jobs that build the base images run in the merge-queue pipeline and on REBUILD_IMAGES.

For Datadog employees:

• If this PR touches code that signs or publishes builds or packages, or handles
  credentials of any kind, I've requested a security review (run the dd:platform-security-review
  skill, or file a request via the PSEC review form).
  bewaire also runs automatically on every PR.
• This PR doesn't touch any of that.

Note for the owner: this PR changes the base images used to build the shipped binaries, so it touches the publishing pipeline; no PSEC review has been requested yet.

• JIRA: APPSEC-71294

Unsure? Have a question? Request a review!

[jbachorik]

@copilot resolve the merge conflicts in this pull request

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-24T12:28:41.298642Z	8715e05	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[datadog-prod-us1-6]

More details

Image-level verification remains incomplete: full builds and certificate scans were not independently confirmed, so this review does not establish that rebuilt CI images contain none of the compromised roots.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 8715e05}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8715e053d4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[eliottness]

@jbachorik thanks for picking this up. Two changes since your merge-queue attempt, both in 2c6ec87 (GitHub-verified):

1. Merge-queue failure: the rejected pipeline failed in benchmarks-inputs-guard. That is the job whose image this PR had moved from registry.ddbuild.io/images/benchmarking-platform-tools-ubuntu:newest to Docker Hub alpine. I can't read that GitLab job's log, so I restored both benchmark files to main's version and listed that image as a rebuild-at-source follow-up. Those jobs only echo variables, so nothing is lost.
2. Codex P1: the five files this PR still modifies now carry # Copyright 2026, Datadog, Inc.. That thread is answered and resolved.

The description now uses the repo template and lists every follow-up in full. GitHub checks are re-running on the new head. Once they are green it can go back into the queue. If benchmarks-inputs-guard fails again on main's image, the cause is not this PR, and the job log should show whether CANDIDATE_VERSION/BASELINE_VERSION forwarding is what broke.

[eliottness]

Merge-queue rejection, root cause from the GitLab logs: the runners only admit images from approved registries.

• First attempt: ValidatingAdmissionPolicy 'third-party-registry' ... Disallowed image(s): alpine:3.24.3@sha256:be2656061b53 (job 2174477532).
• Second attempt: the reliability child pipeline failed, because its skip-reliability job had also been moved to Docker Hub alpine. benchmarks-inputs-guard was only cancelled along with it.

44cbb26 restores .gitlab-ci.yml and .gitlab/build-deploy/.gitlab-ci.yml to main, so the PR now only touches the four base-image Dockerfiles. Moving the default image to Alpine 3.24 is listed as a follow-up (it needs a mirrored alpine 3.24 in registry.ddbuild.io). I'll re-queue once the checks are green.

[eliottness]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-10-26 16:06:39 UTC ℹ️ Start processing command /merge

---

2026-10-26 16:06:43 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 0s (p90).

---

2026-10-26 16:31:11 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 71b6d63:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

• Investigate the failures and when ready, re-add your pull request to the queue!
• If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
• Any question, go check the FAQ.

Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

[eliottness]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-10-26 19:09:33 UTC ℹ️ Start processing command /merge

---

2026-10-26 19:09:37 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 0s (p90).

---

2026-10-26 19:32:03 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 69e36e5:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

• Investigate the failures and when ready, re-add your pull request to the queue!
• If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
• Any question, go check the FAQ.

Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

[eliottness]

Merge-queue attempts 3 and 4 (on 44cbb26) were both rejected in the same way, and I don't think this PR causes it:

• What fails: a child pipeline of the merge-queue run errors before creating any job (147552258 on attempt 3, 147622225 on attempt 4). fuzz-build-and-soak, the integration tests and benchmarks-inputs-guard are then only cancelled along with it.
• The diff can't trigger it: this PR now only changes .gitlab/base/Dockerfile, Dockerfile.musl, el7/Dockerfile and el8/Dockerfile. No CI rule keys on those paths, and .gitlab-ci.yml is identical to main.
• It only happens in the merge queue: in CI Visibility, child pipelines on main (10/10) and on PR branches (45/46) pass over the last 3 days. No java-profiler merge-queue run has passed in the last 14 days, and this PR is the only one that used the queue; recent PRs here are squash-merged directly.

All 84 PR checks pass on 44cbb26. I couldn't read GitLab's own failure reason for the child pipelines (the GitLab API needs AppGate, which is down on my side).