Repository navigation
feat(cws): opt-in Kata guest kernel mounts for THREAD_CONTEXT_SHARING - #8064
Merged
Merged
Conversation
Some tracer pipelines run system-tests against a docker-in-docker service of a Kubernetes pod using a Kata VM runtime. There, the docker daemon's "host" is just a container in the VM: --pid=host only yields the DIND PID namespace, and the VM's debugfs, tracefs and securityfs are not available, so the agent cannot run CWS and the scenario can never pass. The CI runner can fix that without system-tests or Kata doing anything special. On Kata, a hostPath volume under /proc or /sys is not shared from the Kubernetes node: kata-agent mounts the VM's own path instead. A runner that mounts the VM's /proc, /sys/fs/cgroup and kernel filesystems under /vm-host therefore gives the DIND container a procfs numbered in the VM's top-level PID namespace, which is how CWS eBPF numbers the processes in its events. Setting SYSTEM_TESTS_CWS_GUEST_KERNEL_MOUNTS=v1 makes this scenario bind those paths into the agent, in the docker daemon's cgroup namespace. Exposing the VM's kernel interfaces to a privileged agent is only acceptable where the job and its runner explicitly asked for it, so this is an opt-in that leaves the default topology untouched rather than a detection of the environment. Any other value stops pytest with a plain error before anything is prepared, keeping room for a future layout version. The binds use docker's Mounts API, which TestedContainer now forwards along with the cgroup namespace mode, rather than more `volumes` entries: the daemon silently creates an empty directory for a missing Binds source, whereas a missing Mounts source makes container creation fail. On a runner without the /vm-host mounts the agent therefore fails to start instead of running CWS against empty directories. That error, like a CWS self test timeout, is turned into a message naming the variable and the runner setup it needs; the missing-mount one is also written to the console, where the framework would otherwise only say that the agent can't be started. Both new container options default to None, which docker-py omits, so every other container keeps the daemon defaults. In this topology the agent also waits for its CWS self test before the weblog, which depends on it, is started. CWS learns where a process's tracer keeps its thread context either when the tracer publishes it or from a snapshot taken when CWS starts; an Apache worker forked before CWS started and that had not served a request yet has neither, and its first request's events carried no trace context in a validation run. Enforcement is disabled, because cgroup files read through /vm-host are relative to the VM's namespaces rather than the agent's, so the agent cannot reliably identify its own cgroup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
|
cataphract
marked this pull request as ready for review
October 15, 2026 11:53
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…d in configure() always bound the docker host's /sys/kernel/debug and /sys/kernel/security into the agent, and the guest kernel mounts opt-in then deleted both binds again: its Mounts use the same targets, and docker rejects two mounts on one destination. Adding the binds only in the default setup says the same thing without the set-then-delete. Behaviour is unchanged in both modes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cbeauchesne
reviewed
Oct 15, 2026
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 508901a | Docs | View more details | Give us feedback! |
The documented CWS Docker setup passes --cgroupns host alongside --pid host, so set it in every configuration, not only with the Kata guest kernel mounts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A weblog started before CWS was live has produced events without trace context, and nothing makes that specific to the Kata setup, so the agent warmup now waits for the self test in every configuration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cataphract
added a commit
to DataDog/dd-trace-php
that referenced
this pull request
Oct 19, 2026
system-tests no longer takes SYSTEM_TESTS_CWS_GUEST_KERNEL_MOUNTS (review of DataDog/system-tests#8064): THREAD_CONTEXT_SHARING now uses the guest kernel mounts whenever /vm-host exists, which is exactly when the job runs on docker-in-docker:amd64-cws. The runner tag is thus the whole opt-in, and the template only keeps it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review feedback on #8064: users should not have to know which runner their job lands on, nor set a cryptic variable for it. Whether the agent must take the VM's kernel filesystems from /vm-host is fully determined by whether the runner provides /vm-host, so the scenario now checks for it and SYSTEM_TESTS_CWS_GUEST_KERNEL_MOUNTS is gone. GITLAB_CI or CI_RUNNER_* were not used: they say where the job runs, not whether the mounts exist, and a GitLab job on a regular runner must keep the default setup. The check runs in pytest's own container, while the binds are resolved by the docker daemon. That holds because the GitLab Kubernetes executor mounts the runner's hostPath volumes into every container of the job pod, DIND service included. Probing through the daemon (a throwaway container binding /vm-host) would not rely on that, but costs a container start for a layout that today is identical on both sides; the bind mounts still fail the agent creation, with an explanation, if the two ever diverge. A /vm-host lacking one of the five sources stops pytest, rather than falling back to a setup that cannot work there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
danielmercier
approved these changes
Oct 19, 2026
cbeauchesne
approved these changes
Oct 20, 2026
2 tasks
cataphract
added a commit
to DataDog/dd-trace-php
that referenced
this pull request
Oct 20, 2026
THREAD_CONTEXT_SHARING checks that CWS stamps the active trace context
on security events. It is part of the system-tests tracer_release
group, but it cannot get CWS working on the micro-vms runners. Docker
there is a DIND service in a Kata guest: even with --pid=host, the
Agent only sees the DIND PID namespace, and the guest's debugfs,
tracefs and securityfs are absent.
CI Infra now provides docker-in-docker:amd64-cws: the same kata-qemu
DIND runners, plus hostPath volumes of the guest's /proc, /sys/fs/cgroup
and /sys/kernel/{debug,tracing,security} under /vm-host. On Kata those
host paths resolve to the guest VM's, not the node's, so the procfs is
numbered the way CWS numbers processes in its events. When /vm-host
exists, system-tests mounts these into the Agent and waits for the CWS
self test before it starts the weblog (DataDog/system-tests#8064). The
runner tag is therefore the whole opt-in.
Moving the tracer-release jobs to that pool would have been the
smallest diff, since one job runs the whole group. But that is 25
weblogs times two shards of up to four hours each, nearly all of it
scenarios that do not need the pool, which is deliberately small. So
the tracer-release jobs keep their runners and skip
THREAD_CONTEXT_SHARING, and dedicated jobs run just that scenario on
the CWS runners. Unlike tracer-release, they take minutes, so they run
on every pipeline like the other per-scenario system-tests jobs: a PR
that breaks thread context publishing finds out before merging.
They cover a few weblogs rather than all: apache-mod-8.0,
apache-mod-8.2-zts and php-fpm-8.5, i.e. both Apache MPMs and FPM. What
is checked is how the tracer publishes thread context to CWS, which
depends on the SAPI and the threading model much more than on the PHP
version.
Nothing is re-enabled here. The test stays xfail through the
system-tests Agent manifest (APPSEC-70532) until Agent 7.85 fixes the
shutdown crash, and lifting that is up to system-tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Leiyks
added a commit
to DataDog/dd-trace-php
that referenced
this pull request
Oct 22, 2026
* ci: allow system tests to run against a system-tests branch
The system tests jobs always clone system-tests at main, so a change
that needs both repositories, such as the CWS guest kernel mounts, can
only be tried in dd-trace-php CI after the system-tests side has
merged. Testing it the other way round then means merging untested
code into system-tests first.
SYSTEM_TESTS_REF, settable when starting a pipeline, names a branch or
tag to clone instead. It defaults to main, so pipelines that do not set
it behave exactly as before. git clone --branch takes branches and
tags but not commit SHAs, which is enough for trying a pull request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: run THREAD_CONTEXT_SHARING system tests on the CWS runners
THREAD_CONTEXT_SHARING checks that CWS stamps the active trace context
on security events. It is part of the system-tests tracer_release
group, but it cannot get CWS working on the micro-vms runners. Docker
there is a DIND service in a Kata guest: even with --pid=host, the
Agent only sees the DIND PID namespace, and the guest's debugfs,
tracefs and securityfs are absent.
CI Infra now provides docker-in-docker:amd64-cws: the same kata-qemu
DIND runners, plus hostPath volumes of the guest's /proc, /sys/fs/cgroup
and /sys/kernel/{debug,tracing,security} under /vm-host. On Kata those
host paths resolve to the guest VM's, not the node's, so the procfs is
numbered the way CWS numbers processes in its events. When /vm-host
exists, system-tests mounts these into the Agent and waits for the CWS
self test before it starts the weblog (DataDog/system-tests#8064). The
runner tag is therefore the whole opt-in.
Moving the tracer-release jobs to that pool would have been the
smallest diff, since one job runs the whole group. But that is 25
weblogs times two shards of up to four hours each, nearly all of it
scenarios that do not need the pool, which is deliberately small. So
the tracer-release jobs keep their runners and skip
THREAD_CONTEXT_SHARING, and dedicated jobs run just that scenario on
the CWS runners. Unlike tracer-release, they take minutes, so they run
on every pipeline like the other per-scenario system-tests jobs: a PR
that breaks thread context publishing finds out before merging.
They cover a few weblogs rather than all: apache-mod-8.0,
apache-mod-8.2-zts and php-fpm-8.5, i.e. both Apache MPMs and FPM. What
is checked is how the tracer publishes thread context to CWS, which
depends on the SAPI and the threading model much more than on the PHP
version.
Nothing is re-enabled here. The test stays xfail through the
system-tests Agent manifest (APPSEC-70532) until Agent 7.85 fixes the
shutdown crash, and lifting that is up to system-tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Alexandre Rulleau <55387832+Leiyks@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
THREAD_CONTEXT_SHARINGcannot pass in tracer pipelines that run system-tests in a docker-in-docker service of a Kata (kata-qemu) pod, such as dd-trace-php's. There the docker daemon's "host" is only a container of the VM:--pid=hostgives the DIND PID namespace, and the VM's debugfs, tracefs and securityfs are not reachable. CWS eBPF reports PIDs of the VM's top-level PID namespace, so the agent can't resolve the processes it sees.On Kata, a hostPath volume under
/procor/sysresolves to the guest VM's path, not the node's (the Kata shim deliberately doesn't share those from the node, and kata-agent binds the guest's own). CI Infra added a runner pool that mounts the VM's/proc,/sys/fs/cgroupand/sys/kernel/{debug,tracing,security}under/vm-host(docker-in-docker:amd64-cws,micro-vms-cwsin dd-source, DataDog/dd-source#128124). This PR lets the scenario use them.Changes
SYSTEM_TESTS_CWS_GUEST_KERNEL_MOUNTS=v1(opt-in; unset keeps today's setup, any other value stops pytest with an error) makes the agent:/vm-host/proc→/host/proc,/vm-host/sys/fs/cgroup→/host/sys/fs/cgroup(ro), and/vm-host/sys/kernel/{debug,tracing,security}→/sys/kernel/...(rw);--cgroupns host, and with CWS enforcement disabled (cgroup files are relative to the VM's namespaces, so the agent can't reliably identify its own cgroup);/vm-hostthe agent fails to start, with an explanation on the console, instead of silently running against empty directories.TestedContainerforwards two more docker options,cgroupnsandmounts. Both default toNone, which docker-py omits, so other containers are unaffected.docs/understand/scenarios/thread_context_sharing.md.Testing
SYSTEM_TESTS_REF=glopes/cws-test-fix, from DataDog/dd-trace-php branchglopes/cws-test-fix): job 2141686224.test_open_file_carries_trace_contextpassed (forced with-F, since it is still xfail for the Agent, APPSEC-70532); the security event carriesdd.trace_id/dd.span_id. Agent 7.84.1,apache-mod-8.0.TEST_THE_TESTfor the new configuration, and./format.sh --check.Workflow
system-tests-reviewers: ask to one of youre co-worker familiar with the tested feature.system-tests-core(slack)🚀 Once your PR is reviewed and the CI green, you can merge it!
🛟 #apm-shared-testing 🛟
🤖 Generated with Claude Code