AI Guard: tests for backend-provided extra_tags on the ai_guard span by avara1986 · Pull Request #8130 · DataDog/system-tests

avara1986 · 2026-10-21T11:27:54Z

Motivation

Jira: APPSEC-70724, AI Guard: Enrich traces with custom context from AI Guard policies.

AI Guard policies can add their own context to traces without a tracer release per tag. The evaluate response carries two optional attributes, both lists of {key, value} objects (dotted keys such as ai_guard.evaluate_id are not valid JSON:API member names, so they cannot be map keys):

"extra_tags":       [{ "key": "ai_guard.evaluate_id", "value": "<same UUID as data.id>" }],
"extra_metastruct": [{ "key": "evaluate_id", "value": "<same UUID as data.id>" }]

extra_tags: string values, each set as a tag on the ai_guard span under the key exactly as returned.
extra_metastruct: any value (object, array or scalar), each added to the ai_guard meta_struct under the key exactly as returned.
The backend owns the ai_guard. prefix and the namespace. Tracers don't add the prefix and don't filter keys.
A malformed field or entry is skipped and never changes the evaluation.

Implementations:

Python: chore(aiguard): add backend-provided tags to AI Guard spans dd-trace-py#21145 (merged, first map-shaped draft) and chore(aiguard): read backend extra_tags and extra_metastruct as key/value lists dd-trace-py#21252 (list contract, draft).
Node.js: feat(aiguard): apply backend extra_tags and extra_metastruct to AI Guard spans dd-trace-js#10865 (draft).
Backend: ddoghq/dd-source#139566.

Changes

Test_ExtraTags:
- extra_tags entries land as string tags on the span
- extra_metastruct entries (objects, arrays, scalars) land in the meta_struct
- the evaluation ID lands in both, as ai_guard.evaluate_id and evaluate_id, and matches data.id
- keys land verbatim, even outside the ai_guard. namespace (no prefix added, no filtering)
- both kinds are still present on the blocking path (DENY, 403)
- nothing crosses over: a tag never reaches the meta_struct, a meta_struct value is never flattened into a tag
Test_ExtraTagsFailSafe: the action, the span (no error) and the response stay unchanged, while every well-formed part is still applied:
- extra_tags sent as an object (the first draft's map shape) is skipped, extra_metastruct still lands
- extra_metastruct sent as an object is skipped, extra_tags still lands
- malformed entries (not an object, missing / non-string / empty key, non-string extra_tags value) are skipped one by one
Fixtures: the cassettes are written by hand. utils/scripts/gen_extra_tags_cassettes.py generates them together with tests/ai_guard/extra_tags_scenarios.json, which also records what the tracer must apply for each scenario. It reuses the request hashing of gen_redaction_cassettes.py and refuses to overwrite another test's cassette. format.sh checks it for drift, the same way it checks the redaction fixtures.
Manifests:
- Python flask-poc: enabled at v4.17.0-dev.
- Java, Node.js, Ruby, .NET, Go, PHP, cpp_httpd, cpp_nginx: missing_feature (APPSEC-70724).

Testing

Ran the AI_GUARD scenario on flask-poc:

With the dd-trace-py#21252 CI wheel (4173a14c): all 8 new tests pass, and the rest of the AI Guard suite is unaffected.
With dd-trace-py main (ed54859b, map contract from #21145 only): all 8 new tests fail, the fail-safe ones included, so none passes vacuously.

⚠️ Merge after DataDog/dd-trace-py#21252. Otherwise the Python dev CI fails these tests until that PR lands.

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Tests, manifest, weblog are modified -> you'll need a review from system-tests-reviewers: ask to one of youre co-worker familiar with the tested feature.
- Framework is modified, or non obvious usage of it -> get a review from system-tests-core (slack)

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

🤖 Generated with Claude Code

The evaluate response can carry an extra_tags object so AI Guard policies can enrich traces without a tracer release per tag (APPSEC-70724). Scalars must land as ai_guard span tags under the key exactly as returned, and structured values in the ai_guard meta struct; the backend owns the ai_guard prefix, so tracers neither add it nor filter by namespace. A malformed payload must be ignored without changing the evaluation. Cassettes are hand-crafted because the backend does not return the field yet; gen_extra_tags_cassettes.py generates them together with the scenario sidecar and is checked by format.sh. Enabled for python flask-poc (dd-trace-py#21145), missing_feature for every other tracer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-21T11:28:23Z

CODEOWNERS have been resolved as:

tests/ai_guard/extra_tags_scenarios.json                                @DataDog/k9-ai-guard
utils/build/docker/vcr/cassettes/aiguard/aiguard_evaluate_post_19cfc9ce.json  @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/build/docker/vcr/cassettes/aiguard/aiguard_evaluate_post_28917f95.json  @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/build/docker/vcr/cassettes/aiguard/aiguard_evaluate_post_2b336647.json  @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/build/docker/vcr/cassettes/aiguard/aiguard_evaluate_post_354106da.json  @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/build/docker/vcr/cassettes/aiguard/aiguard_evaluate_post_827d1449.json  @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/build/docker/vcr/cassettes/aiguard/aiguard_evaluate_post_bf6868f9.json  @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/build/docker/vcr/cassettes/aiguard/aiguard_evaluate_post_e2c13e67.json  @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/build/docker/vcr/cassettes/aiguard/aiguard_evaluate_post_f0ef2bd8.json  @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/scripts/gen_extra_tags_cassettes.py                               @DataDog/system-tests-core
format.sh                                                               @DataDog/system-tests-core
manifests/cpp_httpd.yml                                                 @DataDog/system-tests-reviewers
manifests/cpp_nginx.yml                                                 @DataDog/system-tests-reviewers
manifests/dotnet.yml                                                    @DataDog/system-tests-reviewers
manifests/golang.yml                                                    @DataDog/system-tests-reviewers
manifests/java.yml                                                      @DataDog/system-tests-reviewers
manifests/nodejs.yml                                                    @DataDog/system-tests-reviewers
manifests/php.yml                                                       @DataDog/system-tests-reviewers
manifests/python.yml                                                    @DataDog/system-tests-reviewers
manifests/ruby.yml                                                      @DataDog/system-tests-reviewers
tests/ai_guard/test_ai_guard_sdk.py                                     @DataDog/k9-ai-guard

datadog-official · 2026-10-21T12:20:28Z

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 6 Pipeline jobs failed

Testing the test | System Tests (java, dev) / End-to-end #1 / spring-boot 1 — ❌ 29 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.ffe.test_dynamic_evaluation.Test_FFE_Flag_Parse_Error_Isolation.test_valid_flag_unaffected[spring-boot] from system_tests_suite

AssertionError: Valid flag corrupted by malformed neighbour: expected &#39;expected&#39;, got &#39;default&#39; (parse error on one flag must not abort processing of the whole config)
assert &#39;default&#39; == &#39;expected&#39;
  - expected
  &#43; default

self = &lt;tests.ffe.test_dynamic_evaluation.Test_FFE_Flag_Parse_Error_Isolation object at 0x7f82cd7be270&gt;

    def test_valid_flag_unaffected(self):
        assert self.r_valid.status_code == 200, f&#34;Valid flag evaluation failed: {self.r_valid.text}&#34;
        result = json.loads(self.r_valid.text)
...

❌ tests.ffe.test_dynamic_evaluation.Test_FFE_RC_Unavailable.test_ffe_rc_unavailable_graceful_degradation[spring-boot] from system_tests_suite

AssertionError: Baseline: expected &#39;on&#39;, got &#39;default_fallback&#39;
assert &#39;default_fallback&#39; == &#39;on&#39;
  - on
  &#43; default_fallback

self = &lt;tests.ffe.test_dynamic_evaluation.Test_FFE_RC_Unavailable object at 0x7f82cd7be090&gt;

    def test_ffe_rc_unavailable_graceful_degradation(self):
        &#34;&#34;&#34;Test that cached flag configs continue working when RC is unavailable.&#34;&#34;&#34;
        # Verify tracer received 503 from RC
...

❌ tests.ffe.test_dynamic_evaluation.Test_FFE_Unknown_Fields_Tolerance.test_unknown_fields_are_ignored[spring-boot] from system_tests_suite

AssertionError: Unknown fields corrupted evaluation: expected &#39;on&#39;, got &#39;default&#39;
assert &#39;default&#39; == &#39;on&#39;
  - on
  &#43; default

self = &lt;tests.ffe.test_dynamic_evaluation.Test_FFE_Unknown_Fields_Tolerance object at 0x7f82cd7be420&gt;

    def test_unknown_fields_are_ignored(self):
        assert self.r.status_code == 200, f&#34;Flag evaluation failed: {self.r.text}&#34;
        result = json.loads(self.r.text)
...

↳ and 26 more — View all

Testing the test | System Tests (java, dev) / parametric / parametric (2) — ❌ 49 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌

tests.parametric.test_ffe.test_configuration_sources.Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery.test_bad_to_good_cold_recovery[server-e...

from system_tests_suite

AssertionError: failed to start FFE provider for bad_to_good
assert False
 &#43;  where False = &lt;bound method ParametricTestClientApi.ffe_start of &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fa61bbe0cb0&gt;&gt;()
 &#43;    where &lt;bound method ParametricTestClientApi.ffe_start of &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fa61bbe0cb0&gt;&gt; = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fa61bbe0cb0&gt;.ffe_start

self = &lt;tests.parametric.test_ffe.test_configuration_sources.Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery object at 0x7fa62952e330&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fa61bbe0cb0&gt;
mocked_backend = &lt;utils.mocked_backend._backend.MockedBackendServer object at 0x7fa61bcd1130&gt;
expected_status_codes = [500, 200]

...

❌

tests.parametric.test_ffe.test_configuration_sources.Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery.test_bad_to_unchanged_cold_preserves_no...

from system_tests_suite

mock FFE agentless backend status did not reach expected state: bad_to_unchanged 500 to 304 cold sequence; status={&#39;requests_total&#39;: 0, &#39;in_flight&#39;: 0, &#39;max_in_flight&#39;: 0, &#39;last_path&#39;: None, &#39;last_if_none_match&#39;: None, &#39;observed_dd_api_key&#39;: None, &#39;last_status_code&#39;: None, &#39;status_codes&#39;: []}

❌

tests.parametric.test_ffe.test_configuration_sources.Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery.test_malformed_cold[library_env0, param...

from system_tests_suite

mock FFE agentless backend status did not reach expected state: malformed_cold response; status={&#39;requests_total&#39;: 0, &#39;in_flight&#39;: 0, &#39;max_in_flight&#39;: 0, &#39;last_path&#39;: None, &#39;last_if_none_match&#39;: None, &#39;observed_dd_api_key&#39;: None, &#39;last_status_code&#39;: None, &#39;status_codes&#39;: []}

↳ and 46 more — View all

Testing the test | System Tests (python, dev) / End-to-end #1 / flask-poc 1 — ❌ 8 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.ai_guard.test_ai_guard_sdk.Test_ExtraTags.test_evaluate_id[flask-poc] from system_tests_suite

AssertionError: Extra tag &#39;ai_guard.evaluate_id&#39; should be &#39;9c9036fc-d7a1-5d5e-bcb3-7d62cc9fa218&#39; on the ai_guard span, got &#39;None&#39;
assert None == &#39;9c9036fc-d7a1-5d5e-bcb3-7d62cc9fa218&#39;
 &#43;  where None = &lt;built-in method get of dict object at 0x7f7a3f414040&gt;(&#39;ai_guard.evaluate_id&#39;)
 &#43;    where &lt;built-in method get of dict object at 0x7f7a3f414040&gt; = {&#39;_dd.svc_src&#39;: &#39;m&#39;, &#39;ai_guard.action&#39;: &#39;ALLOW&#39;, &#39;ai_guard.http.client_ip&#39;: &#39;172.18.0.1&#39;, &#39;ai_guard.http.useragent&#39;: &#39;system_tests rid/FUPMWQFHKGBROUBSDVXBZMYCRXXDVSCZMCDC&#39;, ...}.get

self = &lt;tests.ai_guard.test_ai_guard_sdk.Test_ExtraTags object at 0x7f7a6d695c40&gt;

    def test_evaluate_id(self):
        &#34;&#34;&#34;The evaluation ID lands as ai_guard.evaluate_id and as evaluate_id, correlating the span with the backend.&#34;&#34;&#34;
        assert self.r.status_code == 200
...

❌ tests.ai_guard.test_ai_guard_sdk.Test_ExtraTags.test_extra_metastruct[flask-poc] from system_tests_suite

AssertionError: Extra meta struct key &#39;request_context&#39; not found in the ai_guard meta struct {&#39;messages&#39;: [{&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: &#39;Draft a headline for the city council budget story.&#39;}], &#39;tag_probs&#39;: {&#39;authority-override&#39;: 0.0, &#39;data-exfiltration&#39;: 0.0, &#39;denial-of-service-tool-call&#39;: 0.0, &#39;destructive-tool-call&#39;: 0.0, &#39;indirect-prompt-injection&#39;: 0.0, &#39;instruction-override&#39;: 0.0, &#39;jailbreak&#39;: 0.0, &#39;obfuscation&#39;: 0.0, &#39;role-play&#39;: 0.0, &#39;security-exploit&#39;: 0.0, &#39;system-prompt-extraction&#39;: 0.0}}
assert &#39;request_context&#39; in {&#39;messages&#39;: [{&#39;content&#39;: &#39;Draft a headline for the city council budget story.&#39;, &#39;role&#39;: &#39;user&#39;}], &#39;tag_probs&#39;: {&#39;authority-override&#39;: 0.0, &#39;data-exfiltration&#39;: 0.0, &#39;denial-of-service-tool-call&#39;: 0.0, &#39;destructive-tool-call&#39;: 0.0, ...}}

self = &lt;tests.ai_guard.test_ai_guard_sdk.Test_ExtraTags object at 0x7f7a6d6961e0&gt;

    def test_extra_metastruct(self):
        &#34;&#34;&#34;Each extra_metastruct entry, object, array or scalar, is added to the ai_guard meta struct.&#34;&#34;&#34;
        assert self.r.status_code == 200
&gt;       interfaces.library.validate_one_span(
            self.r, validator=_assert_extra_tags_applied(self.scenario), full_trace=True
...

❌ tests.ai_guard.test_ai_guard_sdk.Test_ExtraTags.test_extra_tags[flask-poc] from system_tests_suite

AssertionError: Extra tag &#39;ai_guard.policy_name&#39; should be &#39;newsroom-default&#39; on the ai_guard span, got &#39;None&#39;
assert None == &#39;newsroom-default&#39;
 &#43;  where None = &lt;built-in method get of dict object at 0x7f7a3f64ff40&gt;(&#39;ai_guard.policy_name&#39;)
 &#43;    where &lt;built-in method get of dict object at 0x7f7a3f64ff40&gt; = {&#39;_dd.svc_src&#39;: &#39;m&#39;, &#39;ai_guard.action&#39;: &#39;ALLOW&#39;, &#39;ai_guard.http.client_ip&#39;: &#39;172.18.0.1&#39;, &#39;ai_guard.http.useragent&#39;: &#39;system_tests rid/PNLWMOSQLHSHFHOYKKMJSWWVYHKXSNCNSWST&#39;, ...}.get

self = &lt;tests.ai_guard.test_ai_guard_sdk.Test_ExtraTags object at 0x7f7a6d696ff0&gt;

    def test_extra_tags(self):
        &#34;&#34;&#34;Each extra_tags entry is set as a string tag on the ai_guard span.&#34;&#34;&#34;
        assert self.r.status_code == 200
...

↳ and 5 more — View all

View all 6 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 2be6bea | Docs | View more details | Give us feedback!}

chatgpt-codex-connector · 2026-10-22T09:42:10Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	🔄 Running since 2026-10-22T09:42:07.280389Z	`1aafd54`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

A tracer without extra_tags support also ignores a malformed payload, so Test_ExtraTagsFailSafe XPASSes under missing_feature (verified on Node.js) and the easy-win bot would wrongly declare the feature supported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Span metrics decode as float64, so a tracer that stores a boolean extra tag as a metric reports 1.0/0.0, which the type guard rejected. Reuse the redacted-tag check, which accepts exactly 1/0 from metrics, for both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The backend contract for APPSEC-70724 changed: the evaluate response carries two separate attributes, both lists of {key, value} objects, since dotted keys such as ai_guard.evaluate_id are not valid JSON:API member names. - extra_tags: string values, set as ai_guard span tags under the key exactly as returned - extra_metastruct: any value, added to the ai_guard meta_struct under the key exactly as returned - the evaluation ID is sent in both, as ai_guard.evaluate_id and evaluate_id, matching data.id Regenerate the hand-written cassettes for the list shape, with a stable UUID as data.id per scenario, and rewrite Test_ExtraTags and Test_ExtraTagsFailSafe accordingly. The fail-safe tests now also assert that the well-formed part of the response is still applied, so they no longer pass vacuously and the easy-win skip entries are dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

avara1986 mentioned this pull request Oct 21, 2026

feat(aiguard): apply backend extra_tags and extra_metastruct to AI Guard spans DataDog/dd-trace-js#10865

Draft

Merge branch 'main' into avara/APPSEC-70724-ai-guard-extra-tags

1aafd54

avara1986 marked this pull request as ready for review October 22, 2026 09:41

avara1986 requested review from a team as code owners October 22, 2026 09:41

avara1986 requested review from christophe-papazian, florentinl and smola October 22, 2026 12:45

florentinl approved these changes Oct 22, 2026

View reviewed changes

Comment thread tests/ai_guard/test_ai_guard_sdk.py Outdated

avara1986 and others added 2 commits October 22, 2026 16:01

cbeauchesne approved these changes Oct 22, 2026

View reviewed changes

watson mentioned this pull request Oct 22, 2026

test(debugger): add snapshot correlation tests with Go, .NET and Node.js weblog endpoints #7425

Merged

avara1986 and others added 2 commits October 23, 2026 09:07

Merge branch 'main' into avara/APPSEC-70724-ai-guard-extra-tags

b11f06c

hhan2024 mentioned this pull request Oct 26, 2026

Download matching OpenFeature provider for Java development tests #8184

Open

Navigation Menu

Repository navigation

AI Guard: tests for backend-provided extra_tags on the ai_guard span - #8130

AI Guard: tests for backend-provided extra_tags on the ai_guard span#8130
avara1986 wants to merge 6 commits into
mainfrom
avara/APPSEC-70724-ai-guard-extra-tags

avara1986 commented Oct 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 22, 2026

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

avara1986 commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Testing

Workflow

Uh oh!

github-actions Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot commented Oct 22, 2026

Codex Review Summary

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

avara1986 commented Oct 21, 2026 •

edited

Loading

github-actions Bot commented Oct 21, 2026 •

edited

Loading

datadog-official Bot commented Oct 21, 2026 •

edited

Loading