Security: No Content Safety Filtering or Output Guardrails on AI Responses

[BenjaminMichaelis]

Summary

The AI chat pipeline returns raw Azure OpenAI output directly to users with no output validation, PII detection, content safety filtering, or structured output enforcement. This means harmful, policy-violating, or sensitive content generated by the model would be passed straight through.

Affected Code

EssentialCSharp.Web/Controllers/ChatController.cs — both /api/chat/message and /api/chat/stream:

// Non-streaming: raw model text returned without any inspection
return Ok(new ChatMessageResponse { Response = response, ... });

// Streaming: raw delta text written directly to SSE stream
var eventData = JsonSerializer.Serialize(new { type = "text", data = text });
await Response.WriteAsync($"data: {eventData}\n\n", cancellationToken);

AIChatService.cs — GetChatCompletionCore returns responseText from the model with no post-processing.

There is also no content safety check on user input before it is sent to the model.

Risk

OWASP AI Agent Security — Risk #5: Output Validation & Guardrails / Risk #8: Sensitive Data Exposure

• A jailbroken or misconfigured model could return harmful content, credentials from its training data, or PII.
• No defense against the model leaking content from its system prompt or injected context back to the user.
• No enforcement that outputs conform to expected shape (e.g., markdown response about C# topics only).

Recommended Mitigations

1. Integrate Azure AI Content Safety for both input (before sending to model) and output (before returning to client):

   // In AIChatService or a middleware wrapper:
   var inputSafety = await contentSafetyClient.AnalyzeTextAsync(prompt);
   if (inputSafety.Value.CategoriesAnalysis.Any(c => c.Severity > 2))
       return ("Content policy violation.", null);

2. Apply a system prompt instruction bounding output scope: "Only respond about C# and Essential C# book topics."
3. Screen output for PII patterns (emails, phone numbers, SSNs) before streaming/returning:

   private static readonly Regex PiiPattern = new(@"\b\d{3}-\d{2}-\d{4}\b|\b[\w.]+@[\w.]+\b", RegexOptions.Compiled);

4. Set max_output_tokens on the ResponseCreationOptions to enforce an output length cap and prevent runaway generation costs.

References

• OWASP AI Agent Security Cheat Sheet — §5 Output Validation & Guardrails
• Azure AI Content Safety
• OWASP LLM Top 10 — LLM02: Insecure Output Handling

[github-actions]

🤖 This is an automated response from Repo Assist.

I reviewed the current chat pipeline in ChatController and AIChatService. The issue remains a valid design gap for input/output moderation: model text is returned or streamed without a content-safety decision, and the streaming path needs a policy for buffering or terminating a response when moderation rejects it.

Some items in the original recommendation are already covered: AIChatService sets MaxOutputTokenCount, MCP tools are allowlisted, and the controller applies rate limiting and CAPTCHA validation. The remaining work is not dependency-neutral, because it requires choosing and configuring a moderation service, defining severity thresholds and user-facing error semantics, and applying the same policy to both /api/chat/message and /api/chat/stream.

Before implementation, maintainers should select the moderation provider and confirm whether streamed responses may be buffered for moderation or must be cancelled on a post-generation check. Once those decisions are tracked, this can be implemented with focused service and controller tests.

Generated by 🌈 Repo Assist, see workflow run. Learn more.
_{Comment /repo-assist to run again}

_{Add this agentic workflow to your repo}

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/repo-assist.md@4bc8419fad05e6b032741cbfd189986700bcf71c