Skip to content

docs: troubleshoot x509 503s and document CA replacement - #142

Merged
ThomasK33 merged 2 commits into
mainfrom
docs/apiserver-tls-troubleshooting
Sep 25, 2026
Merged

ThomasK33 merged 2 commits into
mainfrom
docs/apiserver-tls-troubleshooting

Conversation

@ThomasK33

Copy link
Copy Markdown
Member

Refs #137. Step 3 of 3: docs only.

With #140 and #141, kube-apiserver now verifies the aggregated API server's certificate through the APIService caBundle. This PR documents how to diagnose failures and how to replace the CA.

What changed

  • docs/how-to/troubleshooting.md: a new section, "Proxied requests fail with 503 and an x509 error".
    • It explains that Available=True does not rule this failure out, because kube-apiserver's availability check does not verify the certificate. We found this while testing feat: verify the aggregated API server certificate through the APIService caBundle #141.
    • It shows how to confirm the kube-apiserver x509 log line and how to compare the Secret's ca.crt with the APIService caBundle.
    • It covers four causes, each with a fix: the APIService is opted out, the RBAC is missing, another tool writes caBundle (cert-manager or GitOps), or a CA rotation is in progress.
    • The existing ServiceUnavailable section now tells the server's own specific messages apart from kube-apiserver's generic 503.
  • docs/how-to/deploy-aggregated-apiserver.md: a new "Replace the CA" procedure. It records the fingerprint, deletes the Secret, restarts every replica, then checks the fingerprint, the caBundle and a proxied request. It also states the expected 503 window (about 10 s with two replicas in the feat: verify the aggregated API server certificate through the APIService caBundle #141 dogfood) and covers the opt-out case. Cross-links go both ways.
  • .cspell.json: adds apiservices, kubeadm and noout.

Validation

  • cspell, markdownlint-cli2 0.18.1 and make docs-check (mkdocs build --strict) pass. All new anchors resolve in the built site.
  • The quoted log strings (missing permission …, Set the APIService caBundle …) and the field manager coder-k8s-apiservice-cabundle exist in the code. The replacement and diagnostic commands match the ones run in the feat: verify the aggregated API server certificate through the APIService caBundle #141 Kind dogfood.

Once this lands, #137's acceptance criteria are met. I will close it by hand after checking the landing.


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: medium

Add a troubleshooting section for proxied requests that fail with 503
and an x509 error while the APIService reports Available=True, and a
step-by-step "Replace the CA" procedure with verification commands.

Refs #137

Change-Id: Ia1816aa476aa1723a899e9b02d89b106ea37af1d
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Refs #137

Change-Id: Ia1efd0c5142726674d90bd5bb5d3c350fd2abd52
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-25T11:52:28.884220Z 0bb55df PR opened
🔒 Security Review ✅ Completed 2026-09-25T11:52:25.727029Z 0bb55df Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 0bb55dff2d

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33
ThomasK33 added this pull request to the merge queue Sep 25, 2026
Merged via the queue into main with commit 6a2e09d Sep 25, 2026
13 checks passed
@ThomasK33
ThomasK33 deleted the docs/apiserver-tls-troubleshooting branch September 25, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant