Repository navigation
Proxy exits with wrong code after SIGTERM #1529
Description
Bug Description
Graceful termination seems impossible in v2. Whether I set --max-sigterm-delay or not doesn't matter, process always has exit code 137, even when there are no active connections.
Example code (or command)
- name: cloudsql-proxy
image: {{ $root.Values.sqlProxy.image }}
command:
- '/kubexit/kubexit'
- '/cloud-sql-proxy'
- '--address=127.0.0.1'
- '--max-sigterm-delay=10s'
- '--structured-logs'
- '--prometheus'
{{- range $k, $v := $root.Values.sqlProxy.connections }}
- '{{ $v.dsn }}?port={{ $v.port }}'
{{- end }}
lifecycle:
preStop:
exec:
command: ['sleep', '10']Stacktrace
How to reproduce
- ?
- ?
Environment
- OS type and version: GKE 1.24.4-gke.800
- Cloud SQL Proxy version (
./cloud-sql-proxy --version): 2.0.0-preview.2
Activity
enocom commentedon Nov 6, 2022
Thanks for the issue @cdaguerre.
Looking at these exit codes again, I see that SIGTERM is conventionally 143 and SIGKILL is 137.
I think what this means is that we should be returning exit code 143 instead of 137 here: https://github.com/GoogleCloudPlatform/cloud-sql-proxy/blob/main/cmd/errors.go#L29.
Cf. https://komodor.com/learn/sigterm-signal-15-exit-code-143-linux-graceful-termination/
Is that your expectation?
fix: SIGTERM results in exit code 143
cdaguerre commentedon Nov 7, 2022
Hey @enocom
Actually in v1, even when the proxy received a
SIGTERM, the exit code would be0if there were no more active connections.The use case is for running the proxy as a sidecar in kubernetes
Jobs that run to completion.You need to have a way to kill the cloudsql-proxy sidecar container when the main container finished.
Any exit code other then 0 from one of the containers causes the job to be considered failed.
Basically, ideal behaviour to gracefully terminate on
SIGTERMtaking the--max-sigterm-delayinto account.0as soon as there is no active connection anymore137(SIGKILL) if there is an active connection0if there is no active connection (probably already exited before delay...)EDIT: added link to v1 code:
cloud-sql-proxy/proxy/proxy/client.go
Line 630 in 6af6a92
[-]Graceful termination in v2[/-][+]Proxy exits with wrong code after SIGTERM[/+]enocom commentedon Nov 9, 2022
Thanks for the clarification.
There are two issues here:
Since we've made this worse in v2 with the new non-0 exit code, I'll prioritize adding a
/quitquitquitendpoint to solve this.fix: return correct exit code on SIGTERM (#1530)
adamstrawson commentedon Feb 14, 2023
Hey @enocom Just to dig up this issue again, it seems that we're hitting the same behaviour that @cdaguerre mentions here despite having no open connections (exit code: 143)
We use
cloud-sql-proxyon Kubernetes Jobs, wrapped aroundtimeoutso that it gracefully shuts down the container after completion. Because of this, and the 143 exit code, kubernetes sees the job as failed and puts it into an Error state.terminated - Error (exit code: 143)Prior to v2, this behaved gracefully as expected.
Happy to open as a new issue if you'd prefer.
enocom commentedon Feb 14, 2023
We recently added support for a
/quitquitquitendpoint to support exactly this use case. See here for details: https://github.com/GoogleCloudPlatform/cloud-sql-proxy#localhost-admin-server.