Please, answer some short questions which should help us to understand your problem / question better?
- Which image of the operator are you using? master branch
- Where do you run it - cloud or metal? Kubernetes or OpenShift? Cloud (on-prem K8s) and kind (local)
- Are you running Postgres Operator in production? yes
- Type of issue? Bug report
Description
When a Postgres cluster undergoes an update that requires a rolling update of pods, the operator hits a race condition during the role synchronization phase.
It attempts to execute syncRoles(), but because Postgres is still starting up and temporarily in recovery mode, the database is read-only. This causes the sync to fail.
|
c.logger.Debug("syncing roles") |
|
if err := c.syncRoles(); err != nil { |
|
c.logger.Errorf("could not sync roles: %v", err) |
|
updateFailed = true |
|
} |
Evidence & Logs
The operator logs the following during the rollout with an 25006 (read-only transaction) error:
postgres-operator-758c4b864f-6slj6 postgres-operator time="2026-05-27T19:38:33Z" level=debug msg="syncing roles" cluster-name=default/cafebabe pkg=cluster worker=0
postgres-operator-758c4b864f-6slj6 postgres-operator time="2026-05-27T19:38:33Z" level=debug msg="closing database connection" cluster-name=default/cafebabe pkg=cluster worker=0
postgres-operator-758c4b864f-6slj6 postgres-operator time="2026-05-27T19:38:33Z" level=error msg="could not sync roles: error executing sync statements: could not execute sync requests for users: could not alter user \"postgres\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not alter user \"standby\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not alter user \"monitoring_owner\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not set custom user \"monitoring_owner\" parameters: pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not alter user \"monitoring_reader\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not set custom user \"monitoring_reader\" parameters: pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not alter user \"monitoring_writer\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not set custom user \"monitoring_writer\" parameters: pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not alter user \"monitoring\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)" cluster-name=default/cafebabe pkg=cluster worker=0
Impact
This race condition sets the cluster status to PostgresClusterStatus: UpdateFailed, even though the Kubernetes events confirm the rollout was successful:
Normal Update 2m postgres-operator Rolling update done - pods have been recreated
In the code for the sync of the roles, there is no check for the status code or a retry logic to mark this as transient issue instead of a UpdateFailed error.
|
pgSyncRequests := c.userSyncStrategy.ProduceSyncRequests(dbUsers, newUsers) |
|
if err = c.userSyncStrategy.ExecuteSyncRequests(pgSyncRequests, c.pgDb); err != nil { |
|
return fmt.Errorf("error executing sync statements: %v", err) |
|
} |
During the next reconciliation cycle, the database finishes recovery, syncRoles() succeeds, and the status returns to Running. However, this transient failure causes flapping statuses and false-positive alerts.
Suggested Fix
Since role syncing is non-critical during a brief startup window (especially if roles haven't changed), the operator should check if the database is in recovery mode or catch the 25006 error code and handle it gracefully rather than failing the entire update status.
I am happy to open a Pull Request to add a check for the recovery state if you'd like!
Please, answer some short questions which should help us to understand your problem / question better?
Description
When a Postgres cluster undergoes an update that requires a rolling update of pods, the operator hits a race condition during the role synchronization phase.
It attempts to execute
syncRoles(), but because Postgres is still starting up and temporarily in recovery mode, the database is read-only. This causes the sync to fail.postgres-operator/pkg/cluster/cluster.go
Lines 1147 to 1151 in 4d40270
Evidence & Logs
The operator logs the following during the rollout with an
25006 (read-only transaction)error:Impact
This race condition sets the cluster status to
PostgresClusterStatus: UpdateFailed, even though the Kubernetes events confirm the rollout was successful:Normal Update 2m postgres-operator Rolling update done - pods have been recreatedIn the code for the sync of the roles, there is no check for the status code or a retry logic to mark this as transient issue instead of a UpdateFailed error.
postgres-operator/pkg/cluster/sync.go
Lines 1508 to 1511 in 4d40270
During the next reconciliation cycle, the database finishes recovery,
syncRoles()succeeds, and the status returns toRunning. However, this transient failure causes flapping statuses and false-positive alerts.Suggested Fix
Since role syncing is non-critical during a brief startup window (especially if roles haven't changed), the operator should check if the database is in recovery mode or catch the
25006error code and handle it gracefully rather than failing the entire update status.I am happy to open a Pull Request to add a check for the recovery state if you'd like!