[FIP-28] Support durable freezing and retirement for historical partitions #4165

Search before asking

• I searched in the issues and found nothing similar.

Description

Parent issue: #3631

Follow-up to #4119 and PR #4120. PR #3820 is the current implementation effort related to this lifecycle.

Auto-partition retention must not delete a hot partition until every accepted record is available from a readable lake snapshot. The current transition needs a durable, failover-safe freeze and retirement protocol so that writes cannot race with metadata and replica deletion.

The protocol should:

• Persist a generation-fenced FREEZING state before partition deletion starts.
• Make every bucket leader reject new writes before append, drain writes that were already accepted, and durably acknowledge a final committed barrier offset.
• Recover the state and barriers after Coordinator or bucket-leader failover without reopening the partition for writes.
• Advance only after a readable lake snapshot covers every bucket barrier. A physically tiered offset alone is not sufficient.
• Persist a deterministic RETIRED state, publish historical routing, and only then delete the original partition metadata and replicas.
• Distinguish a retryable FREEZING response from a RETIRED response that permits clients to reroute. A generic missing or freezing response must not trigger historical routing.

This lifecycle should provide the server-authoritative transition required by #4161 and the in-flight write handoff tracked separately for PR #4120.

Willingness to contribute

• I am willing to submit a PR!