Skip to content

fix(kvdb): improve power-loss recovery for sector headers and KV allocation - #440

Open
brightness-810 wants to merge 3 commits into
armink:masterfrom
brightness-810:fix/kvdb-blank-using-sector
Open

brightness-810 wants to merge 3 commits into
armink:masterfrom
brightness-810:fix/kvdb-blank-using-sector

Conversation

@brightness-810

@brightness-810 brightness-810 commented Nov 13, 2026 •

Copy link
Copy Markdown

What this changes

Fix three KVDB issues observed during power-loss recovery:

  • Recover invalid sector headers during initialization when at least one
    valid header remains. Precheck headers before modification, preserve
    valid-header sectors, and reject fully invalid partitions without
    writes or erases.
  • Avoid programming ERR_HDR into fully erased data areas during USING
    sector space calculation and internal USING/FULL sector iteration.
    Preserve the correct free address and capacity.
  • Advance the initial cached free address past a status-only first KV
    header to prevent overlapping KV status tables.

Recovery-policy change: Invalid-header sectors are erased even when
automatic formatting is disabled. Unique data in non-empty affected
sectors may be lost. Earlier repairs are not rolled back if a later
repair fails.

The blank-data guards and allocation-boundary adjustment reuse the
existing FF-boundary helper and do not themselves add erases. Existing
ERR_HDR/DELETED records are not resurrected.

The helper ignores underlying read errors. These changes do not establish
ECC safety or comprehensive GC safety.

Related issue

Closes #439
Closes #441
Closes #442

Test closure

  • Added or updated cases in tests/fdb_kvdb_tc.c or tests/fdb_tsdb_tc.c
  • New cases are registered in the UTEST_UNIT_RUN(...) list of testcase()
  • Cases cover boundaries (sector full / GC triggered / blob crossing write granularity / init check failure)
  • Bug fix: the case failed before the fix and passes after, and its name carries the issue number
  • No new case needed, reason: (pure comment / docs / formatting, or a behaviour-preserving refactor already covered)

Validation currently uses a standalone host RAM-flash harness. The cases
have not yet been ported to the upstream utest files or registered with
UTEST_UNIT_RUN(...), so the corresponding items remain unchecked.
This is a behavior-changing bug fix and requires regression tests.

Local evidence:

  • A historical pre-fix replay reproduced successful initialization
    followed by a failed key lookup. Its complete 32,768-byte output matched
    the captured hardware failure image byte for byte.
  • Nine related tests passed against the current local implementation
    with KV/sector caches both disabled and enabled: 18 replay executions.
  • Coverage includes PRE_DELETE recovery into a blank USING sector, an
    EMPTY-sector control, blank-sector space calculation, internal lookup
    over a blank FULL sector, first-KV creation, a non-erased final byte,
    a partially programmed header, a later valid KV behind an erased first
    slot, and preservation of an already-failed snapshot.
  • Recovery checks include record status, value contents, application CRC,
    generated payload, write/erase counts, and readability after reopening.
  • An iterator-only comparison still programmed ERR_HDR during actual
    space recalculation and advanced the free address by one KV header.
    This supports retaining the space-calculation guard.

Remaining work:

  • Validate the isolated PF-005 patch against the selected upstream base.
  • Port the required regressions to tests/fdb_kvdb_tc.c and register them
    in testcase().
  • Include the actual issue number in the new regression case name.
  • Current results do not establish coverage of all listed boundaries,
    including actual GC triggering, allocation exhaustion, write-granularity
    crossing, and initialization I/O failures.

Write granularity covered

  • CI passed for FDB_WRITE_GRAN = 1 / 8 / 32 / 64 / 128 / 256
  • Local manual verification only, granularities: 128
  • Not verified, reason and risk: 1 / 8 / 32 / 64 / 256 have not been verified. Local tests use 128-bit granularity only. Status-table sizes, header alignment, and FF-boundary behavior at other granularities still require upstream tests and CI.

Data layout and compatibility

  • KV/TSL node format unchanged
  • Node format changed: version bumped, old data migrates by
  • Public struct or API in inc/flashdb.h changed, compatibility impact:
  • API or config macro changed, and docs/ plus docs/zh-cn/ are updated

No changes to persistent status encoding, public structures, APIs, or
configuration macros. No data migration is required.

This fix prevents erroneous status programming into blank data areas.
It does not convert existing ERR_HDR or DELETED records back to WRITE.

Power-loss semantics

  • Not on a write path
  • On a write path, covered by the fdb_xxx_deinit -> fdb_xxx_init reboot simulation
  • On a write path, not coverable by utest, verification method:

These guards affect paths used by allocation and startup recovery.
Although the guards do not write flash themselves, they prevent
read_kv() from programming ERR_HDR and therefore affect persistent
recovery behavior.

Reboot simulation currently runs in the standalone host harness, not
the upstream utest suite:

  1. Construct a valid PRE_DELETE source record and a USING destination
    sector with a fully erased data area in RAM.
  2. Call fdb_kvdb_init() to execute recovery.
  3. Verify a readable WRITE replacement, correct value contents,
    no ERR_HDR programming, and no erases.
  4. Execute fdb_kvdb_deinit() -> fdb_kvdb_init().
  5. Verify that the value remains readable, the image is unchanged, and
    no additional writes or erases occur.

The original hardware image was captured after recovery and verification
had failed. RAM reconstruction establishes a sufficient mechanism that
fully reproduces that image; it does not identify the unique instruction
or execution path at which the original power loss occurred.

These results do not replace a post-fix hardware power-cut campaign and
do not establish safety for every interruption window or GC source erase.

掉电发生在扇区更新为 USING、首条 KV 尚未写入之间时,
可能留下扇区头有效但数据区全 FF 的状态。

恢复 PRE_DELETE 记录时,内部查找可能将已分配的空白目标
误判为损坏 KV 并写入 ERR_HDR。随后搬移仍在同一地址提交
WRITE,并将旧副本标记为 DELETED,导致该 key 无法正常查询。

在空间计算和内部 KV 遍历入口检查整个数据区:
- 空间计算遇到全 FF 数据区时,保留首个空闲地址和完整容量
- 内部遍历跳过数据区全 FF 的 USING/FULL 扇区
- 非空数据区保持原有处理,不改变存储格式或增加擦除
An interrupted first KV write can leave the trailing-FF boundary
inside its status table. Caching this boundary as free space allows
recovery to allocate a replacement inside the incomplete header.

A subsequent search can mark the residual header ERR_HDR at a byte
that also belongs to the replacement's DELETED slot. Committing WRITE
does not clear that marker, and deleting the source leaves no readable
copy.

Advance the initial cached free address past the complete first KV
header when the FF boundary falls within its status-table interval.
Keep the cache-hit condition, capacity calculation, and on-flash
format unchanged, without adding flash reads or writes.
@brightness-810 brightness-810 changed the title fix(kvdb): 避免将空白 USING 扇区误标记为 ERR_HDR fix(kvdb): improve power-loss recovery for sector headers and KV allocation Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant