Skip to content

fix(json): fall back to float for integer literals that overflow int64 - #120

Open
linhongyu510 wants to merge 1 commit into
cloudwego:mainfrom
linhongyu510:fix/json-integer-overflow-fallthrough
Open

linhongyu510 wants to merge 1 commit into
cloudwego:mainfrom
linhongyu510:fix/json-integer-overflow-fallthrough

Conversation

@linhongyu510

@linhongyu510 linhongyu510 commented Oct 8, 2026 •

Copy link
Copy Markdown

What

On the Go fallback JSON decoder (internal/json.DecodeValue, built on
!amd64 || go1.27), an integer literal that does not fit in int64 was
silently decoded as the integer 0.

decodeInt64 scanned the digits, then called strconv.ParseInt(vv, 10, 64).
On ErrRange (overflow) it returned ret >= 0 with v = 0 and a non-nil
error. DecodeValue discards that error and, because ret >= 0, classifies the
token as V_INTEGER{Iv: 0} — the magnitude is rewritten as zero and the error
swallowed.

Scope: which path this fixes

The j2t conversion has two backends:

  • amd64 native asm FSM (internal/native, from native/*.c), built on
    amd64 && !go1.27. It already handled this correctly: vnumber() detects
    overflow (is_overflow) and falls back to V_DOUBLE via atof, never
    emitting 0.
  • Go fallback (impl_fallback.go, !amd64 || go1.27), which calls
    internal/json.DecodeValue. This was the divergent, buggy path.

This PR makes the Go fallback classify overflow exactly like the native FSM
(V_DOUBLE), so the two backends agree. Measured on an i64 field
(Id = 9223372036854775808):

backend before after
amd64 native asm (go1.22) -9223372036854775808 (already correct) unchanged
amd64 Go fallback (go1.27) 0 (silent) -9223372036854775808 (now matches native)
arm64 Go fallback (go1.27) 0 (silent) +9223372036854775807

On amd64 the go<1.27 and go>=1.27 backends now produce byte-identical results.

What the fix does and does not claim

  • thrift double field: the magnitude is no longer lost — an overflowed
    integer is decoded as the nearest float64 instead of 0.
  • thrift integer field: the value still goes through the same narrowing cast
    both backends already share ((intN_t)dv), which is platform-dependent for
    out-of-range values (x86 cvttsd2si "indefinite" vs arm64 saturation). This
    PR does not claim integer output is now exact; it stops the silent 0
    collapse and aligns the fallback with the native reference on the same arch.
  • In-range integers are unchanged.

Reference / oracle

  • The sibling branch for numbers containing ./e/E already returns
    -ERR_INVALID_NUMBER_FMT, which makes DecodeValue fall through to
    decodeFloat64.
  • The native C FSM does the equivalent (is_overflow → set_vt(V_DOUBLE) →
    parse_float).
  • The comment // NOTICE: allow overflow here documents the intent.
  • SkipValue and DecodeValue consume the same number length after the fix.

Fix

Return the ERR_INVALID_NUMBER_FMT sentinel on ErrRange, like the
./e/E branch, so DecodeValue falls back to decodeFloat64. One
behavioral line; the hot path (valid in-range integers) is untouched.

Test

internal/json/decode_number_test.go:

  • TestDecodeIntOverflowFallsBackToDouble — positive/negative overflow,
    MaxInt64+1/MinInt64-1, asserts V_DOUBLE, non-zero magnitude, correct
    sign, and SkipValue/DecodeValue length agreement.
  • TestDecodeInRangeIntegerStaysInteger — guards in-range integers.

Known residual (out of scope)

A number whose magnitude also exceeds float64 range (e.g. 1e400) is still
decoded as V_DOUBLE{0}: decodeFloat64 returns (+Inf, ErrRange) and on
ErrRange discards the +Inf and yields 0. That is pre-existing in the
float path and unchanged by this PR; it can be addressed separately if desired.

Verification

  • go build ./... clean; go vet no new warnings; go test -race ./internal/json clean.
  • go test ./internal/... ./thrift/... pass; conv/j2p, conv/j2t, conv/p2j pass.
  • Cross-version: go1.27/arm64 and go1.22.5/arm64; GOARCH=amd64 cross-compile
    clean, amd64 binaries run under Rosetta for both backends.
  • Hot path: DecodeValue on a normal integer ~5.8 ns/op, 0 allocs (unchanged).
  • Pre-existing, unrelated: conv/t2j TestNobodyRequiredFields/base64_encode
    (hello→helhe) reproduces on clean main.

decodeInt64 returned a successful parse (ret>=0) with value 0 when
strconv.ParseInt hit ErrRange, so DecodeValue classified an overflowed
integer literal (e.g. 9223372036854775808) as V_INTEGER{0}. The error
was discarded and the magnitude was silently rewritten as zero.

Return the ERR_INVALID_NUMBER_FMT sentinel on overflow, matching the
existing '.'/'e'/'E' branch, so DecodeValue falls through to
decodeFloat64. This honors the documented 'allow overflow here' intent
and keeps SkipValue/DecodeValue length agreement.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants