Repository navigation
fix(json): fall back to float for integer literals that overflow int64 - #120
Open
linhongyu510 wants to merge 1 commit into
Open
linhongyu510 wants to merge 1 commit into
linhongyu510 wants to merge 1 commit into
Conversation
decodeInt64 returned a successful parse (ret>=0) with value 0 when
strconv.ParseInt hit ErrRange, so DecodeValue classified an overflowed
integer literal (e.g. 9223372036854775808) as V_INTEGER{0}. The error
was discarded and the magnitude was silently rewritten as zero.
Return the ERR_INVALID_NUMBER_FMT sentinel on overflow, matching the
existing '.'/'e'/'E' branch, so DecodeValue falls through to
decodeFloat64. This honors the documented 'allow overflow here' intent
and keeps SkipValue/DecodeValue length agreement.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
On the Go fallback JSON decoder (
internal/json.DecodeValue, built on!amd64 || go1.27), an integer literal that does not fit inint64wassilently decoded as the integer
0.decodeInt64scanned the digits, then calledstrconv.ParseInt(vv, 10, 64).On
ErrRange(overflow) it returnedret >= 0withv = 0and a non-nilerror.
DecodeValuediscards that error and, becauseret >= 0, classifies thetoken as
V_INTEGER{Iv: 0}— the magnitude is rewritten as zero and the errorswallowed.
Scope: which path this fixes
The j2t conversion has two backends:
internal/native, fromnative/*.c), built onamd64 && !go1.27. It already handled this correctly:vnumber()detectsoverflow (
is_overflow) and falls back toV_DOUBLEviaatof, neveremitting
0.impl_fallback.go,!amd64 || go1.27), which callsinternal/json.DecodeValue. This was the divergent, buggy path.This PR makes the Go fallback classify overflow exactly like the native FSM
(
V_DOUBLE), so the two backends agree. Measured on an i64 field(
Id = 9223372036854775808):-9223372036854775808(already correct)0(silent)-9223372036854775808(now matches native)0(silent)+9223372036854775807On amd64 the go<1.27 and go>=1.27 backends now produce byte-identical results.
What the fix does and does not claim
doublefield: the magnitude is no longer lost — an overflowedinteger is decoded as the nearest
float64instead of0.both backends already share (
(intN_t)dv), which is platform-dependent forout-of-range values (x86
cvttsd2si"indefinite" vs arm64 saturation). ThisPR does not claim integer output is now exact; it stops the silent
0collapse and aligns the fallback with the native reference on the same arch.
Reference / oracle
./e/Ealready returns-ERR_INVALID_NUMBER_FMT, which makesDecodeValuefall through todecodeFloat64.is_overflow→set_vt(V_DOUBLE)→parse_float).// NOTICE: allow overflow heredocuments the intent.SkipValueandDecodeValueconsume the same number length after the fix.Fix
Return the
ERR_INVALID_NUMBER_FMTsentinel onErrRange, like the./e/Ebranch, soDecodeValuefalls back todecodeFloat64. Onebehavioral line; the hot path (valid in-range integers) is untouched.
Test
internal/json/decode_number_test.go:TestDecodeIntOverflowFallsBackToDouble— positive/negative overflow,MaxInt64+1/MinInt64-1, assertsV_DOUBLE, non-zero magnitude, correctsign, and
SkipValue/DecodeValuelength agreement.TestDecodeInRangeIntegerStaysInteger— guards in-range integers.Known residual (out of scope)
A number whose magnitude also exceeds
float64range (e.g.1e400) is stilldecoded as
V_DOUBLE{0}:decodeFloat64returns(+Inf, ErrRange)and onErrRangediscards the+Infand yields0. That is pre-existing in thefloat path and unchanged by this PR; it can be addressed separately if desired.
Verification
go build ./...clean;go vetno new warnings;go test -race ./internal/jsonclean.go test ./internal/... ./thrift/...pass;conv/j2p,conv/j2t,conv/p2jpass.GOARCH=amd64cross-compileclean, amd64 binaries run under Rosetta for both backends.
DecodeValueon a normal integer ~5.8 ns/op, 0 allocs (unchanged).conv/t2j TestNobodyRequiredFields/base64_encode(
hello→helhe) reproduces on cleanmain.