Repository navigation
Description
The no-op value type is somewhat ambiguous.
My current interpretation is:
A. In an array - the element is skipped
[[]
[#][i][2]
[i][1]
[N]
[N]
[i][2]
[1, 2]Note that the container count doesn't include the no-op elements.
B. In an object - the key is discarded
[{]
[#][i][2]
[i][3][foo][i][1]
[N]
[i][3][bar][i][2]
{"foo": 1, "bar": 2}Should the following also be allowed? The merits of it are somewhat questionable, as there is no point in including the key if the value is going to cause it to be removed.
[{]
[#][i][2]
[i][3][foo][i][1]
[i][3][bar][N]
{"foo": 1}C. Top level N - Error or symbol/sentinel value
In languages where it is available: Symbol('noop') (JavaScript, Dart).
Otherwise throw a dedicated error: 'N' can only occur in arrays/objects (distinct from an unexpected byte error!), and ignore if processing a stream.
Importantly, do not skip N before/after a value is found: bjdata.decode("NTN") != bjdata.decode("T").
D. In an un-delimited sequence of BJData objects / an implicit stream of values - the entry is ignored:
Sequence of BJData values in a file or streamed over network - implicitly wrapped in [ and ] so effectively the same as A.
[T]
[i][4]
[N]
[F]
true
4
falseAnother way to interpret it is at the tokenizer level, where it could appear anywhere a token is expected i.e. only excluding strongly-typed contexts where it is impossible to differentiate between the value 0x4E and N:
Surrounding a value:
[N][S][i][3][foo][N][EOF]
"foo" // Not a stream, but doesn't complain because [N] are ignoredWhen waiting for a length:
[S][N][i][3][foo]
"foo"When waiting for a count:
[[]
[$][U]
[N]
[#][N][i][1]
[5]
[5]Within an array:
[[]
[#][i][2]
[i][1]
[N]
[N]
[i][2]
[1, 2]nlohmann/json exhibits some of these behaviors.
Don't think the latter interpretation is ever what was intended, but we would certainly benefit from being more explicit.
To summarize:
- Can
Nappear in arrays? ✅ This is the main purpose - Can
Nappear in objects before keys? ✅ Also potentially useful - Can
Nappear in objects after keys? ❌ Seems redundant - Can
Nappear at top-level? ❌ It is not meaningful in that context. Throw error/special value, and handle in streams. - Can
Nappear in special contexts (between[/$/#,S/i, etc)? ❌ Seems redundant - Does
Ncontribute towards container count? ❌ We might know we have to produce 10 values, but have to send keep-alives in between them because processing is slow. Could also banNin counted containers. - Are leading/trailing
Nignored when looking for a value? ❌
If we can agree on the expected behavior I can make a PR with the appropriate spec changes.
Activity
fangq commentedon Apr 1, 2025
@nebkat, the
no-op/Nmarker was inherited from UBJSON, see spechttps://ubjson.org/type-reference/value-types/#noop
my initial understanding was that, theoretically,
Nis a filler it can appear anywhere, but it will be skipped. But practically, implementing parsers that can detect and skipNanywhere it appears is quite cumbersome and adds unnecessary complexity with diminished returns.The primary use of
Nis in the streaming between elements of array elements and object keys. Therefore, UBJSON spec highlighted the use of fillers in an array, but did not exclude other places.From the Python UBJSON implementation (py-ubjson), it appears that NOOP are only handled inside arrays and objects
https://github.com/search?q=repo%3ANeuroJSON%2Fpybj%20TYPE_NOOP&type=code
however, in my MATLAB/Octave implementation, I skip it in more places
https://github.com/NeuroJSON/jsonlab/blob/da0bdeed1b3d6d82b63383c52ef874e941a280c0/loadbj.m#L367-L370
nebkat commentedon Apr 1, 2025
@fangq right, had that referenced somewhere originally but after many edits it got lost!
If we did want to allow it to appear anywhere, we would treat
[$][N][N][U]as[$][U], but the fact that that is explicitly forbidden even in UBJSON is what made me assume that it was meant to still be treated as a real value - just one that doesn't end up in any containers.I figure in practice it is almost never used, and where it is used it is almost exclusively in un-delimited sequences or arrays. All the other possible locations do not really provide any benefit because they would be immediately following a very short sequence of tokens (e.g. between the
Sand it's length, or between the[$Uand its count), in which case the server should just not send them in the first place.fangq commentedon Apr 2, 2025
can you remind me what is the
Nmarker support in the UBJSON/BJData decoder implementations in nlohmann/json? I haven't revisited it for a few years and can't remember what positions are allowed, what are not.I am not against limiting its use in BJData decoder. There is only one other place I had used
Nhttps://github.com/NeuroJSON/jsonmmap/blob/main/JSON-Mmap_Specification.md#on-disk-value-replacement-with-equal-or-less-bytes
in the above JSON mmap spec, I hope to achieve in-place on-disk value update with memmap-like operations in both JSON and BJData/UBJSON, where JSON files can use white-spaces as fillers to provide extra rooms to allow in-place overwriting of values; and in BJData/UBJSON,
Ncan be used as a filler to expand the data-byte-length without causing compatibility issues. If we limit the places whereNcan be inserted, this may limit what data files can be overwrite on disk.nebkat commentedon Apr 2, 2025
I believe this covers all possible combinations: https://godbolt.org/z/r6z13erbo
I'll give this some more thought tomorrow, but I'm pretty sure this would remain covered if we allow it in arrays/objects/top-level streams. We would only be:
$/#andS/length, which couldn't beN-ified either way.[N][T]at the top level is not equivalent to[T], rather the "ignoring" effect is a function of a "stream parser". This is to counter the way nlohmann currently treats such inputs, which feels wrong to me.In other words, it is a real value (albeit one that encodes the absence of a value), not a magic protocol level skip that can be placed anywhere.
Considering the white-spacing in JSON would not work for
{"foo": "value to be removed"}without also removing the key, I think it would also be reasonable to restrict[{][i][3][foo][N][}].nebkat commentedon Oct 4, 2025
Finally returning to this.
I think by changing
no-opfrom the rather ambiguous "valueless" value to simply a skip signal for containers makes it much easier to understand and doesn't affect any legitimate existing uses.This puts it in a similar category to
$and#which are not in the types table.flynn162 commentedon May 8, 2026
Hi @nebkat and @fangq,
I'd like to use no-ops for aligning data fields inside SoA payloads. This is useful in a Procedural Call system where data packets are serialized into shared memory.
Looking at "Row-Major vs Column-Major Layout", Draft 4 as an example:
Full Text
In the case of Row-Major Layout, no-ops can be interposed between
<id₁>,<active₁>for internal padding, and between<active₁:1>,<x₂:8>for external padding. Bools and fixed-strings can follow 1-byte alignment.In the case of Column-Major Layout, put 0 or more no-ops after
# i 3,<y₃:8>, etc. to enforce a "natural alignment" of thex,y,id, andactivearrays (with caveats below.)Caveat: The schema format should be extended to make parsing no-ops less context-dependent.
# i 3and update a global state to expect further no-ops after<x₃:8>,<y₃:8>, etc.IMO, string fields in the schema have a compact representation:
Perhaps we can extend
field-defandtype-defto do something like this?{x:float64, y:float64, id:uint32, <padding>:uint8x4, active:bool, <padding>:uint8x7}In BNF:
For security, padding length must be positive and cannot exceed 255. IMO, encoders are free to choose between
iorU. The length can be as large as required by SIMD instructions. For example: declaring up to 63 bytes of padding for AVX10/512.nebkat commentedon May 10, 2026
This is an interesting use-case but needs careful consideration to not further complicate the idea of no-ops. Per my previous comment I think no-ops should be a "skip signal" in containers rather than a type, so this might be better solved with a different marker.
The row major case is simple and I wouldn't be opposed to a padding mechanism within schemas for aligning fields. This would allow direct C-style struct memory mapping, and is a pretty minimal change.
There are a number of ways these could be defined:
S+int-type+ lengthZlength key,S+int-type+ length (as above)P(padding) +int-type+ lengthP(padding) + length (implied 1 byte)Hard to say which option is the best, but I would lean towards the
Ptype as the zero length key options are not quite as intuitive.The column major case appears to be more complicated as the intent is explicitly to align the array to N bytes, not to apply a fixed padding. Alignment will depend on where within a file the SoA is written.
If the parser is completely uninvolved in the process then I question whether this is really a concern for the format in the first place? We have adopted proprietary binary formats where we have finer control over alignment and maximum efficiency then use BJData within that for fields where we need the dynamic JSON capabilities.