Skip to content

Clarify no-op usage #15

Open
Open

Description

@nebkat

The no-op value type is somewhat ambiguous.


My current interpretation is:

A. In an array - the element is skipped

[[]
    [#][i][2]
    [i][1]
    [N]
    [N]
    [i][2]
[1, 2]

Note that the container count doesn't include the no-op elements.

B. In an object - the key is discarded

[{]
    [#][i][2]
    [i][3][foo][i][1]
    [N]
    [i][3][bar][i][2]
{"foo": 1, "bar": 2}

Should the following also be allowed? The merits of it are somewhat questionable, as there is no point in including the key if the value is going to cause it to be removed.

[{]
    [#][i][2]
    [i][3][foo][i][1]
    [i][3][bar][N]
{"foo": 1}

C. Top level N - Error or symbol/sentinel value

In languages where it is available: Symbol('noop') (JavaScript, Dart).

Otherwise throw a dedicated error: 'N' can only occur in arrays/objects (distinct from an unexpected byte error!), and ignore if processing a stream.

Importantly, do not skip N before/after a value is found: bjdata.decode("NTN") != bjdata.decode("T").

D. In an un-delimited sequence of BJData objects / an implicit stream of values - the entry is ignored:

Sequence of BJData values in a file or streamed over network - implicitly wrapped in [ and ] so effectively the same as A.

[T]
[i][4]
[N]
[F]
true
4
false

Another way to interpret it is at the tokenizer level, where it could appear anywhere a token is expected i.e. only excluding strongly-typed contexts where it is impossible to differentiate between the value 0x4E and N:

Surrounding a value:

[N][S][i][3][foo][N][EOF]
"foo" // Not a stream, but doesn't complain because [N] are ignored

When waiting for a length:

[S][N][i][3][foo]
"foo"

When waiting for a count:

[[]
    [$][U]
    [N]
    [#][N][i][1]
    [5]
[5]

Within an array:

[[]
    [#][i][2]
    [i][1]
    [N]
    [N]
    [i][2]
[1, 2]

nlohmann/json exhibits some of these behaviors.


Don't think the latter interpretation is ever what was intended, but we would certainly benefit from being more explicit.

To summarize:

  1. Can N appear in arrays? ✅ This is the main purpose
  2. Can N appear in objects before keys? ✅ Also potentially useful
  3. Can N appear in objects after keys? ❌ Seems redundant
  4. Can N appear at top-level? ❌ It is not meaningful in that context. Throw error/special value, and handle in streams.
  5. Can N appear in special contexts (between [/$/#, S/i, etc)? ❌ Seems redundant
  6. Does N contribute towards container count? ❌ We might know we have to produce 10 values, but have to send keep-alives in between them because processing is slow. Could also ban N in counted containers.
  7. Are leading/trailing N ignored when looking for a value? ❌

If we can agree on the expected behavior I can make a PR with the appropriate spec changes.

Activity

  1. fangq commented on Apr 1, 2025

    @fangq
    Member

    @nebkat, the no-op/N marker was inherited from UBJSON, see spec

    https://ubjson.org/type-reference/value-types/#noop

    my initial understanding was that, theoretically, N is a filler it can appear anywhere, but it will be skipped. But practically, implementing parsers that can detect and skip N anywhere it appears is quite cumbersome and adds unnecessary complexity with diminished returns.

    The primary use of N is in the streaming between elements of array elements and object keys. Therefore, UBJSON spec highlighted the use of fillers in an array, but did not exclude other places.

    From the Python UBJSON implementation (py-ubjson), it appears that NOOP are only handled inside arrays and objects

    https://github.com/search?q=repo%3ANeuroJSON%2Fpybj%20TYPE_NOOP&type=code

    however, in my MATLAB/Octave implementation, I skip it in more places

    https://github.com/NeuroJSON/jsonlab/blob/da0bdeed1b3d6d82b63383c52ef874e941a280c0/loadbj.m#L367-L370

  2. nebkat commented on Apr 1, 2025

    @nebkat
    ContributorAuthor

    @fangq right, had that referenced somewhere originally but after many edits it got lost!

    If we did want to allow it to appear anywhere, we would treat [$][N][N][U] as [$][U], but the fact that that is explicitly forbidden even in UBJSON is what made me assume that it was meant to still be treated as a real value - just one that doesn't end up in any containers.

    I figure in practice it is almost never used, and where it is used it is almost exclusively in un-delimited sequences or arrays. All the other possible locations do not really provide any benefit because they would be immediately following a very short sequence of tokens (e.g. between the S and it's length, or between the [$U and its count), in which case the server should just not send them in the first place.

  3. fangq commented on Apr 2, 2025

    @fangq
    Member

    can you remind me what is the N marker support in the UBJSON/BJData decoder implementations in nlohmann/json? I haven't revisited it for a few years and can't remember what positions are allowed, what are not.

    If we did want to allow it to appear anywhere

    I am not against limiting its use in BJData decoder. There is only one other place I had used N

    https://github.com/NeuroJSON/jsonmmap/blob/main/JSON-Mmap_Specification.md#on-disk-value-replacement-with-equal-or-less-bytes

    in the above JSON mmap spec, I hope to achieve in-place on-disk value update with memmap-like operations in both JSON and BJData/UBJSON, where JSON files can use white-spaces as fillers to provide extra rooms to allow in-place overwriting of values; and in BJData/UBJSON, N can be used as a filler to expand the data-byte-length without causing compatibility issues. If we limit the places where N can be inserted, this may limit what data files can be overwrite on disk.

  4. nebkat commented on Apr 2, 2025

    @nebkat
    ContributorAuthor

    can you remind me what is the N marker support in the UBJSON/BJData decoder implementations in nlohmann/json? I haven't revisited it for a few years and can't remember what positions are allowed, what are not.

    I believe this covers all possible combinations: https://godbolt.org/z/r6z13erbo

    in the above JSON mmap spec, I hope to achieve in-place on-disk value update with memmap-like operations in both JSON and BJData/UBJSON, where JSON files can use white-spaces as fillers to provide extra rooms to allow in-place overwriting of values; and in BJData/UBJSON, N can be used as a filler to expand the data-byte-length without causing compatibility issues. If we limit the places where N can be inserted, this may limit what data files can be overwrite on disk.

    I'll give this some more thought tomorrow, but I'm pretty sure this would remain covered if we allow it in arrays/objects/top-level streams. We would only be:

    1. Restricting it is in weird places such as between $/# and S/length, which couldn't be N-ified either way.
    2. Stating explicitly that [N][T] at the top level is not equivalent to [T], rather the "ignoring" effect is a function of a "stream parser". This is to counter the way nlohmann currently treats such inputs, which feels wrong to me.

    In other words, it is a real value (albeit one that encodes the absence of a value), not a magic protocol level skip that can be placed anywhere.

    Considering the white-spacing in JSON would not work for {"foo": "value to be removed"} without also removing the key, I think it would also be reasonable to restrict [{][i][3][foo][N][}].

  5. nebkat commented on Oct 4, 2025

    @nebkat
    ContributorAuthor

    Finally returning to this.

    I think by changing no-op from the rather ambiguous "valueless" value to simply a skip signal for containers makes it much easier to understand and doesn't affect any legitimate existing uses.

    This puts it in a similar category to $ and # which are not in the types table.

  6. flynn162 commented on May 8, 2026

    @flynn162

    Hi @nebkat and @fangq,

    I'd like to use no-ops for aligning data fields inside SoA payloads. This is useful in a Procedural Call system where data packets are serialized into shared memory.

    Looking at "Row-Major vs Column-Major Layout", Draft 4 as an example:

    Full Text

    Row-Major: [$

    [$  {<schema>}  #<count>  <interleaved-payload>
    

    [...] Example: 3 particles with {x:float64, y:float64, id:uint32, active:bool}

    [ $ { i1 x D  i1 y D  i2 id m  i6 active T } # i 3
    <x₁:8><y₁:8><id₁:4><active₁:1>  <x₂:8><y₂:8><id₂:4><active₂:1>  ...
    

    Column-major layout: {$

    {$  {<schema>}  #<count>  <columnar-payload>
    

    [...] Example: Same 3 particles

    { $ { i1 x D  i1 y D  i2 id m  i6 active T } # i 3
     <x₁:8><x₂:8><x₃:8>  <y₁:8><y₂:8><y₃:8>  <id₁:4><id₂:4><id₃:4>  <T><F><T>
    

    In the case of Row-Major Layout, no-ops can be interposed between <id₁>, <active₁> for internal padding, and between <active₁:1>, <x₂:8> for external padding. Bools and fixed-strings can follow 1-byte alignment.

    In the case of Column-Major Layout, put 0 or more no-ops after # i 3, <y₃:8>, etc. to enforce a "natural alignment" of the x, y, id, and active arrays (with caveats below.)

    Caveat: The schema format should be extended to make parsing no-ops less context-dependent.

    • Without a declaration in the schema, a parser will have to look for 1 or more no-ops after the length bytes # i 3 and update a global state to expect further no-ops after <x₃:8>, <y₃:8>, etc.
    • It would be impossible to distinguish between a keep-alive signal and the intention to make all fields aligned to 2^n boundaries.
    • The onus is on the application layer to declare a proper schema that will actually optimize things. We should not put the burden of calculating alignments on the parser.

    IMO, string fields in the schema have a compact representation:

    {
      i4 name S i 32           // 32-byte fixed string
    // ...
    }
    

    Perhaps we can extend field-def and type-def to do something like this?

    {x:float64, y:float64, id:uint32, <padding>:uint8x4, active:bool, <padding>:uint8x7}

    { $ {
    i1 x D          // 8B
    i1 y D          // 8B
    i2 id m         // 4B
    Z S i 4         // declare 4 bytes of internal padding
    i6 active T     // 1B
    Z S i 7         // declare 7 bytes of external padding
    }
    N N ... // zero or more no-ops to align the `x` array
    # i 3
    // start of `x` array, no-ops disallowed
    

    In BNF:

    schema        = '{' 1*(field-def) '}'
    field-def     = name type-spec | padding-spec
    name          = int-type length string-bytes
    type-spec     = ...
    padding-spec  = 'ZS' u8-int-type length
    u8-int-type   = 'U' | 'i'
    

    For security, padding length must be positive and cannot exceed 255. IMO, encoders are free to choose between i or U. The length can be as large as required by SIMD instructions. For example: declaring up to 63 bytes of padding for AVX10/512.

  7. nebkat commented on May 10, 2026

    @nebkat
    ContributorAuthor

    This is an interesting use-case but needs careful consideration to not further complicate the idea of no-ops. Per my previous comment I think no-ops should be a "skip signal" in containers rather than a type, so this might be better solved with a different marker.


    The row major case is simple and I wouldn't be opposed to a padding mechanism within schemas for aligning fields. This would allow direct C-style struct memory mapping, and is a pretty minimal change.

    There are a number of ways these could be defined:

    • Zero length key, S + int-type + length
    • Z length key, S + int-type + length (as above)
    • P (padding) + int-type + length
    • P (padding) + length (implied 1 byte)

    Hard to say which option is the best, but I would lean towards the P type as the zero length key options are not quite as intuitive.


    The column major case appears to be more complicated as the intent is explicitly to align the array to N bytes, not to apply a fixed padding. Alignment will depend on where within a file the SoA is written.

    The onus is on the application layer to declare a proper schema that will actually optimize things. We should not put the burden of calculating alignments on the parser.

    If the parser is completely uninvolved in the process then I question whether this is really a concern for the format in the first place? We have adopted proprietary binary formats where we have finer control over alignment and maximum efficiency then use BJData within that for fields where we need the dynamic JSON capabilities.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions