segmentation fault (out of bounds error) when using BitIntegers.jl

[lampretl]

After starting julia -t 12 --check-bounds=yes I run:

julia> using Random, BitIntegers, Polyester

julia> @define_integers 24;

julia> function f!(x,p) @batch for i=1:length(x) x[i] = p[x[i]] end end;

julia> n=10^6;  p=randperm(n);  x=UInt24.(rand(1:n,10^8));  @assert all(1≤v≤n for v∈x);

julia> f!(x,p)
ERROR: BoundsError: attempt to access 1000000-element StrideArraysCore.PtrArray{Int64, 1, (1,), Tuple{Int64}, Tuple{Nothing}, Tuple{Static.StaticInt{1}}} at index [4235520]
Stacktrace:
  [1] throw_boundserror(A::StrideArraysCore.PtrArray{Int64, 1, (1,), Tuple{Int64}, Tuple{Nothing}, Tuple{Static.StaticInt{1}}}, I::Tuple{Int64})
    @ Base ./essentials.jl:14
  [2] checkbounds
    @ ./abstractarray.jl:699 [inlined]
  [3] getindex
    @ ~/.julia/packages/StrideArraysCore/t77vK/src/ptr_array.jl:1041 [inlined]
  [4] _getindex
    @ ./abstractarray.jl:1336 [inlined]
  [5] getindex
    @ ./abstractarray.jl:1312 [inlined]
  [6] macro expansion
    @ ./REPL[3]:1 [inlined]
  [7] #1
    @ ~/.julia/packages/Polyester/almvr/src/closure.jl:309 [inlined]
  [8] macro expansion
    @ ~/.julia/packages/Polyester/almvr/src/batch.jl:246 [inlined]
  [9] _batch_no_reserve
    @ ~/.julia/packages/Polyester/almvr/src/batch.jl:168 [inlined]
 [10] batch
    @ ~/.julia/packages/Polyester/almvr/src/batch.jl:334 [inlined]
 [11] macro expansion
    @ ~/.julia/packages/Polyester/almvr/src/closure.jl:456 [inlined]
 [12] f!(x::Vector{UInt24}, p::Vector{Int64})
    @ Main ./REPL[3]:1
 [13] top-level scope
    @ REPL[5]:1

My version is:

julia> versioninfo()
Julia Version 1.11.5
Commit 760b2e5b739 (2025-04-14 06:53 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, skylake)
Threads: 12 default, 0 interactive, 13 GC (on 12 virtual cores)

(@v1.11) pkg>  st
  [f517fe37] Polyester v0.7.18
  [c3b6d118] BitIntegers v0.3.5

The error does not appear when I use UInt32 instead. I also reported this in BitIntegers here?

[lampretl]

Any reply would be welcome.

[vchuravy]

Could be #168

[lampretl]

Here is the summary/explanation about this bug by AI (claude code + Fable):

Summary

When a @batch loop reads or writes a Vector{T} where T is a primitive type whose sizeof is not a power of two (so sizeof(T) != Base.aligned_sizeof(T)), the chunks executed by spawned tasks access memory at the wrong addresses and produce silently wrong results — no error, no crash. The chunk run by the calling thread is correct, so small trip counts (or minbatch >= n) mask the bug entirely.

Found in the wild with BitIntegers.UInt24, but it reproduces with a bare primitive type, no packages beyond Polyester needed.

MWE

using Polyester

primitive type UInt24 24 end   # any primitive type with non-power-of-2 sizeof
u24(i::Int) = Core.Intrinsics.trunc_int(UInt24, i)

sizeof(UInt24), Base.aligned_sizeof(UInt24), Base.elsize(Vector{UInt24})  # (3, 4, 4)

n = 100_000
A = u24.(1:n);  B = fill(u24(0), n)

@batch for i = 1:n B[i] = A[i] end
B == A                                  # false (with -t 6; first mismatch at i = 75001)

B .= Ref(u24(0))
@batch minbatch=n for i = 1:n B[i] = A[i] end
B == A                                  # true — single batch, nothing spawned

Diagnosis

Julia stores array elements of such types in slots of Base.aligned_sizeof(T) bytes (4 for a 24-bit type), but the StrideArraysCore.PtrArray that @batch converts captured arrays into for its spawned tasks indexes with a byte stride of sizeof(T) (3). Reading the same memory both ways:

  using StrideArraysCore
  A = u24.(1:8)
  P = StrideArraysCore.PtrArray(A)
  val(x) = Core.Intrinsics.zext_int(Int, x)
  [val(A[i]) for i = 1:8]   # [1, 2, 3, 4, 5, 6, 7, 8]
  [val(P[i]) for i = 1:8]   # [1, 512, 196608, 0, 4, 1280, 393216, 0]

P[2] reads bytes {0x00,0x02,0x00} straddling element 1's padding byte and element 2's low bytes → 512, and each subsequent read drifts one more byte off. Writes mirror this. Because it's raw pointer arithmetic, nothing checks it, and since 3n < 4n the accesses never leave the allocation, so it can't even segfault.

The MWE's failure pattern confirms it: a stride-3 read from A combined with a stride-3 write into B is a verbatim copy of the first 3n bytes, so exactly the first ¾ of the elements (75 000 of 100 000) come out right by accident and the tail is never written. Only the spawned chunks go through the PtrArray bundle — the calling thread's chunk runs on the original Vector — which is why single-batch runs are correct and the bug hides until the loop is big enough to spawn.

Expected behavior

Either index with the true element stride (Base.elsize(Vector{T}) / Base.aligned_sizeof(T) instead of sizeof(T)), or refuse/fall back to capturing the plain array for eltypes where the two differ. Silent corruption is the worst outcome.

Root cause presumably lives in StrideArraysCore's stride computation rather than in Polyester itself — happy to have this transferred if that's the better home.

Proposed fix

The conversion happens in object_and_preserve (StrideArraysCore, thread_compatible.jl), which Polyester calls on every captured variable when packaging work for spawned tasks. That function already knows how to pass an array through untouched when it can't safely take the raw-pointer fast path — it just doesn't recognize this case.

The minimal fix: only convert an array to a PtrArray when its elements are stored without padding, i.e. when sizeof(T) == Base.aligned_sizeof(T); otherwise fall back to passing the original array through, exactly as is already done for non-isbits eltypes.
This changes nothing for all currently-working element types, and turns the silent corruption into correct (just not pointer-optimized) execution. I've verified that passing such arrays through unwrapped makes the MWE above produce correct results.

A fuller fix — teaching PtrArray itself to step by the padded element stride (Base.aligned_sizeof(T) / Base.elsize) instead of sizeof(T) — could restore the fast path for these types later, but the guard above already makes them safe.

(ThreadingUtilities and PolyesterWeave appear uninvolved: the former only stores/loads the already-packaged argument bundle, the latter only manages thread bookkeeping.)

Versions

• Julia 1.12.6, -t 6, x86-64 Linux
• Polyester v0.7.19
• StrideArraysCore v0.5.9
• ManualMemory v0.1.8

[lampretl]

Would it be possible if someone could have a look at the suggested solution and integrate it into the code? @vchuravy @carstenbauer @chriselrod

I can massage my AI to give a more specific refactor, but I myself have not nearly enough experience with threading and Polyester, to be able to come up with a proposed merge request.

[chriselrod]

Ha, the proposed fix is classic AI. Roughly:

PtrArray is broken for these element types. So let's add a workaround that tries to identify these element types and do something else instead.

Obvious the correct fix is to fix PtrArray, not to identify when it's going to be wrong to avoid using it.