Skip to content

GPU backend, NumPy-compatible ops, and cuPyNumeric example ports - #10

Open
AdvaitTahilyani wants to merge 15 commits into
charmtylesfrom
gpu-backend
Open

AdvaitTahilyani wants to merge 15 commits into
charmtylesfrom
gpu-backend

Conversation

@AdvaitTahilyani

Copy link
Copy Markdown

Needs UIUC-PPL/charmtyles#3. Reconverse on Slingshot also needs charmplusplus/reconverse#281.

GPU backend (NVIDIA)

  • The MLIR JIT lowers fused elementwise kernels through MLIR's GPU→NVVM pipeline for the device's actual architecture. Launches go on each chare's own CUDA stream, with no host syncs.
  • GPU-direct messaging between chares, with pooled device buffers.
  • Compiled kernels and cubins are cached on disk (~/.cache/charmnumeric/jit, set with CNUM_JIT_CACHE_DIR). With a warm cache, a kernel's first call takes about 3 ms instead of 30 ms.
  • cuBLAS/cuSOLVER for GEMM and linear algebra; Kokkos for the remaining device code.

NumPy semantics

  • dtypes: bool, int8–64, uint8–64, float16/32/64 and complex64/128, with NEP 50 promotion.
  • 0-d arrays, views and linked copies, reshape and broadcasting, and arrays with more than 3 dims (folded onto 3-D storage).
  • Integer indexing drops a dimension. Results match NumPy for signed zeros, NaN and integer division/modulo.

New operations

  • Elementwise: math, comparison, bitwise and logical ops, where and clip.
  • Reductions: sum, mean, var, std, arg-min/max, bincount, norm and more, along any axis.
  • Scans, sorts and quantiles.
  • Linear algebra: Cholesky, QR, SVD, solve and lstsq.
  • Indexing: gather, scatter, boolean masks, take and put.
  • einsum, tensordot, inner and outer; signal ops such as convolve.
  • RNG and array creation.

cuPyNumeric examples
All 25 examples in examples/cupynumeric are the upstream files unchanged plus a 2-line header. Each one checks its results against NumPy.

Performance (A100)

  • Elementwise add, 1e8 elements: 88 ms → 2.2 ms.
  • Mask indexing, 1e7 elements: 550 ms → 5 ms.
  • cfd: 66 s → 8 s. kmeans_slow: 5.8 s → 1.2 s. linreg/logreg at 4 PEs: about 19 s → 1.6 s.
  • 7 of 10 compared examples match or beat cuPyNumeric on 1 GPU. gemm is still about 3x slower.

Tested

  • Suites:
    • 15 original examples, 25 cuPyNumeric ports and 24 op suites (examples/ops) in every configuration.
    • Single-node runs also ran 42 differential-test repros against NumPy and a 400-seed fuzzer.
  • Charm++ netlrts:
    • CPU at +p1, +p2, and 2 nodes at +p8.
    • GPU on 1×A100, 1×A40, 4×A40 (one process per GPU), and 2 nodes × 2 A40.
    • Everything passes.
  • Charm++ on Reconverse (recharm):
    • CPU with 1–4 PEs, 2 processes, and 2 nodes.
    • GPU on 1 and 4 GPUs.
    • Everything passes except two failures in the Slingshot transport, which reconverse#281 fixes.

Notes

  • netlrts multi-node needs +eth, because 9000-byte datagrams get fragmented on 1500-MTU networks.
  • Multi-GPU nodes need one visible GPU per process.
  • Known limits:
    • A get of an array larger than about 2 GiB fails, because CCS replies carry a 32-bit length.
    • cumsum's first call pays CUDA's lazy module loading; CUDA_MODULE_LOADING=EAGER avoids it.

🤖 Generated with Claude Code

AdvaitTahilyani and others added 5 commits October 8, 2026 23:21
GPU (NVIDIA) execution:
- JIT: lower fused linalg kernels through MLIR's GPU-to-NVVM pipeline into a
  host launcher (PTX, MLIR CUDA runtime); temporaries in managed memory.
- Fix the nvcc build (jit_api.hpp split, nested device lambdas, Eigen on
  device pointers, server_obj defines, charmc link of Kokkos/CUDA libs).
- Device messaging: drop invalid .ci post entries, carry send_id in the
  callback refnum, look up posted receive buffers (Charm++ passes null data).
- Skip the cache_dag plan signature in receive_dag.

Backend/frontend fixes: rank and broadcast flags of shared leaves, size-1
view broadcasting, solver exponential walk, temps still held by user code,
stale in-kernel reads, per-array SUMMA tiles, start epochs of new
partitions, cross-dim set_region offsets, gather offsets, retire ordering,
fused-kernel size cap, kernel-cache collision on operand aliasing.

New ops: elementwise math/comparisons/where/clip (opcodes 20-49), bulk
upload, counter-based RNG and iota creation ops (50-52), full and axis
reductions and bincount (60-61).

Examples: ports of all 25 cuPyNumeric examples (examples/cupynumeric) and
op test suites (examples/ops).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…leanup

New ops:
- transpose/.T/swapaxes (2-D/3-D), vecmat and transposed-operand matmul
- concatenate/hstack/vstack/stack, take (host indices), out= on ufuncs and
  matmul/dot, native eye, newaxis, int32 array-name limit
- astype, NumPy dtype promotion (operators, elementwise ops, assignment),
  cumsum/cumprod/nancumsum/nancumprod, sort/argsort (gathered on one chare)
- cnp.linalg: cholesky, qr, solve, inv, det, slogdet, eigh, eigvalsh, svd,
  svdvals, lstsq, solve_triangular, tssvd (Eigen on CPU, cuSOLVER/cuBLAS on
  NVIDIA; factors on one chare)

Backend cleanup: size-1 operands only computed on owning chares, per-operand
broadcast bookkeeping, uninitialized fields, leaks, kernel cache keyed by
MLIR text with scalars as runtime arguments, one MLIRContext per process;
removed the old string-PTX path.

Ports: cholesky/qr/solve/svd/tssvd/scan/sort/quantiles now mirror the
cuPyNumeric originals; all 25 ports and 15 originals pass on GPU (+p1/+p2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g fixes

- Indexing: native gather/scatter/expand, boolean masks (with slices and
  newaxis), take, fancy setitem, >3-D fancy indexing
- Dtypes: bool, int8-64, uint8-64, float16/32/64, complex64/128 with NumPy
  (NEP 50) promotion; integer true division; shifts
- Reshape/broadcast executors, general broadcasting, in-place operators,
  >3-D arrays (FoldedArray)
- einsum, tensordot, inner, outer, batched matmul; signal ops (flip, convolve)
- 0-d arrays, integer indices drop dims, linked transposes/views, NumPy
  interop (__array__, __array_ufunc__), quantile/percentile/allclose
- All 25 cuPyNumeric examples are the upstream files with only the import
  changed, checked against NumPy through a _benchmark shim
- GPU: scalar code sunk into kernels (fixes MOD/half crashes), loop fusion,
  device temporaries, stream handoff, lazy host mirrors, device-side reshape
  packing, O0 host codegen (add 1e8: 88 -> 2.2 ms on A100)
- Fixes: reduction numbering after partition growth (server crash),
  decomposition offsets, size-1 rank mismatch hangs, ownership-based
  broadcast routing; dead code removed, comments trimmed

Validated: CPU +p1/+p2 and A40 GPU +p1/+p2: 15/15 originals, 25/25 ports,
17/17 op suites, 21 differential-test repros, 400 fuzz seeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he, multi-node fixes

- Boolean mask indexing via new COMPRESS/MASK_SCATTER opcodes (x[x>.5] at 1e7: 550 ms -> 5 ms on A100)
- Persistent on-disk JIT kernel/cubin cache (CNUM_JIT_CACHE_DIR, CNUM_JIT_CACHE=0 to disable)
- MLIR CUDA wrappers use the calling thread's context (fixes multi-node GPU INVALID_HANDLE)
- Async auto-flush every 256 ops, lazy integer-index copies, DAG-scoped decomposition solver
- Pooled device comm buffers, single-chare SUMMA fast path, matvec/reduction rewrites
- Stream-ordering fixes for KokkosBlas gemm on non-cuBLAS dtypes
- cuPyNumeric reference mode for the ports (CNUM_REFERENCE=cupynumeric), comparison scripts
- Multi-node notes (+eth, one GPU per process); test_mask, test_lazy_index, test_multinode_traffic

Code-review fixes:
- JIT: target the device's real sm/PTX, flatten launch grids (no y/z limits), skip empty launches,
  NumPy signed-zero results for sign/maximum/minimum/fmax/fmin/clip
- Reductions: compact partials per kind, sparse/ranged bincount partials, bincount with
  differently laid-out weights, int64 indices, overflow-safe 2-norm
- Uploads: each PE receives only its tiles, chares fetch tiles they lack, payload bounds checks,
  uploads split under the 512 MiB CCS request limit
- Matmul dropped-dim detection, 3-D dim-dropped matvec on GPU, unique in-flight send ids,
  size-1 broadcast set_region across ranks, exact big-int scatter, stale arange/random recipes
- Tests: test_frontend_fixes, test_jit_edges, test_exec_robust, test_upload_split

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Process DAGs and get requests in client epoch order (a get for epoch N+1
  could overtake the DAG of epoch N and hang)
- set_region from a size-1 broadcast source created by a not-yet-executed DAG:
  fall back to array_meta when activating partitions (deadlock)
- SMP: serialize the shared MLIR context and kernel cache, initialize Kokkos
  once per process, lock the device buffer pool, per-PE cuBLAS/cuSOLVER handles

Tested on Charm++ over Reconverse (recharm) and classic netlrts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@adityapb adityapb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes.

I reviewed the frontend (charmnumeric/*.py), the backend core (JIT, executor core, DAG/partition code), all the executors, and the cuPyNumeric ports. This is a static review: I traced the code paths and checked the expected NumPy behaviour with NumPy 1.26. Nothing was run on a GPU. Line numbers are at this PR's head, and "still at #44" means the bug is still there at the top of the stack.

Much of this held up well: the wire encoding against the backend parsers, the JIT promotion table (all 196 pairs), the message protocols (transpose, reshape/broadcast, gather/scatter, reductions, linalg, convolve), and the reduction semantics (NaN, first-occurrence argmax across tiles, Welford var). The problems are mostly silent wrong results in the frontend, plus a few user inputs that kill the server.

Silent wrong results (all still at #44)

  1. Assignment broadcasts against the storage shape, not NumPy's view shape.
    • Where: charmnumeric.py:2427-2428 (_setitem → _fit_assignment at :2343).
    • Cause: shape = region.shape() keeps a size-1 dim for every integer in the key. vshape is computed but only used for host values.
    • Example: x = zeros((4,4,3)); x[:, :, 0] = cnp.array(col) with col.shape == (4,1) writes col[b] instead of col[a], i.e. transposed.
    • Also: x[..., 0] = cnp.arange(3.) on a (2,3,5) array raises, though NumPy accepts it.
  2. Stale caches on linked copies.
    • Where: _cached_transpose (charmnumeric.py:1713-1726) and _size1_to_rank (:588-600).
    • Cause: neither calls _fresh(x) on a cache hit, and writing the parent doesn't clear them.
    • Examples:
      • M = X[0]; cnp.sum(M); y1 = w @ M; X[0] *= 2; y2 = w @ M gives y2 == y1.
      • B = A.T; v @ B; A[0,1] = 100; v @ B is also unchanged.
    • Realistic trigger: Wt = W.T taken once and used in a loop while W is updated in place.
    • Fix: call _fresh(x) before the lookup, or key the cache on the parent's version.
  3. Negative-step writes with None in the key flip the wrong axis.
    • Where: charmnumeric.py:2375-2378.
    • Cause: tdims drops integers but not newaxis entries.
    • Examples:
      • x = zeros(4); x[::-1, None] = v writes v unreversed.
      • Worse: with x = arange(4), r = x[::-1, None]; r[0, 0] = 100 leaves x == [100, 2, 1, 0]. NumPy gives [0, 1, 2, 100].
  4. 3-D matmul squeezes an operand that has exactly one size-1 dim.
    • Where: _needs_batched (charmnumeric.py:1452-1465), which is shared by __matmul__ and cnp.matmul.
    • Examples:
      • (5,1,4) @ (4,6) returns shape (5,6).
      • (3,3,1) @ (3,1,4) returns (3,4) with wrong values.
      • (5,4,1) @ (1,6) raises.
  5. int_array ** 2.0 stays integer and wraps.
    • Where: _power (charmnumeric.py:1172).
    • Cause: exponent == 2 is also true for 2.0, so the fast path keeps the integer dtype.
    • Example: array([200], uint8) ** 2.0 gives 64; NumPy gives 40000.0.
  6. Random arrays with more than 3 dims repeat values.
    • Where: random.py:92 sends the N-D strides, _creation_op (charmnumeric.py:2020-2024) folds only the shape, and executor_creation.cpp:229-230 reads 3 strides.
    • Example: rand(2,3,4,5) has only 77 distinct values out of 120.
    • _rng_reference.py:268 makes the same mistake, so neither CNUM_SIM nor the tests can catch it.
  7. sum/prod ignore dtype= and out= when choosing the accumulator.
    • Where: reductions.py:153-166 always sends a.dtype, and red_acc_kind (executor_reduction.cpp:87-94) picks the accumulator from it.
    • Examples:
      • prod(arange(1,26), dtype=float64) gives 7.03e18; NumPy gives 1.55e25.
      • sum([0.5, 0.5], dtype=int64) gives 1; NumPy gives 0.
    • Fix: cast the input first, as _scan already does (but not for argmax/argmin).

User input that kills the server

  1. clip on complex arrays.
    • Where: CLIP has no complex body (jit.cpp:1875-1887), and a missing body calls llvm::report_fatal_error (jit.cpp:592-594).
    • Effect: cnp.clip(complex_arr, 0, 2) aborts every PE.
    • Fix: add the body. A registry gap should also fail the compile and report back to the client instead of aborting.
    • Still at #44.
  2. A 0-d device index array, e.g. x[cnp.argmax(y)].
    • Cause: indexing.py emits ndE = 0, index_result_meta and dag_group_receive.cpp:327-329 never record the result's metadata, and layout_of aborts (executor_indexing.cpp:566-567).
    • On a 1-D x, the server aborts. Still at #44.
    • On a 2-D x, x[i] and x[i] = v touch only x[i, 0] and leave the rest uninitialised. #39 looks like it fixes this case.
    • Suggestion: reshape 0-d index arrays to (1,) in the frontend and add a test.
  3. indices(dims, sparse=True) with 4 or more dims.
    • Cause: the sparse branch (charmnumeric.py:2908-2918) folds the shape but not the IOTA axis.
    • Effect: k=1 and k=2 come out all zeros, and k=3 hits CkAbort("IOTA: axis 3 out of range").
    • Still at #44.

Backend

  1. 32-bit offsets overflow on tiles of 2 GiB or more.
    • Where:
      • scalar-fill SET_REGION: executor_ast_visitor.cpp:140-153,165 (int flat, flat * esz);
      • halo pack: executor_core.cpp:63-81;
      • dot: int local_size in executor_reduce.cpp:53.
    • Example: x = zeros((20000,20000)); x.fill(1.0) on one GPU writes out of bounds.
    • The general SET_REGION copy became 64-bit later in the stack, but these three are still 32-bit at #44.
  2. SMP device selection.
    • Only the thread that ran Kokkos::initialize selects the device. Other PE threads create their streams on device 0 (partition_lifecycle.cpp:19-23).
    • cnEnsureContext explicitly falls back to device 0 (jit.cpp:1092-1108), although its comment says it uses the chare's device.
    • With +ppn > 1 and a Kokkos device other than 0, kernels launch on the wrong device. This depends on the launch config. Still at #44.
  3. Nits:
    • epoch_arrived (dag_group_receive.cpp:193-207) claims client-reconnect support that isn't implemented.
    • The JIT disk-cache key hashes the host CPU name but not getHostCPUFeatures(). Nodes sharing $HOME where one has AVX-512 masked could load an incompatible object and hit SIGILL.

Fixed later in the stack (for the record)

  • Halo packs on the comm stream race in-place writes on the compute stream, so a[1:] = a[:-1]*2 can send already-updated halos. Fixed in #15.
  • REDUCE (dot) multiplies raw local tiles when its operands are laid out differently. Fixed in #24.
  • Deletions are dropped for ranks a DAG doesn't touch, which leaks device memory. Fixed in #17.
  • Long fused chains exceed the CUDA kernel-parameter limit and abort. Fixed in #38.
  • Device index arithmetic runs in the index array's own dtype: arange(300.)[dev_int8([-1, 5])] gives [43, 5]. Fixed by _index_int64.
  • An out-of-range device index on an inner axis reads the neighbouring row, and negative device repeat counts abort the server. Both fixed later.

Smaller NumPy divergences (still at #44)

  • Zero-size reductions raise (reductions.py:137). NumPy returns 0/1/False/True for sum/prod/any/all, so x[x > t].sum() with no matches raises here.
  • NumPy scalars treated as weak: comparisons and power treat them as weak (_coerce_operand at :1079 and :1167), unlike +.
  • Indexing edge cases:
    • An out-of-range int next to a negative-step slice wraps instead of raising (:2224, int(k) % n).
    • A Python bool index is treated as 0 or 1 (:484).
  • Dtype and shape edge cases: choose ignores scalar choices when picking the result dtype, and atleast_*d and tile mishandle 0-d inputs.
  • Linalg failure reporting: when GPU SVD fails, only S is NaN-filled, so U and Vh come back as finite garbage. CPU BDCSVD info() and the GPU tssvd status are never checked.
  • Loud failures where NumPy succeeds:
    • Empty negative-step slices raise.
    • take, take_along_axis and flip fail on >3-D arrays.
    • quantile imports a NumPy-2-only module, but numpy isn't pinned.
  • Nits:
    • A fractional ddof is truncated to an int.
    • flip's docstring promises a copy, but it returns the input when nothing flips.
    • lstsq returns rank as a float 0-d array.

Tests

  • Most op checks can't see dtype bugs. _ops_common.Checker.check with dtype=None takes the expected dtype from charmnumeric's own result (examples/ops/_ops_common.py:121-135). About 500 of the ~570 checks therefore can't catch dtype errors; item 5, the comparison issue and choose would all pass.
  • The simulator hides stale data. _cnum_sim.py stores results as NumPy views of their inputs, so stale-data bugs like item 2 fix themselves under CNUM_SIM=1.
  • A wrong-result bug is worked around, not tested. examples/ops/test_reductions.py:27-29 says "a fused temporary read twice currently comes back as zeros" and works around it. If that's still true, it's a wrong-result bug that deserves its own test.
  • _benchmark.py checks are loose in places:
    • _comparable skips Python float and complex locals.
    • The tolerance is scaled by the largest magnitude among all locals of a function, so small locals next to large ones get a loose check.
    • It needs Python ≥ 3.12 (sys.monitoring), but nothing declares requires-python.

Examples

  • The ports really are upstream plus a 2-line comment, which is nice.
  • examples/cupynumeric/upstream/ vendors Apache-2.0 code. Please add the LICENSE file next to it, and NOTICE if there is one.
  • examples/cupynumeric/README.md:16,40 points at private paths (/u/atahilyani/cnum-gpu/wt/ports/env_ports.sh, wt/ports/run_suite.sh), and CODEMAP's header still says example/charmnumeric/.

AdvaitTahilyani and others added 4 commits November 9, 2026 22:32
- CLIP gets a complex body: lexicographic order with the clipped value's NaN
  first, as NumPy's complex loop (cnp.clip on complex arrays aborted every PE).
- A registry gap no longer calls report_fatal_error: the kernel writes zeros
  for that op, is kept out of the disk cache, and PE 0 answers the next GET
  with the error, which CharmNumericInterface raises as RuntimeError.
- test_jit_edges: complex clip against NumPy, and a float64 bitwise_and sent
  past the frontend reports the error and leaves the server usable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scalar fill (int flat * esz), the device halo pack and the dot's
local_size used 32-bit offsets, so tiles of 2 GiB or more were written out of
bounds. These kernels now index in 64 bits (CT_*_POLICY64 on the device).
test_jit_edges: CNUM_BIG_SHAPES=1 adds a fill of a 3.2 GB array.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the thread that ran Kokkos::initialize selected the device, so other PE
threads created their streams on device 0. Each DAG group branch now calls
cudaSetDevice(Kokkos::device_id()), and cnEnsureContext falls back to the
thread's runtime device instead of device 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nodes sharing a cache dir with the same CPU name but different features
(e.g. AVX-512 masked) no longer load each other's objects. The epoch_arrived
comment no longer claims client-reconnect support.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AdvaitTahilyani and others added 3 commits November 9, 2026 22:45
- Assignment broadcasts against x[key]'s NumPy shape, not the storage shape.
- _cached_transpose and _size1_to_rank refresh linked copies first, so a
  write to the parent clears their caches.
- Negative-step writes count newaxis entries; linked copies write back
  through the key with its newaxis entries.
- matmul treats every 3-D operand as a batch (N-D @ 2-D runs as one 2-D
  matmul); einsum keeps the internal squeezed-matrix path.
- int ** 2.0 takes the x*x fast path only if NumPy keeps the dtype; NumPy
  scalars stay strong in power and comparisons.
- >3-D random draws fold their index strides like the storage.
- sum/prod accumulate in dtype= (or a wider out=) dtype.
- 0-d device index arrays gather one element (ndE >= 1); sparse indices
  with >3 dims fold the IOTA axis.
- Zero-size reductions return NumPy's identity (or NaN for mean/var/std),
  fractional ddof, choose dtype with scalar choices, atleast_*d/tile of
  0-d, bool and out-of-range indices next to negative steps, empty
  negative-step slices, take/take_along_axis/flip on >3-D, flip always
  copies, lstsq rank is int32.
- SVD, lstsq and tssvd report failures as all-NaN factors (CPU info(),
  GPU devInfo for U/Vh and tssvd).
- examples/ops/test_numpy_semantics.py covers all of the above;
  test_exec_robust checks NumPy's batched matmul.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…enchmark

- Checker.check compares the result dtype with NumPy's (expected's) unless
  dtype= is given; richardson-lucy x2 passes its float32 dtype explicitly.
- _cnum_sim stores copies, so stale-data bugs are not hidden by aliasing.
- test_reductions no longer execute()s fused temporaries before reuse; the
  zeros bug it worked around no longer reproduces.
- _benchmark compares Python float/complex locals and skips only timer
  values (and arithmetic on them).
- setup.py declares python_requires >= 3.12 (sys.monitoring) and
  numpy >= 2.0 (NEP 50 typing, quantile's numpy.lib module).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvaitTahilyani

Copy link
Copy Markdown
Author

Frontend, tests and examples items. Backend items 8, 11, 12 and 13 are covered in a separate note.

Silent wrong results

  1. Assignment broadcast against the storage shape: fixed in 269818c.
  2. Stale _cached_transpose / _size1_to_rank caches: fixed in 269818c. Both now call _fresh(x) first, and a refresh clears the caches.
  3. Negative step with None: fixed in 269818c. Linked copies also write back through the key with its newaxis entries.
  4. 3-D matmul squeeze: fixed in 269818c. Every 3-D operand is now a batch, N-D @ 2-D runs as one 2-D matmul, and mismatched core dims raise. einsum keeps the internal squeezed-matrix path (_matmul_2d).
  5. int ** 2.0: fixed in 269818c.
  6. 3-D random: fixed in 269818c. The strides are folded like the storage, so the simulator reference matches too.

  7. sum/prod dtype= / out=: fixed in 269818c.

Inputs that kill the server
9. 0-d device index: fixed in 269818c. It is gathered as shape (1,) in the frontend. Get and set on 1-D and 2-D arrays are tested.
10. indices(sparse=True) with more than 3 dims: fixed in 269818c.

Smaller NumPy divergences

  • Zero-size reductions: fixed in 269818c. sum/prod/any/all/count_nonzero return the identity, mean/var/std return NaN with a RuntimeWarning, and max/min/arg* raise ValueError.
  • NumPy scalars treated as weak in comparisons and power: fixed in 269818c.
  • Out-of-range int next to a negative-step slice: fixed in 269818c.
  • Python bool index: fixed in 269818c. It now raises NotImplementedError, as indexing.py already did, instead of indexing with 0/1.
  • choose dtype with scalar choices: fixed in 269818c.
  • atleast_*d / tile on 0-d inputs: fixed in 269818c.
  • SVD failure reporting: fixed in 269818c. CPU BDCSVD / SelfAdjointEigenSolver info() and the GPU gesvd and tssvd devInfo are checked, and on failure U, S and Vh are all NaN (also lstsq x and tssvd).
  • Empty negative-step slices: fixed in 269818c.
  • take, take_along_axis and flip on >3-D arrays: fixed in 269818c.
  • quantile's NumPy-2-only import: fixed in 0260f2f. numpy>=2.0 is pinned, which the NEP 50 typing needs anyway.
  • Fractional ddof: fixed in 269818c.
  • flip docstring vs return: fixed in 269818c. It now returns a copy when nothing flips.
  • lstsq rank: fixed in 269818c. It is now a 0-d int32 array, matching NumPy's int32.

Tests

  • Checker ignoring dtypes: fixed in 0260f2f. The expected dtype is now NumPy's. Only two new failures came up: a float64 reference in test_signal's richardson-lucy, which was a test artifact, and test_exec_robust's matmul cases, which encoded the old squeeze (updated in 269818c).
  • _cnum_sim views: fixed in 0260f2f.
  • "Fused temporary read twice": not changed as a bug, because it no longer reproduces at this head (+p1/+p2, every shape and dtype test_reductions uses). The workaround is removed in 0260f2f, so test_reductions now reads fused temporaries twice, and test_numpy_semantics checks it directly.
  • _comparable skipping Python float/complex: fixed in 0260f2f. Only timer values, and arithmetic on them, are skipped now.
  • Record-wide tolerance scale: not changed. Per-local scaling fails cg (r, p, alpha, beta after convergence) and logreg (gradient at the optimum) on rounding noise. Those locals carry the error of the larger values they are computed from.
  • requires-python: fixed in 0260f2f (python_requires='>=3.12').

Examples

  • Apache LICENSE / NOTICE for upstream/: fixed in 4752593.
  • Private paths in README: fixed in 4752593.
  • CODEMAP header: fixed in 4752593.

Validated: CPU build, plus a GPU compile of executor_linalg.cpp. Full CPU regress (orig, ports, ops) passes at +p1 and +p2. The new examples/ops/test_numpy_semantics.py (111 checks) covers every fixed item: it fails on 16b1916 and passes now.

CNUM_BIG_SHAPES fetches arrays over the 512 MB fetch chunk, which this head
does not serve yet; the fill check fetches only a sum and one row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvaitTahilyani

Copy link
Copy Markdown
Author

Backend items 8, 11, 12 and 13 are on branch prfix/be10, pending a merge into gpu-backend:

  • 8 (complex clip aborts every PE): fixed in 166020c. CLIP has a complex body (NumPy's lexicographic order, the clipped value's NaN first). A missing registry body no longer calls report_fatal_error: that op writes zeros, its kernel stays out of the disk cache, and PE 0 answers the next GET with the error, which CharmNumericInterface raises as RuntimeError. Tests in test_jit_edges.py.
  • 11 (32-bit offsets): fixed in 3db1df2 (scalar-fill SET_REGION, halo pack and dot local_size, with 64-bit device range policies for those kernels). 1-D shapes are still int, so the dot's local_size could not actually overflow; it is widened anyway.
  • 12 (SMP device selection): fixed in bf5edab. Every PE thread calls cudaSetDevice(Kokkos::device_id()), and cnEnsureContext falls back to the thread's runtime device, not device 0.
  • 13, epoch_arrived comment: fixed in 14a160c.
  • 13, getHostCPUFeatures() in the JIT disk-cache key: fixed in 14a160c (this also covers Perf phase 1: pre-warm, fewer blocking round trips, bounded JIT cache #14's nit 1).
    Validated: nc build, full CPU regress at +p1 and +p2 (orig 15/15, ports 25/25, ops 24/24 each); test_jit_edges with a 3.2 GB one-tile fill (CNUM_BIG_FILL=1 CNUM_ODF=1); the GPU and Reconverse CUDA builds compile. The GPU run (A100: the new tests, plus 1 process x 2 PE threads with KOKKOS_DEVICE_ID=1 for item 12) is still queued.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvaitTahilyani

Copy link
Copy Markdown
Author

Both fix sets (frontend and backend) are merged into this branch at 3b3b3fd. The merged head builds, and CPU regress passes at +p1 and +p2: orig 15/15, ports 25/25, ops 25/25. A GPU check of the device-only fixes (64-bit fill/halo/dot offsets, SMP device selection, complex clip on GPU) is queued; I'll post the result here.

@AdvaitTahilyani

Copy link
Copy Markdown
Author

GPU check (1 A100, netlrts-cuda): test_jit_edges passes at +p1, including the 3.2 GB fill (CNUM_BIG_FILL=1, item 11), and at +p2 (64 checks). The SMP device-selection check on Reconverse CUDA (1 process × 2 PE threads, KOKKOS_DEVICE_ID=1) was inconclusive: both the old and new builds showed activity on both GPUs, so this test can't tell them apart. Item 12 is therefore validated only by code inspection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvaitTahilyani

Copy link
Copy Markdown
Author

Additional fix (33238cc): bincount aborted the server (std::vector ... max_size()) when an output chare lay entirely above a block's bin range. Example: cnp.bincount(cnp.array([1,2,3,2,9]), minlength=3_000_000). Those chares now get an empty piece. Added a minlength=3_000_000 case to test_reductions, which passes at +p2. The fix is merged up through #44 and #42. #41 rewrites this path (MSG_BIN_EMPTY) and keeps its own version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants