Repository navigation
GPU backend, NumPy-compatible ops, and cuPyNumeric example ports - #10
AdvaitTahilyani wants to merge 15 commits into
Conversation
GPU (NVIDIA) execution: - JIT: lower fused linalg kernels through MLIR's GPU-to-NVVM pipeline into a host launcher (PTX, MLIR CUDA runtime); temporaries in managed memory. - Fix the nvcc build (jit_api.hpp split, nested device lambdas, Eigen on device pointers, server_obj defines, charmc link of Kokkos/CUDA libs). - Device messaging: drop invalid .ci post entries, carry send_id in the callback refnum, look up posted receive buffers (Charm++ passes null data). - Skip the cache_dag plan signature in receive_dag. Backend/frontend fixes: rank and broadcast flags of shared leaves, size-1 view broadcasting, solver exponential walk, temps still held by user code, stale in-kernel reads, per-array SUMMA tiles, start epochs of new partitions, cross-dim set_region offsets, gather offsets, retire ordering, fused-kernel size cap, kernel-cache collision on operand aliasing. New ops: elementwise math/comparisons/where/clip (opcodes 20-49), bulk upload, counter-based RNG and iota creation ops (50-52), full and axis reductions and bincount (60-61). Examples: ports of all 25 cuPyNumeric examples (examples/cupynumeric) and op test suites (examples/ops). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…leanup New ops: - transpose/.T/swapaxes (2-D/3-D), vecmat and transposed-operand matmul - concatenate/hstack/vstack/stack, take (host indices), out= on ufuncs and matmul/dot, native eye, newaxis, int32 array-name limit - astype, NumPy dtype promotion (operators, elementwise ops, assignment), cumsum/cumprod/nancumsum/nancumprod, sort/argsort (gathered on one chare) - cnp.linalg: cholesky, qr, solve, inv, det, slogdet, eigh, eigvalsh, svd, svdvals, lstsq, solve_triangular, tssvd (Eigen on CPU, cuSOLVER/cuBLAS on NVIDIA; factors on one chare) Backend cleanup: size-1 operands only computed on owning chares, per-operand broadcast bookkeeping, uninitialized fields, leaks, kernel cache keyed by MLIR text with scalars as runtime arguments, one MLIRContext per process; removed the old string-PTX path. Ports: cholesky/qr/solve/svd/tssvd/scan/sort/quantiles now mirror the cuPyNumeric originals; all 25 ports and 15 originals pass on GPU (+p1/+p2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g fixes - Indexing: native gather/scatter/expand, boolean masks (with slices and newaxis), take, fancy setitem, >3-D fancy indexing - Dtypes: bool, int8-64, uint8-64, float16/32/64, complex64/128 with NumPy (NEP 50) promotion; integer true division; shifts - Reshape/broadcast executors, general broadcasting, in-place operators, >3-D arrays (FoldedArray) - einsum, tensordot, inner, outer, batched matmul; signal ops (flip, convolve) - 0-d arrays, integer indices drop dims, linked transposes/views, NumPy interop (__array__, __array_ufunc__), quantile/percentile/allclose - All 25 cuPyNumeric examples are the upstream files with only the import changed, checked against NumPy through a _benchmark shim - GPU: scalar code sunk into kernels (fixes MOD/half crashes), loop fusion, device temporaries, stream handoff, lazy host mirrors, device-side reshape packing, O0 host codegen (add 1e8: 88 -> 2.2 ms on A100) - Fixes: reduction numbering after partition growth (server crash), decomposition offsets, size-1 rank mismatch hangs, ownership-based broadcast routing; dead code removed, comments trimmed Validated: CPU +p1/+p2 and A40 GPU +p1/+p2: 15/15 originals, 25/25 ports, 17/17 op suites, 21 differential-test repros, 400 fuzz seeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he, multi-node fixes - Boolean mask indexing via new COMPRESS/MASK_SCATTER opcodes (x[x>.5] at 1e7: 550 ms -> 5 ms on A100) - Persistent on-disk JIT kernel/cubin cache (CNUM_JIT_CACHE_DIR, CNUM_JIT_CACHE=0 to disable) - MLIR CUDA wrappers use the calling thread's context (fixes multi-node GPU INVALID_HANDLE) - Async auto-flush every 256 ops, lazy integer-index copies, DAG-scoped decomposition solver - Pooled device comm buffers, single-chare SUMMA fast path, matvec/reduction rewrites - Stream-ordering fixes for KokkosBlas gemm on non-cuBLAS dtypes - cuPyNumeric reference mode for the ports (CNUM_REFERENCE=cupynumeric), comparison scripts - Multi-node notes (+eth, one GPU per process); test_mask, test_lazy_index, test_multinode_traffic Code-review fixes: - JIT: target the device's real sm/PTX, flatten launch grids (no y/z limits), skip empty launches, NumPy signed-zero results for sign/maximum/minimum/fmax/fmin/clip - Reductions: compact partials per kind, sparse/ranged bincount partials, bincount with differently laid-out weights, int64 indices, overflow-safe 2-norm - Uploads: each PE receives only its tiles, chares fetch tiles they lack, payload bounds checks, uploads split under the 512 MiB CCS request limit - Matmul dropped-dim detection, 3-D dim-dropped matvec on GPU, unique in-flight send ids, size-1 broadcast set_region across ranks, exact big-int scatter, stale arange/random recipes - Tests: test_frontend_fixes, test_jit_edges, test_exec_robust, test_upload_split Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Process DAGs and get requests in client epoch order (a get for epoch N+1 could overtake the DAG of epoch N and hang) - set_region from a size-1 broadcast source created by a not-yet-executed DAG: fall back to array_meta when activating partitions (deadlock) - SMP: serialize the shared MLIR context and kernel cache, initialize Kokkos once per process, lock the device buffer pool, per-PE cuBLAS/cuSOLVER handles Tested on Charm++ over Reconverse (recharm) and classic netlrts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adityapb
left a comment
There was a problem hiding this comment.
Verdict: request changes.
I reviewed the frontend (charmnumeric/*.py), the backend core (JIT, executor core, DAG/partition code), all the executors, and the cuPyNumeric ports. This is a static review: I traced the code paths and checked the expected NumPy behaviour with NumPy 1.26. Nothing was run on a GPU. Line numbers are at this PR's head, and "still at #44" means the bug is still there at the top of the stack.
Much of this held up well: the wire encoding against the backend parsers, the JIT promotion table (all 196 pairs), the message protocols (transpose, reshape/broadcast, gather/scatter, reductions, linalg, convolve), and the reduction semantics (NaN, first-occurrence argmax across tiles, Welford var). The problems are mostly silent wrong results in the frontend, plus a few user inputs that kill the server.
Silent wrong results (all still at #44)
- Assignment broadcasts against the storage shape, not NumPy's view shape.
- Where:
charmnumeric.py:2427-2428(_setitem→_fit_assignmentat:2343). - Cause:
shape = region.shape()keeps a size-1 dim for every integer in the key.vshapeis computed but only used for host values. - Example:
x = zeros((4,4,3)); x[:, :, 0] = cnp.array(col)withcol.shape == (4,1)writescol[b]instead ofcol[a], i.e. transposed. - Also:
x[..., 0] = cnp.arange(3.)on a(2,3,5)array raises, though NumPy accepts it.
- Where:
- Stale caches on linked copies.
- Where:
_cached_transpose(charmnumeric.py:1713-1726) and_size1_to_rank(:588-600). - Cause: neither calls
_fresh(x)on a cache hit, and writing the parent doesn't clear them. - Examples:
M = X[0]; cnp.sum(M); y1 = w @ M; X[0] *= 2; y2 = w @ Mgivesy2 == y1.B = A.T; v @ B; A[0,1] = 100; v @ Bis also unchanged.
- Realistic trigger:
Wt = W.Ttaken once and used in a loop whileWis updated in place. - Fix: call
_fresh(x)before the lookup, or key the cache on the parent's version.
- Where:
- Negative-step writes with
Nonein the key flip the wrong axis.- Where:
charmnumeric.py:2375-2378. - Cause:
tdimsdrops integers but not newaxis entries. - Examples:
x = zeros(4); x[::-1, None] = vwritesvunreversed.- Worse: with
x = arange(4),r = x[::-1, None]; r[0, 0] = 100leavesx == [100, 2, 1, 0]. NumPy gives[0, 1, 2, 100].
- Where:
- 3-D matmul squeezes an operand that has exactly one size-1 dim.
- Where:
_needs_batched(charmnumeric.py:1452-1465), which is shared by__matmul__andcnp.matmul. - Examples:
(5,1,4) @ (4,6)returns shape(5,6).(3,3,1) @ (3,1,4)returns(3,4)with wrong values.(5,4,1) @ (1,6)raises.
- Where:
int_array ** 2.0stays integer and wraps.- Where:
_power(charmnumeric.py:1172). - Cause:
exponent == 2is also true for2.0, so the fast path keeps the integer dtype. - Example:
array([200], uint8) ** 2.0gives64; NumPy gives40000.0.
- Where:
- Random arrays with more than 3 dims repeat values.
- Where:
random.py:92sends the N-D strides,_creation_op(charmnumeric.py:2020-2024) folds only the shape, andexecutor_creation.cpp:229-230reads 3 strides. - Example:
rand(2,3,4,5)has only 77 distinct values out of 120. _rng_reference.py:268makes the same mistake, so neitherCNUM_SIMnor the tests can catch it.
- Where:
sum/prodignoredtype=andout=when choosing the accumulator.- Where:
reductions.py:153-166always sendsa.dtype, andred_acc_kind(executor_reduction.cpp:87-94) picks the accumulator from it. - Examples:
prod(arange(1,26), dtype=float64)gives 7.03e18; NumPy gives 1.55e25.sum([0.5, 0.5], dtype=int64)gives 1; NumPy gives 0.
- Fix: cast the input first, as
_scanalready does (but not for argmax/argmin).
- Where:
User input that kills the server
clipon complex arrays.- Where: CLIP has no complex body (
jit.cpp:1875-1887), and a missing body callsllvm::report_fatal_error(jit.cpp:592-594). - Effect:
cnp.clip(complex_arr, 0, 2)aborts every PE. - Fix: add the body. A registry gap should also fail the compile and report back to the client instead of aborting.
- Still at #44.
- Where: CLIP has no complex body (
- A 0-d device index array, e.g.
x[cnp.argmax(y)].- Cause:
indexing.pyemitsndE = 0,index_result_metaanddag_group_receive.cpp:327-329never record the result's metadata, andlayout_ofaborts (executor_indexing.cpp:566-567). - On a 1-D
x, the server aborts. Still at #44. - On a 2-D
x,x[i]andx[i] = vtouch onlyx[i, 0]and leave the rest uninitialised. #39 looks like it fixes this case. - Suggestion: reshape 0-d index arrays to
(1,)in the frontend and add a test.
- Cause:
indices(dims, sparse=True)with 4 or more dims.- Cause: the sparse branch (
charmnumeric.py:2908-2918) folds the shape but not the IOTA axis. - Effect: k=1 and k=2 come out all zeros, and k=3 hits
CkAbort("IOTA: axis 3 out of range"). - Still at #44.
- Cause: the sparse branch (
Backend
- 32-bit offsets overflow on tiles of 2 GiB or more.
- Where:
- scalar-fill SET_REGION:
executor_ast_visitor.cpp:140-153,165(int flat,flat * esz); - halo pack:
executor_core.cpp:63-81; - dot:
int local_sizeinexecutor_reduce.cpp:53.
- scalar-fill SET_REGION:
- Example:
x = zeros((20000,20000)); x.fill(1.0)on one GPU writes out of bounds. - The general SET_REGION copy became 64-bit later in the stack, but these three are still 32-bit at #44.
- Where:
- SMP device selection.
- Only the thread that ran
Kokkos::initializeselects the device. Other PE threads create their streams on device 0 (partition_lifecycle.cpp:19-23). cnEnsureContextexplicitly falls back to device 0 (jit.cpp:1092-1108), although its comment says it uses the chare's device.- With
+ppn> 1 and a Kokkos device other than 0, kernels launch on the wrong device. This depends on the launch config. Still at #44.
- Only the thread that ran
- Nits:
epoch_arrived(dag_group_receive.cpp:193-207) claims client-reconnect support that isn't implemented.- The JIT disk-cache key hashes the host CPU name but not
getHostCPUFeatures(). Nodes sharing$HOMEwhere one has AVX-512 masked could load an incompatible object and hit SIGILL.
Fixed later in the stack (for the record)
- Halo packs on the comm stream race in-place writes on the compute stream, so
a[1:] = a[:-1]*2can send already-updated halos. Fixed in #15. - REDUCE (dot) multiplies raw local tiles when its operands are laid out differently. Fixed in #24.
- Deletions are dropped for ranks a DAG doesn't touch, which leaks device memory. Fixed in #17.
- Long fused chains exceed the CUDA kernel-parameter limit and abort. Fixed in #38.
- Device index arithmetic runs in the index array's own dtype:
arange(300.)[dev_int8([-1, 5])]gives[43, 5]. Fixed by_index_int64. - An out-of-range device index on an inner axis reads the neighbouring row, and negative device
repeatcounts abort the server. Both fixed later.
Smaller NumPy divergences (still at #44)
- Zero-size reductions raise (
reductions.py:137). NumPy returns 0/1/False/True for sum/prod/any/all, sox[x > t].sum()with no matches raises here. - NumPy scalars treated as weak: comparisons and
powertreat them as weak (_coerce_operandat:1079and:1167), unlike+. - Indexing edge cases:
- An out-of-range int next to a negative-step slice wraps instead of raising (
:2224,int(k) % n). - A Python bool index is treated as 0 or 1 (
:484).
- An out-of-range int next to a negative-step slice wraps instead of raising (
- Dtype and shape edge cases:
chooseignores scalar choices when picking the result dtype, andatleast_*dandtilemishandle 0-d inputs. - Linalg failure reporting: when GPU SVD fails, only S is NaN-filled, so U and Vh come back as finite garbage. CPU BDCSVD
info()and the GPU tssvd status are never checked. - Loud failures where NumPy succeeds:
- Empty negative-step slices raise.
take,take_along_axisandflipfail on >3-D arrays.quantileimports a NumPy-2-only module, but numpy isn't pinned.
- Nits:
- A fractional
ddofis truncated to an int. flip's docstring promises a copy, but it returns the input when nothing flips.lstsqreturnsrankas a float 0-d array.
- A fractional
Tests
- Most op checks can't see dtype bugs.
_ops_common.Checker.checkwithdtype=Nonetakes the expected dtype from charmnumeric's own result (examples/ops/_ops_common.py:121-135). About 500 of the ~570 checks therefore can't catch dtype errors; item 5, the comparison issue andchoosewould all pass. - The simulator hides stale data.
_cnum_sim.pystores results as NumPy views of their inputs, so stale-data bugs like item 2 fix themselves underCNUM_SIM=1. - A wrong-result bug is worked around, not tested.
examples/ops/test_reductions.py:27-29says "a fused temporary read twice currently comes back as zeros" and works around it. If that's still true, it's a wrong-result bug that deserves its own test. _benchmark.pychecks are loose in places:_comparableskips Pythonfloatandcomplexlocals.- The tolerance is scaled by the largest magnitude among all locals of a function, so small locals next to large ones get a loose check.
- It needs Python ≥ 3.12 (
sys.monitoring), but nothing declaresrequires-python.
Examples
- The ports really are upstream plus a 2-line comment, which is nice.
examples/cupynumeric/upstream/vendors Apache-2.0 code. Please add the LICENSE file next to it, and NOTICE if there is one.examples/cupynumeric/README.md:16,40points at private paths (/u/atahilyani/cnum-gpu/wt/ports/env_ports.sh,wt/ports/run_suite.sh), and CODEMAP's header still saysexample/charmnumeric/.
- CLIP gets a complex body: lexicographic order with the clipped value's NaN first, as NumPy's complex loop (cnp.clip on complex arrays aborted every PE). - A registry gap no longer calls report_fatal_error: the kernel writes zeros for that op, is kept out of the disk cache, and PE 0 answers the next GET with the error, which CharmNumericInterface raises as RuntimeError. - test_jit_edges: complex clip against NumPy, and a float64 bitwise_and sent past the frontend reports the error and leaves the server usable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scalar fill (int flat * esz), the device halo pack and the dot's local_size used 32-bit offsets, so tiles of 2 GiB or more were written out of bounds. These kernels now index in 64 bits (CT_*_POLICY64 on the device). test_jit_edges: CNUM_BIG_SHAPES=1 adds a fill of a 3.2 GB array. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the thread that ran Kokkos::initialize selected the device, so other PE threads created their streams on device 0. Each DAG group branch now calls cudaSetDevice(Kokkos::device_id()), and cnEnsureContext falls back to the thread's runtime device instead of device 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nodes sharing a cache dir with the same CPU name but different features (e.g. AVX-512 masked) no longer load each other's objects. The epoch_arrived comment no longer claims client-reconnect support. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Assignment broadcasts against x[key]'s NumPy shape, not the storage shape. - _cached_transpose and _size1_to_rank refresh linked copies first, so a write to the parent clears their caches. - Negative-step writes count newaxis entries; linked copies write back through the key with its newaxis entries. - matmul treats every 3-D operand as a batch (N-D @ 2-D runs as one 2-D matmul); einsum keeps the internal squeezed-matrix path. - int ** 2.0 takes the x*x fast path only if NumPy keeps the dtype; NumPy scalars stay strong in power and comparisons. - >3-D random draws fold their index strides like the storage. - sum/prod accumulate in dtype= (or a wider out=) dtype. - 0-d device index arrays gather one element (ndE >= 1); sparse indices with >3 dims fold the IOTA axis. - Zero-size reductions return NumPy's identity (or NaN for mean/var/std), fractional ddof, choose dtype with scalar choices, atleast_*d/tile of 0-d, bool and out-of-range indices next to negative steps, empty negative-step slices, take/take_along_axis/flip on >3-D, flip always copies, lstsq rank is int32. - SVD, lstsq and tssvd report failures as all-NaN factors (CPU info(), GPU devInfo for U/Vh and tssvd). - examples/ops/test_numpy_semantics.py covers all of the above; test_exec_robust checks NumPy's batched matmul. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…enchmark - Checker.check compares the result dtype with NumPy's (expected's) unless dtype= is given; richardson-lucy x2 passes its float32 dtype explicitly. - _cnum_sim stores copies, so stale-data bugs are not hidden by aliasing. - test_reductions no longer execute()s fused temporaries before reuse; the zeros bug it worked around no longer reproduces. - _benchmark compares Python float/complex locals and skips only timer values (and arithmetic on them). - setup.py declares python_requires >= 3.12 (sys.monitoring) and numpy >= 2.0 (NEP 50 typing, quantile's numpy.lib module). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Frontend, tests and examples items. Backend items 8, 11, 12 and 13 are covered in a separate note. Silent wrong results
Inputs that kill the server Smaller NumPy divergences
Tests
Examples
Validated: CPU build, plus a GPU compile of |
CNUM_BIG_SHAPES fetches arrays over the 512 MB fetch chunk, which this head does not serve yet; the fill check fetches only a sum and one row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Backend items 8, 11, 12 and 13 are on branch
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Both fix sets (frontend and backend) are merged into this branch at 3b3b3fd. The merged head builds, and CPU regress passes at +p1 and +p2: orig 15/15, ports 25/25, ops 25/25. A GPU check of the device-only fixes (64-bit fill/halo/dot offsets, SMP device selection, complex clip on GPU) is queued; I'll post the result here. |
|
GPU check (1 A100, netlrts-cuda): |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Additional fix (33238cc): |
Needs UIUC-PPL/charmtyles#3. Reconverse on Slingshot also needs charmplusplus/reconverse#281.
GPU backend (NVIDIA)
~/.cache/charmnumeric/jit, set withCNUM_JIT_CACHE_DIR). With a warm cache, a kernel's first call takes about 3 ms instead of 30 ms.NumPy semantics
New operations
whereandclip.bincount,normand more, along any axis.takeandput.convolve.cuPyNumeric examples
All 25 examples in
examples/cupynumericare the upstream files unchanged plus a 2-line header. Each one checks its results against NumPy.Performance (A100)
Tested
examples/ops) in every configuration.recharm):Notes
+eth, because 9000-byte datagrams get fragmented on 1500-MTU networks.getof an array larger than about 2 GiB fails, because CCS replies carry a 32-bit length.cumsum's first call pays CUDA's lazy module loading;CUDA_MODULE_LOADING=EAGERavoids it.🤖 Generated with Claude Code