Skip to content

Distribute DeePMD inference through MDCell for UnitCell workflows - #8330

Open
19hello wants to merge 14 commits into
deepmodeling:developfrom
19hello:fix/dp-mdcell-8310
Open

19hello wants to merge 14 commits into
deepmodeling:developfrom
19hello:fix/dp-mdcell-8310

Conversation

@19hello

@19hello 19hello commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Reminder

  • I have read AGENTS.md and docs/developers_guide/agent_governance.md.
  • I have linked an issue.
  • I have added focused unit tests and recorded local model runs.
  • I have listed verification commands, results, and limitations.
  • I have described user-visible behavior and INPUT impact.
  • I have explained the affected core modules.
  • No governance exception is requested.

Linked Issue

Refs #8310. The original 8192-atom run on two A800 80 GB GPUs with DeePMD-kit 3.1.1 remains to be verified before claiming the issue is fully fixed.

What's changed?

  • Add a source_cell MDCell adapter that initializes a domain-decomposed MDCell from STRU or UnitCell, synchronizes coordinates and lattice geometry, returns ghost forces to owners, and gathers a full force matrix for UnitCell optimizers. The driver passes concrete INPUT values to the adapter; the adapter handles restart replication, skin conversion, world-domain construction, atom-count logging, and the UnitCell backing association. It does not receive Parameter or use global PARAM. Run_MD::prepare_mdcell and its MDCell friend declaration were removed.
  • Route both DP cell kinds through one local-neighbor-list inference function. The UnitCell path keeps its distributed MDCell across calculations; relax and cell-relax keep their existing UnitCell-facing optimizer interface. Energy and virial use the MDCell cross-rank reductions.
  • Support dim_aparam > 0 by mapping either one parameter vector or global nat * dim_aparam values to owned atoms and, when the model requires it, ghosts. Keep fparam frame-scoped.
  • Select the DeePMD GPU rank from the node-local MPI rank or ABACUS GPU binding; probe the usable DeePMD C++ interface and communication-capable InputNlist; reject multi-rank inference when the latter is unavailable. Handle ranks with no local atoms.
  • Document distributed DP behavior, atomic-parameter order, and GPU-sharing limits in docs/advanced/md.md.

Unit Tests and/or Case Tests for my changes

  • cmake --build /tmp/abacus-dp-mdcell-build-dpa --target abacus_pw_para MODULE_CELL_mdcell_adapter MODULE_MD_dp_aparam -j8 — passed with local DeePMD 3.2.1.dev26 PyTorch build.
  • ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R '^(MODULE_CELL_mdcell_adapter(_np[24])?|MODULE_MD_dp_aparam|MODULE_MD_run)$' — 5/5 passed outside the restricted sandbox.
  • Non-MPI build of md and domain_decomposition — passed. DeePMD 2.2.11 C-API fallback md compile — passed.
  • With the reporter-provided DPA model: 512-atom Fe, two MPI ranks, one GPU — completed; 128-atom two-rank relax and cell-relax each completed two steps. A sparse two-atom case gave identical energy, forces, and pressure at one and four ranks, including completely empty ranks.
  • With a locally generated dim_aparam=1 DeePMD model: per-atom values 0.1 0.2 and uniform value 0.1 each gave matching one- and two-rank results; changing the parameters changed the energy.
  • python3 tools/03_code_analysis/agent_governance_check.py --base origin/develop --head HEAD --format text — no blockers, only four header-include review warnings. git diff --check origin/develop..HEAD — passed. Code-quality score: 8/8 changed C++ files passed.

Checks not completed or passed:

  • The original 8192-atom/two-A800/DeePMD 3.1.1 configuration is unavailable locally. This workstation has one 11 GB RTX 2080 Ti. At 1024 atoms, two ranks sharing that GPU exhausted its memory; this does not establish the memory limit when ranks use separate A800s.

  • Existing MODULE_MD_DP_potential uses Al-SCAN.pb and failed because this local DeePMD installation lacks libdeepmd_backend_tf.so. The older 2.2.11 installation also lacks required TensorFlow link dependencies. Neither result was counted as a pass.

  • The reporter model and the locally generated atomic-parameter model are not committed to the repository. A committed manual rank-comparison script and fixture now reproduce the reporter-model check; no model-backed Autotest case is registered because the model is an issue attachment.

  • Follow-up initialization refactor: source /home/fy19/abacus-develop/toolchain/install/setup && cmake --build /tmp/abacus-dp-mdcell-build-dpa --target abacus_pw_para MODULE_MD_run MODULE_CELL_mdcell_adapter -j4 — passed. OMP_NUM_THREADS=1 ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R 'MODULE_MD_run|MODULE_CELL_mdcell_adapter' — 4/4 passed (adapter at 1, 2, and 4 ranks, plus Run_MD).

  • After moving setup decisions into the adapter, the executable and MODULE_CELL_mdcell_adapter rebuilt successfully. The adapter regression test checks normal replication (2 atoms), restart without repeated replication (1 atom), and retention of the atom-count log; OMP_NUM_THREADS=1 ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R 'MODULE_MD_run|MODULE_CELL_mdcell_adapter' passed 4/4, including 1/2/4-rank adapter runs.

  • Temporary runtime cases with this build: two-rank LJ Anderson MD with cell_replica 2 1 1 and md_neighbor_skin 1.0 (64 atoms), PW NVT restart (2 atoms), and two-rank LJ NVE restart with cell_replica 2 1 1 (checkpoint remained 32 atoms) — all exited successfully; the atom-count log remained present. These smoke cases were run from /tmp, not committed.

  • python3 tools/03_code_analysis/agent_governance_check.py --staged and git diff --cached --check — passed with no blockers. The final adapter revision has no blockers: the checker reported a reduced global-reference count (1 added, 3 removed), a review warning for <iosfwd> required by the std::ofstream& declaration, and a no-docs-needed warning. This follow-up changes no INPUT keyword, log format, or public user behavior.

Governance Notes

  • INPUT/docs changes: no keyword or parser changes. dp_aparam now works in the MDCell inference path with its existing documented input shapes; docs/advanced/md.md describes the parallel behavior.
  • Core module impact: source_cell owns the adapter and empty-domain neighbor handling; source_md DP uses it. The external ESolver and UnitCell optimizer interfaces are unchanged. New source files are listed in CMake and source/Makefile.Objects.
  • Header dependency warnings: the new adapter and atomic-parameter headers need std::string, std::vector, and std::int64_t in their public declarations; concrete cell types are forward-declared.
  • Exceptions requested: none.

Latest follow-up verification

  • a04ba4094 unified MDCell lattice updates through set_lattice_geometry(latvec, lat0); NHC and MSST pass the existing lat0. The MPI decomposition, adapter, NHC, MSST, and LJ tests passed (6 tests total) after the affected targets built.

  • e442366b6 inlined the output/restart cadence checks without changing their predicates. source /home/fy19/abacus-develop/toolchain/install/setup && cmake --build /tmp/abacus-dp-mdcell-build-dpa --target abacus_pw_para MODULE_MD_func -j 8 passed; ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R '^MODULE_MD_func$' passed. A seven-step LJ MD smoke run with out_freq_ion=3 and md_restartfreq=5 produced STRU1, STRU6 for out_stru=0, and STRU1, STRU4, STRU6, STRU7 for out_stru=1; both wrote restart state at step 6. The earlier predicate-only MODULE_MD_run test target was removed with those predicates. An initial CTest command also selected three targets not built in the DPA build directory, so they were Not Run; this was a build-selection issue, not a test failure.

  • python3 tools/03_code_analysis/agent_governance_check.py --staged --format text passed with only a documentation-sync review warning. git diff --cached --check passed. These follow-up changes do not alter INPUT behavior or user-facing documentation.

  • e0c6d61a1 shortened only comments in run_md.cpp (11 comment lines added, 60 removed); executable statements are unchanged. git diff --cached --check and the staged governance check passed without blockers. No new runtime test or documentation is needed for this comment-only follow-up; the earlier MD runtime checks remain the behavior evidence.

  • 50d03d908 removes redundant int to std::size_t conversions only from standard-container subscripts in the DP inference and MDCell adapter paths. Bounds checks, size comparisons, and non-index conversions remain. source /home/fy19/abacus-develop/toolchain/install/setup && cmake --build /tmp/abacus-dp-mdcell-build-dpa --target abacus_pw_para MODULE_CELL_mdcell_adapter MODULE_MD_dp_aparam -j 8 passed; ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R '^(MODULE_CELL_mdcell_adapter(_np[24])?|MODULE_MD_dp_aparam)$' passed 4/4. The staged governance checker had no blockers, and git diff --cached --check passed. This is an indexing-only cleanup with no INPUT or numerical behavior change, so no new docs or tests are needed.

Conflict resolution and review follow-up (2026-11-19)

  • Merged upstream develop (d75733074), resolving the forcefield source rename and migrated unit-test paths. GitHub reports the PR mergeable against current develop.
  • d11038e21 verifies that libdeepmd_op_pt.so directly depends on libmpi.so before enabling distributed DeePMD inference in both CMake and Make builds. A non-MPI operator configuration now warns and excludes the communication-capable compile definition; runtime rejects multi-rank inference. The MD guide requires the operator and ABACUS to use the same MPI runtime. The resolved MPI library was checked locally with ldd.
  • Added a dedicated LJ output-cadence CTest check for actual STRU frame inventory and restart output with out_stru=0/1; exact file inventory cannot be expressed by the numerical Autotest reference checks. Added a manual model-backed rank-comparison script and fixture, with reproduction instructions in tests/04_FF/22_DP_rank_consistency/README.md. It checks energy, all forces and stress after coordinate and lattice changes, including empty rank partitions. The model binary remains the issue DPA3 relax runs out of GPU memory with two GPUs allocated #8310 attachment.
  • Reused a checked component_count for adapter force allocation and reduction.

Fresh verification after review changes:

  • cmake --build /tmp/abacus-dp-mdcell-build-dpa --target abacus_pw_para MODULE_CELL_mdcell_adapter MODULE_MD_dp_aparam -j 8 passed with local DeePMD 3.2 development build.
  • ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R '^(04_FF_md_output_cadence|MODULE_CELL_mdcell_adapter(_np[24])?|MODULE_MD_dp_aparam)$' passed 5/5.
  • python3 tests/integrate/validation_tools/check_dp_rank_consistency.py --abacus /tmp/abacus-dp-mdcell-build-dpa/abacus_pw_para --mpi /home/fy19/abacus-develop/toolchain/install/openmpi-5.0.10/bin/mpirun --model /tmp/issue8310_repro/frozen_model_op.pth --fixture tests/04_FF/22_DP_rank_consistency/STRU passed for original, displaced and expanded 2-atom cells at 1 and 4 ranks. The displaced cell produced a nonzero force (max 0.7566 eV/Angstrom). Comparison tolerances match the integration-suite values: 1e-5 eV, 1e-4 eV/Angstrom and 1e-3 kbar. The model run required execution outside the restricted sandbox because OpenMPI could not open its interfaces inside it.
  • Configuring against a synthetic operator with no libmpi.so dependency emitted the expected distributed-inference-disabled warning, and the resulting flags omitted __DPMD_COMM_NLIST_ARGS; the real MPI-enabled operator retained __DPMD_COMM_NLIST_ARGS=13.
  • python3 tools/03_code_analysis/agent_governance_check.py --staged reported no findings; git diff --cached --check passed. The full PR diff checker reports only review warnings for migration-neutral global use and header includes, with no blockers.

CI build fix (2026-11-19)

  • The GNU integration/unit-test and CUDA jobs failed during compilation of the migrated test_domain_decomposition.cpp: two tests still called removed MDCell::set_lattice_vectors. Commit 7fd9105b2 updates those calls to set_lattice_geometry(lattice, lat0) while preserving the original lattice constant. This is a test-only call-site correction; no user-facing behavior or documentation changes are needed.
  • cmake --build /tmp/abacus-dp-mdcell-build-dpa --target MODULE_CELL_NEIGHBOR_mdcell_migrate_mpi -j 8 passed.
  • ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R '^MODULE_CELL_NEIGHBOR_(mdcell_migrate_mpi|domain_decomposition_np[14])$' passed 3/3 at 2, 1, and 4 MPI ranks.
  • python3 tools/03_code_analysis/agent_governance_check.py --staged had only the expected documentation-sync review warning for a test-only change; git diff --cached --check passed.

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source inspection identifies a conditional MPI-backend compatibility risk in the new UnitCell inference route. The inline comments also include three non-blocking follow-ups. This was a static review; no builds or runtime tests were run.

ModuleBase::WARNING_QUIT("DP", "This DeePMD model requires dp_aparam.");
}
#endif
#if defined(__DPMD) && (!defined(__DPMD_COMM_NLIST_ARGS) || __DPMD_COMM_NLIST_ARGS == 0)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please enforce the MPI-enabled DeePMD operator prerequisite for distributed message-passing models such as PyTorch DPA-2. DeePMD 3.1.1 exposes this 12-argument InputNlist even when its custom operator was built without MPI; that operator then performs local copies instead of remote embedding exchange. The new UnitCell route therefore needs a safe rejection or supported fallback for that build, and the documentation should require the operator to use the runtime MPI library.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please preserve ordinary TensorFlow-only multi-rank DP runs when enforcing this prerequisite. The new CMake/Make checks remove the communication macro whenever libdeepmd_op_pt.so is absent, and DP::initialize then rejects every multi-rank model, including .pb files dispatched to DeepPotTF. Apply the PyTorch-operator requirement to inference that needs its exchange, or retain a supported fallback for the ordinary backend.

restart_freq,
param_in.inp.out_stru > 0);
const bool do_output_step = eff_output_freq > 0 && current_step % eff_output_freq == 0;
const bool do_restart_step = restart_freq > 0 && current_step % restart_freq == 0;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: could the reported unequal-cadence LJ smoke case be committed with assertions on the actual STRU and restart outputs for out_stru=0 and 1? The current inline predicates preserve the behavior, and these output assertions would retain useful regression coverage after removing MODULE_MD_run.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The committed LJ case now checks the actual STRU inventories for both out_stru settings. Non-blocking: also assert that the first line of Restart_md.txt is step 3, matching STRU4; checking only that the file is nonempty still accepts an initial-only checkpoint or a write on the wrong cadence.

TARGET MODULE_MD_DP_potential
LIBS ${_md_dp_test_libs}
SOURCES test_dp.cpp ../module_dp/dp.cpp
SOURCES test_dp.cpp ../module_dp/dp.cpp ../module_dp/dp_aparam.cpp

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: make the reported model-backed rank comparison reproducible with a small fixture or generation recipe. The registered UnitCell DP test disables MPI and checks only finite energy/forces; compare energy, all forces and stress across ranks after coordinate and lattice updates, including an empty partition.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The committed fixture and script make the three-geometry rank comparison reproducible. Non-blocking: each geometry starts a fresh SCF process, so the script does not test updates to the persistent DP/MDCell instance between ionic steps. Add consecutive evaluations on the same instance or a multi-step relax/cell-relax comparison to cover that remaining path.

Comment thread source/source_cell/mdcell_adapter.cpp Resolved
@mohanchen mohanchen added MD & LAM MD and Larege Atomic Models Refactor Refactor ABACUS codes labels Nov 19, 2026

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static follow-up of 7fd9105b243f; no builds or runtime tests were run. The checked component_count is now reused for allocation and reduction.

TARGET MODULE_MD_dp_forcefield
LIBS ${_md_dp_test_libs}
SOURCES test_dp_forcefield.cpp ../module_dp/dp_forcefield.cpp
../module_dp/dp_aparam.cpp

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With DeePMD enabled, MODULE_MD_dp_forcefield compiles calls to MDCellAdapter, but its sources and libraries provide none of their definitions. The pre-merge dependency list included mdcell_adapter.cpp and mdcell_reader.cpp; both were lost during the test migration, leaving unresolved references. Please restore those dependencies with the test's serial compile settings and build this target with DeePMD enabled.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

MD & LAM MD and Larege Atomic Models Refactor Refactor ABACUS codes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants