Skip to content

Update RunPod management to the V2 API. - #2271

Merged
chensuyue merged 7 commits into
mainfrom
xuehao/runpod
Sep 2, 2026
Merged

chensuyue merged 7 commits into
mainfrom
xuehao/runpod

Conversation

@XuehaoSun

Copy link
Copy Markdown
Contributor

This pull request significantly refactors and improves the reliability, robustness, and maintainability of the RunPod management and GPU monitoring scripts used in the nightly CUDA test pipeline. The most notable changes include a comprehensive rewrite of runpod_manager.py to use the RunPod v2 API with better error handling and dynamic GPU/data center selection, as well as substantial improvements to monitor_gpu.py for more robust metric collection and plotting.

RunPod management improvements:

  • Major rewrite of .azure-pipelines/scripts/cuda_unit_test/runpod_manager.py to use the RunPod v2 API, including:

    • Introduction of a RunPodV2Client class for consistent API requests with retry logic and error handling.
    • Dynamic GPU and data center selection based on live availability and a ban list, replacing static lists.
    • Improved pod creation, status polling, and termination logic with clearer error messages and handling for capacity issues and API errors.
    • Environment variable parsing and payload construction are now more robust and validated.
  • The CUDA_VERSION variable and related logic have been removed from the pipeline YAML, simplifying configuration.

GPU metric collection and plotting improvements:

  • Enhanced parsing and validation in .azure-pipelines/scripts/cuda_unit_test/monitor_gpu.py:

    • Added helper functions for safe float conversion, metric formatting, and dynamic time unit conversion for plotting.
    • Improved error handling and logging for malformed or missing GPU metric data, with detailed reporting of skipped rows.
    • Separate handling and plotting for GPU utilization and memory, with clear messages if data is missing. [1] [2] [3]
  • Device selection in the monitoring daemon now attempts CUDA-visible GPUs first, with fallback to NVML, improving compatibility with different environments.

General codebase quality:

  • Improved logging and error reporting throughout both scripts, making failures easier to diagnose and recover from. [1] [2]
  • Refactored and modularized code for better readability and maintainability, including splitting logic into helper functions and classes. [1] [2]

These changes should result in more reliable nightly test runs, easier debugging, and better future maintainability.

Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
@XuehaoSun

Copy link
Copy Markdown
Contributor Author

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@XuehaoSun
XuehaoSun requested a review from chensuyue September 2, 2026 02:10
@chensuyue
chensuyue merged commit 323e48d into main Sep 2, 2026
25 checks passed
@chensuyue
chensuyue deleted the xuehao/runpod branch September 2, 2026 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants