Higher memory usage on Rust 1.61.0 (compared to 1.60.0) leading to SIGKILL

[badboy]

It seems that memory usage during compilation has increased between 1.60.0 and 1.61.0, enough to trigger an OOM kill on CircleCI when trying to build my project.

Running on a medium+ instance (3 vCPUs, 6 GB RAM).
Command used: cargo test --all --verbose -- --nocapture

  = note: collect2: fatal error: ld terminated with signal 9 [Killed]

Example run: https://app.circleci.com/pipelines/github/mozilla/glean/9430/workflows/8c32e02e-6915-47ee-a587-a0ec098d9afd/jobs/175430
Same on nightly (rustc 1.63.0-nightly (28b891916 2022-05-29))

The same task runs just fine with Rust 1.60.0
I don't yet have a smaller example to showcase this, nor exact memory measurments. I'm willing to explore this further if you let me know how to best record memory usage.

[Stargateur]

try reduce parallel job cargo test --all --verbose -j 1 -- --nocapture

[badboy]

cargo test --all --verbose -j 1 -- --nocapture

That ultimately does make it succeed, but lengthens build time quite a bit

[ethowitz]

FWIW, I'm having this issue too. 1.61.0 seems to use substantially more memory during compilation (specifically, we are running into the issue with cargo clippy), and it's causing CI builds to time out. I'd be happy to provide more detailed information if there's a good way to measure memory usage. Using 1.60.0 solves the issue.

[Stargateur]

I think it's a insolvable problem, thus maybe we could at least be aware of memory increase consomption by rustc when we run crater for a new rustc release, with crater we could see rustc used more or less memory since last time, maybe... but in finite memory is always a hard problem, if you don't have enough memory to build your project the only solution is add memory, or reduce concurrent job, having cpu / 2 job for example. I don't think rustc will have enough contributor to guarantee stable memory usage, that a non goal I think.

@rustbot labels: +regression-from-stable-to-stable -regression-untriaged

(add label thus I'm not very used to this)

[apiraino]

Assigning priority as discussed in the Zulip thread of the Prioritization Working Group.

Ping to WG-compiler-performance for possible insights/suggestions to diagnose the source of the regression

@rustbot label -I-prioritize +P-high +wg-compiler-performance

[ehuss]

@badboy Can you confirm that you have 3 CPUs? In the error message in your log, I see 9 process errors which implies that more than 3 processes were running concurrently. Usually, if Cargo is limited to building 3 jobs in parallel, and one of them fails, it will halt immediately and not try any more. 1.61 changed to use the standard library available_parallelism, which might behaving differently in your setup. There have also been some changes to how it is computed (#92697).

I'm not sure what the best way is to check. Perhaps you can compile a small program and run it to test:

fn main() {
    println!("{}", std::thread::available_parallelism().unwrap());
}

Or if you want to get cargo to tell you directly, the only way I can think of is to create an empty project and get the timing data:

cargo new foo
cd foo
cargo b --timings
grep ncpu target/cargo-timings/cargo-timing.html