Stabilize -Zembed-metadata - #163436
Stabilize -Zembed-metadata#163436Kobzol wants to merge 2 commits into
-Zembed-metadata#163436Conversation
This comment was marked as outdated.
This comment was marked as outdated.
9823ce5 to
9dcacbc
Compare
This comment has been minimized.
This comment has been minimized.
IIRC, this is what we typically did in the past in similar situations. |
There's no waiting-on-cargo label, so marking this as @rustbot blocked. |
Would be fine by me, that would probably be the easiest option, implementation wise. Asked also in https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/How.20to.20land.20-Cembed-metadata/with/627486032 for a vibe-check. |
|
Cargo's stabilization report: rust-lang/cargo#17533. |
But still keep the `-Zembed-metadata` variant for now, to make the migration easier.
9dcacbc to
d59eae9
Compare
|
I implemented keeping both the Since the Cargo stabilization is up, I'm unmarking the PR as blocked. @rustbot ready |
|
@rfcbot merge |
|
@rfcbot fcp merge |
|
@rfcbot fcp merge compiler There are multiple team labels, so I think that we have to specify the team explicitly. |
|
@Kobzol has proposed to merge this. The next step is review by the rest of the tagged team members:
No concerns currently listed. Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up! See this document for info about what commands tagged team members can give me. |
Tracking issue: #139165
This PR proposes to stabilize the
-Zembed-metadatacompiler flag, so that Cargo can start using-Cembed-metadata=noby default on the stable channel.The flag can be used to avoid duplicating the metadata of Rust crates within
.rliband.rmetafiles. This can reduce the size of thetargetdirectory anywhere from 5% to 35% on real-world projects, depending on the chosen compilation profile (1, 2). Additionally,-Cembed-metadata=noalso removes the metadata out of Rustdylibs, and thus reduces their size.The plan is to stabilize this compiler flag so that Cargo can start using it on the stable channel. The stabilization plan thus has two parts:
-Cembed-metadataflag inrustc-Cembed-metadata=noby defaultThe Cargo team has prepared their side of the stabilization report. You can examine it to see the whole picture.
What is being stabilized
The
-Zembed-metadatacompiler flag.Note that
rustcwill still default to-Cembed-metadata=yes, as that is its original behavior since Rust 1.16.0. But it will now be possible to pass-Cembed-metadata=noon stable.Motivation
When compiling Rust crates as libraries, Cargo uses pipelined compilation, which tells
rustc(using--emit=metadata) to output an.rmetafile as soon as possible, so that Cargo can start follow-uprustcinstances sooner and speed up the compilation process. This file contains the Rust-specific metadata of the compiled crate.However, after the compilation of the Rust crate finishes, the compiler will also write an
.rlibfile to disk, which contains both the compiled object code, but also the same metadata that is already contained in the.rmetafile.This means that after the compilation finishes, the metadata of each (non-leaf) compiled crate will be duplicated on disk. The metadata can be quite large, and it is not uncommon that it takes hundreds of megabytes for real-world projects. This duplication thus contributes to increasing the size of the
targetdirectory, which large size is one of the top 3 most pressing challenges that Rust users struggle with.The
-Cembed-metadata=noflag removes this duplication (see below on how it works).Some size benchmark results can be found in (1, 2). It can reach up to 35% size reduction on real-world projects in
releasemode, with the effect being smaller fordevbuilds (because incremental artifacts and debuginfo start to dominate thetargetdirectory size).A slight bonus is that we only write the metadata to disk once, thus resulting in less I/O. I haven't noticed any measurable effect (either positive or negative) on compile times though, at least on Linux. The main motivation is disk size, not compilation speed.
What does the flag do
When you pass
-Cembed-metadata=no,rustcwill not write the full metadata of the compiled crate into the emitted.rlibfile. Instead, it will only write a "metadata stub", which only contains the most basic information (name, target tuple, SVH).Note that this flag should usually be combined with
--emit=metadatato write the metadata to a.rmetafile, otherwise no metadata would be generated at all.When loading crate dependencies passed via
--extern,rustcwill read the metadata of the passed.rlib. If it only contains a stub, it expects that you will also pass another--externflag with a path to a.rmetafile with the full metadata. If you don't, you will be met with the following error:Note that
rustcwill still default to the original behavior of embedding metadata, so build systems that wraprustcwill have to explicitly opt into using-Cembed-metadata=no, and at that point they will also have to start passing--extern=<...>.rmetaflags torustc. That is what Cargo currently does if you usenightlyor if you use the unstable-Zembed-metadata=noCargo flag.In theory, we could relax the requirement to pass both paths to the
.rliband the.rmetafile for each direct dependency in the future, and letrustc --extern=foo/bar.rlibautomatically attempt to load--extern=foo/bar.rmetawhen the.rmetafile path is not passed explicitly..rliband.rmetafiles produced using-Cembed-metadata=noand-Cembed-metadata=yescan be freely combined, and a downstreamrustcinvocation that consumes artifacts produced by=nodoes not need to itself receive the=noflag.Ecosystem impact
The
-Zembed-metadataflag has been used for the standard library that we ship since January 2026 (#145343) and for the compiler itself and codegen backends (cg_clif and cg_gcc) since August 2026 (#151061).The flag has also been used by Cargo by default on the
nightlychannel since the second half of August 2026. A blog post with a call for testing can be found here.I'm not aware of any regressions or issues reported in relation to this, not is the Cargo team (AFAIK). There are some unresolved questions about backwards compability if Cargo starts passing
-Cembed-metadata=noby default, but those are related strictly to Cargo, and not the compiler flag itself (because the compiler flag will remain opt-in).History
The issue of duplicated metadata was originally noted back in 2015 (#23366, #29511), back then because of the metadata being duplicated in Rust
dylibs. Since Cargo started using pipelined compilation by default, it became more acute, because the metadata of all compiled.rlibcrate dependencies started being duplicated in thetargetdirectory.@bjorn3 proposed to add a compiler flag to split out the metadata into a separate file in #57076. The implementation was attempted in #93945 and later in #120855, but there was non-trivial complexity due to Cargo integration.
In 2025, I opened an MCP with the plan to implement this flag in two stages (first in rustc, and only then in Cargo), to reduce the implementation complexity, which was accepted. The implementation in
rustcthen happened in #137535.Implementation history:
-Zembed-metadatato allow omitting full metadata from rlibs and dylibs #137535 (April 2025): Introduction of-Zembed-metadata, reviewed by @petrochenkov-Zembed-metadatacargo#15378 (May 2025): Initial integration in Cargo, reviewed by @epage-Zno-embed-metadatain the standard library #145343 (January 2026): Dogfood the flag for stdlib, reviewed by @bjorn3-Zembed-metadata=noby default on nightly Cargo cargo#17267 (August 2026): Enabled the-Cembed-metadata=noflag by default by Cargo on thenightlychannelUnresolved questions/concerns
Command-line length limit
This is more related to the Cargo side of things, but I also wanted to mention it here, because there is a possible improvement we could do on the compiler side in the future.
As noted above, usage of
-Cembed-metadata=norequires the build system/user to pass two--externflags for each direct dependency of a givenrustcinvocation. This can increase the size of the executed command-line arguments, which can in edge cases start running into operating system limits.Recently, we saw something similar in action with the Cargo's new build dir layout stabilization, where Cargo suddenly started passing a lot more
-Lflags torustc, which caused issues around CLI length limits andPATHenv. var. length limits. Cargo can work around the CLI length limit by using response files (rustc @args), but not all downstream tools (such assccache) support those fully (yet). Based on the experience with the new build dir layout, some work was done to improve this support in the ecosystem though, and to optimize the number of flags that Cargo passes torustc.If this became a problem in practice, we could teach
rustcto start implicitly assuming--extern=foo.rmetawhen--extern=foo.rlibwas passed. This would reduce the number of required--externflags passed to the same count as with-Cembed-metadata=yes(or without using that flag at all).I think that we can always do this retroactively if needed without breaking backwards compatibility, so it does not have to block this stabilization.
Landing this change
Given that this compiler flag is expected to have tight integration with Cargo, and Cargo already uses it on the nightly channel by default, we have to be a bit careful about how we actually land the
-Zembed-metadata=>-Cembed-metadatachange.Problem A: if we simply switched it from
-Zto-Cinrust-lang/rust, then nightly would break from the following day, because Cargo would be still attempting to pass the-Zflag, whilerustcwould already be expecting the-Cvariant of the flag.Problem B: furthermore, bootstrap currently unconditionally passes
-Zembed-metadata=no, both to in-tree Cargo and to beta Cargo, so that also requires some attention.There are several options of how we could resolve this:
Conclude the nightly Cargo experiment, so that it will not pass
-Zembed-metadata=notorustcby default anymore. Then sync Cargo torust-lang/rust, and only then merge this PR. This solves A.Move Cargo from a git submodule to a (Josh) subtree, so that we can atomically change both Cargo and
rustcto use-Cinstead of-Z(inside this PR), to avoid breaking nightly users. This solves A.Do what 2. proposes, but without a subtree. That means that we have to:
-Cin Cargo-Cinrust-lang/rustrust-lang/rustin the same day that we land therustcchangeWhile this solves A, it might run into some staging issues in bootstrap, and it looks very complicated to pull off.
Teach
rustcto accept both the-Zand the-Cvariant of the flag for some time. This solves both A and B, and also gives people who use-Zembed-metadatamanually onnightlysome breathing room to upgrade to the-Cvariant of the flag (but GitHub code search didn't find many usages of this flag, and this is normally an expected breakage for users of unstable compiler flags).I decided to go with 4, because the integration with Cargo is tight, and modifying bootstrap back and forth to support
-Z/no flag /-Cwould be busy work. This solves both A and B and requires little implementation effort.Acknowledgments
CC @bjorn3 @petrochenkov @weihanglo @epage
r? petrochenkov
@rustbot label +needs-fcp +T-compiler