Skip to content

Stabilize -Zembed-metadata - #163436

Open
Kobzol wants to merge 2 commits into
rust-lang:mainfrom
Kobzol:stabilize-embed-metadata
Open

Kobzol wants to merge 2 commits into
rust-lang:mainfrom
Kobzol:stabilize-embed-metadata

Conversation

@Kobzol

@Kobzol Kobzol commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Tracking issue: #139165

This PR proposes to stabilize the -Zembed-metadata compiler flag, so that Cargo can start using -Cembed-metadata=no by default on the stable channel.

The flag can be used to avoid duplicating the metadata of Rust crates within .rlib and .rmeta files. This can reduce the size of the target directory anywhere from 5% to 35% on real-world projects, depending on the chosen compilation profile (1, 2). Additionally, -Cembed-metadata=no also removes the metadata out of Rust dylibs, and thus reduces their size.

The plan is to stabilize this compiler flag so that Cargo can start using it on the stable channel. The stabilization plan thus has two parts:

  • Stabilize the -Cembed-metadata flag in rustc
  • Change Cargo to use -Cembed-metadata=no by default

The Cargo team has prepared their side of the stabilization report. You can examine it to see the whole picture.

What is being stabilized

The -Zembed-metadata compiler flag.

Note that rustc will still default to -Cembed-metadata=yes, as that is its original behavior since Rust 1.16.0. But it will now be possible to pass -Cembed-metadata=no on stable.

Motivation

When compiling Rust crates as libraries, Cargo uses pipelined compilation, which tells rustc (using --emit=metadata) to output an .rmeta file as soon as possible, so that Cargo can start follow-up rustc instances sooner and speed up the compilation process. This file contains the Rust-specific metadata of the compiled crate.

However, after the compilation of the Rust crate finishes, the compiler will also write an .rlib file to disk, which contains both the compiled object code, but also the same metadata that is already contained in the .rmeta file.

This means that after the compilation finishes, the metadata of each (non-leaf) compiled crate will be duplicated on disk. The metadata can be quite large, and it is not uncommon that it takes hundreds of megabytes for real-world projects. This duplication thus contributes to increasing the size of the target directory, which large size is one of the top 3 most pressing challenges that Rust users struggle with.

The -Cembed-metadata=no flag removes this duplication (see below on how it works).

Some size benchmark results can be found in (1, 2). It can reach up to 35% size reduction on real-world projects in release mode, with the effect being smaller for dev builds (because incremental artifacts and debuginfo start to dominate the target directory size).

A slight bonus is that we only write the metadata to disk once, thus resulting in less I/O. I haven't noticed any measurable effect (either positive or negative) on compile times though, at least on Linux. The main motivation is disk size, not compilation speed.

What does the flag do

When you pass -Cembed-metadata=no, rustc will not write the full metadata of the compiled crate into the emitted .rlib file. Instead, it will only write a "metadata stub", which only contains the most basic information (name, target tuple, SVH).

Note that this flag should usually be combined with --emit=metadata to write the metadata to a .rmeta file, otherwise no metadata would be generated at all.

When loading crate dependencies passed via --extern, rustc will read the metadata of the passed .rlib. If it only contains a stub, it expects that you will also pass another --extern flag with a path to a .rmeta file with the full metadata. If you don't, you will be met with the following error:

error: only metadata stub found for `rlib` dependency `foo`. Please provide path to the corresponding .rmeta file with full metadata

Note that rustc will still default to the original behavior of embedding metadata, so build systems that wrap rustc will have to explicitly opt into using -Cembed-metadata=no, and at that point they will also have to start passing --extern=<...>.rmeta flags to rustc. That is what Cargo currently does if you use nightly or if you use the unstable -Zembed-metadata=no Cargo flag.

In theory, we could relax the requirement to pass both paths to the .rlib and the .rmeta file for each direct dependency in the future, and let rustc --extern=foo/bar.rlib automatically attempt to load --extern=foo/bar.rmeta when the .rmeta file path is not passed explicitly.

.rlib and .rmeta files produced using -Cembed-metadata=no and -Cembed-metadata=yes can be freely combined, and a downstream rustc invocation that consumes artifacts produced by =no does not need to itself receive the =no flag.

Ecosystem impact

The -Zembed-metadata flag has been used for the standard library that we ship since January 2026 (#145343) and for the compiler itself and codegen backends (cg_clif and cg_gcc) since August 2026 (#151061).

The flag has also been used by Cargo by default on the nightly channel since the second half of August 2026. A blog post with a call for testing can be found here.

I'm not aware of any regressions or issues reported in relation to this, not is the Cargo team (AFAIK). There are some unresolved questions about backwards compability if Cargo starts passing -Cembed-metadata=no by default, but those are related strictly to Cargo, and not the compiler flag itself (because the compiler flag will remain opt-in).

History

The issue of duplicated metadata was originally noted back in 2015 (#23366, #29511), back then because of the metadata being duplicated in Rust dylibs. Since Cargo started using pipelined compilation by default, it became more acute, because the metadata of all compiled .rlib crate dependencies started being duplicated in the target directory.

@bjorn3 proposed to add a compiler flag to split out the metadata into a separate file in #57076. The implementation was attempted in #93945 and later in #120855, but there was non-trivial complexity due to Cargo integration.

In 2025, I opened an MCP with the plan to implement this flag in two stages (first in rustc, and only then in Cargo), to reduce the implementation complexity, which was accepted. The implementation in rustc then happened in #137535.

Implementation history:

Unresolved questions/concerns

Command-line length limit

This is more related to the Cargo side of things, but I also wanted to mention it here, because there is a possible improvement we could do on the compiler side in the future.

As noted above, usage of -Cembed-metadata=no requires the build system/user to pass two --extern flags for each direct dependency of a given rustc invocation. This can increase the size of the executed command-line arguments, which can in edge cases start running into operating system limits.

Recently, we saw something similar in action with the Cargo's new build dir layout stabilization, where Cargo suddenly started passing a lot more -L flags to rustc, which caused issues around CLI length limits and PATH env. var. length limits. Cargo can work around the CLI length limit by using response files (rustc @args), but not all downstream tools (such as sccache) support those fully (yet). Based on the experience with the new build dir layout, some work was done to improve this support in the ecosystem though, and to optimize the number of flags that Cargo passes to rustc.

If this became a problem in practice, we could teach rustc to start implicitly assuming --extern=foo.rmeta when --extern=foo.rlib was passed. This would reduce the number of required --extern flags passed to the same count as with -Cembed-metadata=yes (or without using that flag at all).

I think that we can always do this retroactively if needed without breaking backwards compatibility, so it does not have to block this stabilization.

Landing this change

Given that this compiler flag is expected to have tight integration with Cargo, and Cargo already uses it on the nightly channel by default, we have to be a bit careful about how we actually land the -Zembed-metadata => -Cembed-metadata change.

Problem A: if we simply switched it from -Z to -C in rust-lang/rust, then nightly would break from the following day, because Cargo would be still attempting to pass the -Z flag, while rustc would already be expecting the -C variant of the flag.

Problem B: furthermore, bootstrap currently unconditionally passes -Zembed-metadata=no, both to in-tree Cargo and to beta Cargo, so that also requires some attention.

There are several options of how we could resolve this:

  1. Conclude the nightly Cargo experiment, so that it will not pass -Zembed-metadata=no to rustc by default anymore. Then sync Cargo to rust-lang/rust, and only then merge this PR. This solves A.

  2. Move Cargo from a git submodule to a (Josh) subtree, so that we can atomically change both Cargo and rustc to use -C instead of -Z (inside this PR), to avoid breaking nightly users. This solves A.

    • Even though I would like to do this migration anyway, I don't think that we need to tie it to this stabilization, as it can take some time before we resolve all concerns about the migration.
  3. Do what 2. proposes, but without a subtree. That means that we have to:

    • Change to -C in Cargo
    • Change to -C in rust-lang/rust
    • Sync the Cargo change back to rust-lang/rust in the same day that we land the rustc change

    While this solves A, it might run into some staging issues in bootstrap, and it looks very complicated to pull off.

  4. Teach rustc to accept both the -Z and the -C variant of the flag for some time. This solves both A and B, and also gives people who use -Zembed-metadata manually on nightly some breathing room to upgrade to the -C variant of the flag (but GitHub code search didn't find many usages of this flag, and this is normally an expected breakage for users of unstable compiler flags).

I decided to go with 4, because the integration with Cargo is tight, and modifying bootstrap back and forth to support -Z/no flag /-C would be busy work. This solves both A and B and requires little implementation effort.

Acknowledgments

  • @bjorn3 for coming up with the idea, providing a reference implementation and mentoring me to finish up the implementation
  • @petrochenkov for reviewing the compiler side of things
  • @epage and @weihanglo for reviewing the Cargo side of things

CC @bjorn3 @petrochenkov @weihanglo @epage

r? petrochenkov

@rustbot label +needs-fcp +T-compiler

@rustbot rustbot added A-run-make Area: port run-make Makefiles to rmake.rs A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. needs-fcp This change is insta-stable, or significant enough to need a team FCP to proceed. labels Sep 28, 2026
@rust-log-analyzer

This comment was marked as outdated.

@Kobzol
Kobzol force-pushed the stabilize-embed-metadata branch from 9823ce5 to 9dcacbc Compare September 28, 2026 12:17
@rust-log-analyzer

This comment has been minimized.

@petrochenkov

Copy link
Copy Markdown
Contributor
  1. Teach rustc to accept both the -Z and the -C variant of the flag for some time. This solves both A and B, and also gives people who use -Zembed-metadata manually on nightly some breathing room to upgrade to the -C variant of the flag (but GitHub code search didn't find many usages of this flag, and this is normally an expected breakage for users of unstable compiler flags).

IIRC, this is what we typically did in the past in similar situations.

@petrochenkov petrochenkov added T-cargo Relevant to the cargo team, which will review and decide on the PR/issue. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 28, 2026
@petrochenkov

Copy link
Copy Markdown
Contributor

The Cargo team is currently preparing a stabilization report of the Cargo side, once it is ready, I will link it here, so that you can see the whole picture.

There's no waiting-on-cargo label, so marking this as @rustbot blocked.

@rustbot rustbot added the S-blocked Status: Blocked on something else such as an RFC or other implementation work. label Sep 28, 2026
@Kobzol

Kobzol commented Sep 28, 2026

Copy link
Copy Markdown
Member Author

IIRC, this is what we typically did in the past in similar situations.

Would be fine by me, that would probably be the easiest option, implementation wise. Asked also in https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/How.20to.20land.20-Cembed-metadata/with/627486032 for a vibe-check.

@weihanglo

Copy link
Copy Markdown
Member

Cargo's stabilization report: rust-lang/cargo#17533.

But still keep the `-Zembed-metadata` variant for now, to make the migration easier.
@Kobzol
Kobzol force-pushed the stabilize-embed-metadata branch from 9dcacbc to d59eae9 Compare September 29, 2026 06:17
@Kobzol

Kobzol commented Sep 29, 2026

Copy link
Copy Markdown
Member Author

I implemented keeping both the -Z and -C option, because it really seems like the easiest way to resolve the tight Cargo integration. Passing both flags is an error.

Since the Cargo stabilization is up, I'm unmarking the PR as blocked.

@rustbot ready

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Sep 29, 2026
@petrochenkov

Copy link
Copy Markdown
Contributor

@rfcbot merge

@petrochenkov

Copy link
Copy Markdown
Contributor

@rfcbot fcp merge

@Kobzol

Kobzol commented Sep 29, 2026

Copy link
Copy Markdown
Member Author

@rfcbot fcp merge compiler

There are multiple team labels, so I think that we have to specify the team explicitly.

@rust-rfcbot

rust-rfcbot commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

@Kobzol has proposed to merge this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

@rust-rfcbot rust-rfcbot added the proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. label Sep 29, 2026
@rust-rfcbot rust-rfcbot added disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. and removed needs-fcp This change is insta-stable, or significant enough to need a team FCP to proceed. labels Sep 29, 2026
@petrochenkov petrochenkov added S-waiting-on-t-compiler Status: Awaiting decision from T-compiler S-waiting-on-fcp Status: PR is in FCP and is awaiting for FCP to complete. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-run-make Area: port run-make Makefiles to rmake.rs A-testsuite Area: The testsuite used to check the correctness of rustc disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. S-waiting-on-fcp Status: PR is in FCP and is awaiting for FCP to complete. S-waiting-on-t-compiler Status: Awaiting decision from T-compiler T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-cargo Relevant to the cargo team, which will review and decide on the PR/issue. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Projects

Status: FCP merge

Development

Successfully merging this pull request may close these issues.

6 participants