Automatic checking of cfgs at compile-time

The Cargo and Compiler team are delighted to announce that starting with Rust 1.80 (or nightly-2024-05-05) every reachable #[cfg] will be automatically checked that they match the expected config names and values.

This can help with verifying that the crate is correctly handling conditional compilation for different target platforms or features. It ensures that the cfg settings are consistent between what is intended and what is used, helping to catch potential bugs or errors early in the development process.

This addresses a common pitfall for new and advanced users.

This is another step to our commitment to provide user-focused tooling and we are eager and excited to finally see it fixed, after more than two years since the original RFC 30131.

A look at the feature

Every time a Cargo feature is declared that feature is transformed into a config that is passed to rustc (the Rust compiler) so it can verify with it along with well known cfgs if any of the #[cfg], #![cfg_attr] and cfg! have unexpected configs and report a warning with the unexpected_cfgs lint.

Cargo.toml:

[package]
name = "foo"

[features]
lasers = []
zapping = []

src/lib.rs:

#[cfg(feature = "lasers")]  // This condition is expected
                            // as "lasers" is an expected value
                            // of the `feature` cfg
fn shoot_lasers() {}

#[cfg(feature = "monkeys")] // This condition is UNEXPECTED
                            // as "monkeys" is NOT an expected
                            // value of the `feature` cfg
fn write_shakespeare() {}

#[cfg(windosw)]             // This condition is UNEXPECTED
                            // it's supposed to be `windows`
fn win() {}

cargo check:

Custom cfgs and build scripts

In Cargo point-of-view: a custom cfg is one that is neither defined by rustc nor by a Cargo feature. Think of tokio_unstable, has_foo, ... but not feature = "lasers", unix or debug_assertions

Some crates use custom cfgs that they either expected from the environment (RUSTFLAGSor other means) or is enabled by some logic in the crate build.rs. For those crates Cargo provides a new instruction: cargo::rustc-check-cfg2 (or cargo:rustc-check-cfg for older Cargo version).

The syntax to use is described in the rustc book section checking configuration, but in a nutshell the basic syntax of --check-cfg is:

cfg(name, values("value1", "value2", ..., "valueN"))

Note that every custom cfgs must always be expected, regardless if the cfg is active or not!

build.rs example

build.rs:

fn main() {
    println!("cargo::rustc-check-cfg=cfg(has_foo)");
    //        ^^^^^^^^^^^^^^^^^^^^^^ new with Cargo 1.80
    if has_foo() {
        println!("cargo::rustc-cfg=has_foo");
    }
}

Each cargo::rustc-cfg should have an accompanying unconditional cargo::rustc-check-cfg directive to avoid warnings like this: unexpected cfg condition name: has_foo.

Equivalence table

cargo::rustc-cfg

cargo::rustc-check-cfg

foo

cfg(foo) or cfg(foo, values(none()))

foo=""

cfg(foo, values(""))

foo="bar"

cfg(foo, values("bar"))

foo="1" and foo="2"

cfg(foo, values("1", "2"))

foo="1" and bar="2"

cfg(foo, values("1")) and cfg(bar, values("2"))

foo and foo="bar"

cfg(foo, values(none(), "bar"))

More details can be found in the rustc book.

Frequently asked questions

Can it be disabled?

For Cargo users, the feature is always on and cannot be disabled, but like any other lints it can be controlled: #![warn(unexpected_cfgs)].

Does the lint affect dependencies?

No, like most lints, unexpected_cfgs will only be reported for local packages thanks to cap-lints.

How does it interact with the RUSTFLAGS env?

You should be able to use the RUSTFLAGS environment variable like it was before.Currently --cfg arguments are not checked, only usage in code are.

This means that doing RUSTFLAGS="--cfg tokio_unstable" cargo check will not report any warnings, unless tokio_unstable is used within your local crates, in which case crate author will need to make sure that that custom cfg is expected with cargo::rustc-check-cfg in the build.rs of that crate.

How to expect custom cfgs without a build.rs?

There is currently no way to expect a custom cfg other than with cargo::rustc-check-cfg in a build.rs.

Crate authors that don't want to use a build.rs are encouraged to use Cargo features instead.

How does it interact with other build systems?

Non-Cargo based build systems are not affected by the lint by default. Build system authors that wish to have the same functionality should look at the rustc documentation for the --check-cfg flag for a detailed explanation of how to achieve the same functionality.

1. The stabilized implementation and RFC 3013 diverge significantly, in particular there is only one form for --check-cfg: cfg() (instead of values() and names() being incomplete and subtlety incompatible with each other). ↩
2. cargo::rustc-check-cfg will start working in Rust 1.80 (or nightly-2024-05-05). From Rust 1.77 to Rust 1.79 (inclusive) it is silently ignored. In Rust 1.76 and below a warning is emitted when used without the unstable Cargo flag -Zcheck-cfg. ↩

Continue Reading…

Announcing Rustup 1.27.1

The Rustup team is happy to announce the release of Rustup version 1.27.1.Rustup is the recommended tool to install Rust, a programming language that is empowering everyone to build reliable and efficient software.

If you have a previous version of Rustup installed, getting Rustup 1.27.1 is as easy as stopping any programs which may be using Rustup (e.g. closing your IDE) and running:

$ rustup self update

Rustup will also automatically update itself at the end of a normal toolchain update:

$ rustup update

If you don't have it already, you can get Rustup from the appropriate page on our website.

What's new in Rustup 1.27.1

This new Rustup release involves some minor bug fixes.

The headlines for this release are:

1. Prebuilt Rustup binaries should be working on older macOS versions again.
2. rustup-init will no longer fail when fish is installed but ~/.config/fish/conf.d hasn't been created.
3. Regressions regarding symlinked RUSTUP_HOME/(toolchains|downloads|tmp) have been addressed.

Full details are available in the changelog!

Rustup's documentation is also available in the Rustup Book.

Thanks

Thanks again to all the contributors who made Rustup 1.27.1 possible!

• Anas (0x61nas)
• cuiyourong (cuiyourong)
• Dirkjan Ochtman (djc)
• Eric Huss (ehuss)
• eth3lbert (eth3lbert)
• hev (heiher)
• klensy (klensy)
• Chih Wang (ongchi)
• Adam (pie-flavor)
• rami3l (rami3l)
• Robert (rben01)
• Robert Collins (rbtcollins)
• Sun Bin (shandongbinzhou)
• Samuel Moelius (smoelius)
• vpochapuis (vpochapuis)
• Renovate Bot (renovate)

Continue Reading…

Announcing Rust 1.78.0

The Rust team is happy to announce a new version of Rust, 1.78.0. Rust is a programming language empowering everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, you can get 1.78.0 with:

$ rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website, and check out the detailed release notes for 1.78.0.

If you'd like to help us out by testing future releases, you might consider updating locally to use the beta channel (rustup default beta) or the nightly channel (rustup default nightly). Please report any bugs you might come across!

What's in 1.78.0 stable

Diagnostic attributes

Rust now supports a #[diagnostic] attribute namespace to influence compiler error messages. These are treated as hints which the compiler is not required to use, and it is also not an error to provide a diagnostic that the compiler doesn't recognize. This flexibility allows source code to provide diagnostics even when they're not supported by all compilers, whether those are different versions or entirely different implementations.

With this namespace comes the first supported attribute, #[diagnostic::on_unimplemented], which can be placed on a trait to customize the message when that trait is required but hasn't been implemented on a type. Consider the example given in the stabilization pull request:

#[diagnostic::on_unimplemented(
    message = "My Message for `ImportantTrait<{A}>` is not implemented for `{Self}`",
    label = "My Label",
    note = "Note 1",
    note = "Note 2"
)]
trait ImportantTrait<A> {}

fn use_my_trait(_: impl ImportantTrait<i32>) {}

fn main() {
    use_my_trait(String::new());
}

Previously, the compiler would give a builtin error like this:

error[E0277]: the trait bound `String: ImportantTrait<i32>` is not satisfied
  --> src/main.rs:12:18
   |
12 |     use_my_trait(String::new());
   |     ------------ ^^^^^^^^^^^^^ the trait `ImportantTrait<i32>` is not implemented for `String`
   |     |
   |     required by a bound introduced by this call
   |

With #[diagnostic::on_unimplemented], its custom message fills the primary error line, and its custom label is placed on the source output. The original label is still written as help output, and any custom notes are written as well. (These exact details are subject to change.)

error[E0277]: My Message for `ImportantTrait<i32>` is not implemented for `String`
  --> src/main.rs:12:18
   |
12 |     use_my_trait(String::new());
   |     ------------ ^^^^^^^^^^^^^ My Label
   |     |
   |     required by a bound introduced by this call
   |
   = help: the trait `ImportantTrait<i32>` is not implemented for `String`
   = note: Note 1
   = note: Note 2

For trait authors, this kind of diagnostic is more useful if you can provide a better hint than just talking about the missing implementation itself. For example, this is an abridged sample from the standard library:

#[diagnostic::on_unimplemented(
    message = "the size for values of type `{Self}` cannot be known at compilation time",
    label = "doesn't have a size known at compile-time"
)]
pub trait Sized {}

For more information, see the reference section on the diagnostic tool attribute namespace.

Asserting unsafe preconditions

The Rust standard library has a number of assertions for the preconditions of unsafe functions, but historically they have only been enabled in #[cfg(debug_assertions)] builds of the standard library to avoid affecting release performance. However, since the standard library is usually compiled and distributed in release mode, most Rust developers weren't ever executing these checks at all.

Now, the condition for these assertions is delayed until code generation, so they will be checked depending on the user's own setting for debug assertions -- enabled by default in debug and test builds. This change helps users catch undefined behavior in their code, though the details of how much is checked are generally not stable.

For example, slice::from_raw_parts requires an aligned non-null pointer. The following use of a purposely-misaligned pointer has undefined behavior, and while if you were unlucky it may have appeared to "work" in the past, the debug assertion can now catch it:

fn main() {
    let slice: &[u8] = &[1, 2, 3, 4, 5];
    let ptr = slice.as_ptr();

    // Create an offset from `ptr` that will always be one off from `u16`'s correct alignment
    let i = usize::from(ptr as usize & 1 == 0);
    
    let slice16: &[u16] = unsafe { std::slice::from_raw_parts(ptr.add(i).cast::<u16>(), 2) };
    dbg!(slice16);
}

thread 'main' panicked at library/core/src/panicking.rs:220:5:
unsafe precondition(s) violated: slice::from_raw_parts requires the pointer to be aligned and non-null, and the total size of the slice not to exceed `isize::MAX`
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread caused non-unwinding panic. aborting.

Deterministic realignment

The standard library has a few functions that change the alignment of pointers and slices, but they previously had caveats that made them difficult to rely on in practice, if you followed their documentation precisely. Those caveats primarily existed as a hedge against const evaluation, but they're only stable for non-const use anyway. They are now promised to have consistent runtime behavior according to their actual inputs.

• pointer::align_offset computes the offset needed to change a pointer to the given alignment. It returns usize::MAX if that is not possible, but it was previously permitted to always return usize::MAX, and now that behavior is removed.
• slice::align_to and slice::align_to_mut both transmute slices to an aligned middle slice and the remaining unaligned head and tail slices. These methods now promise to return the largest possible middle part, rather than allowing the implementation to return something less optimal like returning everything as the head slice.

Stabilized APIs

• impl Read for &Stdin
• Accept non 'static lifetimes for several std::error::Error related implementations
• Make impl<Fd: AsFd> impl take ?Sized
• impl From for io::Error

These APIs are now stable in const contexts:

• Barrier::new()

Compatibility notes

• As previously announced, Rust 1.78 has increased its minimum requirement to Windows 10 for the following targets:
  • x86_64-pc-windows-msvc
  • i686-pc-windows-msvc
  • x86_64-pc-windows-gnu
  • i686-pc-windows-gnu
  • x86_64-pc-windows-gnullvm
  • i686-pc-windows-gnullvm
• Rust 1.78 has upgraded its bundled LLVM to version 18, completing the announced u128/i128 ABI change for x86-32 and x86-64 targets. Distributors that use their own LLVM older than 18 may still face the calling convention bugs mentioned in that post.

Other changes

Check out everything that changed in Rust, Cargo, and Clippy.

Contributors to 1.78.0

Many people came together to create Rust 1.78.0. We couldn't have done it without all of you. Thanks!

Continue Reading…

Announcing Google Summer of Code 2024 selected projects

The Rust Project is participating in Google Summer of Code (GSoC) 2024, a global program organized by Google which is designed to bring new contributors to the world of open-source.

In February, we published a list of GSoC project ideas, and started discussing these projects with potential GSoC applicants on our Zulip. We were pleasantly surprised by the amount of people that wanted to participate in these projects and that led to many fruitful discussions with members of various Rust teams. Some of them even immediately began contributing to various repositories of the Rust Project, even before GSoC officially started!

After the initial discussions, GSoC applicants prepared and submitted their project proposals. We received 65 (!) proposals in total. We are happy to see that there was so much interest, given that this is the first time the Rust Project is participating in GSoC.

A team of mentors primarily composed of Rust Project contributors then thoroughly examined the submitted proposals. GSoC required us to produce a ranked list of the best proposals, which was a challenging task in itself since Rust is a big project with many priorities! We went through many rounds of discussions and had to consider many factors, such as prior conversations with the given applicant, the quality and scope of their proposal, the importance of the proposed project for the Rust Project and its wider community, but also the availability of mentors, who are often volunteers and thus have limited time available for mentoring.

In many cases, we had multiple proposals that aimed to accomplish the same goal. Therefore, we had to pick only one per project topic despite receiving several high-quality proposals from people we'd love to work with. We also often had to choose between great proposals targeting different work within the same Rust component to avoid overloading a single mentor with multiple projects.

In the end, we narrowed the list down to twelve best proposals, which we felt was the maximum amount that we could realistically support with our available mentor pool. We submitted this list and eagerly awaited how many of these twelve proposals would be accepted into GSoC.

Selected projects

On the 1st of May, Google has announced the accepted projects. We are happy to announce that 9 proposals out of the twelve that we have submitted were accepted by Google, and will thus participate in Google Summer of Code 2024! Below you can find the list of accepted proposals (in alphabetical order), along with the names of their authors and the assigned mentor(s):

• Adding lint-level configuration to cargo-semver-checks by Max Carr, mentored by Predrag Gruevski
• Implementation of a Faster Register Allocator For Cranelift by d-sonuga, mentored by Chris Fallin and Amanieu d'Antras
• Improve Rust benchmark suite by s7tya, mentored by Jakub Beránek
• Move cargo shell completions to Rust by shanmu, mentored by Ed Page
• Rewriting Esoteric, Error-Prone Makefile Tests Using Robust Rust Features by Julien Robert, mentored by Jieyou Xu
• Rewriting the Rewrite trait by SeoYoung Lee, mentored by Yacin Tmimi
• Rust to .NET compiler - add support for compiling & running cargo tests by Fractal Fir, mentored by Jack Huey
• Sandboxed and Deterministic Proc Macro using Wasm by Apurva Mishra, mentored by David Lattimore
• Tokio async support in Miri by Tiffany Pek Yuan, mentored by Oli Scherer

Congratulations to all applicants whose project was selected! The mentors are looking forward to working with you on these exciting projects to improve the Rust ecosystem. You can expect to hear from us soon, so that we can start coordinating the work on your GSoC projects.

We would also like to thank all the applicants whose proposal was sadly not accepted, for their interactions with the Rust community and contributions to various Rust projects. There were some great proposals that did not make the cut, in large part because of limited review capacity. However, even if your proposal was not accepted, we would be happy if you would consider contributing to the projects that got you interested, even outside GSoC! Our project idea list is still actual, and could serve as a general entry point for contributors that would like to work on projects that would help the Rust Project maintainers and the Rust ecosystem.

Assuming our involvement in GSoC 2024 is successful, there's a good chance we'll participate next year as well (though we can't promise anything yet) and we hope to receive your proposals again in the future! We also are planning to participate in similar programs in the very near future. Those announcements will come in separate blog posts, so make sure to subscribe to this blog so that you don't miss anything.

The accepted GSoC projects will run for several months. After GSoC 2024 finishes (in autumn of 2024), we plan to publish a blog post in which we will summarize the outcome of the accepted projects.

Continue Reading…

Security advisory for the standard library (CVE-2024-24576)

The Rust Security Response WG was notified that the Rust standard library did not properly escape arguments when invoking batch files (with the bat andcmd extensions) on Windows using the Command API. An attacker able to control the arguments passed to the spawned process could execute arbitrary shell commands by bypassing the escaping.

The severity of this vulnerability is critical if you are invoking batch files on Windows with untrusted arguments. No other platform or use is affected.

This vulnerability is identified by CVE-2024-24576.

Overview

The Command::arg and Command::args APIs state in their documentation that the arguments will be passed to the spawned process as-is, regardless of the content of the arguments, and will not be evaluated by a shell. This means it should be safe to pass untrusted input as an argument.

On Windows, the implementation of this is more complex than other platforms, because the Windows API only provides a single string containing all the arguments to the spawned process, and it's up to the spawned process to split them. Most programs use the standard C run-time argv, which in practice results in a mostly consistent way arguments are splitted.

One exception though is cmd.exe (used among other things to execute batch files), which has its own argument splitting logic. That forces the standard library to implement custom escaping for arguments passed to batch files. Unfortunately it was reported that our escaping logic was not thorough enough, and it was possible to pass malicious arguments that would result in arbitrary shell execution.

Mitigations

Due to the complexity of cmd.exe, we didn't identify a solution that would correctly escape arguments in all cases. To maintain our API guarantees, we improved the robustness of the escaping code, and changed the Command API to return an InvalidInput error when it cannot safely escape an argument. This error will be emitted when spawning the process.

The fix will be included in Rust 1.77.2, to be released later today.

If you implement the escaping yourself or only handle trusted inputs, on Windows you can also use the CommandExt::raw_arg method to bypass the standard library's escaping logic.

Affected Versions

All Rust versions before 1.77.2 on Windows are affected, if your code or one of your dependencies executes batch files with untrusted arguments. Other platforms or other uses on Windows are not affected.

Acknowledgments

We want to thank RyotaK for responsibly disclosing this to us according to theRust security policy, and Simon Sawicki (Grub4K) for identifying some of the escaping rules we adopted in our fix.

We also want to thank the members of the Rust project who helped us disclose the vulnerability: Chris Denton for developing the fix; Mara Bos for reviewing the fix; Pietro Albini for writing this advisory; Pietro Albini, Manish Goregaokar and Josh Stone for coordinating this disclosure; Amanieu d'Antras for advising during the disclosure.

Continue Reading…

Announcing Rust 1.77.2

The Rust team has published a new point release of Rust, 1.77.2. Rust is a programming language that is empowering everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, getting Rust 1.77.2 is as easy as:

rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website.

What's in 1.77.2

This release includes a fix for CVE-2024-24576.

Before this release, the Rust standard library did not properly escape arguments when invoking batch files (with the bat and cmd extensions) on Windows using the Command API. An attacker able to control the arguments passed to the spawned process could execute arbitrary shell commands by bypassing the escaping.

This vulnerability is CRITICAL if you are invoking batch files on Windows with untrusted arguments. No other platform or use is affected.

You can learn more about the vulnerability in the dedicated advisory.

Contributors to 1.77.2

Many people came together to create Rust 1.77.2. We couldn't have done it without all of you. Thanks!

Continue Reading…

Changes to Rust's WASI targets

WASI 0.2 was recently stabilized, and Rust has begun implementing first-class support for it in the form of a dedicated new target. Rust 1.78 will introduce new wasm32-wasip1 (tier 2) and wasm32-wasip2 (tier 3) targets. wasm32-wasip1 is an effective rename of the existing wasm32-wasitarget, freeing the target name up for an eventual WASI 1.0 release. Starting Rust 1.78 (May 2nd, 2024), users of WASI 0.1 are encouraged to begin migrating to the new wasm32-wasip1 target before the existing wasm32-wasi target is removed in Rust 1.84 (January 5th, 2025).

In this post we'll discuss the introduction of the new targets, the motivation behind it, what that means for the existing WASI targets, and a detailed schedule for these changes. This post is about the WASI targets only; the existing wasm32-unknown-unknown and wasm32-unknown-emscripten targets are unaffected by any changes in this post.

Introducing wasm32-wasip2

After nearly five years of work the WASI 0.2 specificationwas recently stabilized. This work builds on WebAssembly Components (think: strongly-typed ABI for Wasm), providing standard interfaces for things like asynchronous IO, networking, and HTTP. This will finally make it possible to write asynchronous networked services on top of WASI, something which wasn't possible using WASI 0.1.

People interested in compiling Rust code to WASI 0.2 today are able to do so using the cargo-componenttool. This tool is able to take WASI 0.1 binaries, and transform them to WASI 0.2 Components using a shim. It also provides native support for common cargo commands such as cargo build, cargo test, and cargo run. While it introduces some inefficiencies because of the additional translation layer, in practice this already works really well and people should be enough able to get started with WASI 0.2 development.

We're however keen to begin making that translation layer obsolete. And for that reason we're happy to share that Rust has made its first steps towards that with the introduction of the tier 3 wasm32-wasip2target landing in Rust 1.78. This will initially miss a lot of expected features such as stdlib support, and we don't recommend people use this target quite yet. But as we fill in those missing features over the coming months, we aim to eventually hit meet the criteria to become a tier 2 target, at which point the wasm32-wasip2 target would be considered ready for general use. This work will happen through 2024, and we expect for this to land before the end of the calendar year.

Renaming wasm32-wasi to wasm32-wasip1

The original name for what we now call WASI 0.1 was "WebAssembly System Interface, snapshot 1". Rust shipped support for this in 2019, and we did so knowing the target would likely undergo significant changes in the future. With the knowledge we have today though, we would not have chosen to introduce the "WASI, snapshot 1" target as wasm32-wasi. We should have instead chosen to add some suffix to the initial target triple so that the eventual stable WASI 1.0 target can just be called wasm32-wasi.

In anticipation of both an eventual WASI 1.0 target, and to preserve consistency between target names, we'll begin rolling out a name change to the existing WASI 0.1 target. Starting in Rust 1.78 (May 2nd, 2024) a new wasm32-wasip1 target will become available. Starting Rust 1.81 (September 5th, 2024) we will begin warning existing users of wasm32-wasi to migrate to wasm32-wasip1. And finally in Rust 1.84 (January 9th, 2025) the wasm32-wasi target will no longer be shipped on the stable release channel. This will provide an 8 month transition period for projects to switch to the new target name when they update their Rust toolchains.

The name wasip1 can be read as either "WASI (zero) point one" or "WASI preview one". The official specification uses the "preview" moniker, however in most communication the form "WASI 0.1" is now preferred. This target triple was chosen because it not only maps to both terms, but also more closely resembles the target terminology used in other programming languages. This is something the WASI Preview 2 specification also makes note of.

Timeline

This table provides the dates and cut-offs for the target rename fromwasm32-wasi to wasm32-wasip1. The dates in this table do not apply to the newly-introduced wasm32-wasi-preview1-threads target; this will be renamed towasm32-wasip1-threads in Rust 1.78 without going through a transition period. The tier 3 wasm32-wasip2 target will also be made available in Rust 1.78.

date

Rust Stable

Rust Beta

Rust Nightly

Notes

2024-02-08

1.76

1.77

1.78

wasm32-wasip1 available on nightly

2024-03-21

1.77

1.78

1.79

wasm32-wasip1 available on beta

2024-05-02

1.78

1.79

1.80

wasm32-wasip1 available on stable

2024-06-13

1.79

1.80

1.81

warn if wasm32-wasi is used on nightly

2024-07-25

1.80

1.81

1.82

warn if wasm32-wasi is used on beta

2024-09-05

1.81

1.82

1.83

warn if wasm32-wasi is used on stable

2024-10-17

1.82

1.83

1.84

wasm32-wasi unavailable on nightly

2024-11-28

1.83

1.84

1.85

wasm32-wasi unavailable on beta

2025-01-09

1.84

1.85

1.86

wasm32-wasi unavailable on stable

Conclusion

In this post we've discussed the upcoming updates to Rust's WASI targets. Come Rust 1.78 the wasm32-wasip1 (tier 2) and wasm32-wasip2 (tier 3) targets will be added. In Rust 1.81 we will begin warning if wasm32-wasi is being used. And in Rust 1.84, the existing wasm32-wasi target will be removed. This will free up wasm32-wasi to eventually be used for a WASI 1.0 target. Users will have 8 months to switch to the new target name when they update their Rust toolchains.

The wasm32-wasip2 target marks the start of native support for WASI 0.2. In order to target it today from Rust, people are encouraged to usecargo-component tool instead. The plan is to eventually graduate wasm32-wasip2 to a tier-2 target, at which point cargo-component will be upgraded to support it natively instead.

With WASI 0.2 finally stable, it's an exciting time for WebAssembly development. We're happy for Rust to begin implementing native support for WASI 0.2, and we're excited for what this will enable people to build.

Continue Reading…

Changes to `u128`/`i128` layout in 1.77 and 1.78

Rust has long had an inconsistency with C regarding the alignment of 128-bit integers on the x86-32 and x86-64 architectures. This problem has recently been resolved, but the fix comes with some effects that are worth being aware of.

As a user, you most likely do not need to worry about these changes unless you are:

1. Assuming the alignment of i128/u128 rather than using align_of
2. Ignoring the improper_ctypes* lints and using these types in FFI

There are also no changes to architectures other than x86-32 and x86-64. If your code makes heavy use of 128-bit integers, you may notice runtime performance increases at a possible cost of additional memory use.

This post documents what the problem was, what changed to fix it, and what to expect with the changes. If you are already familiar with the problem and only looking for a compatibility matrix, jump to the Compatibility section.

Background

Data types have two intrinsic values that relate to how they can be arranged in memory; size and alignment. A type's size is the amount of space it takes up in memory, and its alignment specifies which addresses it is allowed to be placed at.

The size of simple types like primitives is usually unambiguous, being the exact size of the data they represent with no padding (unused space). For example, an i64 always has a size of 64 bits or 8 bytes.

Alignment, however, can vary. An 8-byte integer could be stored at any memory address (1-byte aligned), but most 64-bit computers will get the best performance if it is instead stored at a multiple of 8 (8-byte aligned). So, like in other languages, primitives in Rust have this most efficient alignment by default. The effects of this can be seen when creating composite types (playground link):

use core::mem::{align_of, offset_of};

#[repr(C)]
struct Foo {
    a: u8,  // 1-byte aligned
    b: u16, // 2-byte aligned
}

#[repr(C)]
struct Bar {
    a: u8,  // 1-byte aligned
    b: u64, // 8-byte aligned
}

println!("Offset of b (u16) in Foo: {}", offset_of!(Foo, b));
println!("Alignment of Foo: {}", align_of::<Foo>());
println!("Offset of b (u64) in Bar: {}", offset_of!(Bar, b));
println!("Alignment of Bar: {}", align_of::<Bar>());

Output:

Offset of b (u16) in Foo: 2
Alignment of Foo: 2
Offset of b (u64) in Bar: 8
Alignment of Bar: 8

We see that within a struct, a type will always be placed such that its offset is a multiple of its alignment - even if this means unused space (Rust minimizes this by default when repr(C) is not used).

These numbers are not arbitrary; the application binary interface (ABI) says what they should be. In the x86-64 psABI (processor-specific ABI) for System V (Unix & Linux),Figure 3.1: Scalar Types tells us exactly how primitives should be represented:

C type

Rust equivalent

sizeof

Alignment (bytes)

char

i8

1

1

unsigned char

u8

1

1

short

i16

2

2

unsigned short

u16

2

2

long

i64

8

8

unsigned long

u64

8

8

The ABI only specifies C types, but Rust follows the same definitions both for compatibility and for the performance benefits.

The Incorrect Alignment Problem

If two implementations disagree on the alignment of a data type, they cannot reliably share data containing that type. Rust had inconsistent alignment for 128-bit types:

println!("alignment of i128: {}", align_of::<i128>());

// rustc 1.76.0
alignment of i128: 8

printf("alignment of __int128: %zu\n", _Alignof(__int128));

// gcc 13.2
alignment of __int128: 16

// clang 17.0.1
alignment of __int128: 16

(Godbolt link) Looking back at the psABI, we can see that Rust has the wrong alignment here:

C type

Rust equivalent

sizeof

Alignment (bytes)

__int128

i128

16

16

unsigned __int128

u128

16

16

It turns out this isn't because of something that Rust is actively doing incorrectly: layout of primitives comes from the LLVM codegen backend used by both Rust and Clang, among other languages, and it has the alignment for i128 hardcoded to 8 bytes.

Clang uses the correct alignment only because of a workaround, where the alignment is manually set to 16 bytes before handing the type to LLVM. This fixes the layout issue but has been the source of some other minor problems.2Rust does no such manual adjustement, hence the issue reported athttps://github.com/rust-lang/rust/issues/54341.

The Calling Convention Problem

There is an additional problem: LLVM does not always do the correct thing when passing 128-bit integers as function arguments. This was a known issue in LLVM, before itsrelevance to Rust was discovered.

When calling a function, the arguments get passed in registers (special storage locations within the CPU) until there are no more slots, then they get "spilled" to the stack (the program's memory). The ABI tells us what to do here as well, in the section 3.2.3 Parameter Passing:

Arguments of type __int128 offer the same operations as INTEGERs, yet they do not fit into one general purpose register but require two registers. For classification purposes __int128 is treated as if it were implemented as:

typedef struct {
   long low, high;
} __int128;

with the exception that arguments of type __int128 that are stored in memory must be aligned on a 16-byte boundary.

We can try this out by implementing the calling convention manually. In the below C example, inline assembly is used to call foo(0xaf, val, val, val) with val as0x0x11223344556677889900aabbccddeeff.

x86-64 uses the registers rdi, rsi, rdx, rcx, r8, and r9 to pass function arguments, in that order (you guessed it, this is also in the ABI). Each register fits a word (64 bits), and anything that doesn't fit gets pushed to the stack.

/* full example at <https://godbolt.org/z/5c8cb5cxs> */

/* to see the issue, we need a padding value to "mess up" argument alignment */
void foo(char pad, __int128 a, __int128 b, __int128 c) {
    printf("%#x\n", pad & 0xff);
    print_i128(a);
    print_i128(b);
    print_i128(c);
}

int main() {
    asm(
        /* load arguments that fit in registers */
        "movl    $0xaf, %edi \n\t"                /* 1st slot (edi): padding char (`edi` is the
                                                   * same as `rdi`, just a smaller access size) */
        "movq    $0x9900aabbccddeeff, %rsi \n\t"  /* 2rd slot (rsi): lower half of `a` */
        "movq    $0x1122334455667788, %rdx \n\t"  /* 3nd slot (rdx): upper half of `a` */
        "movq    $0x9900aabbccddeeff, %rcx \n\t"  /* 4th slot (rcx): lower half of `b` */
        "movq    $0x1122334455667788, %r8  \n\t"  /* 5th slot (r8):  upper half of `b` */
        "movq    $0xdeadbeef4c0ffee0, %r9  \n\t"  /* 6th slot (r9):  should be unused, but
                                                   * let's trick clang! */

        /* reuse our stored registers to load the stack */
        "pushq   %rdx \n\t"                       /* upper half of `c` gets passed on the stack */
        "pushq   %rsi \n\t"                       /* lower half of `c` gets passed on the stack */

        "call    foo \n\t"                        /* call the function */
        "addq    $16, %rsp \n\t"                  /* reset the stack */
    );
}

Running the above with GCC prints the following expected output:

0xaf
0x11223344556677889900aabbccddeeff
0x11223344556677889900aabbccddeeff
0x11223344556677889900aabbccddeeff

But running with Clang 17 prints:

0xaf
0x11223344556677889900aabbccddeeff
0x11223344556677889900aabbccddeeff
0x9900aabbccddeeffdeadbeef4c0ffee0
//^^^^^^^^^^^^^^^^ this should be the lower half
//                ^^^^^^^^^^^^^^^^ look familiar?

Surprise!

This illustrates the second problem: LLVM expects an i128 to be passed half in a register and half on the stack when possible, but this is not allowed by the ABI.

Since the behavior comes from LLVM and has no reasonable workaround, this is a problem in both Clang and Rust.

Solutions

Getting these problems resolved was a lengthy effort by many people, starting with a patch by compiler team member Simonas Kazlauskas in 2017: D28990. Unfortunately, this wound up reverted. It was later attempted again in D86310 by LLVM contributor Harald van Dijk, which is the version that finally landed in October 2023.

Around the same time, Nikita Popov fixed the calling convention issue with D158169. Both of these changes made it into LLVM 18, meaning all relevant ABI issues will be resolved in both Clang and Rust that use this version (Clang 18 and Rust 1.78 when using the bundled LLVM).

However, rustc can also use the version of LLVM installed on the system rather than a bundled version, which may be older. To mitigate the chance of problems from differing alignment with the same rustc version, a proposal was introduced to manually correct the alignment like Clang has been doing. This was implemented by Matthew Maurer in #11672.

Since these changes, Rust now produces the correct alignment:

println!("alignment of i128: {}", align_of::<i128>());

// rustc 1.77.0
alignment of i128: 16

As mentioned above, part of the reason for an ABI to specify the alignment of a datatype is because it is more efficient on that architecture. We actually got to see that firsthand: the initial performance run with the manual alignment change showed nontrivial improvements to compiler performance (which relies heavily on 128-bit integers to work with integer literals). The downside of increasing alignment is that composite types do not always fit together as nicely in memory, leading to an increase in usage. Unfortunately this meant some of the performance wins needed to be sacrificed to avoid an increased memory footprint.

Compatibility

The most imporant question is how compatibility changed as a result of these fixes. In short, i128 and u128 with Rust using LLVM 18 (the default version starting with 1.78) will be completely compatible with any version of GCC, as well as Clang 18 and above (released March 2024). All other combinations have some incompatible cases, which are summarized in the table below:

Compiler 1

Compiler 2

status

Rust ≥ 1.78 with bundled LLVM (18)

GCC (any version)

Fully compatible

Rust ≥ 1.78 with bundled LLVM (18)

Clang ≥ 18

Fully compatible

Rust ≥ 1.77 with LLVM ≥ 18

GCC (any version)

Fully compatible

Rust ≥ 1.77 with LLVM ≥ 18

Clang ≥ 18

Fully compatible

Rust ≥ 1.77 with LLVM ≥ 18

Clang < 18

Storage compatible, has calling bug

Rust ≥ 1.77 with LLVM < 18

GCC (any version)

Storage compatible, has calling bug

Rust ≥ 1.77 with LLVM < 18

Clang (any version)

Storage compatible, has calling bug

Rust < 1.773

GCC (any version)

Incompatible

Rust < 1.773

Clang (any version)

Incompatible

GCC (any version)

Clang ≥ 18

Fully compatible

GCC (any version)

Clang < 18

Storage compatible with calling bug

Effects & Future Steps

As mentioned in the introduction, most users will notice no effects of this change unless you are already doing something questionable with these types.

Starting with Rust 1.77, it will be reasonably safe to start experimenting with 128-bit integers in FFI, with some more certainty coming with the LLVM update in 1.78. There is ongoing discussion about lifting the lint in an upcoming version, but we want to be cautious and avoid introducing silent breakage for users whose Rust compiler may be built with an older LLVM.

1. https://bugs.llvm.org/show_bug.cgi?id=50198 ↩
2. https://github.com/llvm/llvm-project/issues/20283 ↩
3. Rust < 1.77 with LLVM 18 will have some degree of compatibility, this is just an uncommon combination. ↩ ↩2

Continue Reading…

Announcing Rust 1.77.1

The Rust team has published a new point release of Rust, 1.77.1. Rust is a programming language that is empowering everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, getting Rust 1.77.1 is as easy as:

rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website.

What's in 1.77.1

Cargo enabled stripping of debuginfo in release builds by defaultin Rust 1.77.0. However, due to a pre-existing issue, debuginfo stripping does not behave in the expected way on Windows with the MSVC toolchain.

Rust 1.77.1 therefore disables the new Cargo behavior on Windows for targets that use MSVC. There are no changes for other targets. We plan to eventually re-enable debuginfo stripping in release mode in a later Rust release.

Contributors to 1.77.1

Many people came together to create Rust 1.77.1. We couldn't have done it without all of you. Thanks!

Continue Reading…

Announcing Rust 1.77.0

The Rust team is happy to announce a new version of Rust, 1.77.0. Rust is a programming language empowering everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, you can get 1.77.0 with:

$ rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website, and check out the detailed release notes for 1.77.0.

If you'd like to help us out by testing future releases, you might consider updating locally to use the beta channel (rustup default beta) or the nightly channel (rustup default nightly). Please report any bugs you might come across!

What's in 1.77.0 stable

This release is relatively minor, but as always, even incremental improvements lead to a greater whole. A few of those changes are highlighted in this post, and others may yet fill more niche needs.

C-string literals

Rust now supports C-string literals (c"abc") which expand to a nul-byte terminated string in memory of type &'static CStr. This makes it easier to write code interoperating with foreign language interfaces which require nul-terminated strings, with all of the relevant error checking (e.g., lack of interior nul byte) performed at compile time.

Support for recursion in async fn

Async functions previously could not call themselves due to a compiler limitation. In 1.77, that limitation has been lifted, so recursive calls are permitted so long as they use some form of indirection to avoid an infinite size for the state of the function.

This means that code like this now works:

async fn fib(n: u32) -> u32 {
   match n {
       0 | 1 => 1,
       _ => Box::pin(fib(n-1)).await + Box::pin(fib(n-2)).await
   }
}

offset_of!

1.77.0 stabilizes offset_of! for struct fields, which provides access to the byte offset of the relevant public field of a struct. This macro is most useful when the offset of a field is required without an existing instance of a type. Implementing such a macro is already possible on stable, but without an instance of the type the implementation would require tricky unsafe code which makes it easy to accidentally introduce undefined behavior.

Users can now access the offset of a public field with offset_of!(StructName, field). This expands to a usize expression with the offset in bytes from the start of the struct.

Enable strip in release profiles by default

Cargo profileswhich do not enable debuginfo in outputs (e.g., debug = 0) will enable strip = "debuginfo" by default.

This is primarily needed because the (precompiled) standard library ships with debuginfo, which means that statically linked results would include the debuginfo from the standard library even if the local compilations didn't explicitly request debuginfo.

Users which do want debuginfo can explicitly enable it with thedebugflag in the relevant Cargo profile.

Clippy adds a new incompatible_msrv lint

The Rust project only supports the latest stable release of Rust. Some libraries aim to have an older minimum supported Rust version (MSRV), typically verifying this support by compiling in CI with an older release. However, when developing new code, it's convenient to use latest documentation and the latest toolchain with fixed bugs, performance improvements, and other improvements. This can make it easy to accidentally start using an API that's only available on newer versions of Rust.

Clippy has added a new lint, incompatible_msrv, which will inform users if functionality being referenced is only available on newer versions than theirdeclared MSRV.

Stabilized APIs

• array::each_ref
• array::each_mut
• core::net
• f32::round_ties_even
• f64::round_ties_even
• mem::offset_of!
• slice::first_chunk
• slice::first_chunk_mut
• slice::split_first_chunk
• slice::split_first_chunk_mut
• slice::last_chunk
• slice::last_chunk_mut
• slice::split_last_chunk
• slice::split_last_chunk_mut
• slice::chunk_by
• slice::chunk_by_mut
• Bound::map
• File::create_new
• Mutex::clear_poison
• RwLock::clear_poison

Other changes

Check out everything that changed in Rust, Cargo, and Clippy.

Contributors to 1.77.0

Many people came together to create Rust 1.77.0. We couldn't have done it without all of you. Thanks!

Continue Reading…

Announcing Rustup 1.27.0

The rustup team is happy to announce the release of rustup version 1.27.0.Rustup is the recommended tool to install Rust, a programming language that is empowering everyone to build reliable and efficient software.

If you have a previous version of rustup installed, getting rustup 1.27.0 is as easy as stopping any programs which may be using Rustup (e.g. closing your IDE) and running:

$ rustup self update

Rustup will also automatically update itself at the end of a normal toolchain update:

$ rustup update

If you don't have it already, you can get rustup from the appropriate page on our website.

What's new in rustup 1.27.0

This long-awaited Rustup release has gathered all the new features and fixes since April 2023. These changes include improvements in Rustup's maintainability, user experience, compatibility and documentation quality.

Also, it's worth mentioning that Dirkjan Ochtman (djc) and rami3l (rami3l) have joined the team and are coordinating this new release.

At the same time, we have granted Daniel Silverstone (kinnison) and 二手掉包工程师 (hi-rustin) their well-deserved alumni status in this release cycle. Kudos for your contributions over the years and your continuous guidance on maintaining the project!

The headlines for this release are:

1. Basic support for the fish shell has been added. If you're using fish, PATH configs for your Rustup installation will be added automatically from now on.
   Please note that this will only take effect on installation, so if you have already installed Rustup on your machine, you will need to reinstall it. For example, if you have installed Rustup via rustup.rs, simply follow rustup.rs's instructions again; if you have installed Rustup using some other method, you might want to reinstall it using that same method.
2. Rustup support for loongarch64-unknown-linux-gnu as a host platform has been added. This means you should be able to install Rustup via rustup.rs and no longer have to rely on loongnix.cn or self-compiled installations.
   Please note that as of March 2024, loongarch64-unknown-linux-gnu is a "tier 2 platform with host tools", so Rustup is guaranteed to build for this platform. According to Rust's target tier policy, this does not imply that these builds are also guaranteed to work, but they often work to quite a good degree and patches are always welcome!

Full details are available in the changelog!

Rustup's documentation is also available in the rustup book.

Thanks

Thanks again to all the contributors who made rustup 1.27.0 possible!

• Anthony Perkins (acperkins)
• Tianqi (airstone42)
• Alex Gaynor (alex)
• Alex Hudspith (alexhudspith)
• Alan Somers (asomers)
• Brett (brettearle)
• Burak Emir (burakemir)
• Chris Denton (ChrisDenton)
• cui fliter (cuishuang)
• Dirkjan Ochtman (djc)
• Dezhi Wu (dzvon)
• Eric Swanson (ericswanson-dfinity)
• Prikshit Gautam (gautamprikshit1)
• hev (heiher)
• 二手掉包工程师 (hi-rustin)
• Kamila Borowska (KamilaBorowska)
• klensy (klensy)
• Jakub Beránek (Kobzol)
• Kornel (kornelski)
• Matt Harding (majaha)
• Mathias Brossard (mbrossard)
• Christian Thackston (nan60)
• Ruohui Wang (noirgif)
• Olivier Lemasle (olivierlemasle)
• Chih Wang (ongchi)
• Pavel Roskin (proski)
• rami3l (rami3l)
• Robert Collins (rbtcollins)
• Sandesh Pyakurel (Sandesh-Pyakurel)
• Waffle Maybe (WaffleLapkin)
• Jubilee (workingjubilee)
• WÁNG Xuěruì (xen0n)
• Yerkebulan Tulibergenov (yerke)
• Renovate Bot (renovate)

Continue Reading…

crates.io: Download changes

Like the rest of the Rust community, crates.io has been growing rapidly, with download and package counts increasing 2-3x year-on-year. This growth doesn't come without problems, and we have made some changes to download handling on crates.io to ensure we can keep providing crates for a long time to come.

The Problem

This growth has brought with it some challenges. The most significant of these is that all download requests currently go through the crates.io API, occasionally causing scaling issues. If the API is down or slow, it affects all download requests too. In fact, the number one cause of waking up our crates.io on-call team is "slow downloads" due to the API having performance issues.

Additionally, this setup is also problematic for users outside of North America, where download requests are slow due to the distance to the crates.io API servers.

The Solution

To address these issues, over the last year we have decided to make some changes:

Starting from 2024-03-12, cargo will begin to download crates directly from our static.crates.io CDN servers.

This change will be facilitated by modifying the config.json file on the package index. In other words: no changes to cargo or your own system are needed for the changes to take effect. The config.json file is used by cargo to determine the download URLs for crates, and we will update it to point directly to the CDN servers, instead of the crates.io API.

Over the past few months, we have made several changes to the crates.io backend to enable this:

• We announced the deprecation of "non-canonical" downloads, which would be harder to support when downloading directly from the CDN.
• We changed how downloads are counted. Previously, downloads were counted directly on the crates.io API servers. Now, we analyze the log files from the CDN servers to count the download requests.

The latter change has caused the download numbers of most crates to increase, as some download requests were not counted before. Specifically, crates.io mirrors were often downloading directly from the CDN servers already, and those downloads had previously not been counted. For crates with a lot of downloads these changes will be barely noticeable, but for smaller crates, the download numbers have increased quite a bit over the past few weeks since we enabled this change.

Expected Outcomes

We expect these changes to significantly improve the reliability and speed of downloads, as the performance of the crates.io API servers will no longer affect the download requests. Over the next few weeks, we will monitor the performance of the system to ensure that the changes have the expected effects.

We have noticed that some non-cargo build systems are not using the config.json file of the index to build the download URLs. We will reach out to the maintainers of those build systems to ensure that they are aware of the change and to help them update their systems to use the new download URLs. The old download URLs will continue to work, but these systems will be missing out on the potential performance improvement.

We are excited about these changes and believe they will greatly improve the reliability of crates.io. We look forward to hearing your feedback!

Continue Reading…

Clippy: Deprecating `feature = "cargo-clippy"`

Since Clippy v0.0.97 and before it was shipped with rustup, Clippy implicitly added a feature = "cargo-clippy" config1 when linting your code with cargo clippy.

Back in the day (2016) this was necessary to allow, warn or deny Clippy lints using attributes:

#[cfg_attr(feature = "cargo-clippy", allow(clippy_lint_name))]

Doing this hasn't been necessary for a long time. Today, Clippy users will set lint levels with tool lint attributes using the clippy:: prefix:

#[allow(clippy::lint_name)]

The implicit feature = "cargo-clippy" has only been kept for backwards compatibility, but will now be deprecated.

Alternative

As there is a rare use case for conditional compilation depending on Clippy, we will provide an alternative. So in the future you will be able to use:

#[cfg(clippy)]

Transitioning

Should you have instances of feature = "cargo-clippy" in your code base, you will see a warning from the new Clippy lintclippy::deprecated_clippy_cfg_attr. This lint can automatically fix your code. So if you should see this lint triggering, just run:

cargo clippy --fix -- -Aclippy::all -Wclippy::deprecated_clippy_cfg_attr

This will fix all instances in your code.

In addition, check your .cargo/config file for:

[target.'cfg(feature = "cargo-clippy")']
rustflags = ["-Aclippy::..."]

If you have this config, you will have to update it yourself, by either changing it to cfg(clippy) or taking this opportunity to transition to setting lint levels in Cargo.toml directly.

Motivation for Deprecation

Currently, there's a call for testing, in order to stabilize checking conditional compilation at compile time, aka cargo check -Zcheck-cfg. If we were to keep the feature = "cargo-clippy" config, users would start seeing a lot of warnings on their feature = "cargo-clippy"conditions. To work around this, they would either need to allow the lint or have to add a dummy feature to their Cargo.toml in order to silence those warnings:

[features]
cargo-clippy = []

We didn't think this would be user friendly, and decided that instead we want to deprecate the implicit feature = "cargo-clippy" config and replace it with theclippy config.

1. It's likely that you didn't even know that Clippy implicitly sets this config (which was not a Cargo feature). This is intentional, as we stopped advertising and documenting this a long time ago. ↩

Continue Reading…

Updated baseline standards for Windows targets

The minimum requirements for Tier 1 toolchains targeting Windows will increase with the 1.78 release (scheduled for May 02, 2024). Windows 10 will now be the minimum supported version for the *-pc-windows-* targets. These requirements apply both to the Rust toolchain itself and to binaries produced by Rust.

Two new targets have been added with Windows 7 as their baseline: x86_64-win7-windows-msvc and i686-win7-windows-msvc. They are starting as Tier 3 targets, meaning that the Rust codebase has support for them but we don't build or test them automatically. Once these targets reach Tier 2 status, they will be available to use via rustup.

Affected targets

• x86_64-pc-windows-msvc
• i686-pc-windows-msvc
• x86_64-pc-windows-gnu
• i686-pc-windows-gnu
• x86_64-pc-windows-gnullvm
• i686-pc-windows-gnullvm

Why are the requirements being changed?

Prior to now, Rust had Tier 1 support for Windows 7, 8, and 8.1 but these targets no longer meet our requirements. In particular, these targets could no longer be tested in CI which is required by the Target Tier Policy and are not supported by their vendor.

Continue Reading…

Rust participates in Google Summer of Code 2024

We're writing this blog post to announce that the Rust Project will be participating in Google Summer of Code (GSoC) 2024. If you're not eligible or interested in participating in GSoC, then most of this post likely isn't relevant to you; if you are, this should contain some useful information and links.

Google Summer of Code (GSoC) is an annual global program organized by Google that aims to bring new contributors to the world of open-source. The program pairs organizations (such as the Rust Project) with contributors (usually students), with the goal of helping the participants make meaningful open-source contributions under the guidance of experienced mentors.

As of today, the organizations that have been accepted into the program have been announced by Google. The GSoC applicants now have several weeks to send project proposals to organizations that appeal to them. If their project proposal is accepted, they will embark on a 12-week journey during which they will try to complete their proposed project under the guidance of an assigned mentor.

We have prepared a list of project ideas that can serve as inspiration for potential GSoC contributors that would like to send a project proposal to the Rust organization. However, applicants can also come up with their own project ideas. You can discuss project ideas or try to find mentors in the #gsoc Zulip stream. We have also prepared a proposal guide that should help you with preparing your project proposals.

You can start discussing the project ideas with Rust Project maintainers immediately. The project proposal application period starts on March 18, 2024, and ends on April 2, 2024 at 18:00 UTC. Take note of that deadline, as there will be no extensions!

If you are interested in contributing to the Rust Project, we encourage you to check out our project idea list and send us a GSoC project proposal! Of course, you are also free to discuss these projects and/or try to move them forward even if you do not intend to (or cannot) participate in GSoC. We welcome all contributors to Rust, as there is always enough work to do.

This is the first time that the Rust Project is participating in GSoC, so we are quite excited about it. We hope that participants in the program can improve their skills, but also would love for this to bring new contributors to the Project and increase the awareness of Rust in general. We will publish another blog post later this year with more information about our participation in the program.

Continue Reading…

2023 Annual Rust Survey Results

Hello, Rustaceans!

The Rust Survey Team is excited to share the results of our 2023 survey on the Rust Programming language, conducted between December 18, 2023 and January 15, 2024. As in previous years, the 2023 State of Rust Survey was focused on gathering insights and feedback from Rust users, and all those who are interested in the future of Rust more generally.

This eighth edition of the survey surfaced new insights and learning opportunities straight from the global Rust language community, which we will summarize below. In addition to this blog post, this year we have also prepared a report containing charts with aggregated results of all questions in the survey. Based on feedback from recent years, we have also tried to provide more comprehensive and interactive charts in this summary blog post. Let us know what you think!

Our sincerest thanks to every community member who took the time to express their opinions and experiences with Rust over the past year. Your participation will help us make Rust better for everyone.

There's a lot of data to go through, so strap in and enjoy!

Participation

Survey

Started

Completed

Completion rate

Views

2022

11 482

9 433

81.3%

25 581

2023

11 950

9 710

82.2%

16 028

As shown above, in 2023, we have received 37% fewer survey views in vs 2022, but saw a slight uptick in starts and completions. There are many reasons why this could have been the case, but it’s possible that because we released the 2022 analysis blog so late last year, the survey was fresh in many Rustaceans’ minds. This might have prompted fewer people to feel the need to open the most recent survey. Therefore, we find it doubly impressive that there were more starts and completions in 2023, despite the lower overall view count.

Community

This year, we have relied on automated translations of the survey, and we have asked volunteers to review them. We thank the hardworking volunteers who reviewed these automated survey translations, ultimately allowing us to offer the survey in seven languages: English, Simplified Chinese, French, German, Japanese, Russian, and Spanish. We decided not to publish the survey in languages without a translation review volunteer, meaning we could not issue the survey in Portuguese, Ukrainian, Traditional Chinese, or Korean.

The Rust Survey team understands that there were some issues with several of these translated versions, and we apologize for any difficulty this has caused. We are always looking for ways to improve going forward and are in the process of discussing improvements to this part of the survey creation process for next year.

We saw a 3pp increase in respondents taking this year’s survey in English – 80% in 2023 and 77% in 2022. Across all other languages, we saw only minor variations – all of which are likely due to us offering fewer languages overall this year due to having fewer volunteers.

Rust user respondents were asked which country they live in. The top 10 countries represented were, in order: United States (22%), Germany (12%), China (6%), United Kingdom (6%), France (6%), Canada (3%), Russia (3%), Netherlands (3%), Japan (3%), and Poland (3%) . We were interested to see a small reduction in participants taking the survey in the United States in 2023 (down 3pp from the 2022 edition) which is a positive indication of the growing global nature of our community! You can try to find your country in the chart below:

[PNG] [SVG]

Once again, the majority of our respondents reported being most comfortable communicating on technical topics in English at 92.7% — a slight difference from 93% in 2022. Again, Chinese was the second-highest choice for preferred language for technical communication at 6.1% (7% in 2022).

[PNG] [SVG]

We also asked whether respondents consider themselves members of a marginalized community. Out of those who answered, 76% selected no, 14% selected yes, and 10% preferred not to say.

We have asked the group that selected “yes” which specific groups they identified as being a member of. The majority of those who consider themselves a member of an underrepresented or marginalized group in technology identify as lesbian, gay, bisexual, or otherwise non-heterosexual. The second most selected option was neurodivergent at 41% followed by trans at 31.4%. Going forward, it will be important for us to track these figures over time to learn how our community changes and to identify the gaps we need to fill.

[PNG] [SVG]

As Rust continues to grow, we must acknowledge the diversity, equity, and inclusivity (DEI)-related gaps that exist in the Rust community. Sadly, Rust is not unique in this regard. For instance, only 20% of 2023 respondents to this representation question consider themselves a member of a racial or ethnic minority and only 26% identify as a woman. We would like to see more equitable figures in these and other categories. In 2023, the Rust Foundation formed a diversity, equity, and inclusion subcommittee on its Board of Directors whose members are aware of these results and are actively discussing ways that the Foundation might be able to better support underrepresented groups in Rust and help make our ecosystem more globally inclusive. One of the central goals of the Rust Foundation board's subcommittee is to analyze information about our community to find out what gaps exist, so this information is a helpful place to start. This topic deserves much more depth than is possible here, but readers can expect more on the subject in the future.

Rust usage

In 2023, we saw a slight jump in the number of respondents that self-identify as a Rust user, from 91% in 2022 to 93% in 2023.

[PNG] [SVG]

Of those who used Rust in 2023, 49% did so on a daily (or nearly daily) basis — a small increase of 2pp from the previous year.

[PNG] [SVG]

31% of those who did not identify as Rust users cited the perception of difficulty as the primary reason for not having used it, with 67% reporting that they simply haven’t had the chance to prioritize learning Rust yet, which was once again the most common reason.

[PNG] [SVG] [Wordcloud of open answers]

Of the former Rust users who participated in the 2023 survey, 46% cited factors outside their control (a decrease of 1pp from 2022), 31% stopped using Rust due to preferring another language (an increase of 9pp from 2022), and 24% cited difficulty as the primary reason for giving up (a decrease of 6pp from 2022).

[PNG] [SVG] [Wordcloud of open answers]

Rust expertise has generally increased amongst our respondents over the past year! 23% can write (only) simple programs in Rust (a decrease of 6pp from 2022), 28% can write production-ready code (an increase of 1pp), and 47% consider themselves productive using Rust — up from 42% in 2022. While the survey is just one tool to measure the changes in Rust expertise overall, these numbers are heartening as they represent knowledge growth for many Rustaceans returning to the survey year over year.

[PNG] [SVG]

In terms of operating systems used by Rustaceans, the situation is very similar to the results from 2022, with Linux being the most popular choice of Rust users, followed by macOS and Windows, which have a very similar share of usage.

[PNG] [SVG] [Wordcloud of open answers]

Rust programmers target a diverse set of platforms with their Rust programs, even though the most popular target by far is still a Linux machine. We can see a slight uptick in users targeting WebAssembly, embedded and mobile platforms, which speaks to the versatility of Rust.

[PNG] [SVG] [Wordcloud of open answers]

We cannot of course forget the favourite topic of many programmers: which IDE (developer environment) do they use. Visual Studio Code still seems to be the most popular option, with RustRover (which was released last year) also gaining some traction.

[PNG] [SVG] [Wordcloud of open answers]

You can also take a look at the linked wordcloud that summarizes open answers to this question (the "Other" category), to see what other editors are also popular.

Rust at Work

We were excited to see a continued upward year-over-year trend of Rust usage at work. 34% of 2023 survey respondents use Rust in the majority of their coding at work — an increase of 5pp from 2022. Of this group, 39% work for organizations that make non-trivial use of Rust.

[PNG] [SVG]

Once again, the top reason employers of our survey respondents invested in Rust was the ability to build relatively correct and bug-free software at 86% — a 4pp increase from 2022 responses. The second most popular reason was Rust’s performance characteristics at 83%.

[PNG] [SVG]

We were also pleased to see an increase in the number of people who reported that Rust helped their company achieve its goals at 79% — an increase of 7pp from 2022. 77% of respondents reported that their organization is likely to use Rust again in the future — an increase of 3pp from the previous year. Interestingly, we saw a decrease in the number of people who reported that using Rust has been challenging for their organization to use: 34% in 2023 and 39% in 2022. We also saw an increase of respondents reporting that Rust has been worth the cost of adoption: 64% in 2023 and 60% in 2022.

[PNG] [SVG]

There are many factors playing into this, but the growing awareness around Rust has likely resulted in the proliferation of resources, allowing new teams using Rust to be better supported.

In terms of technology domains, it seems that Rust is especially popular for creating server backends, web and networking services and cloud technologies.

[PNG] [SVG] [Wordcloud of open answers]

You can scroll the chart to the right to see more domains. Note that the Database implementation and Computer Games domains were not offered as closed answers in the 2022 survey (they were merely submitted as open answers), which explains the large jump.

It is exciting to see the continued growth of professional Rust usage and the confidence so many users feel in its performance, control, security and safety, enjoyability, and more!

Challenges

As always, one of the main goals of the State of Rust survey is to shed light on challenges, concerns, and priorities on Rustaceans’ minds over the past year.

Of those respondents who shared their main worries for the future of Rust (9,374), the majority were concerned about Rust becoming too complex at 43% — a 5pp increase from 2022. 42% of respondents were concerned about a low level of Rust usage in the tech industry. 32% of respondents in 2023 were most concerned about Rust developers and maintainers not being properly supported — a 6pp increase from 2022.

We saw a notable decrease in respondents who were not at all concerned about the future of Rust, 18% in 2023 and 30% in 2022.

Thank you to all participants for your candid feedback which will go a long way toward improving Rust for everyone.

[PNG] [SVG] [Wordcloud of open answers]

Closed answers marked with N/A were not present in the previous (2022) version of the survey.

In terms of features that Rust users want to be implemented, stabilized or improved, the most desired improvements are in the areas of traits (trait aliases, associated type defaults, etc.), const execution (generic const expressions, const trait methods, etc.) and async (async closures, coroutines).

[PNG] [SVG] [Wordcloud of open answers]

It is interesting that 20% of respondents answered that they wish Rust to slow down the development of new features, which likely goes hand in hand with the previously mentioned worry that Rust becomes too complex.

The areas of Rust that Rustaceans seem to struggle with the most seem to be asynchronous Rust, the traits and generics system and also the borrow checker.

[PNG] [SVG] [Wordcloud of open answers]

Respondents of the survey want the Rust maintainers to mainly prioritize fixing compiler bugs (68%), improving the runtime performance of Rust programs (57%) and also improving compile times (45%).

[PNG] [SVG]

Same as in recent years, respondents noted that compilation time is one of the most important areas that should be improved. However, it is interesting to note that respondents also seem to consider runtime performance to be more important than compile times.

Looking ahead

Each year, the results of the State of Rust survey help reveal the areas that need improvement in many areas across the Rust Project and ecosystem, as well as the aspects that are working well for our community.

We are aware that the survey has contained some confusing questions, and we will try to improve upon that in the next year's survey. If you have any suggestions for the Rust Annual survey, please let us know!

We are immensely grateful to those who participated in the 2023 State of Rust Survey and facilitated its creation. While there are always challenges associated with developing and maintaining a programming language, this year we were pleased to see a high level of survey participation and candid feedback that will truly help us make Rust work better for everyone.

If you’d like to dig into more details, we recommend you to browse through the full survey report.

Continue Reading…

Announcing Rust 1.76.0

The Rust team is happy to announce a new version of Rust, 1.76.0. Rust is a programming language empowering everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, you can get 1.76.0 with:

rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website, and check out the detailed release notes for 1.76.0.

If you'd like to help us out by testing future releases, you might consider updating locally to use the beta channel (rustup default beta) or the nightly channel (rustup default nightly). Please report any bugs you might come across!

What's in 1.76.0 stable

This release is relatively minor, but as always, even incremental improvements lead to a greater whole. A few of those changes are highlighted in this post, and others may yet fill more niche needs.

ABI compatibility updates

A new ABI Compatibility section in the function pointer documentation describes what it means for function signatures to be ABI-compatible. A large part of that is the compatibility of argument types and return types, with a list of those that are currently considered compatible in Rust. For the most part, this documentation is not adding any new guarantees, only describing the existing state of compatibility.

The one new addition is that it is now guaranteed that char and u32 are ABI compatible. They have always had the same size and alignment, but now they are considered equivalent even in function call ABI, consistent with the documentation above.

Type names from references

For debugging purposes, any::type_name::() has been available since Rust 1.38 to return a string description of the type T, but that requires an explicit type parameter. It is not always easy to specify that type, especially for unnameable types like closures or for opaque return types. The new type_name_of_val(&T) offers a way to get a descriptive name from any reference to a type.

fn get_iter() -> impl Iterator<Item = i32> {
    [1, 2, 3].into_iter()
}

fn main() {
    let iter = get_iter();
    let iter_name = std::any::type_name_of_val(&iter);
    let sum: i32 = iter.sum();
    println!("The sum of the `{iter_name}` is {sum}.");
}

This currently prints:

The sum of the `core::array::iter::IntoIter<i32, 3>` is 6.

Stabilized APIs

• Arc::unwrap_or_clone
• Rc::unwrap_or_clone
• Result::inspect
• Result::inspect_err
• Option::inspect
• type_name_of_val
• std::hash::{DefaultHasher, RandomState}These were previously available only through std::collections::hash_map.
• ptr::{from_ref, from_mut}
• ptr::addr_eq

Other changes

Check out everything that changed in Rust, Cargo, and Clippy.

Contributors to 1.76.0

Many people came together to create Rust 1.76.0. We couldn't have done it without all of you. Thanks!

Continue Reading…

crates.io: API status code changes

Cargo and crates.io were developed in the rush leading up to the Rust 1.0 release to fill the needs for a tool to manage dependencies and a registry that people could use to share code. This rapid work resulted in these tools being connected with an API that initially didn't return the correct HTTP response status codes. After the Rust 1.0 release, Rust's stability guarantees around backward compatibility made this non-trivial to fix, as we wanted older versions of Cargo to continue working with the current crates.io API.

When an old version of Cargo receives a non-"200 OK" response, it displays the raw JSON body like this:

error: failed to get a 200 OK response, got 400
headers:
    HTTP/1.1 400 Bad Request
    Content-Type: application/json; charset=utf-8
    Content-Length: 171

body:
{"errors":[{"detail":"missing or empty metadata fields: description, license. Please see https://doc.rust-lang.org/cargo/reference/manifest.html for how to upload metadata"}]}

This was improved in pull request #6771, which was released in Cargo 1.34 (mid-2019). Since then, Cargo has supported receiving 4xx and 5xx status codes too and extracts the error message from the JSON response, if available.

On 2024-03-04 we will switch the API from returning "200 OK" status codes for errors to the new 4xx/5xx behavior. Cargo 1.33 and below will keep working after this change, but will show the raw JSON body instead of a nicely formatted error message. We feel confident that this degraded error message display will not affect very many users. According to the crates.io request logs only very few requests are made by Cargo 1.33 and older versions.

This is the list of API endpoints that will be affected by this change:

• GET /api/v1/crates
• PUT /api/v1/crates/new
• PUT /api/v1/crates/:crate/:version/yank
• DELETE /api/v1/crates/:crate/:version/unyank
• GET /api/v1/crates/:crate/owners
• PUT /api/v1/crates/:crate/owners
• DELETE /api/v1/crates/:crate/owners

All other endpoints have already been using regular HTTP status codes for some time.

If you are still using Cargo 1.33 or older, we recommend upgrading to a newer version to get the improved error messages and all the other nice things that the Cargo team has built since then.

Continue Reading…

Announcing Rust 1.75.0

The Rust team is happy to announce a new version of Rust, 1.75.0. Rust is a programming language empowering everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, you can get 1.75.0 with:

rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website, and check out the detailed release notes for 1.75.0.

If you'd like to help us out by testing future releases, you might consider updating locally to use the beta channel (rustup default beta) or the nightly channel (rustup default nightly). Please report any bugs you might come across!

What's in 1.75.0 stable

async fn and return-position impl Trait in traits

As announcedlast week, Rust 1.75 supports use of async fn and -> impl Trait in traits. However, this initial release comes with some limitations that are described in the announcement post.

It's expected that these limitations will be lifted in future releases.

Pointer byte offset APIs

Raw pointers (*const T and *mut T) used to primarily support operations operating in units of T. For example, <*const T>::add(1) would addsize_of::<T>() bytes to the pointer's address. In some cases, working with byte offsets is more convenient, and these new APIs avoid requiring callers to cast to *const u8/*mut u8 first.

• pointer::byte_add
• pointer::byte_offset
• pointer::byte_offset_from
• pointer::byte_sub
• pointer::wrapping_byte_add
• pointer::wrapping_byte_offset
• pointer::wrapping_byte_sub

Code layout optimizations for rustc

The Rust compiler continues to get faster, with this release including the application ofBOLT to our binary releases, bringing a 2% mean wall time improvements on our benchmarks. This tool optimizes the layout of the librustc_driver.so library containing most of the rustc code, allowing for better cache utilization.

We are also now building rustc with -Ccodegen-units=1, which provides more opportunity for optimizations in LLVM. This optimization brought a separate 1.5% wall time mean win to our benchmarks.

In this release these optimizations are limited to x86_64-unknown-linux-gnucompilers, but we expect to expand that over time to include more platforms.

Stabilized APIs

• Atomic*::from_ptr
• FileTimes
• FileTimesExt
• File::set_modified
• File::set_times
• IpAddr::to_canonical
• Ipv6Addr::to_canonical
• Option::as_slice
• Option::as_mut_slice
• pointer::byte_add
• pointer::byte_offset
• pointer::byte_offset_from
• pointer::byte_sub
• pointer::wrapping_byte_add
• pointer::wrapping_byte_offset
• pointer::wrapping_byte_sub

These APIs are now stable in const contexts:

• Ipv6Addr::to_ipv4_mapped
• MaybeUninit::assume_init_read
• MaybeUninit::zeroed
• mem::discriminant
• mem::zeroed

Other changes

Check out everything that changed in Rust, Cargo, and Clippy.

Contributors to 1.75.0

Many people came together to create Rust 1.75.0. We couldn't have done it without all of you. Thanks!

Continue Reading…

Announcing `async fn` and return-position `impl Trait` in traits

The Rust Async Working Group is excited to announce major progress towards our goal of enabling the use of async fn in traits. Rust 1.75, which hits stable next week, will include support for both -> impl Trait notation and async fn in traits.

This is a big milestone, and we know many users will be itching to try these out in their own code. However, we are still missing some important features that many users need. Read on for recommendations on when and how to use the stabilized features.

What's stabilizing

Ever since the stabilization of RFC #1522 in Rust 1.26, Rust has allowed users to write impl Trait as the return type of functions (often called "RPIT"). This means that the function returns "some type that implements Trait". This is commonly used to return closures, iterators, and other types that are complex or impossible to write explicitly.

/// Given a list of players, return an iterator
/// over their names.
fn player_names(
    players: &[Player]
) -> impl Iterator<Item = &String> {
    players
        .iter()
        .map(|p| &p.name)
}

Starting in Rust 1.75, you can use return-position impl Trait in trait (RPITIT) definitions and in trait impls. For example, you could use this to write a trait method that returns an iterator:

trait Container {
    fn items(&self) -> impl Iterator<Item = Widget>;
}

impl Container for MyContainer {
    fn items(&self) -> impl Iterator<Item = Widget> {
        self.items.iter().cloned()
    }
}

So what does all of this have to do with async functions? Well, async functions are "just sugar" for functions that return -> impl Future. Since these are now permitted in traits, we also permit you to write traits that use async fn.

trait HttpService {
    async fn fetch(&self, url: Url) -> HtmlBody;
//  ^^^^^^^^ desugars to:
//  fn fetch(&self, url: Url) -> impl Future<Output = HtmlBody>;
}

Where the gaps lie

-> impl Trait in public traits

The use of -> impl Trait is still discouraged for general use in public traits and APIs for the reason that users can't put additional bounds on the return type. For example, there is no way to write this function in a way that is generic over the Container trait:

fn print_in_reverse(container: impl Container) {
    for item in container.items().rev() {
        // ERROR:                 ^^^
        // the trait `DoubleEndedIterator`
        // is not implemented for
        // `impl Iterator<Item = Widget>`
        eprintln!("{item}");
    }
}

Even though some implementations might return an iterator that implements DoubleEndedIterator, there is no way for generic code to take advantage of this without defining another trait. In the future we plan to add a solution for this. For now, -> impl Trait is best used in internal traits or when you're confident your users won't need additional bounds. Otherwise you should consider using an associated type.1

async fn in public traits

Since async fn desugars to -> impl Future, the same limitations apply. In fact, if you use bare async fn in a public trait today, you'll see a warning.

warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified
 --> src/lib.rs:7:5
  |
7 |     async fn fetch(&self, url: Url) -> HtmlBody;
  |     ^^^^^
  |
help: you can desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change
  |
7 -     async fn fetch(&self, url: Url) -> HtmlBody;
7 +     fn fetch(&self, url: Url) -> impl std::future::Future<Output = HtmlBody> + Send;
  |

Of particular interest to users of async are Send bounds on the returned future. Since users cannot add bounds later, the error message is saying that you as a trait author need to make a choice: Do you want your trait to work with multithreaded, work-stealing executors?

Thankfully, we have a solution that allows using async fn in public traits today! We recommend using the trait_variant::make proc macro to let your users choose. This proc macro is part of the trait-variant crate, published by the rust-lang org. Add it to your project with cargo add trait-variant, then use it like so:

#[trait_variant::make(HttpService: Send)]
pub trait LocalHttpService {
    async fn fetch(&self, url: Url) -> HtmlBody;
}

This creates two versions of your trait: LocalHttpService for single-threaded executors and HttpService for multithreaded work-stealing executors. Since we expect the latter to be used more commonly, it has the shorter name in this example. It has additional Send bounds:

pub trait HttpService: Send {
    fn fetch(
        &self,
        url: Url,
    ) -> impl Future<Output = HtmlBody> + Send;
}

This macro works for async because impl Future rarely requires additional bounds other than Send, so we can set our users up for success. See the FAQ below for an example of where this is needed.

Dynamic dispatch

Traits that use -> impl Trait and async fn are not object-safe, which means they lack support for dynamic dispatch. We plan to provide utilities that enable dynamic dispatch in an upcoming version of the trait-variant crate.

How we hope to improve in the future

In the future we would like to allow users to add their own bounds to impl Trait return types, which would make them more generally useful. It would also enable more advanced uses of async fn. The syntax might look something like this:

trait HttpService = LocalHttpService<fetch(): Send> + Send;

Since these aliases won't require any support on the part of the trait author, it will technically make the Send variants of async traits unnecessary. However, those variants will still be a nice convenience for users, so we expect that most crates will continue to provide them.

Of course, the goals of the Async Working Group don't stop with async fn in traits. We want to continue building features on top of it that enable more reliable and sophisticated use of async Rust, and we intend to publish a more extensive roadmap in the new year.

Frequently asked questions

Is it okay to use -> impl Trait in traits?

For private traits you can use -> impl Trait freely. For public traits, it's best to avoid them for now unless you can anticipate all the bounds your users might want (in which case you can use #[trait_variant::make], as we do for async). We expect to lift this restriction in the future.

Should I still use the #[async_trait] macro?

There are a couple of reasons you might need to continue using async-trait:

• You want to support Rust versions older than 1.75.
• You want dynamic dispatch.

As stated above, we hope to enable dynamic dispatch in a future version of the trait-variant crate.

Is it okay to use async fn in traits? What are the limitations?

Assuming you don't need to use #[async_trait] for one of the reasons stated above, it's totally fine to use regular async fn in traits. Just remember to use #[trait_variant::make] if you want to support multithreaded runtimes.

The biggest limitation is that a type must always decide if it implements the Send or non-Send version of a trait. It cannot implement the Send version conditionally on one of its generics. This can come up in the middleware pattern, for example, RequestLimitingService<T> that is HttpService if T: HttpService.

Why do I need #[trait_variant::make] and Send bounds?

In simple cases you may find that your trait appears to work fine with a multithreaded executor. There are some patterns that just won't work, however. Consider the following:

fn spawn_task(service: impl HttpService + 'static) {
    tokio::spawn(async move {
        let url = Url::from("https://rust-lang.org");
        let _body = service.fetch(url).await;
    });
}

Without Send bounds on our trait, this would fail to compile with the error: "future cannot be sent between threads safely". By creating a variant of your trait with Send bounds, you avoid sending your users into this trap.

Note that you won't see a warning if your trait is not public, because if you run into this problem you can always add the Send bounds yourself later.

For a more thorough explanation of the problem, see this blog post.2

Can I mix async fn and impl trait?

Yes, you can freely move between the async fn and -> impl Future spelling in your traits and impls. This is true even when one form has a Send bound.3 This makes the traits created by trait_variant nicer to use.

trait HttpService: Send {
    fn fetch(&self, url: Url)
    -> impl Future<Output = HtmlBody> + Send;
}

impl HttpService for MyService {
    async fn fetch(&self, url: Url) -> HtmlBody {
        // This works, as long as `do_fetch(): Send`!
        self.client.do_fetch(url).await.into_body()
    }
}

Why don't these signatures use impl Future + '_?

For -> impl Trait in traits we adopted the 2024 Capture Rules early. This means that the + '_ you often see today is unnecessary in traits, because the return type is already assumed to capture input lifetimes. In the 2024 edition this rule will apply to all function signatures. See the linked RFC for more.

Why am I getting a "refine" warning when I implement a trait with -> impl Trait?

If your impl signature includes more detailed information than the trait itself, you'll get a warning:

pub trait Foo {
    fn foo(self) -> impl Debug;
}

impl Foo for u32 {
    fn foo(self) -> String {
//                  ^^^^^^
//  warning: impl trait in impl method signature does not match trait method signature
        self.to_string()
    }
}

The reason is that you may be leaking more details of your implementation than you meant to. For instance, should the following code compile?

fn main() {
    // Did the implementer mean to allow
    // use of `Display`, or only `Debug` as
    // the trait says?
    println!("{}", 32.foo());
}

Thanks to refined trait implementations it does compile, but the compiler asks you to confirm your intent to refine the trait interface with #[allow(refining_impl_trait)] on the impl.

Conclusion

The Async Working Group is excited to end 2023 by announcing the completion of our primary goal for the year! Thank you to everyone who helpfully participated in design, implementation, and stabilization discussions. Thanks also to the users of async Rust who have given great feedback over the years. We're looking forward to seeing what you build, and to delivering continued improvements in the years to come.

1. Note that associated types can only be used in cases where the type is nameable. This restriction will be lifted once impl_trait_in_assoc_type is stabilized. ↩
2. Note that in that blog post we originally said we would solve the Send bound problem before shipping async fn in traits, but we decided to cut that from the scope and ship the trait-variant crate instead. ↩
3. This works because of auto-trait leakage, which allows knowledge of auto traits to "leak" from an item whose signature does not specify them. ↩

Continue Reading…

Launching the 2023 State of Rust Survey

It’s time for the 2023 State of Rust Survey!

Since 2016, the Rust Project has collected valuable information and feedback from the Rust programming language community through our annual State of Rust Survey. This tool allows us to more deeply understand how the Rust Project is performing, how we can better serve the global Rust community, and who our community is composed of.

Like last year, the 2023 State of Rust Survey will likely take you between 10 and 25 minutes, and responses are anonymous. We will accept submissions until Monday, January 15th, 2024. Trends and key insights will be shared on blog.rust-lang.org as soon as possible in 2024.

We invite you to take this year’s survey whether you have just begun using Rust, you consider yourself an intermediate to advanced user, or you have not yet used Rust but intend to one day. Your responses will help us improve Rust over time by shedding light on gaps to fill in the community and development priorities, and more.

Once again, we are offering the State of Rust Survey in the following languages (if you speak multiple languages, please pick one). Language options are available on the main survey page:

• English
• Simplified Chinese
• French
• German
• Japanese
• Russian
• Spanish

Please help us spread the word by sharing the survey link via your social media networks, at meetups, with colleagues, and in any other community that makes sense to you.

This survey would not be possible without the time, resources, and attention of members of the Survey Working Group, the Rust Foundation, and other collaborators. Thank you!

If you have any questions, please see our frequently asked questions.

We appreciate your participation!

Click here to read a summary of last year's survey findings.

Continue Reading…

A Call for Proposals for the Rust 2024 Edition

The year 2024 is soon to be upon us, and as long-time Rust aficionados know, that means that a new Edition of Rust is on the horizon!

What is an Edition?

You may be aware that a new version of Rust is released every six weeks. New versions of the language can both add things as well as change things, but only in backwards-compatible ways, according to Rust's1.0 stability guarantee.

But does that mean that Rust can never make backwards-incompatible changes? Not quite! This is what an Edition is: Rust's mechanism for introducing backwards-incompatible changes in a backwards-compatible way. If that sounds like a contradiction, there are three key properties of Editions that preserve the stability guarantee:

1. Editions are opt-in; crates only receive breaking changes if its authors explicitly ask for them.
2. Crates that use older editions never get left behind; a crate written for the original Rust 2015 Edition is still supported by every Rust release, and can still make use of all the new goodies that accompany each new version, e.g. new library APIs, compiler optimizations, etc.
3. An Edition never splits the library ecosystem; crates using new Editions can depend on crates using old Editions (and vice-versa!), so nobody ever has to worry about Edition-related incompatibility.

In order to keep churn to a minimum, a new Edition of Rust is only released once every three years. We've had the 2015 Edition, the 2018 Edition, the 2021 Edition, and soon, the 2024 Edition. And we could use your help!

A call for proposals for the Rust 2024 Edition

We know how much you love Rust, but let's be honest, no language is perfect, and Rust is no exception. So if you've got ideas for how Rust could be better if only that pesky stability guarantee weren't around, now's the time to share! Also note that potential Edition-related changes aren't just limited to the language itself: we'll also consider changes to both Cargo and rustfmt as well.

Please keep in mind that the following criteria determine the sort of changes we're looking for:

1. A change must be possible to implement without violating the strict properties listed in the prior section. Specifically, the ability of crates to have cross-Edition dependencies imposes restrictions on changes that would take effect across crate boundaries, e.g. the signatures of public APIs. However, we will occasionally discover that an Edition-related changethat was once thought to be impossible actually turns out to be feasible, so hope is not lost if you're not sure if your idea meets this standard; propose it just to be safe!
2. We strive to ensure that nearly all Edition-related changes can be applied to existing codebases automatically (via tools like cargo fix), in order to make upgrading to a new Edition as painless as possible.
3. Even if an Edition could make any given change, that doesn't mean that it should. We're not looking for hugely-invasive changes or things that would fundamentally alter the character of the language. Please focus your proposals on things like fixing obvious bugs, changing annoying behavior, unblocking future feature development, and making the language easier and more consistent.

To spark your imagination, here's a real-world example. In the 2015 and 2018 Editions, iterating over a fixed-length array via [foo].into_iter()will yield references to the iterated elements; this is is surprising because, on other types, calling .into_iter() produces an iteratorthat yields owned values rather than references. This limitation existed because older versions of Rust lacked the ability to implement traits for all possible fixed-length arrays in a generic way. Once Rust finally became able to express this,all Editions at last gained the ability to iterate over owned values in fixed-length arrays; however, in the specific case of [foo].into_iter(), altering the existing behavior would have broken lots of code in the wild. Therefore, we used the 2021 Edition to fix this inconsistency for the specific case of [foo].into_iter(), allowing us to address this long-standing issue while preserving Rust's stability guarantees.

How to contribute

Just like other changes to Rust, Edition-related proposals follow the RFC process, as documented in the Rust RFCs repository. Please follow the process documented there, and please consider publicizing a draft of your RFC to collect preliminary feedback before officially submitting it, in order to expedite the RFC process once you've filed it for real! (And in addition to the venues mentioned in the prior link, please feel free to announce your pre-RFC to our Zulip channel.)

Please file your RFCs as soon as possible! Our goal is to release the 2024 Edition in the second half of 2024, which means we would like to get everything implemented(not only the features themselves, but also all the Edition-related migration tooling) by the end of May, which means that RFCs should be accepted by the end of February. And since RFCs take time to discuss and consider, we strongly encourage you to have your RFC filed by the end of December, or the first week of January at the very latest.

We hope to have periodic updates on the ongoing development of the 2024 Edition. In the meantime, if you have any questions or if you would like to help us make the new Edition a reality, we invite you to come chat in the #edition channel in the Rust Zulip.

Continue Reading…

Cargo cache cleaning

Cargo has recently gained an unstable feature on the nightly channel (starting with nightly-2023-11-17) to perform automatic cleaning of cache content within Cargo's home directory. This post includes:

• A description of what this means (What is this feature?)
• A call for help (What to watch out for and Request for feedback)
• Implementation details (Design considerations and implementation details)
• An outline of what is planned for the future (Plan for the future)

In short, we are asking people who use the nightly channel to enable this feature and report any issues you encounter on the Cargo issue tracker. To enable it, place the following in your Cargo config file (typically located in ~/.cargo/config.toml or %USERPROFILE%\.cargo\config.toml for Windows):

[unstable]
gc = true

Or set the CARGO_UNSTABLE_GC=true environment variable or use the -Zgc CLI flag to turn it on for individual commands.

We'd particularly like people who use unusual filesystems or environments to give it a try, since there are some parts of the implementation which are sensitive and need battle testing before we turn it on for everyone.

What is this feature?

Cargo keeps a variety of cached data within the Cargo home directory. This cache can grow unbounded and can get quite large (easily reaching many gigabytes). Community members have developed tools to manage this cache, such as cargo-cache, but cargo itself never exposed any ability to manage it.

This cache includes:

• Registry index data, such as package dependency metadata from crates.io.
• Compressed .crate files downloaded from a registry.
• The uncompressed contents of those .crate files, which rustc uses to read the source and compile dependencies.
• Clones of git repositories used by git dependencies.

The new garbage collection ("GC") feature adds tracking of this cache data so that cargo can automatically or manually remove unused files. It keeps an SQLite database which tracks the last time the various cache elements have been used. Every time you run a cargo command that reads or writes any of this cache data, it will update the database with a timestamp of when that data was last used.

What isn't yet included is cleaning of target directories, see Plan for the future.

Automatic cleaning

When you run cargo, once a day it will inspect the last-use cache tracker, and determine if any cache elements have not been used in a while. If they have not, then they will be automatically deleted. This happens with most commands that would normally perform significant work, like cargo build or cargo fetch.

The default is to delete data that can be locally recreated if it hasn't been used for 1 month, and to delete data that has to be re-downloaded after 3 months.

Automatic deletion is disabled if cargo is offline such as with --offline or --frozen to avoid deleting artifacts that may need to be used if you are offline for a long period of time.

The initial implementation has exposed a variety of configuration knobs to control how automatic cleaning works. However, it is unlikely we will expose too many low-level details when it is stabilized, so this may change in the future (see issue #13061). See the Automatic garbage collection section for more details on this configuration.

Manual cleaning

If you want to manually delete data from the cache, several options have been added under the cargo clean gc subcommand. This subcommand can be used to perform the normal automatic daily cleaning, or to specify different options on which data to remove. There are several options for specifying the age of data to delete (such as --max-download-age=3days) or specifying the maximum size of the cache (such as --max-download-size=1GiB). See the Manual garbage collection section or run cargo clean gc --help for more details on which options are supported.

This CLI design is only preliminary, and we are looking at determining what the final design will look like when it is stabilized, see issue #13060.

What to watch out for

After enabling the gc feature, just go about your normal business of using cargo. You should be able to observe the SQLite database stored in your cargo home directory at ~/.cargo/.global-cache.

After the first time you use cargo, it will populate the database tracking all the data that already exists in your cargo home directory. Then, after 1 month, cargo should start deleting old data, and after 3 months will delete even more data.

The end result is that after that period of time you should start to notice the home directory using less space overall.

You can also try out the cargo clean gc command and explore some of its options if you want to try to manually delete some data.

If you run into problems, you can disable the gc feature and cargo should return to its previous behavior. Please let us know on the issue tracker if this happens.

Request for feedback

We'd like to hear from you about your experience using this feature. Some of the things we are interested in are:

• Have you run into any bugs, errors, issues, or confusing problems? Please file an issue over at https://github.com/rust-lang/cargo/issues/.
• The first time that you use cargo with GC enabled, is there an unreasonably long delay? Cargo may need to scan your existing cache data once to detect what already exists from previous versions.
• Do you notice unreasonable delays when it performs automatic cleaning once a day?
• Do you have use cases where you need to do cleaning based on the size of the cache? If so, please share them at #13062.
• If you think you would make use of manually deleting cache data, what are your use cases for doing that? Sharing them on #13060 about the CLI interface might help guide us on the overall design.
• Does the default of deleting 3 month old data seem like a good balance for your use cases?

Or if you would prefer to share your experiences on Zulip, head over to the #t-cargo stream.

Design considerations and implementation details

(These sections are only for the intently curious among you.)

The implementation of this feature had to consider several constraints to try to ensure that it works in nearly all environments, and doesn't introduce a negative experience for users.

Performance

One big focus was to make sure that the performance of each invocation of cargo is not significantly impacted. Cargo needs to potentially save a large chunk of data every time it runs. The performance impact will heavily depend on the number of dependencies and your filesystem. Preliminary testing shows the impact can be anywhere from 0 to about 50ms.

In order to minimize the performance impact of actually deleting files, the automatic GC runs only once a day. This is intended to balance keeping the cache clean without impacting the performance of daily use.

Locking

Another big focus is dealing with cache locking. Previously, cargo had a single lock on the package cache, which cargo would hold while downloading registry data and performing dependency resolution. When cargo is actually running rustc, it previously did not hold a lock under the assumption that existing cache data will not be modified.

However, now that cargo can modify or delete existing cache data, it needs to be careful to coordinate with anything that might be reading from the cache, such as if multiple cargo commands are run simultaneously. To handle this, cargo now has two separate locks, which are used together to provide three separate locking states. There is a shared read lock, which allows multiple builds to run in parallel and read from the cache. There is a write lock held while downloading registry data, which is independent of the read lock which allows concurrent builds to still run while new packages are downloaded. The third state is a write lock that prevents either of the two previous locks from being held, and ensures exclusive access while cleaning the cache.

Versions of cargo before 1.75 don't know about the exclusive write lock. We are hoping that in practice it will be rare to concurrently run old and new cargo versions, and that it is unlikely that the automatic GC will need to delete data that is concurrently in use by an older version.

Error handling and filesystems

Because we do not want problems with GC from disrupting users, the implementation silently skips the GC if it is unable to acquire an exclusive lock on the package cache. Similarly, when cargo saves the timestamp data on every command, it will silently ignore errors if it is unable to open the database, such as if it is on a read-only filesystem, or it is unable to acquire a write lock. This may result in the last-use timestamps becoming stale, but hopefully this should not impact most usage scenarios. For locking, we are paying special attention to scenarios such as Docker container mounts and network filesystems with questionable locking support.

Backwards compatibility

Since the cache is used by any version of cargo, we have to pay close attention to forwards and backwards compatibility. We benefit from SQLite's particularly stable on-disk data format which has been stable since 2004. Cargo has support to do schema migrations within the database that stay backwards compatible.

Plan for the future

A major aspect of this endeavor is to gain experience with using SQLite in a wide variety of environments, with a plan to extend its usage in several other parts of cargo.

Registry index metadata

One place where we are looking to introduce SQLite is for the registry index cache. When cargo downloads registry index data, it stores it in a custom-designed binary file format to improve lookup performance. However, this index cache uses many small files, which may not perform well on some filesystems.

Additionally, the index cache grows without bound. Currently the automatic cache cleaning will only delete an entire index cache if the index itself hasn't been used, which is rarely the case for crates.io. We may also need to consider finer-grained timestamp tracking or some mechanism to periodically purge this data.

Target directory change tracking and cleaning

Another place we are looking to introduce SQLite is for managing the target directory. In cargo's target directory, cargo keeps track of information about each crate that has been built with what is called a fingerprint. These fingerprints help cargo know if it needs to recompile something. Each artifact is tracked with a set of 4 files, using a mixture of custom formats.

We are looking to replace this system with SQLite which will hopefully bring about several improvements. A major focus will be to provide cleaning of stale data in the target directory, which tends to use substantial amount of disk space. Additionally we are looking to implement other improvements, such as more accurate fingerprint tracking, provide information about why cargo thinks something needed to be recompiled, and to hopefully improve performance. This will be important for the script feature, which uses a global cache for build artifacts, and the future implementation of a globally-shared build cache.

Continue Reading…

A Call for Proposals for the Rust 2024 Edition

The year 2024 is soon to be upon us, and as long-time Rust aficionados know, that means that a new Edition of Rust is on the horizon!

What is an Edition?

You may be aware that a new version of Rust is released every six weeks. New versions of the language can both add things as well as change things, but only in backwards-compatible ways, according to Rust's1.0 stability guarantee.

But does that mean that Rust can never make backwards-incompatible changes? Not quite! This is what an Edition is: Rust's mechanism for introducing backwards-incompatible changes in a backwards-compatible way. If that sounds like a contradiction, there are three key properties of Editions that preserve the stability guarantee:

1. Editions are opt-in; crates only receive breaking changes if its authors explicitly ask for them.
2. Crates that use older editions never get left behind; a crate written for the original Rust 2015 Edition is still supported by every Rust release, and can still make use of all the new goodies that accompany each new version, e.g. new library APIs, compiler optimizations, etc.
3. An Edition never splits the library ecosystem; crates using new Editions can depend on crates using old Editions (and vice-versa!), so nobody ever has to worry about Edition-related incompatibility.

In order to keep churn to a minimum, a new Edition of Rust is only released once every three years. We've had the 2015 Edition, the 2018 Edition, the 2021 Edition, and soon, the 2024 Edition. And we could use your help!

A call for proposals for the Rust 2024 Edition

We know how much you love Rust, but let's be honest, no language is perfect, and Rust is no exception. So if you've got ideas for how Rust could be better if only that pesky stability guarantee weren't around, now's the time to share! Also note that potential Edition-related changes aren't just limited to the language itself: we'll also consider changes to both Cargo and rustfmt as well.

Please keep in mind that the following criteria determine the sort of changes we're looking for:

1. A change must be possible to implement without violating the strict properties listed in the prior section. Specifically, the ability of crates to have cross-Edition dependencies imposes restrictions on changes that would take effect across crate boundaries, e.g. the signatures of public APIs. However, we will occasionally discover that an Edition-related changethat was once thought to be impossible actually turns out to be feasible, so hope is not lost if you're not sure if your idea meets this standard; propose it just to be safe!
2. We strive to ensure that nearly all Edition-related changes can be applied to existing codebases automatically (via tools like cargo fix), in order to make upgrading to a new Edition as painless as possible.
3. Even if an Edition could make any given change, that doesn't mean that it should. We're not looking for hugely-invasive changes or things that would fundamentally alter the character of the language. Please focus your proposals on things like fixing obvious bugs, changing annoying behavior, unblocking future feature development, and making the language easier and more consistent.

To spark your imagination, here's a real-world example. In the 2015 and 2018 Editions, iterating over a fixed-length array via [foo].into_iter()will yield references to the iterated elements; this is is surprising because, on other types, calling .into_iter() produces an iteratorthat yields owned values rather than references. This limitation existed because older versions of Rust lacked the ability to implement traits for all possible fixed-length arrays in a generic way. Once Rust finally became able to express this,all Editions at last gained the ability to iterate over owned values in fixed-length arrays; however, in the specific case of [foo].into_iter(), altering the existing behavior would have broken lots of code in the wild. Therefore, we used the 2021 Edition to fix this inconsistency for the specific case of [foo].into_iter(), allowing us to address this long-standing issue while preserving Rust's stability guarantees.

How to contribute

Just like other changes to Rust, Edition-related proposals follow the RFC process, as documented in the Rust RFCs repository. Please follow the process documented there, and please consider publicizing a draft of your RFC to collect preliminary feedback before officially submitting it, in order to expedite the RFC process once you've filed it for real! (And in addition to the venues mentioned in the prior link, please feel free to announce your pre-RFC to our Zulip channel.)

Please file your RFCs as soon as possible! Our goal is to release the 2024 Edition in the second half of 2024, which means we would like to get everything implemented(not only the features themselves, but also all the Edition-related migration tooling) by the end of May, which means that RFCs should be accepted by the end of February. And since RFCs take time to discuss and consider, we strongly encourage you to have your RFC filed by the end of December, or the first week of January at the very latest.

We hope to have periodic updates on the ongoing development of the 2024 Edition. In the meantime, if you have any questions or if you would like to help us make the new Edition a reality, we invite you to come chat in the #edition channel in the Rust Zulip.

Continue Reading…

Announcing Rust 1.74.1

The Rust team has published a new point release of Rust, 1.74.1. Rust is a programming language that is empowering everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, getting Rust 1.74.1 is as easy as:

rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website.

What's in 1.74.1

1.74.1 resolves a few regressions introduced in 1.74.0:

• Resolved spurious STATUS_ACCESS_VIOLATIONs in LLVM
• Clarify guarantees for std::mem::discriminant
• Fix some subtyping-related regressions

Contributors to 1.74.1

Many people came together to create Rust 1.74.1. We couldn't have done it without all of you. Thanks!

Continue Reading…

Announcing Rust 1.74.0

The Rust team is happy to announce a new version of Rust, 1.74.0. Rust is a programming language empowering everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, you can get 1.74.0 with:

rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website, and check out the detailed release notes for 1.74.0 on GitHub.

If you'd like to help us out by testing future releases, you might consider updating locally to use the beta channel (rustup default beta) or the nightly channel (rustup default nightly). Please report any bugs you might come across!

What's in 1.74.0 stable

Lint configuration through Cargo

As proposed in RFC 3389, the Cargo.toml manifest now supports a [lints] table to configure the reporting level (forbid, deny, warn, allow) for lints from the compiler and other tools. So rather than setting RUSTFLAGS with -F/-D/-W/-A, which would affect the entire build, or using crate-level attributes like:

#![forbid(unsafe_code)]
#![deny(clippy::enum_glob_use)]

You can now write those in your package manifest for Cargo to handle:

[lints.rust]
unsafe_code = "forbid"

[lints.clippy]
enum_glob_use = "deny"

These can also be configured in a [workspace.lints] table, then inherited by[lints] workspace = true like many other workspace settings. Cargo will also track changes to these settings when deciding which crates need to be rebuilt.

For more information, see the lints and workspace.lints sections of the Cargo reference manual.

Cargo Registry Authentication

Two more related Cargo features are included in this release: credential providers and authenticated private registries.

Credential providers allow configuration of how Cargo gets credentials for a registry. Built-in providers are included for OS-specific secure secret storage on Linux, macOS, and Windows. Additionally, custom providers can be written to support arbitrary methods of storing or generating tokens. Using a secure credential provider reduces risk of registry tokens leaking.

Registries can now optionally require authentication for all operations, not just publishing. This enables private Cargo registries to offer more secure hosting of crates. Use of private registries requires the configuration of a credential provider.

For further information, see theCargo docs.

Projections in opaque return types

If you have ever received the error that a "return type cannot contain a projection or Self that references lifetimes from a parent scope," you may now rest easy! The compiler now allows mentioning Self and associated types in opaque return types, like async fn and -> impl Trait. This is the kind of feature that gets Rust closer to how you might just_expect_ it to work, even if you have no idea about jargon like "projection".

This functionality had an unstable feature gate because its implementation originally didn't properly deal with captured lifetimes, and once that was fixed it was given time to make sure it was sound. For more technical details, see the stabilization pull request, which describes the following examples that are all now allowed:

struct Wrapper<'a, T>(&'a T);

// Opaque return types that mention `Self`:
impl Wrapper<'_, ()> {
    async fn async_fn() -> Self { /* ... */ }
    fn impl_trait() -> impl Iterator<Item = Self> { /* ... */ }
}

trait Trait<'a> {
    type Assoc;
    fn new() -> Self::Assoc;
}
impl Trait<'_> for () {
    type Assoc = ();
    fn new() {}
}

// Opaque return types that mention an associated type:
impl<'a, T: Trait<'a>> Wrapper<'a, T> {
    async fn mk_assoc() -> T::Assoc { /* ... */ }
    fn a_few_assocs() -> impl Iterator<Item = T::Assoc> { /* ... */ }
}

Stabilized APIs

• core::num::Saturating
• impl Fromio::Stdout for std::process::Stdio
• impl Fromio::Stderr for std::process::Stdio
• impl From for std::process::Child{Stdin, Stdout, Stderr}
• impl From for std::process::Child{Stdin, Stdout, Stderr}
• std::ffi::OsString::from_encoded_bytes_unchecked
• std::ffi::OsString::into_encoded_bytes
• std::ffi::OsStr::from_encoded_bytes_unchecked
• std::ffi::OsStr::as_encoded_bytes
• std::io::Error::other
• impl TryFrom for u16
• impl<T: Clone, const N: usize> From<&[T; N]> for Vec
• impl<T: Clone, const N: usize> From<&mut [T; N]> for Vec
• impl<T, const N: usize> From<[T; N]> for Arc<[T]>
• impl<T, const N: usize> From<[T; N]> for Rc<[T]>

These APIs are now stable in const contexts:

• core::mem::transmute_copy
• str::is_ascii
• [u8]::is_ascii

Compatibility notes

• As previously announced, Rust 1.74 has increased its requirements on Apple platforms. The minimum versions are now:
  • macOS: 10.12 Sierra (First released 2016)
  • iOS: 10 (First released 2016)
  • tvOS: 10 (First released 2016)

Other changes

Check out everything that changed in Rust, Cargo, and Clippy.

Contributors to 1.74.0

Many people came together to create Rust 1.74.0. We couldn't have done it without all of you. Thanks!

Continue Reading…

Faster compilation with the parallel front-end in nightly

The Rust compiler's front-end can now use parallel execution to significantly reduce compile times. To try it, run the nightly compiler with the -Z threads=8 option. This feature is currently experimental, and we aim to ship it in the stable compiler in 2024.

Keep reading to learn why a parallel front-end is needed and how it works, or just skip ahead to the How to use itsection.

Compile times and parallelism

Rust compile times are a perennial concern. The Compiler Performance Working Grouphas continually improved compiler performance for several years. For example, in the first 10 months of 2023, there were mean reductions in compile time of13%, in peak memory use of15%, and in binary size of7%, as measured by our performance suite.

However, at this point the compiler has been heavily optimized and new improvements are hard to find. There is no low-hanging fruit remaining.

But there is one piece of large but high-hanging fruit: parallelism. Current Rust compiler users benefit from two kinds of parallelism, and the newly parallel front-end adds a third kind.

Existing interprocess parallelism

When you compile a Rust program, Cargo launches multiple rustc processes, compiling multiple crates in parallel. This works well. Try compiling a large Rust program with the -j1 flag to disable this parallelization and it will take a lot longer than normal.

You can visualise this parallelism if you build with Cargo's--timings flag, which produces a chart showing how the crates are compiled. The following image shows the timeline when building ripgrep on a machine with 28 virtual cores.

There are 60 horizontal lines, each one representing a distinct process. Their durations range from a fraction of a second to multiple seconds. Most of them are rustc, and the few orange ones are build scripts. The first twenty run all start at the same time. This is possible because there are no dependencies between the relevant crates. But further down the graph, parallelism reduces as crate dependencies increase. Although the compiler can overlap compilation of dependent crates somewhat thanks to a feature called pipelined compilation, there is much less parallel execution happening towards the end of compilation, and this is typical for large Rust programs. Interprocess parallelism is not enough to take full advantage of many cores. For more speed, we need parallelism within each process.

Existing intraprocess parallelism: the back-end

The compiler is split into two halves: the front-end and the back-end.

The front-end does many things, including parsing, type checking, and borrow checking. Until this week, it could not use parallel execution.

The back-end performs code generation. It generates code in chunks called "codegen units" and then LLVM processes these in parallel. This is a form of coarse-grained parallelism.

We can visualize the difference between the serial front-end and the parallel back-end. The following image shows the output of a profiler calledSamply measuring rustc as it does a release build of the final crate in Cargo. The image is superimposed with markers that indicate front-end and back-end execution.

Each horizontal line represents a thread. The main thread is labelled "rustc" and is shown at the bottom. It is busy for most of the execution. The other 16 threads are LLVM threads, labelled "opt cgu.00" through to "opt cgu.15". There are 16 threads because 16 is the default number of codegen units for a release build.

There are several things worth noting.

• Front-end execution takes 10.2 seconds.
• Back-end execution occurs takes 6.2 seconds, and the LLVM threads are running for 5.9 seconds of that.
• The parallel code generation is highly effective. Imagine if all those LLVM executed one after another!
• Even though there are 16 LLVM threads, at no point are all 16 executing at the same time, despite this being run on a machine with 28 cores. (The peak is 14 or 15.) This is because the main thread translates its internal code representation (MIR) to LLVM's code representation (LLVM IR) in serial. This takes a brief period for each codegen unit, and explains the staircase shape on the left-hand side of the code generation threads. There is some room for improvement here.
• The front-end is entirely serial. There is a lot of room for improvement here.

New intraprocess parallelism: the front-end

The front-end is now capable of parallel execution. It usesRayon to perform compilation tasks using fine-grained parallelism. Many data structures are synchronized by mutexes and read-write locks, atomic types are used where appropriate, and many front-end operations are made parallel. The addition of parallelism was done by modifying a relatively small number of key points in the code. The vast majority of the front-end code did not need to be changed.

When the parallel front-end is enabled and configured to use eight threads, we get the following Samply profile when compiling the same example as before.

Again, there are several things worth nothing.

• Front-end execution takes 5.9 seconds (down from 10.2 seconds).
• Back-end execution takes 5.3 seconds (down from 6.2 seconds), and the LLVM threads are running for 4.9 seconds of that (down from 5.9 seconds).
• There are seven additional threads labelled "rustc" operating in the front-end. The reduced front-end time shows they are reasonably effective, but the thread utilization is patchy, with the eight threads all having periods of inactivity. There is room for significant improvement here.
• Eight of the LLVM threads start at the same time. This is because the eight "rustc" threads create the LLVM IR for eight codegen units in parallel. (For seven of those threads that is the only work they do in the back-end.) After that, the staircase effect returns because only one "rustc" thread does LLVM IR generation while seven or more LLVM threads are active. If the number of threads used by the front-end was changed to 16 the staircase shape would disappear entirely, though in this case the final execution time would barely change.

Putting it all together

Rust compilation has long benefited from interprocess parallelism, via Cargo, and from intraprocess parallelism in the back-end. It can now also benefit from intraprocess parallelism in the front-end.

You might wonder how interprocess parallelism and intraprocess parallelism interact. If we have 20 parallel rustc invocations and each one can have up to 16 threads running, could we end up with hundreds of threads on a machine with only tens of cores, resulting in inefficient execution as the OS tries its best to schedule them?

Fortunately no. The compiler uses the jobserver protocolto limit the number of threads it creates. If a lot of interprocess parallelism is occuring, intraprocess parallelism will be limited appropriately, and the number of threads will not exceed the number of cores.

How to use it

The nightly compiler is now shipping with the parallel front-end enabled. However, by default it runs in single-threaded mode and won't reduce compile times.

Keen users can opt into multi-threaded mode with the -Z threads option. For example:

$ RUSTFLAGS="-Z threads=8" cargo build --release

Alternatively, to opt in from aconfig.toml file (for one or more projects), add these lines:

[build]
rustflags = ["-Z", "threads=8"]

It may be surprising that single-threaded mode is the default. Why parallelize the front-end and then run it in single-threaded mode? The answer is simple: caution. This is a big change! The parallel front-end has a lot of new code. Single-threaded mode exercises most of the new code, but excludes the possibility of threading bugs such as deadlocks that can affect multi-threaded mode. Even in Rust, parallel programs are harder to write correctly than serial programs. For this reason the parallel front-end also won't be shipped in beta or stable releases for some time.

Performance effects

When the parallel front-end is run in single-threaded mode, compilation times are typically 0% to 2% slower than with the serial front-end. This should be barely noticeable.

When the parallel front-end is run in multi-threaded mode with -Z threads=8, our measurements on real-world code show that compile times can be reduced by up to 50%, though the effects vary widely and depend on the characteristics of the code and its build configuration. For example, dev builds are likely to see bigger improvements than release builds because release builds usually spend more time doing optimizations in the back-end. A small number of cases compile more slowly in multi-threaded mode than single-threaded mode. These are mostly tiny programs that already compile quickly.

We recommend eight threads because this is the configuration we have tested the most and it is known to give good results. Values lower than eight will see smaller benefits. Values greater than eight will give diminishing returns and may even give worse performance.

If a 50% improvement seems low when going from one to eight threads, recall from the explanation above that the front-end only accounts for part of compile times, and the back-end is already parallel. You can't beat Amdahl's Law.

Memory usage can increase significantly in multi-threaded mode. We have seen increases of up to 35%. This is unsurprising given that various parts of compilation, each of which requires a certain amount of memory, are now executing in parallel.

Correctness

Reliability in single-threaded mode should be high.

In multi-threaded mode there are some known bugs, including deadlocks. If compilation hangs, you have probably hit one of them.

Feedback

If you have any problems with the parallel front-end, please check the issues marked with the "WG-compiler-parallel" label. If your problem does not match any of the existing issues, please file a new issue.

For more general feedback, please start a discussion on the wg-parallel-rustc Zulip channel. We are particularly interested to hear the performance effects on the code you care about.

Future work

We are working to improve the performance of the parallel front-end. As the graphs above showed, there is room to improve the utilization of the threads in the front-end. We are also ironing out the remaining bugs in multi-threaded mode.

We aim to stabilize the -Z threads option and ship the parallel front-end running by default in multi-threaded mode on stable releases in 2024.

Acknowledgments

The parallel front-end has been under development for a long time. It was started by @Zoxc, who also did most of the work for several years. After a period of inactivity, the project was revived this year by @SparrowLii, who led the effort to get it shipped. Other members of the Parallel Rustc Working Group have also been involved with reviews and other activities. Many thanks to everyone involved.

Continue Reading…

crates.io: Dropping support for non-canonical downloads

TL;DR

• We want to improve the reliability and performance of crate downloads.
• "Non-canonical downloads" (that use URLs containing hyphens or underscores where the crate published uses the opposite) are blocking these plans.
• On 2023-11-20 support for "non-canonical downloads" will be disabled.
• cargo users are unaffected.

What are "non-canonical downloads"?

The "non-canonical downloads" feature allows everyone to download the serde_derive crate from https://crates.io/api/v1/crates/serde%5Fderive/1.0.189/download, but also from https://crates.io/api/v1/crates/SERDE-derive/1.0.189/download, where the underscore was replaced with a hyphen (crates.io normalizes underscores and hyphens to be the same for uniqueness purposes, so it isn't possible to publish a crate named serde-derive because serde_derive exists) and parts of the crate name are using uppercase characters. The same also works vice versa, if the canonical crate name uses hyphens and the download URL uses underscores instead. It even works with any other combination for crates that have multiple such characters (please don't mix them…!).

Why remove it?

Supporting such non-canonical download requests means that the crates.io server needs to perform a database lookup for every download request to figure out the canonical crate name. The canonical crate name is then used to construct a download URL and the client is HTTP-redirected to that URL.

While we have introduced a caching layer some time ago to address some of the performance concerns, having all download requests go through our backend servers has still started to become problematic and at the current rate of growth will not become any easier in the future.

Having to support "non-canonical downloads" however prevents us from using CDNs directly for the download requests, so if we can remove support for non-canonical download requests, it will unlock significant performance and reliability gains.

Who is using "non-canonical downloads"?

cargo always uses the canonical crate name from the package index to construct the corresponding download URLs. If support was removed for this on the crates.io side then cargo would still work exactly the same as before.

Looking at the crates.io request logs, the following user-agents are currently relying on "non-canonical downloads" support:

• cargo-binstall/1.1.2
• Faraday v0.17.6
• Go-http-client/2.0
• GNU Guile
• python-requests/2.31.0

Three of these are just generic HTTP client libraries. GNU Guile is apparently a programming language, so most likely this is also a generic user-agent from a custom user program.

cargo-binstall is a tool enabling installation of binary artifacts of crates. The maintainer is already aware of the upcoming change and confirmed that more recent versions of cargo-binstall should not be affected by this change.

We recommend that any scripts relying on non-canonical downloads be adjusted to use the canonical names from the package index, the database dump, or the crates.io API instead. If you don't know which data source is best suited for you, we welcome you to take a look at the crates.io data access page.

What is the plan?

1. Today: Announce the removal of support for non-canonical downloads on the main Rust blog.
2. 2023-11-20: Disable support for non-canonical downloads and return a migration error message instead, to alert remaining users of this feature of the need to migrate. This still needs to put load on the application to detect a request is using a non-canonical download URL.
3. 2023-12-18: Return a regular 404 error instead of the migration error message, allowing us to get rid of (parts of) the database query.

Note that we will still need the database query for download counting purposes for now. We have plans to remove this requirement as well, but those efforts are blocked by us still supporting non-canonical downloads.

If you want to follow the progress on implementing these changes or if you have comments you can subscribe to the corresponding tracking issue. Related discussions are also happening on the crates.io Zulip stream.

Continue Reading…

A tale of broken badges and 23,000 features

Around mid-October of 2023 the crates.io team was notified by one of our users that a shields.io badge for their crate stopped working. The issue reporter was kind enough to already debug the problem and figured out that the API request that shields.io sends to crates.io was most likely the problem. Here is a quote from the original issue:

This crate makes heavy use of feature flags which bloat the response payload of the API.

Apparently the API response for this specific crate had broken the 20 MB mark and shields.io wasn't particularly happy with this. Interestingly, this crate only had 9 versions published at this point in time. But how do you get to 20 MB with only 9 published versions?

As the quote above already mentions, this crate is using features… a lot of features… almost 23,000! 😱

What crate needs that many features? Well, this crate provides SVG icons for Rust-based web applications… and it uses one feature per icon so that the payload size of the final WebAssembly bundle stays small.

At first glance there should be nothing wrong with this. This seems like a reasonable thing to do from a crate author perspective and neither cargo, nor crates.io, were showing any warnings about this. Unfortunately, some of the internals are not too happy about such a high number of features…

The first problem that was already identified by the crate author: the API responses from crates.io are getting veeeery large. Adding to the problem is the fact that the crates.io API currently does not paginate the list of published versions. Changing this is obviously a breaking change, so our team had been a bit reluctant to change the behavior of the API in that regard, though this situation has shown that we will likely have to tackle this problem in the near future.

The next problem is that the index file for this crate is also getting large. With 9 published versions it already contains 11 MB of data. And just like the crates.io API, there is currently no pagination built into the package index file format.

Now you may ask, why do the package index and cargo need to know about features? Well, the easy answer is: for dependency resolution. Features can enable optional dependencies, so when a dependency feature is used it might influence the dependency resolution. Our initial thought was that we could at least drop all empty feature declarations from the index file (e.g. foo = []), but the cargo team informed us that cargo relies on them being available there too, and so for backwards-compatibility reasons this is not an option.

On the bright side, most Rust users are on cargo versions these days that use the sparse package index by default, which only downloads index files for packages actually being used. In other words: only users of this icon crate need to pay the price for downloading all the metadata. On the flipside, this means users who are still using the git-based index are all paying for this one crate using 23,000 features.

So, where do we go from here? 🤔

While we believe that supporting such a high number of features is conceptually a valid request, with the current implementation details in crates.io and cargo we cannot support this. After analyzing all of these downstream effects from a single crate having that many features, we realized we need some form of restriction on crates.io to keep the system from falling apart.

Now comes the important part: on 2023-10-16 the crates.io team deployed a change limiting the number of features a crate can have to 300 for any new crates/versions being published.

… for now, or at least until we have found solutions for the above problems.

We are aware of a couple of crates that also have legitimate reasons for having more than 300 features, and we have granted them appropriate exceptions to this rule, but we would like to ask everyone to be mindful of these limitations of our current systems.

We also invite everyone to participate in finding solutions to the above problems. The best place to discuss ideas is the crates.io Zulip stream, and once an idea is a bit more fleshed out it will then be transformed into an RFC.

Finally, we would like to thank Charles Edward Gagnon for making us aware of this problem. We also want to reiterate that the author and their crate are not to blame for this. It is hard to know of these crates.io implementation details when developing crates, so if anything, the blame would be on us, the crates.io team, for not having limits on this earlier. Anyway, we have them now, and now you all know why! 👋

Continue Reading…

Announcing the New Rust Project Directors

We are happy to announce that we have completed the process to elect new Project Directors.

The new Project Directors are:

• Scott McMurray
• Jakob Degen
• Santiago Pastorino

They will join Ryan Levick and Mark Rousskov to make up the five members of the Rust Foundation Board of Directors who represent the Rust Project.

The board is made up of Project Directors, who come from and represent the Rust Project, and Member Directors, who represent the corporate members of the Rust Foundation.

Both of these director groups have equal voting power.

We look forward to working with and being represented by this new group of project directors.

We were fortunate to have a number of excellent candidates and this was a difficult decision. We wish to express our gratitude to all of the candidates who were considered for this role! We also extend our thanks to the project as a whole who participated by nominating candidates and providing additional feedback once the nominees were published. Finally, we want to share our appreciation for the Project Director Elections Subcommittee for working to design and facilitate running this election process.

This was a challenging decision for a number of reasons.

This was also our first time doing this process and we learned a lot to use to improve it going forward. The Project Director Elections Subcommittee will be following up with a retrospective outlining how well we achieved our goals with this process and making suggestions for future elections. We are expecting another election next year to start a rotating cadence of 2-year terms. Project governance is about iterating and refining over time.

Once again, we thank all who were involved in this process and we are excited to welcome our new Project Directors.

Continue Reading…