Rust is getting more popular among software developers, and the Python community is no exception. I started learning Rust a few years ago, but after some point, I began to lose motivation because most of my exercises were toy examples and far away from the real applications. So I questioned myself:

  • Can I use Rust in my day-to-day job as a Python developer?
  • Can I build something that will benefit the projects I am working with?

Probably, many of us, people coming from the Python background, had similar thoughts. In a 3 chapter series, I will share my experience with embedding Rust into Python projects and try to give you some options that may answer such questions.

Update (2020-08-06): More links to popular Rust-powered tools & thread-safety note

ANNOUNCE: I build a service for API fuzzing. Sign up to check your API now!

In this series we'll talk about:

  • Why?

    • Reasons to use Rust in Python projects;
    • What to think about when considering it;

  • When?

    • Looking for spots where it makes sense to use Rust;

  • How?

    • Building a Rust library;
    • How Python interacts with other languages;
    • Adding Python bindings;
    • Usability improvements;
    • Configuring tests, builds, benchmarks, and CI;
    • Packaging & releasing

This part covers "Why?" and "When?". Other chapters:

Why?

Performance

Python values development speed and simplicity, but often it comes as a trade-off with performance. And when your application needs to run faster or consume less memory, you may think to rewrite some bottlenecks in a language with better performance characteristics than Python.

When appropriately applied, Rust can yield performance improvements in orders of magnitude and provide speed levels comparable with C/C++. For example, source maps processing in Sentry went from >20 seconds to <0.5 sec after they replaced Python with Rust.

But will using a different language improve the performance of your program?

Maybe.

If your program spends its time waiting for IO or uses a not-efficient algorithm, then using Rust may not give you much improvement or, if misapplied, even can make it slower.

Review your algorithms, data structures, and data access patterns first!

Reliability

Rust's ownership model empowers many language's features. It was very novel to me when I saw it for the first time, and its benefits were not immediately clear. It turns out that it is possible to eliminate whole classes of errors at compile time:

Another point is that Rust is a statically typed language with a rich type system. Even though that Python's approach to typing is disparate, optional type annotations and type-checking tools (e.g. mypy) become more popular among Python users. Often they are in CI as a part of static checks, but with Rust, it is a part of the language, that can't be opt-out. In this case, the compiler will catch many errors, and they won't appear in runtime.

For example, missing check for None value:

def get_name(first, middle, last):
    return f"{first} {middle} {last}"

>>> get_name("robin", None, "doe")
'robin None doe'

In this case, you'd probably want to create a string without a middle name at all, but instead, you got robin None doe. The example is naive, but when your codebase grows, and you have dozens of input sources, it becomes harder to track such cases, and many of them may stay unnoticed to developers. There could be runtime checks and tests, but when you encode this information in types, you can verify the code validity, before running it.

Similar Rust code:

fn get_name(first: &str, middle: &str, last: &str) -> String {
    format!("{} {} {}", first, middle, last)
}

fn main() {
    let f = get_name("robin", None, "doe");
    println!("{}", f)
}

And such program won't even compile:

error[E0308]: mismatched types
 --> src/main.rs:6:45
  |
6 |     let f = get_name("robin", None, "doe");
  |                               ^^^^ expected `&str`, found enum `std::option::Option`

The Rust compiler can guard you against many common pitfalls and will guide you for fixing them. It could be annoying, though, but it forces you to design your code more robustly, and there are ways to opt-out some restrictions with unsafe, and it means that you, as a developer, take the responsibility to uphold safety guarantees.

Rust's type system often implies verbose code, but at the same time this code is more explicit about edge cases

There is also another vital difference - how Python and Rust approach thread safety. Python has a Global Interpreter Lock (GIL) - a mutex that prevents multiple native OS threads from running Python bytecode simultaneously, hence avoiding potential race conditions. On the other side, in most cases, Rust can statically check if your code is safe to run concurrently and/or in parallel by leveraging its type system.

Besides that, with the rayon crate, it is straightforward to convert your sequential code into parallel. The sequential version:

fn process_files(filenames: &[&str]) -> io::Result<()> {
    let processed = filenames
        .iter()
        .map(|&name| handle_file(name));
    println!("Files processed: {}", processed.count());
    Ok(())
}

To get the parallel version you need to add use rayon::prelude::*; and change .iter() to .par_iter(), that simple:

use rayon::prelude::*;

fn process_files(filenames: &[&str]) -> io::Result<()> {
    let processed = filenames
        .par_iter()
        .map(|&name| handle_file(name));
    println!("Files processed: {}", processed.count());
    Ok(())
}

And you get each chunk of work processed by a different thread, which potentially leads to more efficient multicore systems utilization.

Costs

But does your program even need that possible performance level? Does it matter for you if your program runs 100 milliseconds or 3 milliseconds? How beneficial that improvement will be?

You need to answer these questions if you want to consider Rust because everything has a cost.

It introduces extra tools and steps in your workflow. You'll need to compile your Rust code, which is known to be slow, you'll need to run more tools in your pipeline - clippy, rustfmt, cargo check, etc. They will require extra effort from your side, especially if you are new to them.

Rust has a notoriously steep learning curve, and it might take some time to be as productive with it as with Python. For example, things that "just work" in Python may require extra care in Rust.

What if before creating a "full name", we want to apply a title case to each name component? With Python, it is straightforward:

>>> "robin".title()
'Robin'

In Rust, &str and String types do not have such a method. And somewhat sufficient implementation could look like this:

trait TitleCase {
    fn title(&self) -> String;
}

impl TitleCase for &str {
    fn title(&self) -> String {
        let mut c = self.chars();
        match c.next() {
            None => String::new(),
            Some(f) => f.to_uppercase().collect::<String>() + c.as_str(),
        }
    }
}

fn main() {
    assert_eq!("robin".title(), "Robin");
}

The complexity behind a seemingly simple problem is beautifully explained in this StackOverflow answer. However, there is a crate for it.

Consider if benefits of introducing Rust will outweigh short and long-term costs for your project

However, I like to view it as a long-term investment - eventually, these costs will pay off, especially in large projects.

When?

As I already mentioned, Rust could be used to improve performance & reliability, and I'd like to note a few areas where Rust can completely replace Python or complement existing projects.

Command Line tools

Rust has rapid startup time and excellent overall performance, which makes it great for CLI applications. It enables a lot of awesome use cases, for example, you can run your code formatters and linters on saving a file without noticing delays.

I use black to format Python code in my projects and rustfmt for Rust code. To compare their startup time, we can use hyperfine (a command-line benchmarking tool written in Rust) and run both tools on empty files to avoid work other than starting the program.

Here are the results from my local environment (Intel 8700K, 32GB DDR4 3200 MHz, Samsung SSD 850 EVO):

$ hyperfine --warmup 5  "black empty.py" "rustfmt empty.rs"

Benchmark #1: black empty.py
  Time (mean ± σ):      88.2 ms ±   0.7 ms    [User: 78.7 ms, System: 9.6 ms]
  Range (min … max):    86.8 ms …  90.1 ms    32 runs

Benchmark #2: rustfmt empty.rs
  Time (mean ± σ):       2.7 ms ±   0.2 ms    [User: 1.6 ms, System: 1.6 ms]
  Range (min … max):     2.5 ms …   3.9 ms    646 runs

  Warning: Command took less than 5 ms to complete. Results might be inaccurate.

Summary
  'rustfmt empty.rs' ran
   32.63 ± 2.22 times faster than 'black empty.py'

As you see, the difference is significant - rustfmt is >30 times faster in startup time. There are many reasons for that and importing modules is one of the most significant contributors to it - the interpreter parses the source code and executes it, which means that any slow operation placed on the module-level will slow down the startup time.

For real-life projects, the startup time is higher, for example, Mercurial developers report hg startup time could take over 100ms just to get to a Mercurial command's main function. They plan to make hg a Rust binary - it is not there yet (the last update of the document mentioned above happened in 2018), but the current source code (5.5rc0 at the time of writing) contains a significant amount of Rust code.

Another benefit that with Rust, you have a single binary, nothing else - your users don't have to install Python & dependencies. If you have a Python CLI tool, you could look at PyOxidizer, which helps embed Python into a binary.

As a quintessence of Rust CLI tools, check ripgrep, that is generally faster than other command-line search tools (like grep or ag). Read more in the Andrew Gallant's blog post.

You could try building a CLI tool in Rust yourself, take a look at the CLI Book.

WebAssembly

Rust also can compile to WebAssembly via wasm-bindgen. It might sound like something more relevant for JavaScript developers, but imagine if you have a backend service that does some standalone CPU-intensive work without many dependencies on other services. For example, a service for generating a greeting card image from the user's input.

There are many similar scenarios, and with WebAssembly, you can offload such computations to the user's browser, reduce the number of services you have, and reduce network traffic. The user won't need to download an image from the server - it also may improve the web page responsiveness.

As an example, see a parallel raytracing demo just in your browser (should work in the latest Firefox and Chrome versions).

Data processing

As Rust gives you a low-level control over memory allocations, there are a lot of highly efficient parsers, many of those don't heap allocate.

For example, serde allows you to define various serializing and deserializing formats very efficiently and flexibly. If you want more fine-tuning, you could build a custom parser with nom or use any of already existing ones (you could, also, read sources of this CSV parser built by BurntSushi for inspiration).

If you want to try Rust-powered tools in your projects, then I can suggest the following ones:

orjson

This JSON library is built on top of serde and vastly outperforms the stdlib json module. It also beats the popular ujson library, especially in the encoding (up to ~7x), according to benchmarks in both libraries.

fastuuid

It is a fast alternative to the standard uuid module. In some cases, it is faster than the stdlib one up to 15x.

jsonschema-rs

A shameless plug: Implementation of JSON Schema validation (written by me) that aims for complete spec support and faster validation - in some benchmarks validation is up to 280x faster than with the pure-Python jsonschema library.

However, some of the libraries above imply certain usage restrictions - no pickle support, limited integer precision, and similar. Read through their docs and decide if they fit your use-case.

Another example is data science, where Python is dominant. However, the Weld project claims order-of-magnitude speedups for existing Pandas + NumPy code by providing a common runtime for data libraries. Check this YouTube video for more information.

Summary

Rust is a delightful and effective language that empowers developers to build fast and reliable systems. It can improve projects written in Python in many aspects, especially when applied thoughtfully. However all these benefits have costs that should be accounted for when you consider Rust as a part of your ecosystem.

To illustrate how Rust handles data processing, we will build a library for CSS inlining in the next chapter. It is a common task when you send HTML emails or embed HTML into third party resources - style tags are often stripped away. Then in the third chapter, we'll add Python bindings to this crate. In the end, we will have a simpler alternative to packages like premailer or pynliner.

Chapters:

Cheers,

Dmitry

❤ ❤ ❤