Sometimes we need to extend existing Python projects with other languages to solve performance issues or to include some functionality already written in another language. In this chapter, we will create a full-featured Python integration for a Rust crate we built in the previous article. As the primary goal, I want to focus on a pleasant end-user experience and share tips on improving the usability and debuggability of the resulting library.

Overview:

Low-level details of Rust / Python integration
- Python C API;
- Rust / C interoperability;
Applying PyO3 to the CSS inlining crate
- Functions & classes;
- Compilation;
Usability
- Documentation & function signatures
- Custom exceptions
- Debugging
Packaging, testing, benchmarking and releasing

Target audience: People who are looking for ways to connect existing Rust code with Python. Some general knowledge of Rust syntax would be useful.

ANNOUNCE: I build a service for API fuzzing. Sign up to check your API now!

It is the last part of the 3 chapter series about Rust for Python users. Other chapters:

Python C API

NOTE. In this article, I will imply using CPython, which is the de-facto reference Python implementation.

As the first step, let's check how Python can use code written in other languages. The most popular way to do so is to use Python's C API. It allows programmers to access Python interpreter internals and defines an API that the interpreter will expect from an extension.

Alternatively, it is possible to use ctypes, but it may perform significantly worse. See this example of integration

The cornerstone of Python's C API is PyObject. It is a struct containing information about the object's type and the number of references to this object (which is needed for garbage collection). Python objects are mostly moved around via PyObject * pointers, and you could see them in many C API function signatures. Accessing the concrete type is done via casting such pointer to a pointer to a specific type.

A Python module is also an object under the hood and has the PyModuleObject type. If you want to create a new one, let's say css_inline you need to define a few things:

an array of methods definitions for your module object;
a module definition struct;
a function to initialize the module;

The initialization function should be named PyInit_<modulename> for ASCII-only names. Therefore we can create a css_inline.c file as follows:

// Include all function, type, and macro definitions
#define PY_SSIZE_T_CLEAN
#include <Python.h>

// A wrapper function to call the Rust one
static PyObject *inline_css(PyObject *self, PyObject *args) {
  return PyLong_FromLong(42);
}

// Methods definition
static PyMethodDef css_inlineMethods[] = {
    {
        "inline_css", // Name of the method
        inline_css,   // Pointer to the implementation
        METH_VARARGS, // Calling convention (*args)
        "Inlines CSS" // Method docstring
    },
    {
        // Sentinel value
        NULL, NULL, 0, NULL
    }
};

// Module definition
static struct PyModuleDef css_inline_module = {
    PyModuleDef_HEAD_INIT, // Special value to initialize modules
    "css_inline",          // Name of the module
    "CSS inlining module", // Docstring
    -1,                    // Indicates if the module can be used
                           // safely with sub-interpreters
    css_inlineMethods      // Methods defined in the module
};

// PyMODINIT_FUNC is a special macro that sets the return type to `PyObject*`
// and does a few other things
PyMODINIT_FUNC PyInit_css_inline(void) {
  // Single-phase initialization
  return PyModule_Create(&css_inline_module);
}

It is possible to create a module with a non-ASCII name. See this chapter of the official documentation

In the extension, we also added the css_inline stub function that will work as a gateway to our Rust code. Since we chose METH_VARARGS as the calling convention, this function will accept two values, the first one is the module object, and the second one is a tuple with all arguments. The function's goal is to process these arguments, pass them to the Rust part, and convert the return value to a type that is understandable by the Python C API.

To build this extension, we may use setuptools with the following setup.py file:

from setuptools import setup, Extension

setup(
    name="css_inline",
    version="1.0",
    ext_modules=[Extension("css_inline", ["css_inline.c"])],
)

The python setup.py install command will create a usable .so library and install it:

import css_inline

assert css_inline.inline_css() == 42  # Works!

When Python interpreter tries to import this module for the first time, ExtensionFileLoader will find the extension file, open it with the dlopen syscall (on Linux), and call the PyInit_css_inline function. After a few additional steps, the module is ready to use.

Here is a detailed guide on Python C extensions, or you could refer to the official documentation.

Rust / C interoperability

The integration of a Rust extension works similarly to the C-one. To integrate a Rust project with the Python C API, we need to expose a PyInit_css_inline function that the Python interpreter will call on the C level. Also, we'd like to call C API functions to work with Python objects. Happily, Rust can communicate with C in both directions - Rust code can call C code and vice versa.

In Rust the initialization function might look like this:

#[no_mangle]
pub unsafe extern "C" fn PyInit_css_inline() -> *mut PyObject {
    // Initialization code
}

Let's break it down!

#[no_mangle]

Many programming languages use a technique named "name mangling", which, for example, serves to resolve function naming ambiguity and encode useful information into the final symbol name. And Rust is no exception, but the #[no_mangle] annotation tells the Rust compiler not to change the name of this function, so the consumer of this function (in our case CPython interpreter) can use exactly this name.

Python uses name mangling too! Create a class C with the __foo method - an instance of this class will have _C__foo method, but no __foo.

extern "C"

In this case, the extern keyword implies creating a function that can be called from other languages using the default C ABI (Application Binary Interface) on the specific platform.

An extern function is not necessarily unsafe, but we plan to call C code there; therefore, we marked it with the unsafe keyword.

This function should return a type that mirrors the PyObject struct from the Python C API to match the Python interpreter's expectations. It means that all dependent types should also be defined to compile this code.

The Rust definition of the PyObject struct might look like this:

#[repr(C)]
pub struct PyObject {
    pub ob_refcnt: Py_ssize_t,
    pub ob_type: *mut PyTypeObject,
}

pub type Py_ssize_t = ::libc::ssize_t;

#[repr(C)]
pub struct PyTypeObject {
   // ...
}

#[repr(C)] makes those structs to use the same memory layout as C uses.

The necessity to have all dependent types implies a lot of boilerplate, which we can avoid by using such projects as rust-cpython or PyO3. The latter one is a fork of the former one, but over time they diverged a lot. Recently, PyO3 became available for stable Rust, and I am going to use it for the rest of the article.

You could read more about the differences between rust-cpython and PyO3 on this page.

PyO3 provides a lot of helpers. For example, there are already FFI definitions for various structs like PyObject and PyTypeObject. To simplify integration, PyO3 heavily relies on procedural macros. You don't have to specify extern "C" function manually because the pymodule macro will take care of it:

use pyo3::prelude::*;

// This docstring in the "///" block will be used for the Python module
// and will be available as `css_inline.__doc__`
/// Module docs!
#[pymodule]
fn css_inline(py: Python, module: &PyModule) -> PyResult<()> {
    Ok(())
}

Here you need to specify a Rust function that will initialize your module. Before we get to this function, let's look at what code the compiler will get after macro expansion by using cargo-expand:

// Prelude imports go here ...

// Our initialization function
fn css_inline(py: Python, module: &PyModule) -> PyResult<()> {
    Ok(())
}

// Python interpreter expects this function
#[no_mangle]
#[allow(non_snake_case)]
pub unsafe extern "C" fn PyInit_css_inline() -> *mut pyo3::ffi::PyObject {
    use pyo3::derive_utils::ModuleDef;
    const NAME: &'static str = "css_inline\u{0}";
    static MODULE_DEF: ModuleDef = unsafe { ModuleDef::new(NAME) };
    // Our `css_inline` initialization function is called inside
    match MODULE_DEF.make_module("Module docs!", css_inline) {
      Ok(m) => m,
      Err(e) => e.restore_and_null(unsafe {pyo3::Python::assume_gil_acquired()}),
    }
}

This initialization code inside PyInit_css_inline will create a Python module with the specified docstring and call our css_inline function as the last step. It will also take care of errors that might occur inside css_inline. The PyResult type is essentially an alias to Result<T, PyErr>, where the PyErr type represents a Python exception that will be propagated to the Python side.

As you see, the css_inline function accepts two parameters:

Python. It is a special marker that indicates that the GIL (Global Interpreter Lock) is currently acquired.
PyModule. A wrapper around the C API's PyModuleObject struct.

The module initialization approach taken by PyO3 differs from the C example above (which follows the reference Python C API docs) - it creates an empty module with the given name but allows the developer to customize this module via an initialization function.

use pyo3::wrap_pyfunction;

#[pyfunction]
fn inline(html: &str) -> PyResult<String> {
    todo!()
}

// ...
#[pymodule]
fn css_inline(py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pyfunction!(inline))?;
    Ok(())
}

Adding each item boils down to:

Appending the new item to the __all__ module's attribute using the PyObject_GetAttr and PyList_Append C API calls;
Setting a new attribute on the module via the PyObject_SetAttr C API call.

This approach follows the single-phase way to initialize extensions by using PyModule_Create internally. For a small performance penalty, you gain a very flexible way to initialize an extension.

The official documentation has a chapter about extension initialization

Now, let's use PyO3 in the css_inline project from the previous article!

Python bindings for the CSS inlining crate

To simplify integration, we can put bindings to the crate's repository as a sub-project. Probably, we could add more bindings in the future, so let's create Python-specific ones inside the bindings/python directory. By having such a layout, we can test bindings against any change in the main crate, making incompatibilities more visible.

$ mkdir bindings
$ cargo new --lib --name css-inline-python bindings/python
     Created library `css-inline-python` package

As the library will be used from C, we need to use cdylib as the crate-type in our Cargo.html. By using this option, the resulting library won't have Rust-specific information inside, and can be used from another language:

[lib]
name = "css_inline"
crate-type = ["cdylib"]

On the Python side, we'd probably like to use css_inline name instead of css_inline_python for clarity. Still, our CSS inlining crate already uses this name, therefore adding another crate with the same name will cause name collision during compilation. To avoid this, we can explicitly set the final library name in the [lib] section, as in the example above.

These bindings have minimal dependencies - our original CSS inlining crate, which is available locally (we can specify a relative path to it) and PyO3:

[dependencies]
css-inline = { path = "../../css_inline", version = "*" }
pyo3 = { version = "0.11.1", features = ["extension-module"]}

By default, PyO3 builds a binary and requires the extension-module feature as an indicator for building a library. See more in this pull request

The inline function could serve as the main entry point for our bindings. It takes an HTML document as a string, configuration options, and returns inlined HTML.

use pyo3::{prelude::*, wrap_pyfunction};

/// Inline CSS!
#[pyfunction]
fn inline(html: &str, remove_style_tags: Option<bool>) -> PyResult<String> {
    todo!()
}

/// Module docs!
#[pymodule]
fn css_inline(py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pyfunction!(inline))?;
    Ok(())
}

This function takes two arguments of native Rust types. Still, under the hood, the resulting C-facing function follows the METH_VARARGS | METH_KEYWORDS calling convention, it accepts three *mut PyObject values - the module object, *args (PyTuple), and **kwargs (PyDict). Then PyO3 tries to extract the desired arguments from these values and cast them to Rust types.

Check out how argument processing looks like by using cargo-expand!

Then to process options, we can create InlineOptions and pass it to the inliner.

/// Inline CSS!
#[pyfunction]
fn inline(html: &str, remove_style_tags: Option<bool>) -> PyResult<String> {
    let options = css_inline::InlineOptions {
        remove_style_tags: remove_style_tags.unwrap_or(false)
    };
    let inliner = css_inline::CSSInliner::new(options);
    inliner.inline(html)  // Compilation error!
}

The return type of this function implies the PyErr error type, and we have to convert the original InlineError to it. The canonical way to do so in Rust is the newtype pattern:

Create a simple tuple struct InlineErrorWrapper that wraps the foreign type 'InlineError';
Implement From<InlineErrorWrapper> for PyErr that will convert the wrapper to PyErr.

use pyo3::exceptions::ValueError;

struct InlineErrorWrapper(css_inline::InlineError);

impl From<InlineErrorWrapper> for PyErr {
    fn from(error: InlineErrorWrapper) -> Self {
        let message = match error.0 {
            css_inline::InlineError::IO(inner) => inner.to_string(),
            css_inline::InlineError::ParseError(message) => message,
        };
        ValueError::py_err(message)
    }
}

Note, newtypes are zero-cost abstractions - there is no runtime overhead for using them.

Conversion from one error type to another could be done with the map_err function, which maps the Err value, which is css_inline::InlineError. In our case we can use the type constructor directly instead of a closure:

#[pyfunction]
fn inline(html: &str, remove_style_tags: Option<bool>) -> PyResult<String> {
    // ...
    // In more complex cases we could use a closure:
    // .map_err(|e| InlineErrorWrapper(e))
    Ok(inliner.inline(html).map_err(InlineErrorWrapper)?)
}

Besides the inline function, there is also the CSSInliner struct, which could be exposed as a Python class using a few more procedural macros:

/// Customizable CSS inliner.
#[pyclass(module="css_inline")]
struct CSSInliner {
    inner: css_inline::CSSInliner,  // Only C-like structs are supported!
}

In our case, the original css_inline::CSSInliner struct is wrapped with another one under the same name. This wrapper has the #[pyclass] macro that generates code to make this struct representable as a Python class. Note that the module argument should be passed manually; otherwise, the CSSInliner.__module__ attribute will be equal to builtins. The struct docstring will become the class docstring, similarly to the #[pyfunction] behavior.

Then we need to specify a couple of methods for this Python class, which could be done via the #[pymethods] macro:

#[pymethods]
impl CSSInliner {
    #[new]
    fn new(remove_style_tags: Option<bool>) -> Self {
        let options = css_inline::InlineOptions {
            remove_style_tags: remove_style_tags.unwrap_or(false),
        };
        Self {
            inner: css_inline::CSSInliner::new(options),
        }
    }

    /// Inline CSS in the given HTML document
    fn inline(&self, html: &str) -> PyResult<String> {
        Ok(self.inner.inline(html).map_err(InlineErrorWrapper)?)
    }
}

The constructor is marked with the #[new] macro and will generate a proper __new__ Python method (__init__ is not supported). Other methods could be specified as regular Rust struct's methods.

Then, we can add this class to the module via module.add_class:

#[pymodule]
fn css_inline(py: Python, module: &PyModule) -> PyResult<()> {
    module.add_class::<CSSInliner>()?;
    // ...
}

See more about the "turbofish" notation (::<T>) in this blog post

At this point, the code is ready for compilation! There are many things we need to implement and improve, but now we need a simple way to use our extension from Python.

The PyO3 team offers two tools for building and developing Python packages - maturin and setuptools-rust. The former requires no configuration and slightly more straightforward to use than the latter, but not as capable. For this project, I am going to use setuptools-rust.

This approach is similar to the one we have seen before with setuptools and a C extension. The setup.py file:

from setuptools import setup
from setuptools_rust import Binding, RustExtension

setup(
    name="css-inline",
    version="0.1",
    rust_extensions=[RustExtension("css_inline", binding=Binding.PyO3)],
    zip_safe=False,
)

After installing setuptools-rust, we can install this package with python setup.py install and use it:

import css_inline

html = """
<html><head>
    <style>h1 { color:blue; }</style>
</head>
<body>
    <h1>Big Text</h1>
</body></html>"""

print(css_inline.inline(html, True))

The code above will output HTML with inlined styles as expected:

<html><head>

</head>
<body>
    <h1 style=" color:blue; ">Big Text</h1>
</body></html>

It is a minimum to get a working Rust extension and is not enough for a full-featured package, but before diving into packaging, let's improve the code itself!

Usability

The end-user won't see the source code, and it might be harder to debug the compiled Rust extension than a pure-Python library. For this reason, it is essential to make the extension behavior explicit and well-documented.

At the current stage, the inline function documentation is almost empty:

In [1]: import css_inline
In [2]: css_inline.inline?
Docstring: Inline CSS!
Type:      builtin_function_or_method

It is not clear how this function could be used - there is no signature. PyO3 provides #[text_signature] for this purpose:

/// Inline CSS!
#[pyfunction]
#[text_signature = "(html, remove_style_tags=False)"]
fn inline(html: &str, remove_style_tags: Option<bool>) -> PyResult<String> {
    // ... implementation
}

Now the signature is visible:

In [2]: css_inline.inline?
Signature: css_inline.inline(html, remove_style_tags=False)
Docstring: Inline CSS!
Type:      builtin_function_or_method

At the moment PyO3 doesn't support type annotations in function signatures, but there is an open feature request for that

Unfortunately, this signature doesn't work in all cases; for example, PyCharm doesn't recognize it, but there is a workaround. We can add a text signature as the first line of the docstring:

/// inline(html, remove_style_tags=False)
///
/// Inline CSS!
#[pyfunction]
#[text_signature = "(html, remove_style_tags=False)"]
fn inline(html: &str, remove_style_tags: Option<bool>) -> PyResult<String> {
    // ...
}

And then PyCharm identifies the signature correctly and you can use the "Go to Declaration or Usages" feature and see the following:

def inline(html, remove_style_tags=False): # real signature unknown; restored from __doc__
    """
    inline(html, remove_style_tags=False)

    Inline CSS!
    """
    return ""

Autocomplete will also work; however, type annotations won't work as well as with the text_signature macro. This approach works similarly with structs - you need to put text_signature and docstring above the struct definition.

I didn't find any way to avoid duplication of signatures with the latest (0.11.1 at the time of writing) version of PyO3. Please, let me know if it is possible to do it without much hacking!

There are certain limitations on the CPython internals level as well. Learn more in this StackOverflow answer and this CPython issue

Another vital aspect is error handling - errors happen, and there should be a decent way to debug them. Now, in case, if we pass invalid CSS to the inline function, we'll encounter a ValueError. It might be better to define a custom exception class to distinguish inlining errors from other ValueError around:

use pyo3::{create_exception, exceptions::ValueError};

const INLINE_ERROR_DOCSTRING: &str = "An error that can occur during inlining";

create_exception!(css_inline, InlineError, ValueError);

impl From<InlineErrorWrapper> for PyErr {
    fn from(error: InlineErrorWrapper) -> Self {
        let message = match error.0 {
            css_inline::InlineError::IO(inner) => inner.to_string(),
            css_inline::InlineError::ParseError(message) => message,
        };
        InlineError::py_err(message)
    }
}

/// Module docs!
#[pymodule]
fn css_inline(py: Python, module: &PyModule) -> PyResult<()> {
    // ...

    let inline_error = py.get_type::<InlineError>();
    inline_error.setattr("__doc__", INLINE_ERROR_DOCSTRING)?;
    module.add("InlineError", inline_error)?;
    Ok(())
}

PyO3 provides the create_exception macro that accepts the module where the exception is defined, the new exception struct name, and the exception base class.

Unfortunately, exceptions defined with this macro don't have a docstring, but we can add a docstring manually by modifying the __doc__ attribute. Note, we need to update the From<InlineErrorWrapper> implementation to use the custom exception class.

Now inlining exceptions have InlineError type:

In [1]: import css_inline

In [2]: html = """
<html>
  <head>
    <style>@wrong { color:blue; }</style>
  </head>
  <body>
      <h1>Big Text</h1>
  </body>
</html>"""

In [3]: css_inline.inline(html)
---------------------------------------------------------------------------
InlineError                               Traceback (most recent call last)
<ipython-input-3-a0a2e78bb61b> in <module>
----> 1 css_inline.inline(html)

InlineError: Invalid @ rule: wrong

But what if something unexpected happens? Let's say we used unwrap somewhere in the Rust code, and we hit panic. For example, if during the development, we didn't want to implement error conversion via map_err at the moment and just used unwrap:

#[pyfunction]
fn inline(html: &str, remove_style_tags: Option<bool>) -> PyResult<String> {
    // ...
    // Just to postpone error handling implementation
    Ok(inliner.inline(html).unwrap())
}

In this case, PyO3 will handle the error and raise a PanicException with information about where the error happened in the source code, which is quite helpful:

In [1]: import css_inline

In [2]: html = """
<html>
  <head>
    <style>@wrong { color:blue; }</style>
  </head>
  <body>
      <h1>Big Text</h1>
  </body>
</html>"""

In [3]: css_inline.inline(html)
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: ParseError("Invalid @ rule: wrong")', src/lib.rs:27:8
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---------------------------------------------------------------------------
PanicException                            Traceback (most recent call last)
<ipython-input-3-a0a2e78bb61b> in <module>
----> 1 css_inline.inline(html)

PanicException: called `Result::unwrap()` on an `Err` value: ParseError("Invalid @ rule: wrong")

However, if the end-user will hit such an error, it will be nice to have more information about the package's build environment to reproduce the error easier. The pyo3-built crate can do this - it provides the build meta-information, which can be included with the extension. First, we need to add new dependencies to the bindings' Cargo.toml file:

[build-dependencies]
# `built` requires `chrono` to provide the built-time in UTC
built = { version = "0.4", features = ["chrono"] }

[dependencies]
pyo3-built = "0.4"

Then a special build.rs file in the bindings/python directory with the following content:

fn main() {
    let src = std::env::var("CARGO_MANIFEST_DIR").unwrap();
    let dst = std::path::Path::new(
        &std::env::var("OUT_DIR").unwrap()
    ).join("built.rs");
    let mut opts = built::Options::default();
    opts.set_dependencies(true).set_compiler(true).set_env(true);
    built::write_built_file_with_opts(&opts, std::path::Path::new(&src), &dst)
        .expect("Failed to acquire build-time information");
}

Here is a reference for build scripts from the Cargo Book

And finally, include the build-time information to the Python module itself:

#[allow(dead_code)]
mod build {
    // Include the content of the `built.rs` file here
    include!(concat!(env!("OUT_DIR"), "/built.rs"));
}

#[pymodule]
fn css_inline(py: Python, module: &PyModule) -> PyResult<()> {
    // ...
    module.add("__build__", pyo3_built::pyo3_built!(py, build))?;
    Ok(())
}

This information includes the compiler version, build time, dependencies, build target and is available in the css_inline.__build__ attribute:

In [2]: css_inline.__build__
Out[2]:
{'build': {'rustc': 'rustc',
  'rustc-version': 'rustc 1.45.2 (d3fb005a3 2020-07-31)',
  'opt-level': '3',
  'debug': False,
  'jobs': 12},
 'info-time': datetime.datetime(2020, 8, 21, 12, 11),
 'dependencies': {'autocfg': '1.0.1',
 ...
 }
}

But what if we faced a segfault? For example, we used the C API directly like this:

use pyo3::ffi::{PyList_GET_ITEM, PyLong_AsLongLong};
use pyo3::AsPyPointer;

#[pyfunction]
fn first_item(object: &PyAny) -> i64 {
    let item = unsafe { PyList_GET_ITEM(object.as_ptr(), 0) };
    unsafe { PyLong_AsLongLong(item) }
}

#[pymodule]
fn css_inline(py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pyfunction!(first_item))?;
    Ok(())
}

In this silly example, the function gets the first element of a list and then converts it to i64. It works fine with non-empty lists and even throws exceptions when we pass a list of strings. However, it crashes the interpreter if we pass an empty list or some other object:

In [1]: import css_inline

In [2]: css_inline.first_item({})
[1]    24317 segmentation fault (core dumped)  ipython

LLDB can help us to debug the problem. We'll need a build with debug symbols - could be obtained with pip install -e .. And then we can reproduce the error:

$ lldb python
(lldb) target create "python"
Current executable set to 'python' (x86_64).
(lldb) r -c 'import css_inline; css_inline.first_item({})'
Process 25992 launched: '/home/stranger6667/.virtualenvs/css-inline-example/bin/python' (x86_64)
Process 25992 stopped
* thread #1, name = 'python', stop reason = signal SIGSEGV: invalid address (fault address: 0x340a)
    frame #0: 0x00007ffff71b037f css_inline.cpython-38-x86_64-linux-gnu.so`pyo3::ffi::listobject::PyList_GET_ITEM::h6e8d7eff02d88fcf(op=0x00007ffff75178c0, i=0) at listobject.rs:35:5
   32   #[cfg(not(Py_LIMITED_API))]
   33   #[inline]
   34   pub unsafe fn PyList_GET_ITEM(op: *mut PyObject, i: Py_ssize_t) -> *mut PyObject {
-> 35       *(*(op as *mut PyListObject)).ob_item.offset(i as isize)
   36   }
   37
   38   #[cfg(not(Py_LIMITED_API))]

You could also see where SIGSEGV happened and use lldb commands like bt to show the call stack and v to check the variable values. LLDB is an excellent tool that provides rich capabilities for debugging and might give a lot of insights into why a particular segmentation fault happened.

Packaging

Now, when the extension provides an interface to a Rust crate, we can appropriately pack it for easier installation. All the standard Python package metadata (as per PEP 566) could be added to the setup call as usual:

from setuptools import setup
from setuptools_rust import Binding, RustExtension

setup(
    name="css-inline",
    version="0.1",
    python_requires=">=3.5",
    license="MIT",
    author="Dmitry Dygalo",
    author_email="dadygalo@gmail.com",
    rust_extensions=[
        RustExtension("css_inline", binding=Binding.PyO3)
    ],
    zip_safe=False,
    # More metadata here
)

See more metadata in the core metadata specifications

To reduce the size of the resulting binary, you might want to strip not necessary symbols from it. It is possible by passing the strip argument with an integer value to the RustExtension class:

0 - Default. No symbols will be stripped
1 - Strip debug symbols
2 - Strip all symbols

Using two latter options under the hood translates to calling the strip utility with -S and -x, respectively. For comparison, the unstripped version of these bindings results in a ~1.2 MB wheel file, and the version without debug symbols is ~600 Kb.

Then to support PEP 518, we need to specify the build requirements in the pyproject.toml file:

[build-system]
requires = ["setuptools", "wheel", "setuptools-rust"]

At this point, we can adequately build a wheel package by running python setup.py bdist_wheel. It will create a whl package for your platform. To make binary wheels for other platforms, we'll use Docker and go through it in detail in the Releasing section.

However, for platforms, we don't build wheels for we can provide a source distribution, so the extension still could be compiled manually. The css_inline crate is specified via a local path, and its sources should be available during installation, it means they should be included in the distribution. As a workaround, we could do the following:

Temporarily put the css_inline sources to the bindings/python directory (e.g., via a link);
Change the dependency in the Cargo.toml file, so it points to the new css_inline location;
Create a source distribution;
Undo Cargo.toml changes and remove a link to the css_inline sources.

A small bash script will do the job, let's create a build-sdist.sh file in the bindings root:

#!/bin/bash
set -ex

# Create a symlink for css_inline
ln -sf ../../css_inline css_inline
# Modify Cargo.toml to include this symlink
cp Cargo.toml Cargo.toml.orig
sed -i 's/\.\.\/\.\.\/css_inline/\.\/css_inline/' Cargo.toml
# Build the source distribution
python setup.py sdist
# Undo changes
rm css_inline
mv Cargo.toml.orig Cargo.toml

I found this solution in the tokenizers repository. Let me know if there are better alternatives

We need MANIFEST.in to list what should be in the source distribution:

include Cargo.toml
include pyproject.toml
include rust-toolchain
include build.rs
recursive-include src *
recursive-include css_inline *
recursive-exclude css_inline/target *

It includes the rust-toolchain file, that indicates to cargo, that the stable toolchain is required. This file contains only the "stable" string.

Now the python setup.py sdist will produce a .tar.gz file that will contain everything needed to build an extension from source code.

Testing

To verify that our extension works as expected we can write tests on the Python level inside the tests directory:

# test_inlining.py
import css_inline

def test_simple():
    html = """
<html>
<head>
    <style>h1 { color:blue; }</style>
</head>
<body>
    <h1>Big Text</h1>
</body>
</html>"""
    expected = """<html><head>

</head>
<body>
    <h1 style=" color:blue; ">Big Text</h1>

</body></html>"""
    assert css_inline.inline(html, True) == expected

This test checks if the "style" tag is removed and the styles are inlined, it doesn't cover everything, but represents some baseline of functionality level we want to achieve. Note, we don't use the unittest framework in favor of a more straightforward pytest-style approach, but it could work here.

PyO3 supports Python 3.5 - 3.8, and the most popular way to run tests in multiple environments is tox. However, having local dependency implies a limitation - the sdist package should include the local dependency as we saw in the Packaging section. A simple workaround is to skip the sdist step and perform an editable install. Using this approach, we'll have faster compile time because setuptools_rust will create a debug build instead of the release one.

For this purpose, we could use the following tox.ini:

[tox]
skipsdist = True
envlist = py{35,36,37,38}

[testenv]
deps =
  setuptools_rust
  pytest
commands =
  pip install -e .
  python -m pytest tests {posargs:}

Then, running tox will execute tests for all declared Python versions. You could add -p command-line option there to run all jobs in parallel. Another useful addition would be to use the hypothesis library for property-based testing, which can discover many edge-cases in your code. A simple smoke test would look like this:

from hypothesis import strategies as st, given

@given(html=st.text(), remove_style_tags=st.booleans())
def test_smoke(html, remove_style_tags):
    with suppress(css_inline.InlineError):
        css_inline.inline(html, remove_style_tags)

This test will generate random strings and booleans and fail if something else than the InlineError occurs. I, personally, found a couple of segfaults in my own Rust / Python project by utilizing property-based tests and found them incredibly helpful.

The st.text() strategy is quite simple and is not likely to generate valid HTML at all, but hypothesis is highly configurable and provides us with a lot of ways to craft test data we need. See the documentation to learn how to do it

Benchmarking

Benchmarking is an important topic that often when done without care, may lead to imprecise results and wrong impressions. Many things could go wrong - input data, the way code is called, number of features used, the test machine configuration, and so on. At the same time, it might give at least a rough performance characteristic of different libraries. Take it with a massive grain of salt.

I am going to give an example of how we can compare different libraries. The candidates besides our library:

premailer. The most popular Python library for CSS inlining from all three. Uses Cython-based lxml package for HTML processing;
pynliner. Uses beautifulsoup4 without explicit lxml support, but uses lxml if installed;
inlinestyler. It also uses lxml.

Of course, the comparison is not fair, because our library has fewer features and therefore performs much less work than the others. To run benchmarks, we'll use the pytest-benchmark library and tox. Let's create a new benches/inliner.py file:

import inlinestyler.utils
import premailer
import pynliner
import pytest

import css_inline

SIMPLE_HTML = """<html>
<head>
    <title>Test</title>
    <style>
        h1, h2 { color:blue; }
        strong { text-decoration:none }
        p { font-size:2px }
        p.footer { font-size: 1px}
    </style>
</head>
<body>
    <h1>Big Text</h1>
    <p>
        <strong>Solid</strong>
    </p>
    <p class="footer">Foot notes</p>
</body>
</html>"""

@pytest.mark.parametrize(
    "func",
    (
        css_inline.inline,
        premailer.transform,
        pynliner.fromString,
        inlinestyler.utils.inline_css,
    ),
    ids=("css_inline", "premailer", "pynliner", "inlinestyler"),
)
@pytest.mark.benchmark(group="simple")
def test_simple(benchmark, func):
    benchmark(func, SIMPLE_HTML)

We can run it via tox as well:

[testenv:bench]
basepython = python3.8
deps =
  premailer
  pynliner
  inlinestyler
  setuptools_rust
  pytest
  pytest-benchmark
commands =
  # Release build
  python setup.py install
  python -m pytest benches/inliner.py {posargs:--benchmark-columns=mean}

And the output will look like this:

---------- benchmark 'simple': 4 tests -----------
Name (time in us)                   Mean
--------------------------------------------------
test_simple[css_inline]          23.5992 (1.0)
test_simple[premailer]          347.7987 (14.74)
test_simple[inlinestyler]     2,527.3290 (107.09)
test_simple[pynliner]         2,884.2000 (122.22)
--------------------------------------------------

Using a fair benchmark (not like the one above) on meaningful inputs, we can see how our code performs and what other libraries may do better in different circumstances.

Can inlining performance be improved? Sure! Check the lol-html crate for streaming HTML rewriting

Continuous integration

To ensure that everything works on each change, we could use a CI system, and the following example will cover GitHub Actions, but the approach could be applied for other CI providers as well.

At this point, we might want to run the following:

cargo clippy. To ensure our code follows common Rust idioms & lints;
cargo fmt. Provides consistent code formatting;
tox. Runs all Python tests.

A GitHub Actions workflow requires a yml file inside .github/workflows in the project root. Let's create build.yml file and add cargo jobs there:

name: ci

on:
  pull_request: {}
  push:
    branches:
      - master

jobs:

  fmt:
    name: Rustfmt
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: stable
          override: true
      - run: rustup component add rustfmt
      - uses: actions-rs/cargo@v1
        with:
          command: fmt
          args: --manifest-path bindings/python/Cargo.toml --all -- --check

  clippy:
    name: Clippy
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: stable
          override: true
      - run: rustup component add clippy
      - uses: actions-rs/cargo@v1
        with:
          command: clippy
          args: --manifest-path bindings/python/Cargo.toml -- -D warnings

These jobs use actions-rs and will fail on any formatting or linting errors. Python tests job goes under the jobs key and includes testing on Linux, Windows and Mac OS X:

# ...
jobs:
  # ...
  test-python:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        python-version: ['3.5', '3.6', '3.7', '3.8']

    name: Python ${{ matrix.python-version }} on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v2
        with:
          submodules: true
      - uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
          architecture: x64

      - run: python -m pip install tox
        working-directory: ./bindings/python

      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
          override: true

      - name: Run ${{ matrix.python }} tox job
        run: tox -e py
        working-directory: ./bindings/python

Releasing

When all regular things are covered, we are ready to publish our package. As the end-goal we need to:

Build wheels for all supported Python versions on Linux, Mac OS X, and Windows;
Build a source distribution;
Upload all artifacts to PyPI.

To centralize commands configuration and dependencies, we could extend our tox.ini:

[testenv:build-sdist]
deps =
  setuptools_rust
commands =
  ./build-sdist.sh

[testenv:build-wheel]
deps =
  setuptools_rust
  wheel
commands =
  python setup.py bdist_wheel

For release, we need another workflow, that will only run when we need to make a release - I prefer to distinguish such cases by git tags:

# .github/workflows/python-release.yml
name: Python Release

on:
  push:
    tags:
      - python-v*

This workflow will run only on git tags, that start with "python-v", for example, "python-v0.1.0"

The most simple step is creating a source distribution:

# ...
jobs:
  create_source_dist:
    name: Create sdist package
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: 3.7
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: stable
          override: true
      - name: Install Tox
        run: pip install tox
      - name: Build sdist
        working-directory: ./bindings/python
        run: tox -e build-sdist
      - uses: actions/upload-artifact@v2
        with:
          name: Distribution Artifacts
          path: bindings/python/dist/

This job will run tox -e build-sdist and store the resulting .tar.gz file in temporary per-build storage.

Building wheels for Windows and Mac OS X is straightforward - we need to call python setup.py bdist_wheel and store the created wheel file:

# ...
jobs:
  # ...
  create_macos_and_windows_wheels:
    name: Wheels for Python ${{ matrix.python-version }} / ${{ matrix.os }}
    strategy:
      matrix:
        os: [macos-latest, windows-latest]
        python-version: ['3.5', '3.6', '3.7', '3.8']
        architecture: [x64]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
          architecture: ${{ matrix.architecture }}
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: stable
          override: true
      - name: Install Tox
        run: pip install tox
      - name: Build wheel
        working-directory: ./bindings/python
        run: tox -e build-wheel
      - uses: actions/upload-artifact@v2
        with:
          name: Distribution Artifacts
          path: bindings/python/dist/

We build wheels only for 64-bit platforms for simplicity, but it is possible to build against more target architectures. Use this workflow as an example.

Learn more about the rustc platform support in this guide

Building wheels for Linux is slightly more complicated because, in general, compiled modules built on one Linux distribution might not work on other Linux distributions or even on the same distributions when they have different system libraries installed. To solve this problem, PEP 513 defines a set of rules, so the packages conforming to this PEP will work on many Linux systems (therefore, it is called "manylinux" policy).

Python Packaging Authority (PyPA) provides a set of Docker images to simplify building manylinux-compatible wheels. We will use the manylinux2014_x86_64 image, based on CentOS 7 (supported until 2024):

# ...
jobs:
  create_wheels_manylinux:
    name: Wheels for Python ${{ matrix.PYTHON_IMPLEMENTATION_ABI }} / Linux
    strategy:
      fail-fast: false
      matrix:
        # List of the language version - ABI pairs to publish wheels for
        # The list of supported is obtainable by running
        # `docker run quay.io/pypa/manylinux2014_x86_64 ls /opt/python`
        # Read more about compatibility tags in PEP 425
        # https://www.python.org/dev/peps/pep-0425/
        PYTHON_IMPLEMENTATION_ABI: [cp35-cp35m, cp36-cp36m, cp37-cp37m, cp38-cp38]
    runs-on: ubuntu-latest
    container: quay.io/pypa/manylinux2014_x86_64
    env:
      # Variable needed for PyO3 to properly identify the python interpreter
      PYTHON_SYS_EXECUTABLE: /opt/python/${{ matrix.PYTHON_IMPLEMENTATION_ABI }}/bin/python
    steps:
      - uses: actions/checkout@v2
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: stable
          override: true
      - name: Install Tox
        run: ${{ env.PYTHON_SYS_EXECUTABLE }} -m pip install tox
      - name: Build wheel
        working-directory: ./bindings/python
        run: |
          ${{ env.PYTHON_SYS_EXECUTABLE }} -m tox -e build-wheel
          # Ensure that the wheel is tagged as manylinux2014 platform
          auditwheel repair \
            --wheel-dir=./dist \
            --plat manylinux2014_x86_64 \
            ./dist/css_inline-*-${{ matrix.PYTHON_IMPLEMENTATION_ABI }}-linux_x86_64.whl
          # Remove `linux_x86_64` tagged wheels as they are not supported by PyPI
          rm ./dist/css_inline-*-${{ matrix.PYTHON_IMPLEMENTATION_ABI }}-linux_x86_64.whl
      - uses: actions/upload-artifact@v2
        with:
          name: Distribution Artifacts
          path: bindings/python/dist/

There are two essential distinctions from the Windows / Mac OS X job:

We need to explicitly specify the Python executable path since there are many of them installed;
auditwheel tool relabels wheels with the proper manylinux tag;

We prepared wheels for all major platforms and a source distribution, and then the last step is to upload these artifacts to PyPI:

# ...
jobs:
  # ...
  upload_to_pypi:
    needs:
    - create_macos_and_windows_wheels
    - create_wheels_manylinux
    - create_source_dist
    name: Upload Artifacts to PyPi
    runs-on: ubuntu-latest
    steps:
    - uses: actions/download-artifact@v2
      with:
        name: Distribution Artifacts
        path: bindings/python/dist/
    - name: Publish distribution package to PyPI
      uses: pypa/gh-action-pypi-publish@v1.2.2
      with:
        user: ${{ secrets.PYPI_USERNAME }}
        password: ${{ secrets.PYPI_PASSWORD }}
        packages_dir: bindings/python/dist/

This job requires you to specify two secrets in the repo settings - PYPI_USERNAME and PYPI_PASSWORD, and it will be executed after all artifacts are ready.

Big thanks to Samuele Maci who contributed the major part of these workflows

Summary

Of course, there are many more things we can improve here, but I hope that I shed some light on how Rust / Python projects may work and what are their standard components. The topics we covered could be applied to a broad range of projects, and I encourage you to try building your own so you can learn more in practice.

The project we used today is on GitHub. Also, you could find more examples in the PyO3 repo.

See the complete CSS inlining implementation in this GitHub repo

It is the last chapter of the "Rust for a Pythonista" series, where I tried to cover various aspects of Rust and Python integration and share my experience with it. If you have any comments, questions, or suggestions, feel free to reach me on Twitter.

Chapters:

Rust for a Pythonista #1: Why and when?
Rust for a Pythonista #2: Building a Rust crate for CSS inlining
Rust for a Pythonista #3: Python bindings

Sincerely,

Dmitry

❤ ❤ ❤

Rust for a Pythonista #3: Python bindings