Librsvg's test suite is now in Rust

Some important changes are afoot in librsvg.

Changes to continuous integration

Some days ago, Dunja Lalic rewrote the continuous integration scripts to be much faster. A complete pipeline used to take about 90 minutes to run, now it takes about 15 minutes on average.

Graph with pipeline timings, which shrink drastically

The description of changes is interesting. The idea is to make tests fail as fast as possible, close to the beginning of the pipeline. To speed up the whole pipeline, Dunja did the following:

Move the cargo check stage to the beginning. This test means, "does this even have a chance of compiling?".
Have the code style and formatting tests, cargo clippy and cargo fmt, run in parallel with the unit tests. These lints can fail, but they are easy to fix after one is finished modifying the main code.
Run the unit tests and the smoke tests in debug mode, so they compile quickly.
Run the complete integration test suite in release mode. This takes longer to compile, but there are some slow tests that benefit a lot from faster execution.
Move the release tests until the end, and only run them once a week — or whenever, by hand. These take a good amount of time to run, because they do a full make distcheck and autotools is slow. Even then, now these tests take 30-40 minutes, instead of the 90 from before.
Between each stage of the pipeline, don't cache what doesn't help reduce compilation time. It seems that keeping around a big cache, with the whole build target, between each pipeline stage can be worse than not having one at all.

Complete pipeline with all the stages

Test suite in Rust

Beteen Sven Neumann, Dunja Lalic, and myself we have finally ported the test suite to Rust: all of librsvg's tests are now in Rust, except for the C API tests. We had to do a few things:

Review the old tests and remove some obsolete ones.
Port each of the test modules to Rust. They are small, but each one has special little things — test for crashes in the XML loading code, test for crashes during rendering, test the library's security limits.
Fix the small tests that come as part of the documentation.
Untangle the reference tests and port them to Rust.
Move little chunks of code around so the unit tests and integration tests can share utilities to compare images, compute file paths for test fixtures, etc.

The most complicated thing to port was the reference tests. These are the most important ones; each test loads an SVG document, renders it, and compares the result to a reference PNG image. There are some complications in the tests; they have to create a special configuration for Fontconfig and Pango, so as to have reproducible font rendering. The pango-rs bindings do not cover this part of Pango, so we had to do some things by hand.

Anyway, the tests are now in Rust. One nice thing is that now the tests run automatically in parallel, across all CPU cores, so we save on total testing time.

What's next: cargo-c and publish to crates.io

We want to be able to publish librsvg in crates.io as a normal crate; this implies being able to compile, test, and publish entirely from Cargo. The compilation and testing part is done.

Now, we have to reorganize the code so it can be published to crates.io. Librsvg comes in three parts, rsvg_internals with the implementation of the library, librsvg with the traditional C API, and librsvg_crate with the Rust API. However, to publish the Rust API to crates.io, it would be more convenient to have a single crate instead of one with the internals and one with the API.

The next step is thus to reorganize the code:

Make it possible to implement the C API as a compile-time option on top of the normal Rust code. We want to use cargo-c to compile the traditional shared library librsvg.so, instead of depending on C tools for compiling and linking.
Combine rsvg_internals and librsvg_crate in a single crate, to publish them together. Crates.io has a 10 MB limit per crate; now that the test suite lives in a separate tests crate, this shouldn't be a problem.
I would like to polish the public error types before publishing the Rust API; right now they expose some implementation details that are of no interest to callers of the library.

What remains to be ported to Rust?

Only two things, which amount to less than 900 lines of C code:

rsvg-convert - the command-line program that everyone uses to convert SVG to PNG and other formats. Fortunately, Sven Neumann wrote some fantastic tests for rsvg-convert, as it is like an API that we need to keep stable: if we change the command-line options or the program's behavior, we would break everyone's scripts.
The gdk-pixbuf module for loading SVG. Alberto Ruiz has started porting it to Rust. The generic part of this code could later serve to wrap other Rust image codecs and plug them to gdk-pixbuf.