Benchmarks on open-source CI

We took one CI job from each of several well-known open-source projects and ran the project's own job, its steps unchanged, on GitHub's hosted runner and on a Skipbox own machine. Every run is public in the skipbox-dev organization on GitHub, so you can open each log and read the time of every step.

Results

Each project's own job with its steps unchanged on GitHub's ubuntu-24.04 and a Skipbox own machine. Both 4 vCPU, one commit run again.

The table has every project we measured; the home page shows three of them. Warm is the second and later runs, cold the first. Each time in bold is the median of that side's runs, and every run behind it is linked under it with its own time. A time runs from the job's start to its end as GitHub's jobs API reports it, so time spent waiting in the queue isn't counted. A run that failed isn't counted either.

What was compared

• GitHub: the standard ubuntu-24.04 runner, in a public fork of the project. GitHub gives a public repository's standard runner 4 CPUs and 16 GB, twice the 8 GB the Skipbox machine had in these runs. Larger runners were never used.
• Skipbox: a VM made from Skipbox's production runner image, with 4 vCPU and 8 GB, the smallest plan's size on the days measured, on one of Skipbox's own servers. Since 14 November 2026 that plan has 16 GB, the same as GitHub's runner here. Between jobs it is set up and wiped exactly as an own machine is: credentials go after every job, and caches, Docker images, the checkout and Cargo's target/ stay.

The Skipbox server is in Japan and has an Intel Core Ultra 7 265F. The VM had 4 of its efficiency cores, which run at up to 4.6 GHz. Production jobs on that server get its performance cores, up to 5.3 GHz, first. Own machines on Hetzner Cloud run on other CPUs and weren't measured here, so the part of a gain that comes from these cores holds only on that server.

How each project was run

1. The commit is the one of the project's last successful run of that job on its own repository. Every run of a project, on both sides, uses that one commit.
2. The job's steps are copied from the project's workflow byte for byte. Only what the bench needs changes: the trigger is a manual dispatch, runs-on names the side, the matrix keeps its one Ubuntu entry, and secrets are removed.
3. Each side runs it three times in a row. The first run starts with no cache at all. The second and third run the same commit again, so they find everything the earlier runs built: the best case for both sides.

What makes the difference

Two things, and the table shows which one each project gains from.

• The kept disk. On GitHub every job starts on a new VM with an empty disk, and restores its caches over the network. On a Skipbox own machine the previous job's caches, Docker layers, checkout and build output are already on the disk, so an incremental Rust build compiles only what changed. tauri gains most this way: its warm runs are far faster than its cold ones on Skipbox, and the gap between the sides is much wider warm than cold. astro too: Turborepo's cache stays (turbo 2), so its warm runs replay every task instead of running it. See VM isolation for what stays and what is wiped.
• Faster cores. The same job runs faster on the Skipbox server's cores even with nothing kept on its disk, as each project's cold columns show. pytest gains mostly this way: its cold run is faster on Skipbox too, with nothing kept.

Publish your own benchmarks

You may benchmark Skipbox and publish the results without asking us. Name these beside the results:

• the plan, such as S (4 vCPU / 16 GB), or the trial
• the runs-on label, such as skipbox
• the dates of the runs