Skip to main content

Walltime

Open Beta

This feature is currently in open beta and is subject to change. You can test it out for free. If you have any feedback, please reach out to us via Discord or email our support.

The walltime instruments allow measuring the walltime of your benchmarks directly in the CI. It leverages bare-metal runners managed and provided by CodSpeed to measure the performance of your benchmarks with low noise and high precision.

Example of a walltime benchmark run

Supported languages and integrations​

At the moment, the walltime instrument is supported for the following languages and integrations:

  • Python
    • pytest-codspeed >= 3.0.0
  • Rust
    • codspeed-divan-compat >= 2.8.0
    • codspeed-criterion-compat >= 2.9.1

If you want to use the walltime instrument with a different language or integration, please reach out on Discord or email our support.

Usage with GitHub Actions​

Requirements:

  • CodSpeedHQ/action >= 3.1.0

The CI setup is exactly the same as the one with the instrumentation but instead of running on a GitHub-hosted runner, you'll need to request a "codspeed-macro" runner. This runner is a bare-metal machine managed by CodSpeed and will provide you with a more stable and precise environment to run your benchmarks:

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: ubuntu-latest
    runs-on: codspeed-macro
    steps:
      - uses: actions/checkout@v4
      # ...
      # Setup your environment here:
      #  - Configure your Python/Rust/Node version
      #  - Install your dependencies
      #  - Build your benchmarks (if using a compiled language)
      # ...
      - name: Run benchmarks
        uses: CodSpeedHQ/action@v3
        with:
          token: ${{ secrets.CODSPEED_TOKEN }}
          run: "<Insert your benchmark command here>"

Your benchmarks will now run on a CodSpeed-managed runner, the action and the benchmark integration will automatically collect walltime data and you'll be able to see the new measurements in the CodSpeed dashboard.

Caches

If you're using caches in the GitHub action workflow, make sure your cache keys include runner.arch to avoid cache misses since CodSpeed Macro runners are running on the ARM64 architecture.

For example:

- uses: actions/cache@v4
  with:
    path: /home/.cache/pip
    key: pip-${{ hashFiles('pyproject.toml') }}
    key: pip-${{ runner.arch }}-${{ hashFiles('pyproject.toml') }}
Multiple benchmark processes

With the walltime instrument, you should try and avoid running multiple benchmark processes in parallel since this can lead to noisy measurements.

Thus, using pytest-xdist or similar tools is not recommended.