Walltime
This feature is currently in beta and is subject to change. If you are interested in testing it out, please reach out to us via Discord or email out support.
The walltime instruments allow measuring the walltime of your benchmarks directly in the CI. It leverages bare-metal runners managed and provided by CodSpeed to measure the performance of your benchmarks with low noise and high precision.
Usage with GitHub Actions
Requirements:
pytest-codspeed >= 3.0.0(only Python is supported for now)CodSpeedHQ/action >= 3.1.0
The CI setup is exactly the same as the one with the instrumentation but instead of running on a GitHub-hosted runner, you'll need to request a "codspeed-macro" runner. This runner is a bare-metal machine managed by CodSpeed and will provide you with a more stable and precise environment to run your benchmarks:
jobs:
benchmarks:
name: Run benchmarks
runs-on: ubuntu-latest
runs-on: codspeed-macro
steps:
- uses: actions/checkout@v4
# ...
# Setup your environment here:
# - Configure your Python/Rust/Node version
# - Install your dependencies
# - Build your benchmarks (if using a compiled language)
# ...
- name: Run benchmarks
uses: CodSpeedHQ/action@v3
with:
token: ${{ secrets.CODSPEED_TOKEN }}
run: "<Insert your benchmark command here>"
Your benchmarks will now run on a CodSpeed-managed runner, the action and the benchmark integration will automatically collect walltime data and you'll be able to see the new measurements in the CodSpeed dashboard.
By default, the macro runners are only available for the private repositories of your organization. If you want to use them on a public repository, you'll need to explicitly allow them from your GitHub organization settings (under Organization Settings > Actions > Runner groups > Default).
