Skip to main content

Ignoring a Benchmark

Reasons to ignore a benchmark​

Ignoring a benchmark should be a last resort when the source of flakiness cannot be immediately addressed. Some reasons to ignore a benchmark include:

  • The benchmark is I/O bound (network access, file-system access, etc.), leading to inconsistent results
  • The benchmarked source code has a non-deterministic behavior (random number generation, memory allocation, etc.)
  • The benchmark is consistently flaky for no apparent reason

Remember, the goal of CodSpeed is to provide accurate and reliable measurements. Ignoring a benchmark should be a temporary measure until the source of flakiness can be addressed.

Effects of ignoring a benchmark​

An ignored benchmark will be excluded from the overall measure of a report.

However, it will still be accessible at the bottom of the report, in the Ignored section. This allows you to keep track of ignored benchmarks, and keep access to their execution profiles and flamegraphs and re-enable them once the source of flakiness has been addressed.

Ignoring or un-ignoring a benchmark will regenerate and potentially modify the reports of the latest commits of the default branch and branches with opened pull requests.

How to ignore a benchmark​

Ignoring Permission

When working with an organization's repository, only the admins are allowed to ignore or un-ignore benchmarks.

To ignore a benchmark, follow these steps:

  1. Navigate to the benchmark's dashboard. You can do this by clicking on the button in the report or directly through the benchmarks page

  2. Once on the benchmark's dashboard, click on the Ignore button

  3. Enter a reason for ignoring the benchmark. This information will be used by CodSpeed to prioritize areas where we can reduce the flakiness of benchmarks.

  4. Click on the Ignore button to confirm the action

How to un-ignore a benchmark​

You can reverse the effects of ignoring a benchmark by simply clicking on the Un-ignore button on an ignored benchmark's page.