Impact Metrics

Benchmark performance impact

The performance impact denotes an improvement or regression in performance of a benchmark. It is calculated by comparing the benchmark time measurement of the head commit with the time measurement of the base commit.

$$impact = \frac{speed - baseSpeed}{baseSpeed}$$

A negative performance metric means that the benchmark is slower than the previous commit. The closer its value is to -1, the slower it is.

$$-1 \lt impact < 0$$

A positive performance metric means that the benchmark is faster than the previous commit. Its value can go up to +Infinity to denote massive speed improvements.

$$0 \lt impact \lt +\infty$$

Naturally, when the benchmark is as fast as the previous commit, the performance metric is 0.

Regression threshold

On the settings page of a project, you can set a threshold for a regression to be considered a regression. By default, this value is set to 10% (which is equivalent to 0.1). The value can be set from 0% to 50% by an admin of the project. More information about setting the threshold can be found in the customization documentation.

Commit performance impact

To get the overall performance impact of a commit, we aggregate all the benchmark impacts. n being the number of benchmarks.

Regression threshold exceeded

If there is a regression above the threshold, the overall commit impact will be the biggest regression impact.

$$commitImpact = \min_{0\leq i\lt n} impact[i]$$

example

With impacts: [0.1, 0, -0.3] and a threshold of 0.25, the overall commit impact will be -0.3.

Improvement threshold exceeded

If there is an improvement above the threshold, the overall commit impact will be the maximum improvement impact.

$$commitImpact = \max_{0\leq i\lt n} impact[i]$$

example

With impacts: [0.1, 0.3, -0.2] and a threshold of 0.25, the overall commit impact will be 0.3.

No threshold exceeded

Finally, in the remaining cases, a geometric mean is calculated from all the benchmark performance impacts.

$$commitImpact = \left(\prod_{i=0}^{n-1} 1+impact[i]\right)^{\frac{1}{n}} - 1$$

example

With impacts: [0.1, 0.3, -0.2] and a threshold of 0.5, the overall commit impact will be approximately 0.0459.

info

A geometric mean will give more relevant results than an arithmetic mean for this kind of measure as it will be less sensitive to outliers.

Baseline report selection

To create a performance impact, we need to compare the execution speed of the benchmarks against a baseline of those benchmarks' execution speed. Depending on the context of the run, the baseline report can be different.

Pull Request

When triggering a CodSpeed run on a pull request between a head branch and a base branch, the baseline report will be the report of the latest commit of the base branch with a CodSpeed run.

info

Checked-out commit of a Pull Request in GitHub Actions By default, when using the action/checkout in GitHub Actions, the checked-out commit of a pull request will be the merge commit of the pull request. This means that GitHub will create a merge commit M between the base branch and the head branch, and use it as the checked-out commit.

In this case, the report will be between the HEAD commit C of the base branch and the merge commit M. The report will thus only include the changes of commits B1 and B2.

info

Every time a new commit is pushed on the feat-branch, the baseline report will be updated to the latest commit of the base branch with a CodSpeed run.

Branch

When triggering a CodSpeed run following a push on a branch, the baseline report will be the report of the closest commit of the branch.

info

In this example, a report already exists for the commit B of the main branch. A new commit C is pushed on the main branch. The baseline report for C will be the report of the commit B.

Performance impact Gauge

The performance impact gauge is a visual representation of the performance impact, displayed in multiple places of the CodSpeed UI.

Some examples of the gauge with their corresponding impact values:

-0.75

-0.2

0

0.3

1.5

info

To make it easier to spot regressions and improvements, the mapping between the actual performance impact value and the gauge is not linear.