C++

To use CodSpeed in your C++ codebase, you can use CodSpeed's google_benchmark library, which is a compatibility layer to run both instrumented and walltime CodSpeed benchmarks.

Installation

The recommended way to use CodSpeed's google_benchmark is through cmake. Add the following to your CMakeLists.txt:

CMakeLists.txt

cmake_minimum_required(VERSION 3.12)
include(FetchContent)

project(my_codspeed_project VERSION 0.0.0 LANGUAGES CXX)

# Enable release mode with debug symbols to display useful profiling data
set(CMAKE_BUILD_TYPE RelWithDebInfo)

set(BENCHMARK_DOWNLOAD_DEPENDENCIES ON)

FetchContent_Declare(
    google_benchmark
    GIT_REPOSITORY https://github.com/CodSpeedHQ/codspeed-cpp # Target the codspeed cpp repository
    SOURCE_SUBDIR google_benchmark # Make sure to target the google_benchmark subdirectory
    GIT_TAG main # Or chose a specific version or git ref, check the releases page on the repository
)

FetchContent_MakeAvailable(google_benchmark)

# Declare your benchmark executable and its sources here
add_executable(my_benchmark_executable benches/bench.cpp)

# Link your executable against the `benchmark::benchmark`, the `google_benchmark` library
# Note: the first argument must match the first argument of the `add_executable` call
target_link_libraries(my_benchmark_executable benchmark::benchmark)

note

Checkout the releases page if you want to target a specific version of the library.

tip

This examples is a dedicated CMakeLists.txt file for the benchmark executable.
You can also add an executable target to your existing project's CMakeLists.txt. Make sure to link this target against the benchmark::benchmark library.

Usage

Creating benchmark

Let's create an example benchmark

main.cpp

// Define the function under test
static void BM_StringCopy(benchmark::State &state) {
  std::string x = "hello";

  // Google benchmark relies on state.begin() and state.end() to run the benchmark and count iterations
  for (auto _ : state) {
    std::string copy(x);
  }
}
// Register the benchmarked to be called by the executable
BENCHMARK(BM_StringCopy);

static void BM_memcpy(benchmark::State &state) {
  char *src = new char[state.range(0)];
  char *dst = new char[state.range(0)];
  memset(src, 'x', state.range(0));
  for (auto _ : state)
    memcpy(dst, src, state.range(0));
  delete[] src;
  delete[] dst;
}

BENCHMARK(BM_memcpy)->Range(8, 8 << 10);

// Entrypoint of the benchmark executable
BENCHMARK_MAIN();

Checkout the Google benchmark user guide for more advanced usage of the library.

Compiling and running benchmarks

To build the benchmark executable, run

The CODSPEED_MODE flag

Please note the -DCODSPEED_MODE=instrumentation flag in the cmake command.
This will enable the CodSpeed instrumentation mode for the benchmark executable, where each benchmark is run only once on a simulated CPU.

You can also use -DCODSPEED_MODE=walltime if you are building for walltime codspeed reports, see dedicated documentation for more information.

If you omit the CODSPEED_MODE cmake flag, CodSpeed will not be enabled in the benchmark executable, and it will run as a regular benchmark.

Debug symbols

In order to get the most out of CodSpeed reports, debug symbols need to be enabled within your executable.
In the example above, this is done by setting CMAKE_BUILD_TYPE to RelWithDebInfo.

Testing the benchmarks locally

Simply execute the compiled binary to run the benchmarks.

Congratulations ! 🎉 You can now run those benchmark in your CI to get the actual performance measurements.

Running the benchmarks in your CI

To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to detect the CI environment and properly configure the instrumented environment.

tip

If you want more details on how to configure the CodSpeed action, you can check out the Continuous Reporting section.

Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the main branch and every pull request:

.github/workflows/codspeed.yml

name: CodSpeed

on:
  push:
    branches:
      - "main" # or "master"
  pull_request:
  # `workflow_dispatch` allows CodSpeed to trigger backtest
  # performance analysis in order to generate initial data.
  workflow_dispatch:

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build the benchmark target(s)
        run: |
          mkdir build
          cd build
          cmake -DCODSPEED_MODE=instrumentation ..
          make -j

      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v3
        with:
          run: ./build/my_benchmark_executable # Replace with the proper executable path
          token: ${{ secrets.CODSPEED_TOKEN }}

Usage without cmake

While cmake is the recommended way of using the library, it should be usable by directly including the code in your project through a submodule. If you run into issues integrating CodSpeed's google_benchmark library with your project, please reach out and open an issue on the codspeed-cpp repository.

Recipes

Running benchmarks in parallel

If your benchmarks are taking too much time to run under the CodSpeed action, you can run them in parallel to speed up the execution.

Running benchmarks in parallel CI jobs

To parallelize your benchmarks, first split them in multiple executables that each run a subset of your benches.

CMakelists.txt

# Create individual benchmark executables
set(BENCHMARKS first_bench second_bench third_bench)

# Add `bench_name` target with `bench_name.cpp` source for each bench listed above
foreach(benchmark IN LISTS BENCHMARKS)
  add_executable(${benchmark} benches/${benchmark}.cpp)
  target_link_libraries(${benchmark}
    benchmark::benchmark
  )
endforeach()

# Create a custom target to run all benchmarks locally
add_custom_target(run_all_benchmarks
  COMMAND ${CMAKE_COMMAND} -E echo "Running all benchmarks..."
)
# Register each benchmark target as a dependency of
foreach(benchmark IN LISTS BENCHMARKS)
  add_custom_command(
    TARGET run_all_benchmarks
    POST_BUILD
    COMMAND ${CMAKE_COMMAND} -E echo "Running ${benchmark}..."
    COMMAND $<TARGET_FILE:${benchmark}>
    WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
  )
endforeach()

Then update your CI workflow to run benchmarks executable by executable

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: ubuntu-latest
    strategy:
      matrix:
        target: [first_bench, second_bench, third_bench]
    steps:
      - uses: actions/checkout@v4

      - name: Build the benchmark target
        run: |
          mkdir build
          cd build
          cmake -DCODSPEED_MODE=instrumentation ..
          make -j ${{ matrix.target }}

      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v3
        with:
          run: ./build/${{ matrix.target }}
          token: ${{ secrets.CODSPEED_TOKEN }}