CPU
CodSpeed instruments your benchmarks to measure the performance of your code. A benchmark will be run only once and the CPU behavior will be simulated. This ensures that the measurement is as accurate as possible, taking into account not only the instructions executed but also the cache and memory access patterns. The simulation gives us an equivalent of the CPU cycles that includes cache and memory access.
Once we have the number of cycles for a benchmark, we transform it into an execution time measurement by using the following formula, where
FREQUENCY is a constant set to the frequency (number of instructions executed per second) of a real CPU:
We then calculate the execution speed of the benchmark by taking the inverse of the execution time:
This is the displayed metric in the CodSpeed reports.
A performance increase of a benchmark will increase its execution speed. Same for a performance regression. However, if execution time was used, a performance increase of a benchmark would result in a decrease in its execution time. This would be counter-intuitive.
System Calls
System calls play a critical role in the performance of software, but they present unique challenges for accurate measurement. A system call is a request made by a program to the operating system's kernel, typically for I/O operations such as reading from or writing to files, communicating over a network, or interacting with hardware devices.
Due to their nature, system calls introduce variability in execution time. This variability is influenced by several factors, including system load, network latency, and disk I/O performance. As a result, the execution time of system calls can fluctuate significantly, making them the most inconsistent part of a program's execution time.
To ensure that our execution speed measurements are as stable and reliable as possible, CodSpeed does not include system calls in the measurement. Instead, we focus solely on the code executed within user space(the code you wrote), excluding any time spent in system calls. This approach allows us to provide a clear and consistent metric for the execution speed of your code, independent of your hardware and all variability that it can create.
Still, the wall time spent on system calls is recorded and this data is available in the profile view, providing insight into how much time is consumed by system interactions. While these times are not included in the overall execution speed metric, they offer valuable information for performance analysis.
In the future, we plan to enhance CodSpeed by emulating system calls. This will allow us to more accurately anticipate the performance impact of system calls, further improving the reliability and comprehensiveness of our performance measurements.