Mind the scaling governor while timing code

Here’s a small bit of info that could save you some sweat and tears: when timing your code on Linux, make sure your performance scaling governor is set appropriately.

This came up back when I was trying to do frame synchronization in software for desert train. In general, when you are trying to measure how long a specific portion of your code takes to execute (maybe using something like POSIX’s clock_gettime()), you may see measurements oscillate around multiple points in a strange way, in a multimodal distribution.

In the case I mentioned, and in multiple cases since then, I was measuring the same piece of code that ran the same workload each time. For a few frames, the execution time tripled or even quadrupled, until it went back to the one I would expect for a few frames, moving back and forth between the two.

What I saw was the result of an otherwise useful and desirable behavior of the CPU frequency scaling subsystem of Linux. It seems that each frame (and their succession) took a short enough time to compute that the scaling governor kept dropping the CPU to a lower frequency-voltage configuration. While this is great when you want execution to be efficient, it’s no good for measuring wall-clock time during profiling.

Specifically, this was a result of using the schedutil scaling governor, which implements “scheduler-driven CPU frequency selection”.

Configuration

Before we proceed, I’m going to echo the warning from the related page of the Arch Linux wiki: Use CPU monitoring tools (for temperatures, voltage, etc.) when changing the default governor.

The following is one way to monitor your current core frequencies in real time.

$ watch cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_freq

To monitor your temperatures, you could use the following instead.

$ watch cat /sys/class/thermal/thermal_zone*/temp

One way to get consistent timings is to temporarily set the performance governor, which will run the CPU at the maximum frequency set by the scaling_max_freq policy limit. You can activate a governor on every available CPU manually in the following way. If you are not running this as a superuser, you have to add your “super user do” before tee, which, for me, is sudo.

# echo <governor> | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

This setting is temporary, and will reset to the default cpufreq.default_governor kernel option on boot.

Even with these fairly standard tweaks, you shouldn’t just believe random people on the internet about their safety. With that said, I personally only saw my temperatures increase appreciably when I ran a heavier workload, long after I set my scaling governor to performance. The frequencies were consistently higher though, so it may have just come down to cooling.

After you’re done, don’t forget to set it back to your default, to let your processor kick back again.