Accuracy and precision

As I mentioned at the top, throughout this article we are going to looks at the accuracy and precision of power meters. Therefore, before we get too much further, it’s a good idea to start with a quick explanation of these two concepts

The dartboard below can best illustrate accuracy:

All the darts are close to the centre but they are scattered around the bullseye.  The darts are accurate in that none of them are far away from the bullseye, but they aren’t very precise as the darts are scattered.

Accuracy is important when it comes to power meters as it allows one person’s power data to be compared to another. It also allows those who are lucky enough to have power meters on multiple bikes to compare their output on one bike to the other.

Average power comparison and accuracy

Normally when we compare power outputs, it is over a certain period of time – maybe the length of a climb, a TT or even an interval in training. This is exactly what happened in the research. The average power for a certain period of time was compared with the true power value

The accuracy of a power meter gives us an understanding of how close it is measuring to the true power value. In our dartboard example this is how close each dart is to the bullseye. In power terms, essentially we are asking if the 300w being measured on a power meter is the same as a true 300w.

We can see the results in the chart below.

What we need to look at in the chart is the mean deviation (%), which represents the average accuracy across all power meters from the specific brand. Essentially it is a comparison of the average position of the darts to the bullseye across all tests of all models of one brand. Let’s use SRM as an example, which shows values of -0.5% ±2.4%. This means across the test, SRM power meters overall under read by -0.5% – however, there was a range of accuracies; if we look at the average difference between this average and each individual SRM power meter (the standard deviation for those who remember GCSE maths!), we get a figure of ±2.4%.

We’re quite used to seeing power meters manufacturers stating their claimed accuracy – for example, Stages power meters are advertised as ±2%. However, what we can see from the study is that most power meters actually fell outside of their advertised level of accuracy in terms of mean deviation, given here in brackets: Stages (±2%), Power2max (±2%), Polar (±2.5%), Verve (±0.37%) and Rotor (±1%). Only SRM (±1%), Powertap (±1.5%), Quarq (±1.5%) and Garmin (±2%) were within their stated level of accuracy.

I should note here that the Stages discrepancy could be explained by the fact that it only measures on one side. Therefore, if the riders who tested the Stages had a right leg bias this may explain why the Stages under reads slightly.

Precision

Despite all that, the precision of a power meter is arguably of much greater importance than how accurate is it.

In this research, the coefficient of variance can be considered a good measure of precision. Basically what this is measuring in our dartboard example is how far the darts are from one another.

Precision essentially means that the numbers are reliable test after test. So the 300w your power meter is measuring one day is exactly the same as the 300w the next day. To be able to track your performance and train effectively, precision is more important than accuracy.

When considering the coefficient of variance numbers, you need to keep in mind that a lower number is better. For example, overall across the three Verve power meters tested the coefficient of variance was only 0.6%. On average, across those three Verve power meters, if you measure 300w on one day the most you could expect that to differ by on another day is 1.8w for the same power.

The only unit that comes out badly in terms of average precision is the Stages power meter, which had a coefficient of variance of two per cent. Here, 300w might be 300w on one day and 294w or 306w the next day.

Now just as with the average accuracy, the average precision doesn’t tell the entire story. We can also look at the precision of the individual power meters across multiple tests (figure three).

This graph shows some interesting results. To interpret the results, if the dots are closer to the x axis then the precision of that unit is good, so the higher the dot the poorer the precision.

A number of things jump out from this.

Firstly, the SRM, Powertap, Rotor, Quarq and Verve power meters all display good precision and, with each of these, the values are under 1.5% – on any given day 300w may range from 295.5 to 304.5w. That’s not bad at all and probably less than the daily variation in your form.

However, we should also look at where those dots fall. With SRM, Powertap and Verve the dots take a pyramid shape, so most power meters are in the 0-1% range rather than the 1-2% range. So, if you were to buy a power meter from these brands, you are more likely to get a power meter with a coeffecient of variance of less than one per cent, as opposed to one with between one and two per cent variance.

The worst performer on the test was the Stages power meter. One unit showed a coefficient of variance of 6% – that means your 300w could be as much as 282-318w, which is well outside of what is useful when it comes to trying to ride to power or monitor your performance. What is also strange about the Stages power meters is that while some units were very precise, others were very imprecise.

Let me put this into context. With the least accurate power meter with the worst precision, your true 300w may be being measured as 256-298w on any given day. On the other hand, with the most accurate and precise power meter a true 300w could be measured as 297-300w.

Second-to-second comparison

So far we have looked at the average power over a longer period of time and how accurate and precise that measurement is. However when we ride we don’t maintain a perfectly smooth power output. Instead, power output on the road is what we call ‘stochastic’ – it varies greatly from second to second.

To put this into context, a 300w average power output for 10 seconds may look something like this on a second-to-second basis:

300w, 305w, 295w, 298w, 307w, 295w, 302w, 300w, 294w, 304w

What we have to consider is that the relevant precision and accuracy of power meters is affecting the recorded power numbers on a measurement-by-measurement basis. So for the same power 300w power output for 10 seconds the recorded power might look something like this:

307w, 298w, 304w, 293w, 305w, 291w, 295w, 306w, 311w, 290w

Overall the average power for the 10 seconds is identical, but if we were looking for a peak one-second power within the effort we would get big differences between the true value and the measured value.

What have we learnt from this?

The quick conclusion to draw is that your power meter might not be telling you exactly what you think it is. Indeed, the 300w you are seeing on your computer might not be quite the same as the 300w your ride partner is apparently producing. On top of this, with some power meters it appears that 300w today might not even be the same as 300w tomorrow.

In short, some power meters are more accurate and precise than others. When choosing a power meter, I would urge you to look at the data presented in this article about the accuracy and precision of various power meters, as well as considering other factors that may affect your purchasing decision, like budget, bike compatibility and aesthetics. At the end of the day your power meter is a training aid, however to be useful in that it needs to be providing you with, at the very least, precise data.

One final thing to mention is that all power meters in this study were calibrated via a zero offset before each test (and where possible), so the levels of accuracy and precision we see here represent the best performance you are going to get from a power meter. It is likely that once you add in real word conditions, the levels of error will increase.

Therefore, to maximise the accuracy and precision of any power meter, make sure you look after it. That means calibrating it where possible and making sure it is well charged before each ride. Another tip is to let it acclimitise to the outside temperature before starting your training. Finally, a great pro tip and something I get the riders I coach to do is to recalibrate your power meter before the first effort of the day. That means you know you are getting the most accurate data you can, when it really matters.