How to Interpret Open Oximetry Testing Results on the Device Data Page

When you click on a device to view its standard performance details, some of the terms can be unfamiliar. To help, we’ve put together this guide to explain some of these terms and concepts. 

NOTE: If you come across something that isn’t explained here, you can simply hover over the dark grey “i” button for a quick explanation. If you’re still unsure, feel free to contact us, and we’ll clarify!

——————————————————————————————————————————————————–

How do we determine how accurate the oxygen saturation measured by the pulse oximeters is? 

Pulse oximeters estimate oxygen saturation (SpO2), which is the percentage of hemoglobin in the blood that is bound to oxygen. Pulse oximeters do this non-invasively by shining light through the skin. The most accurate way to measure oxygen saturation, however, involves taking a blood sample from an artery and analyzing it with a specialized device called a blood gas analyzer.

To assess the accuracy of pulse oximeters, we compare the oxygen saturation from the pulse oximeter to the oxygen saturation from the blood gas analyzer. We do this in healthy adults, in a controlled lab setting, by gradually and safely lowering their oxygen levels from a saturation of 100% down to 70%. Specialized statistical methods like ‘ARMS’ (described elsewhere in this FAQ) are then used to evaluate the pulse oximeter’s performance. For more information about our process, please refer to our study protocol.

——————————————————————————————————————————————————–

Can you explain what the manufacturer-claimed ARMS (Root Mean Square Error) for the SpO2 range of 70-100% refers to?

The ARMS can be a confusing term to describe pulse oximeter performance, but it is the most common metric used and is required by regulatory bodies. ARMS stands for accuracy root mean square. It tells you, on average, how close the device’s readings are to your true blood oxygen levels, when the oxygen levels are between 70% and 100%, based on tests in healthy adults in controlled laboratory studies.

It’s helpful to understand that a lower ARMS means the device readings are more accurate and precise. For example, with an ARMS of 2%, the device’s readings would typically be within approximately 2-4% of the true oxygen level. So, if your true oxygen saturation is 94%, then the pulse oximeter reading may be between 90% and 98%. 

Under the standard performance information for a particular device, there are two different ARMS values listed: 1) An ARMS that’s reported in the product manual from the manufacture (Manufacturer claimed Arms), 2) The Independent ARMS that we calculate from our testing here at the UCSF Hypoxia Lab using our study protocol.

We follow current FDA guidance and ISO standards for testing and also add more elements to our testing protocols to better account for diversity of multiple factors like different skin colors and different levels of blood flow to the hands (factors that may impact performance of pulse oximeters).

For more information about ARMS, check out this detailed FAQ.

——————————————————————————————————————————————————–

What does the “Independent ARMS Study Cohort Size” mean?

This refers to how many healthy adults were tested using the OpenOximetry.org Protocol for this particular device. We share preliminary results after testing in at least 10 people (which is the minimum number required by the FDA guidance and ISO standards as of October 2024), but we continue testing the devices on as many different participants as possible to improve the diversity of people included. We are also waiting for updated guidelines to know the best number of participants for these tests.

——————————————————————————————————————————————————–

How is skin pigment defined?

Skin pigment refers specifically to the amount and type of melanin in your skin. People with darker skin have more melanin, while those with lighter skin have less.

We’re working hard to understand how skin pigment affects pulse oximetry readings. To learn more about how we measure skin pigment in our studies, check out our Skin Color Quantification page and our FAQ on skin pigment.

——————————————————————————————————————————————————–

Why might openoximetry.org results be different from what a manufacturer or other testing has reported?

While our testing follows FDA guidance and ISO standards for evaluating pulse oximeter performance, some aspects of our procedures may differ from those used in other testing labs. For instance, we test devices on a different group of people than those used by other labs, and device performance can vary between groups. Additionally, one way our testing may differ, while still adhering to ISO standards and FDA guidelines, is in how we modify participants’ physiology. For example, in pulse oximeter testing on the fingers, it is common practice (and allowed under current regulations as of October 2024) to warm participants’ hands to improve circulation and enhance the signal. This may result in better performance for some devices during warming than without warming. However, for the Open Oximetry Project, we do not routinely warm participants’ hands during testing.

——————————————————————————————————————————————————–

Why might openoximetry.org results change over time?

Our results will continue to evolve as we continue testing devices in more people,  learning more about how to best measure performance, and as regulatory recommendations for testing change. This is an ongoing process, and the science is still developing, so it’s not perfect yet. There are many factors that affect how pulse oximeters perform, and we can’t account for all of them with the current testing protocols. As we collect more data from a wider range of devices and diverse groups of people, our understanding and results may change. So, it’s a good idea to check back regularly for updated information!

——————————————————————————————————————————————————–

What does pulsatility amplitude mean?

Pulse oximeters look for the pulsatile (i.e. beating or changing) signal coming from the blood vessels to measure oxygen levels in the blood.  Some devices report the pulsatility in various ways like perfusion index, pulsatility amplitude, percent modulation or other ways. This gives us an idea of how strong the blood flow signal is in the area being measured. For more information, check out our FAQ on perfusion.

——————————————————————————————————————————————————–

What is differential bias?

Differential bias (sometimes called disparate bias) refers to how much a pulse oximeter’s performance can vary between people with light skin and those with dark skin. To measure this, we compare how much the device over- or under-estimates blood oxygen levels (SpO2) in people with very light skin versus those with very dark skin*.

*Using modeling of skin pigment and SpO2 data, very light skin and very dark skin are defined as having an difference in ITA of 100. Read more about differential bias here.