How to “validate” newly developed cardiac output monitoring devices

In the past decade, technological advances have catalyzedthe development of (advanced) hemodynamic monitoringdevices from invasive towards less invasive methods.Almost every month a new device appears on the marketclaiming to measure or estimate hemodynamic variables ina minimal or non-invasive fashion. These devices allowclinicians to broaden the scope beyond traditional pressure-based hemodynamic monitoring and by now, flow(-related)variables such as cardiac output (CO) can be assessed thisway in almost all patients in the anesthetic or critical caresetting. Hopefully, these technological advancements proveadvantageous in terms of patient outcome in the (near)future. However, an accurate and precise estimation of theassumed-to-be-measured hemodynamic variable is anabsolute prerequisite before such new devices can beclinically implemented, or even before outcome-relatedstudies can be performed. Additionally, given the (hemo-dynamic) heterogeneity of various patient populations (e.g.patients with septic shock versus patients with cardiogenicshock), both accuracy and precision should be investigatedin the relevant patient population(s). Therefore, researchersall over the world are stimulated to perform comparisonstudies in which new devices, software versions or sensorrevisions are compared with their clinical reference meth-ods or ‘‘gold standards’’, and journals are overflooded withmanuscripts on such evaluation studies.The basis for statistical reporting for such studies wasset almost 30 years ago by Bland and Altman in their land-mark paper [1], in which they introduced their famous‘‘Bland–Altman plot’’. In this (scatter)plot, agreementbetween two measurement methods is assessed by plottingthe mean of the measurements against the difference of themeasurements and it allows the calculation of the limits ofagreement (LOA; 1.96 9 SD of the bias) as a measure ofreproducibility.Assuch,theBland–Altmanplotappreciatestwo important aspects: it assesses method agreement basedon the ‘‘closeness’’ of individual data points and it does notrequire the definition of a gold standard, i.e. it assumes nofixed‘‘true’’value.Sinceitsintroduction,theBland–Altmanplot (initially developed for comparing CO measurements)has become the minimal standard for statistical reporting ofmethod comparison studies in general [2].While the Bland–Altman plot remains highly popular, itbears some important limitations that are especially rele-vant for CO method agreement studies. At first, the plotonly provides an estimation of agreement in the light of alinear relationship between the two measurement methods,and it does not take the magnitude of these observationsinto consideration, while this is highly important as a‘‘closer’’ agreement is required for a CO value of 1.5 L/mincompared to a CO of 3.5 L/min. To overcome this issue,Critchley and Critchley [3] introduced percentage errors(calculated as the LOA divided by the mean of the mea-surements) to compensate agreement for the magnitude ofmeasurement. In CO agreement studies, the calculation ofpercentage errors is well adopted and there is generalconsensus for accepting new CO measurement methods ifthese devices meet the so-called Critchley criteria, whichmeans that the percentage error is below 30 %.As all clinicians will be aware of, CO is not a staticvariable and changes continuously secondary to complex