Why evaluate AI products before use?
There are over 1000 AI products available for sale in radiology (FDA)
An ever-increasing number of AI products offers the potential to bring huge benefits to healthcare providers and patients.
But how can you be confident that a product works for your needs and patient populations?
And how can you objectively choose between different options on the market?
The need for local validation
Even when an AI product has been assessed in regulation, it is still important to check how it works for your local use case before putting it into use on patients. Read more in a Case Study on our blog.
Unlike conventional software, AI products consist of models created by learning from training datasets. The performance of these models depends heavily on the training data as well as the learning process, but these are often unavailable in commercial products, resulting in a risk of ‘black-box’ behaviour.
In particular, it is important that the training data scans are representative of the population scans the AI is to be used on. Variations could be caused by differences in the demographics of the population, the pathologies assessed or the devices and operating workflows used to acquire the images.
However, without access to this knowledge, it is essential to be able to validate these products for the local population before procurement, and before use on patients, to ensure their safety and efficacy.
What to assess in a technical evaluation?
Software for technical evaluation can reduce the workload required by you team and allow for more scalable validation. It also creates the standardisation needed to be able to easily compare different products on the same local site data.
Performance
Is the diagnostic accuracy sufficient for your clinical workflow?
Fairness
Are there any biases in the product outputs? Are all demographics treated equally?
Robustness
Does the product work consistently across a range of different imaging acquisitions (e.g. sites, scanners, parameters)?