AI hybrid strategy improves mammogram interpretation

A hybrid reading strategy for screening mammography, developed by Dutch researchers and deployed retrospectively to more than 40,000 exams, reduced radiologist workload by 38% without changing recall or cancer detection rates.

The study, which emphasizes AI confidence, was published in Radiology.

“Although the overall performance of state-of-the-art AI models is very high, AI sometimes makes mistakes,” said Sarah D. Verboom, M.Sc., a doctoral candidate in the Department of Medical Imaging at Radboud University Medical Center in the Netherlands.

“Identifying exams in which AI interpretation is unreliable is crucial to allow for and optimize use of AI models in breast cancer screening programs.”

The hybrid reading strategy involves using a combination of radiologist readers and a stand-alone AI interpretation of cases in which the AI model performs as well as, or better than, the radiologist.

“We can achieve this performance level if the AI model provides not only an assessment of the probability of malignancy (PoM) for a case but also a rating of its certainty of that assessment,” Verboom said.

“Unfortunately, the PoM itself is not always a good predictor of certainty because deep neural networks tend to be overconfident in their predictions.”

To develop and evaluate a hybrid reading strategy, the researchers used a dataset of 41,469 screening mammography exams from 15,522 women (median age 59 years) with 332 screen-detected cancers and 34 interval cancers. The exams were performed between 2003 and 2018 in Utrecht, Netherlands, as part of the Dutch National Breast Cancer Screening Program.

The dataset was divided at the patient level into two equal groups with identical cancer detection, recall and interval cancer rates. The first group was used to determine the optimal thresholds for the hybrid reading strategy, while the second group was used to evaluate the reading strategies.

Of the uncertainty metrics evaluated by the researchers, the entropy of the mean PoM score of the most suspicious region produced a cancer detection rate of 6.6 per 1,000 cases and a recall rate of 23.7 per 1,000 cases, similar to rates of standard double-reading by radiologists.

The final hybrid reading strategy involved AI evaluating every screening mammogram to produce two outputs: the PoM and an uncertainty estimate of that prediction. When AI determined the PoM was below the established threshold with certainty, the case was considered normal.

When AI detected a PoM above the established threshold, women were recalled for further testing, but only when that prediction was deemed confident. Otherwise, the exam was double-read by radiologists.

Although the majority of AI decisions were uncertain and deferred to a human reader, 38% were classified as certain and could be read solely by AI. Using the researchers’ strategy reduced radiologist reading workload to 61.9% without changing recall (23.6‰ vs. 23.9‰) or cancer detection (6.6‰ vs. 6.7‰) rates, both of which are comparable to those of standard double-reading.

When the AI model was certain, the area under the curve (AUC) was higher (0.96 vs. 0.87). Its sensitivity nearly matched that of double radiologist reading (85.4% vs. 88.9%). Younger women with dense breasts were more likely to have an uncertain AI score.

“The key component of our study isn’t necessarily that this is the best way to split the workload, but that it’s helpful to have uncertainty quantification built into AI models,” Verboom said. “I hope commercial products integrate this into their models, because I think it’s a very useful metric.”

Verboom noted that if the study results occurred in clinical practice, the decision to recall 19% of women would be made by AI without the intervention of a radiologist.

“Several studies have shown that women participating in breast cancer screening programs have positive attitudes about the use of AI,” she said. “However, most women prefer their mammogram to be read by at least one radiologist.”

She said it may be more acceptable for radiologists to review exams deemed uncertain by AI, as well as AI recall cases.

“The use of AI with uncertainty quantification can be a possible solution for workforce shortages and could help build trust in the implementation of AI,” Verboom said.

Verboom said further research, ideally a prospective trial, is needed to determine how the workload reduction achieved by the hybrid reading strategy could decrease radiologist reading time.

“I think in the future, we could get to a point where a portion of women are sent home without ever having a radiologist look at their mammogram because AI will determine that their exam is normal,” she said. “We’re not there yet, but I think we could get there with this uncertainty metric and quality control.”

A hybrid reading strategy for screening mammography, developed by Dutch researchers and deployed retrospectively to more than 40,000 exams, reduced radiologist workload by 38% without changing recall or cancer detection rates.

The study, which emphasizes AI confidence, was published in Radiology.

“Although the overall performance of state-of-the-art AI models is very high, AI sometimes makes mistakes,” said Sarah D. Verboom, M.Sc., a doctoral candidate in the Department of Medical Imaging at Radboud University Medical Center in the Netherlands.

“Identifying exams in which AI interpretation is unreliable is crucial to allow for and optimize use of AI models in breast cancer screening programs.”

The hybrid reading strategy involves using a combination of radiologist readers and a stand-alone AI interpretation of cases in which the AI model performs as well as, or better than, the radiologist.

“We can achieve this performance level if the AI model provides not only an assessment of the probability of malignancy (PoM) for a case but also a rating of its certainty of that assessment,” Verboom said.

“Unfortunately, the PoM itself is not always a good predictor of certainty because deep neural networks tend to be overconfident in their predictions.”

To develop and evaluate a hybrid reading strategy, the researchers used a dataset of 41,469 screening mammography exams from 15,522 women (median age 59 years) with 332 screen-detected cancers and 34 interval cancers. The exams were performed between 2003 and 2018 in Utrecht, Netherlands, as part of the Dutch National Breast Cancer Screening Program.

The dataset was divided at the patient level into two equal groups with identical cancer detection, recall and interval cancer rates. The first group was used to determine the optimal thresholds for the hybrid reading strategy, while the second group was used to evaluate the reading strategies.

Of the uncertainty metrics evaluated by the researchers, the entropy of the mean PoM score of the most suspicious region produced a cancer detection rate of 6.6 per 1,000 cases and a recall rate of 23.7 per 1,000 cases, similar to rates of standard double-reading by radiologists.

The final hybrid reading strategy involved AI evaluating every screening mammogram to produce two outputs: the PoM and an uncertainty estimate of that prediction. When AI determined the PoM was below the established threshold with certainty, the case was considered normal.

When AI detected a PoM above the established threshold, women were recalled for further testing, but only when that prediction was deemed confident. Otherwise, the exam was double-read by radiologists.

Although the majority of AI decisions were uncertain and deferred to a human reader, 38% were classified as certain and could be read solely by AI. Using the researchers’ strategy reduced radiologist reading workload to 61.9% without changing recall (23.6‰ vs. 23.9‰) or cancer detection (6.6‰ vs. 6.7‰) rates, both of which are comparable to those of standard double-reading.

When the AI model was certain, the area under the curve (AUC) was higher (0.96 vs. 0.87). Its sensitivity nearly matched that of double radiologist reading (85.4% vs. 88.9%). Younger women with dense breasts were more likely to have an uncertain AI score.

“The key component of our study isn’t necessarily that this is the best way to split the workload, but that it’s helpful to have uncertainty quantification built into AI models,” Verboom said. “I hope commercial products integrate this into their models, because I think it’s a very useful metric.”

Verboom noted that if the study results occurred in clinical practice, the decision to recall 19% of women would be made by AI without the intervention of a radiologist.

“Several studies have shown that women participating in breast cancer screening programs have positive attitudes about the use of AI,” she said. “However, most women prefer their mammogram to be read by at least one radiologist.”

She said it may be more acceptable for radiologists to review exams deemed uncertain by AI, as well as AI recall cases.

“The use of AI with uncertainty quantification can be a possible solution for workforce shortages and could help build trust in the implementation of AI,” Verboom said.

Verboom said further research, ideally a prospective trial, is needed to determine how the workload reduction achieved by the hybrid reading strategy could decrease radiologist reading time.

“I think in the future, we could get to a point where a portion of women are sent home without ever having a radiologist look at their mammogram because AI will determine that their exam is normal,” she said. “We’re not there yet, but I think we could get there with this uncertainty metric and quality control.”