Quick Buzz Feed

Machine Learning Has High Accuracy, Little Prospective Validation in MPNs

Gary Lloyd | Oct 31,26 | 01:36 EST

Others

AI in myeloproliferative neoplasms shows high discrimination in classifying disease subtypes, grading marrow fibrosis, and estimating survival within research cohorts. However, a narrative review highlights a critical need for prospective, outcome-focused validation trials before these tools can significantly impact treatment decisions or patient experiences.

Gaps AI Targets in Myeloproliferative Neoplasms This section delves into the inherent difficulties in diagnosing and managing myeloproliferative neoplasms (MPN). It specifically addresses the three classic Philadelphia-negative MPN subtypes: polycythemia vera, myelofibrosis (MF), and essential thrombocythemia (ET), which are primarily driven by JAK2, CALR, or MPL mutations. The article notes that patients with MF face a median overall survival of 3 to 7 years and a 10% to 20% risk of progressing to a blast phase, with existing treatments like JAK inhibitors offering only modest improvements. A significant diagnostic challenge lies in accurately differentiating ET from prefibrotic primary MF (pre-PMF), even among expert hematopathologists, with agreement levels as low as a kappa of 0.47. The rarity of MPN further complicates AI model development due to limited sample sizes available for training and validation. Bone Marrow Image Analysis Leads AI Diagnostics in MPN The review highlights digital pathology as the most advanced and mature application of artificial intelligence (AI) in the diagnostic landscape of MPN. It references a multicenter study involving 1051 patients across seven hospitals, where a deep learning model, integrating whole-slide images with laboratory data, achieved an impressive area under the curve (AUC) of 0.931 for differentiating MPN from other conditions. In a smaller 100-case reader study, this model's performance was not statistically different from that of three experienced hematopathologists. However, the review authors caution that the reader sample size was small, leading to considerable uncertainty in the estimates, and that statistical non-significance does not equate to equivalence. Another platform demonstrated an AUC of 0.94 for subtyping in an external cohort, and a foundation model, trained on 200 biopsies from Florence, Italy, showed an AUC of 0.89 with a sensitivity of 66.6% for distinguishing pre-PMF from ET. Blood Count Models Reach Prospective Testing Among the various AI applications discussed, only tools estimating marrow fibrosis from standard blood counts have undergone prospective testing. The FIBOM-AI model, which utilizes 27 complete blood count variables in conjunction with patient age to identify grade 2 to 3 reticulin fibrosis, was developed across six French centers and externally validated in seven others. This model achieved an AUC of 0.92 in the external cohort and an accuracy of 85.2% when applied prospectively in routine clinical practice. Its creators intend for it to serve as a triage tool, rather than a definitive replacement for biopsy, as it does not assign an MPN subtype or detect pre-PMF. In terms of prognosis, the AI-based prognostic score AIPSS-MF, a random survival forest model incorporating eight clinical variables, demonstrated a concordance index of 0.814 among 511 patients, surpassing the performance of the International Prognostic Scoring System (0.724) and MIPSS70 (0.654). Nevertheless, the authors point out that these established legacy scores were not recalibrated. The area of therapy response prediction remains significantly less developed, with no validated machine learning models for predicting JAK inhibitor response in MF, and a phase 4 study on hydroxyurea-resistance predictors in polycythemia vera was terminated without conclusive results. Review Limitations and Research Priorities The authors acknowledge that this was a narrative, nonsystematic review, which inherently carries the potential for bias in the selection and interpretation of studies. They highlight that no meta-analysis was conducted, and performance metrics were reported as published without recalculation. A significant concern raised is the infrequent reporting of critical details such as calibration, decision-curve analysis, or performance stratification by age, sex, ethnicity, and care setting in the foundational studies. This omission leaves unquantified the risk of inconsistent performance across diverse patient groups. To advance the field, the authors strongly advocate for prospective multicenter validation trials that include head-to-head comparisons against established standard scores. They also call for clinical-utility trials, the adoption of federated learning approaches, and the establishment of shared benchmark datasets to overcome limitations imposed by small sample sizes. Furthermore, they emphasize the necessity for AI software used in diagnosis or treatment, typically regulated as medical devices, to be continuously monitored for performance drift post-approval.

Gaps AI Targets in Myeloproliferative Neoplasms

This section delves into the inherent difficulties in diagnosing and managing myeloproliferative neoplasms (MPN). It specifically addresses the three classic Philadelphia-negative MPN subtypes: polycythemia vera, myelofibrosis (MF), and essential thrombocythemia (ET), which are primarily driven by JAK2, CALR, or MPL mutations. The article notes that patients with MF face a median overall survival of 3 to 7 years and a 10% to 20% risk of progressing to a blast phase, with existing treatments like JAK inhibitors offering only modest improvements. A significant diagnostic challenge lies in accurately differentiating ET from prefibrotic primary MF (pre-PMF), even among expert hematopathologists, with agreement levels as low as a kappa of 0.47. The rarity of MPN further complicates AI model development due to limited sample sizes available for training and validation.

Bone Marrow Image Analysis Leads AI Diagnostics in MPN

The review highlights digital pathology as the most advanced and mature application of artificial intelligence (AI) in the diagnostic landscape of MPN. It references a multicenter study involving 1051 patients across seven hospitals, where a deep learning model, integrating whole-slide images with laboratory data, achieved an impressive area under the curve (AUC) of 0.931 for differentiating MPN from other conditions. In a smaller 100-case reader study, this model's performance was not statistically different from that of three experienced hematopathologists. However, the review authors caution that the reader sample size was small, leading to considerable uncertainty in the estimates, and that statistical non-significance does not equate to equivalence. Another platform demonstrated an AUC of 0.94 for subtyping in an external cohort, and a foundation model, trained on 200 biopsies from Florence, Italy, showed an AUC of 0.89 with a sensitivity of 66.6% for distinguishing pre-PMF from ET.

Blood Count Models Reach Prospective Testing

Among the various AI applications discussed, only tools estimating marrow fibrosis from standard blood counts have undergone prospective testing. The FIBOM-AI model, which utilizes 27 complete blood count variables in conjunction with patient age to identify grade 2 to 3 reticulin fibrosis, was developed across six French centers and externally validated in seven others. This model achieved an AUC of 0.92 in the external cohort and an accuracy of 85.2% when applied prospectively in routine clinical practice. Its creators intend for it to serve as a triage tool, rather than a definitive replacement for biopsy, as it does not assign an MPN subtype or detect pre-PMF. In terms of prognosis, the AI-based prognostic score AIPSS-MF, a random survival forest model incorporating eight clinical variables, demonstrated a concordance index of 0.814 among 511 patients, surpassing the performance of the International Prognostic Scoring System (0.724) and MIPSS70 (0.654). Nevertheless, the authors point out that these established legacy scores were not recalibrated. The area of therapy response prediction remains significantly less developed, with no validated machine learning models for predicting JAK inhibitor response in MF, and a phase 4 study on hydroxyurea-resistance predictors in polycythemia vera was terminated without conclusive results.

Review Limitations and Research Priorities

The authors acknowledge that this was a narrative, nonsystematic review, which inherently carries the potential for bias in the selection and interpretation of studies. They highlight that no meta-analysis was conducted, and performance metrics were reported as published without recalculation. A significant concern raised is the infrequent reporting of critical details such as calibration, decision-curve analysis, or performance stratification by age, sex, ethnicity, and care setting in the foundational studies. This omission leaves unquantified the risk of inconsistent performance across diverse patient groups. To advance the field, the authors strongly advocate for prospective multicenter validation trials that include head-to-head comparisons against established standard scores. They also call for clinical-utility trials, the adoption of federated learning approaches, and the establishment of shared benchmark datasets to overcome limitations imposed by small sample sizes. Furthermore, they emphasize the necessity for AI software used in diagnosis or treatment, typically regulated as medical devices, to be continuously monitored for performance drift post-approval.