Applying machine learning to medicine is hindered by the sensitivity of the data used to train the models. However, a new approach called “federated” training aims to address this issue. The non-profit MLCommons Association has formed a group called MedPerf, which benchmarks computer chips for their performance on AI tasks. In a position paper published in Nature, MedPerf proposes a solution to the data impasse by sending AI models to clinicians who have access to private datasets. The clinicians then report back on how the models perform on the data, allowing developers to benefit from real data without compromising privacy. The goal is to promote wider adoption of medical AI and improve patient outcomes.
The MedPerf benchmark has initially focused on radiology and surgery, but it can be applied to other biomedical tasks as well. The platform consists of MLCubs, which are secure application containers that handle data preparation, model hosting, and performance evaluation. MedPerf has collaborated with Hugging Face, a popular repository of AI models, and Sage Bionetworks, which develops the Synapse platform for data sharing. The approach has already been tested in challenges involving brain tumor segmentation and has undergone pilot studies with academic groups.
MedPerf is now transitioning from an alpha to a beta stage and plans to expand its platform to more participants. The group is calling on healthcare stakeholders to form benchmark committees and for data owners to register their data on the platform. The code for MedPerf is available on GitHub.
Overall, MedPerf aims to facilitate the evaluation of healthcare AI models and promote collaboration in the field. By addressing the challenges associated with sensitive medical data, the group hopes to accelerate the adoption of AI in clinical practice and ultimately improve patient care.
