NA-MIC Project Weeks

Back to Projects List

Training deep learning models for B-line detection and localization in lung ultrasound videos using crowdsourced labels

Key Investigators

• Mike Jin (Brigham and Women's Hospital, USA)
• Tamas Ungi (Queen's University, Canada)
• Ameneh Asgari-Targhi (Brigham and Women's Hospital, USA)
• Tina Kapur (Brigham and Women's Hospital, USA)

Presenter location: In-person

Project Description

AI models for medical imaging need labeled data, which can be hard to obtain with the required volume and/or accuracy.

Crowdsourced labels on medical imaging data can help bridge that gap, as suggested by our data showing that crowdsourced classifications and segmentations for B-lines on lung ultrasound videos have comparable or better accuracy than annotations from medical experts with advanced training in lung ultrasound (Duggan 2023) (Jin 2023).

To demonstrate this, we will train deep learning B-line classification and segmentation models using crowdsourced annotations on lung ultrasound videos collected from 483 patients and compare their performance to models trained on manual annotations from experts.

Objective

Produce models for classification and segmentation of lung ultrasound videos with comparable or improved performance to models trained on manual annotations from experts.

Approach and Plan

1. Using crowdsourced labels, train CNN-RNN deep learning models to classify lung ultrasound videos as having no B-lines, discrete B-lines, or confluent B-lines.
2. Using crowdsourced labels, train deep learning models to segment individual frames within lung ultrasound videos for the presence and location of individual B-lines using open-source code from (Lucassen 2023).
3. Compare the performance of crowdlabel-trained models to models trained on manual annotations from experts.

Progress and Next Steps

Collected 330,000 (177,000 at start of PW40) crowdsourced segmentation opinions to form high-quality segmentations of 21,000 (8,500 at the start of PW40) frames within the videos from 483 patients.

In progress:

1. Collecting ~3,000 additional high-quality frame segmentations per day with a target of 20,000 frames remaining which will be completed this month (41,000 total).
2. Training video classification models on high-quality classifications of 4,030 ultrasound videos from 483 patients.
3. Preparing training data for segmentation models.

Illustrations

What are B-lines in lung ultrasound? Here is a picture. The white beams are B-lines, the dark sectors are shadows from ribs.

For crowdsourcing, we collect opinions from 5 experts and combine their opinions, like so. Yellow is the expert consensus:

Then, we collect opinions from crowd using a gamified system very similar to Google’s RECAPTCHA. We take the most reliable opinions and combine them to get a high-quality consensus. Here are the crowd opinions for the same image frame, with the expert consensus in yellow:

For this image, here are the crowd consensus and expert consensus both overlaid:

We will be continuing to crowdsource B-line segmentations throughout Project Week! Here is the visualization of the progress so far (mid-day 2024-01-29):

Background and References

No response