Challenge Description

This challenge is an extension of Left Ventricle Full Quantification Challenge MICCAI 2018 (LVQuan18), the main difference is that this challenge (LVQuan19) will provide original data without preprocessing for training and testing phases, which is more clinical than the data providing by LVQuan18. The challenge addresses the analysis of left ventricles (LV) in patients undergoing cardiac MRI.

To assess the heart's function, the LV's function, morphology and temporal dynamics is of clinical interest. This analysis in clinical routine is time consuming and prone to error and inter-observer variability. In this challenge the extraction of the LV's cavity and myocardium and subsequently the regression of regional wall thicknesses, LV dimensions and the classification of the phase of the cardiac cycle are to be performed. These are common and significant parameters to assess the LV function.

The aim of this challenge is to learn effective machine learning models that can estimate a set of clinical significant LV indices (regional wall thicknesses, cavity dimensions, area of cavity and myocardium, cardiac phase) directly from MR images. No intermediate segmentation is required in the whole procedure.

The training dataset is available via E-mail: lvquan19@outlook.com

Test Dataset Release: Google Drive, Baidu Pan, Password：s0cv

DATA DESCRIPTION

Features of Training and test dataset will be descripted below.

Training Dataset

A training dataset with processed SAX MR sequences of 56 subjects from clinical environment is used for model learning and validation. For each subject, 20 frames are included for the whole cardiac cycle. All ground truth values of the above-mentioned LV indices are provided for every single frame. With the training dataset, we encourage the participants to report the performance of their algorithms with N-fold cross validation, where the following configuration are used :

N=3, the number of each folds are (18, 19, 19);
N=5, the number of each folds are (11, 11, 11, 11, 12).

In such a way, we can have initial impression of the algorithms’ stability. The training dataset is available via
E-mail: lvquan19@outlook.com (in order to count the number of participants)

Test Dataset

A test dataset with processed SAX MR sequences of 30 subjects will be provided to participants according to the challenge schedule for algorithms assessment and ranking. The processing procedure is the same as that in the training dataset. For each subject, only the SAX image sequences of 20 frames are provided, while their ground truth values are not.

Important dates

(may change according to the schedule of STACOM 2019)

Training/Validation data release: 5 March 2019

Test data release: 1 July 2019

Abstract registration: 1 July 2019

Test results submission: 4 July 2019

Test performance feedback: 10 July 2019

Paper submission deadline: 15 July 2019

Workshop paper notification: 1 August 2019

Final version included in the PDF (distributed by MICCAI): 15 August 2019

Camera ready version submission: mid November

Challenge day: 13 October 2019 (in conjunction with STACOM 2019)

Objective

Developing reliable and accurate automatic method for full quantification of cardiac LV for each frame in the short axis view cardiac MR sequences. The quantifications include the following LV indices.

Name	Description
A1, A2	cavity area and myocardium area, in mm^2.
D1~D3	dimensions of cavity of three directions (IS-AL, I-A, and IL-AS) in mm.
RWT1~RWT6	regional wall thickness in mm, staring from the anterior-septal segment in counter clockwise direction, i.e., IS, I, IL, AL, A, AS.
Phase	binary variable for cardiac phase, 1 means systolic phase and 0 means diastolic.

For more details about left ventricle quantification, the following two references are recommended:

[1] Wufeng Xue, Gary Brahm, Sachin Pandey, Stephanie Leung, and Shuo Li. Full left ventricle quantification via deep multitask relationships learning. Medical Image Analysis, 43:54–65, Jan. 2018. [pdf]
[2] Wufeng Xue, Andrea Lum, Ashley Mercado, Mark Landis, James Warringto, and Shuo Li. Full quantification of left ventricle via deep multitask learning network respecting intra-and inter-task relatedness. MICCAI, 2017. [pdf]

Evaluation Criterion

Metrics

Mean Absolute Error (MAE) and Pearson correlation coefficient (PCC) are used to assess the performance of the algorithms for estimation of areas, dimensions and regional wall thicknesses. For the 30 subjects in the test dataset, there are a total of 600 images. For any LV indices in {A1, A2, D1~D3, RWT1~RWT6}, the MAE and PCC can be computed as:

${\scriptstyle MAE_{ind}=\frac{\sum_{i=1}^{600}|x_{ind}(i)-y_{ind}(i)|}{600}}$

${\scriptstyle PCC_{ind}=\frac{\sum_{i=1}^{600}[x_{ind}(i)-\bar{x}_{ind}][y_{ind}(i)-\bar{y}_{ind}]}{\sqrt{\sum_{i=1}^{600}[x_{ind}(i)-\bar{x}_{ind}]^2\sum_{i=1}^{600}[y_{ind}(i)-\bar{y}_{ind}]^2}}}$

where ${\scriptstyle ind\in \{A1,A2, D1~D3, RWT1~RWT6\}}$ , ${\scriptstyle x_{ind}}$ is the groud truth value and ${\scriptstyle y_{ind}}$ is the estimated value, ${\scriptstyle \bar{x}_{ind}}$ and ${\scriptstyle \bar{y}_{ind}}$ are their mean values.
Error Rate (ER) is used to assess the performance of the algorithms for cardiac phase identification. It can be computed as:

${\scriptstyle ER_{phase}=\frac{\sum_{i=1}^{600}\mathbf{1}(x_{phase}(i)\neq y_{phase}(i))}{600}}$

where ${\scriptstyle \mathbf{1}()}$ is the indication function, ${\scriptstyle x_{phase}}$ and ${\scriptstyle y_{phase}}$ are the ground truth value and the estimated value of cardiac phase, respectively.

Ranking

MAE and ER are used for the final ranking of all participants.

Initialization:

1. For each participant, compute ${\scriptstyle MAE_{ind}}$ for ${\scriptstyle ind\in \{A1,A2, D1\sim D3, RWT1\sim RWT6\}}$ and ${\scriptstyle ER_{phase}}$ .

2. Compute the mean MAE values for areas, dimensions, and regional wall thicknesses, and obtain ${\scriptstyle MAE_{area}}$ , ${\scriptstyle MAE_{dim}}$ , ${\scriptstyle MAE_{rwt}}$ .

3. Build four ranking lists ( ${\scriptstyle R_{area}}$ , ${\scriptstyle R_{dim}}$ , ${\scriptstyle R_{rwt}}$ , ${\scriptstyle R_{phase}}$ ) of all participants according to ${\scriptstyle MAE_{area}}$ , ${\scriptstyle MAE_{dim}}$ , ${\scriptstyle MAE_{rwt}}$ and ${\scriptstyle ER_{phase}}$ in ascending order, respectively. Statistical test is conducted for areas, dimensions, and regional wall thicknesses to test the significance of difference between two participants. If no significance exists, the two participants will have the same rank in the list.

4. The final ranking list ${\scriptstyle R_{ALL}}$ is computed by sorting the summation of the four ranking lists above in ascending order.

Paper Submission

Workshop paper

All participants are encouraged to submit a full workshop paper describing their algorithms and results. Manuscript up to 8 pages should follow the template of main conferences’ paper and be submitted via the STACOM submission system.

One-page abstract

Participants who cannot finish the task on time are encouraged to complete the task before 6 September 2019, with the one-page abstract submitted to the organizer via email (lvquan19@outlook.com). The results on test data should be sent to the organizer for performance evaluation before 1 September 2019.

Challenge ranking

Successful participants with workshop paper will be ranked according the assessment criterion.

Full ranking

A full ranking list including all the participants who completed the LV quantification task, with either workshop paper or one-page abstract, will also be announced during the workshop.

Result Submission

Details about the final result submission.

How

Send algorithm output on the test dataset to organizers via email (lvquan19@outlook.com).

Format

Results of test dataset can be submitted in a MATLAB data file (.mat file), with the following protocols.

Pay attention! Don't forget to convert the results to real physical values(with rwt,dims in mm, and areas in mm^2).

physical value of rwt= rwt*pix_spacing*80

physical value of dims= dims*pix_spacing*80

physical value of areas= areas*(pix_spacing*80)^2

File Name: LVQUAN_XXX.mat. XXX is the initials of the participant’s name.

MATALB data: the submitted results should be in the same format as the ground truth in the training dataset. Four variables should be contained in the LVQUAN_XXX.mat file:

areas_hat:
- array of size of 2×600.
- areas(1,:) is for cavity area and areas(2,:) is for myocardium area.
- For the 20 frames of subject s, the results should be areas(:, (s-1)×20:s×20).

dims_hat:
- array of size of 3×600.
- dims(1,:) is for direction of IS-AL, dims(2,:) is for direction of I-A, and dims(3,:) is for direction of IL-AS.
- For the 20 frames of subject s, the results should be dims(:, (s-1)×20:s×20).

rwt_hat:
- array of size of 6×600.
- areas(1,:)~areas(6,:) are for segments IS, I, IL, AL, A and AS, accordingly.
- For the 20 frames of subject s, the results should be rwt(:, (s-1)×20:s×20).

lv_phase_hat:
- array of size of 1×600.
- For the 20 frames of subject s, the results should be lv_phase(:, (s-1)×20:s×20).

Left Ventricle Full Quantification Challenge MICCAI 2019