<abstract xmlns="http://www.w3.org/1999/xhtml"><sec id="j_JoU.2022.0013_s_008"><h3>Aim of the study</h3><p>Deep neural networks have achieved good performance in breast mass classification in ultrasound imaging. However, their usage in clinical practice is still limited due to the lack of explainability of decisions conducted by the networks. In this study, to address the explainability problem, we generated saliency maps indicating ultrasound image regions important for the network’s classification decisions.</p></sec><sec id="j_JoU.2022.0013_s_009"><h3>Material and methods</h3><p>Ultrasound images were collected from 272 breast masses, including 123 malignant and 149 benign. Transfer learning was applied to develop a deep network for breast mass classification. Next, the class activation mapping technique was used to generate saliency maps for each image. Breast mass images were divided into three regions: the breast mass region, the peritumoral region surrounding the breast mass, and the region below the breast mass. The pointing game metric was used to quantitatively assess the overlap between the saliency maps and the three selected US image regions.</p></sec><sec id="j_JoU.2022.0013_s_010"><h3>Results</h3><p>Deep learning classifier achieved the area under the receiver operating characteristic curve, accuracy, sensitivity, and specificity of 0.887, 0.835, 0.801, and 0.868, respectively. In the case of the correctly classified test US images, analysis of the saliency maps revealed that the decisions of the network could be associated with the three selected regions in 71% of cases.</p></sec><sec id="j_JoU.2022.0013_s_011"><h3>Conclusions</h3><p>Our study is an important step toward better understanding of deep learning models developed for breast mass diagnosis. We demonstrated that the decisions made by the network can be related to the appearance of certain tissue regions in breast mass US images.</p></sec></abstract>

Aim of the studyDeep neural networks have achieved good performance in breast mass classification in ultrasound imaging. However, their usage in clinical practice is still limited due to the lack of explainability of decisions conducted by the networks. In this study, to address the explainability problem, we generated saliency maps indicating ultrasound image regions important for the network’s classification decisions.Material and methodsUltrasound images were collected from 272 breast masses, including 123 malignant and 149 benign. Transfer learning was applied to develop a deep network for breast mass classification. Next, the class activation mapping technique was used to generate saliency maps for each image. Breast mass images were divided into three regions: the breast mass region, the peritumoral region surrounding the breast mass, and the region below the breast mass. The pointing game metric was used to quantitatively assess the overlap between the saliency maps and the three selected US image regions.ResultsDeep learning classifier achieved the area under the receiver operating characteristic curve, accuracy, sensitivity, and specificity of 0.887, 0.835, 0.801, and 0.868, respectively. In the case of the correctly classified test US images, analysis of the saliency maps revealed that the decisions of the network could be associated with the three selected regions in 71% of cases.ConclusionsOur study is an important step toward better understanding of deep learning models developed for breast mass diagnosis. We demonstrated that the decisions made by the network can be related to the appearance of certain tissue regions in breast mass US images.


<div xmlns="http://www.w3.org/1999/xhtml"><sec id="j_JoU.2022.0013_s_001"><div>Introduction</div><p>Breast cancer is the most frequent cancer in women worldwide<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_001">1</a>)</sup>. Ultrasound (US) imaging is commonly used by radiologists to characterize breast masses and conduct a diagnosis. However, accurate differentiation between malignant and benign breast masses requires knowledge about the characteristic US image features associated with the malignancy, which is why it is considered difficult. Malignant breast masses are commonly characterized by indistinct and highly variable contours as well as the presence of shadowing artifacts and calcifications. Various deep learning methods have been proposed over the last years to help with the breast mass classification<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_002">2</a>,<a ref-type="bibr" href="#j_JoU.2022.0013_ref_003">3</a>)</sup>. Deep convolutional neural networks (CNNs) can automatically process input US images to determine classification decisions. Although deep networks achieved excellent performance in breast mass classification, their usage in clinical practice is still limited due to the lack of model interpretability and explainability. Neural networks are commonly perceived as ‘black-box’ models, which raises questions about their applicability in medicine<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_004">4</a>,<a ref-type="bibr" href="#j_JoU.2022.0013_ref_005">5</a>)</sup>.</p><p>In this study, we address the problem of explainability of deep networks developed for breast mass classification in US imaging. First, we developed a CNN for breast mass classification using transfer learning. Second, we used the class activation mapping (CAM) technique to generate the network’s saliency maps for each breast mass US image. Saliency maps indicate which image regions activated the network. Therefore, these maps can be used to visually explain the decisions conducted by the model. In this study, we divided the US images into three regions commonly adopted by radiologists to visually assess breast masses. Next, we utilized the pointing game metric to quantitatively associate the three regions with the saliency maps generated by the network<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_006">6</a>)</sup>.</p></sec><sec id="j_JoU.2022.0013_s_002"><div>Materials and methods</div><sec id="j_JoU.2022.0013_s_002_s_001"><div>Dataset</div><p>The study was approved by the Institutional Review Board. All medical procedures involving human participants were performed in accordance with the guidelines stated in the Declaration of Helsinki and its later amendments or comparable ethical standards. US images were collected from 272 breast masses, of which 123 masses were malignant and 149 were benign. The average age of the patients diagnosed with malignant and benign masses was 54.4 and 40.8 years, respectively. Two orthogonal scans (longitudinal and transverse) were collected for each breast mass using the Ultrasonix SonixTouch Research US scanner (Ultrasonix Inc., Canada) equipped with the L14-5/38 linear probe operating at 10 MHz<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_007">7</a>)</sup>. The radiologists who collected the data assessed each case using the Breast Imaging-Reporting and Data System (BI-RADS). A total of 88 masses corresponded to BI-RADS category 3, 123 to BI-RADS 4, and 61 to BI-RADS 5 according to the American College of Radiology Atlas and the guidelines of the Polish Ultrasound Society<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_008">8</a>,<a ref-type="bibr" href="#j_JoU.2022.0013_ref_009">9</a>).</sup> Lesions corresponding to the BI-RADS categories 4 and 5 were subjected to core needle biopsy. In the case of BI-RADS 3 masses, part of them were assessed using fine needle aspiration biopsy, while the remainder were followed up over a two-year period. Manual segmentations indicating areas of breast masses were prepared by an experienced radiologist. Exemplary US images are presented in <a ref-type="fig" href="#j_JoU.2022.0013_fig_001">Fig. 1</a>. To develop the classification model, patient data were divided into training and test sets with a 204/68 split. The ratio of malignant to benign masses, equal to approximately 34%, was maintained for each set.</p><figure id="j_JoU.2022.0013_fig_001" position="float" fig-type="figure"><h2>Fig. 1</h2><figCaption><p>Exemplary US images presenting benign and malignant breast masses</p></figCaption><img xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="graphic/j_JoU.2022.0013_fig_001.jpg" src="https://sciendo-parsed.s3.eu-central-1.amazonaws.com/64722e5c215d2f6c89dbec5f/j_JoU.2022.0013_fig_001.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20240630T133622Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=18000&amp;X-Amz-Credential=AKIA6AP2G7AKP25APDM2%2F20240630%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=328f2ed4f57c58c981e3e4c2b2f0dc47b437f86f3af3fc399826f7d9694cc814" class="mw-100"/></figure></sec></sec><sec id="j_JoU.2022.0013_s_003"><div>Deep learning methods</div><p>To differentiate between malignant and benign breast masses, we used a recently proposed deep learning method<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_010">10</a>)</sup>. ResNet CNN pre-trained on the ImageNet dataset served as the backbone for the classification model<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_011">11</a>,<a ref-type="bibr" href="#j_JoU.2022.0013_ref_012">12</a>)</sup>. The last dense layer of the pre-trained network was replaced with a single output dense layer equipped with the sigmoid activation function suitable for the binary classification. The weights of the dense layer were randomly initialized. Transfer learning technique based on the scaling of deep representations was used to adjust the pre-trained network to process breast mass US images. More details about the deep learning model and the transfer learning technique utilized can be found in our previous paper<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_010">10</a>)</sup>. Binary cross-entropy and the Adam optimizer were used to train the network. The learning rate and batch size were set to 0.001 and 12, respectively. All US images were resized to the dimensions of 224x224 to match the original resolution of the ImageNet images used for the pre-training. Additionally, image augmentation was applied to generate more data for the training. The training of the network was terminated if no improvement with respect to the loss function on the test set was observed after 8 epochs.</p><p>Following the training, we used the CAM technique to generate image saliency maps (see <a ref-type="fig" href="#j_JoU.2022.0013_fig_002">Fig. 2</a>). The CAM technique utilizes the weights of the last dense classification layer to combine feature maps extracted before the global average pooling (GAP) layer and generate a low resolution saliency map<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_013">13</a>)</sup>. Next, the saliency map is resized to match the resolution of the original US image. The saliency map indicates what image regions activated the network. In the case of the binary classification, the average value of the saliency map (the overall activation) is directly related to the classification decision made by the model<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_014">14</a>)</sup>. In this study, we coded the positive and negative activation regions with red and blue colors, respectively. A saliency map dominated by a red color would probably indicate a malignant breast mass, while a saliency map dominated by a blue color would correspond to a benign mass.</p><figure id="j_JoU.2022.0013_fig_002" position="float" fig-type="figure"><h2>Fig. 2</h2><figCaption><p>Scheme presenting the calculations of a saliency map. Weights of the linear dense classification layer are utilized to combine feature maps extracted before the global average pooling (GAP) layer</p></figCaption><img xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="graphic/j_JoU.2022.0013_fig_002.jpg" src="https://sciendo-parsed.s3.eu-central-1.amazonaws.com/64722e5c215d2f6c89dbec5f/j_JoU.2022.0013_fig_002.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20240630T133622Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=18000&amp;X-Amz-Credential=AKIA6AP2G7AKP25APDM2%2F20240630%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=0186dd991bfa351cbeb095fe0d83f18423817b9226821881fb3e4801eacbf1d6" class="mw-100"/></figure></sec><sec id="j_JoU.2022.0013_s_004"><div>Evaluations</div><p>Classification performance was assessed using the receiver operating characteristic curve (ROC) and the area under the ROC curve (AUC). Accuracy, sensitivity, and specificity were calculated based on the point on the ROC curve that was the closest to the left upper corner of the curve<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_015">15</a>)</sup>. The pointing game metric was used to evaluate the saliency maps in a quantitative way<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_006">6</a>)</sup>. The aim of the pointing game score is to assess what image region is highlighted by the saliency map. In our case, we used the manual breast mass segmentations to pre-determine three regions in the US images:</p><list list-type="order"><list-item><p>The breast mass region, the appearance of which should naturally be taken into account by the network.</p></list-item><list-item><p>The peritumoral region, specified as a 5 mm ring around the breast mass, which was calculated with the morphological operations based on the breast mass manual segmentation. The appearance of this region is related to the characteristics of the mass boundary; blurry and indistinct margins are commonly encountered in malignant masses, while benign masses exhibit well-defined borders.</p></list-item><list-item><p>The region below the breast mass. Malignant masses commonly have a larger attenuation coefficient than benign masses, which makes the region below the malignant mass less bright in comparison to adjacent tissues, eventually causing the shadowing artifact. Therefore, the appearance of this region may also be important for the deep classifier.</p></list-item></list><p><a ref-type="fig" href="#j_JoU.2022.0013_fig_003">Figure 3</a> presents a breast mass US image and the three regions selected to evaluate the saliency maps. In the case of the pointing game metric, we considered that the saliency map hit the region if the maximal or minimal (or both) point(s) in the saliency map was (were) contained within the particular region. As mentioned in the previous subsection, saliency maps in the case of the binary classification may show regions of both positive and negative activations. Therefore, to explain the decisions conducted by the model, it is important to assess both the location of the maximal and minimal activation areas.</p><figure id="j_JoU.2022.0013_fig_003" position="float" fig-type="figure"><h2>Fig. 3</h2><figCaption><p><bold>A</bold>. <italic>Exemplary US image and the three regions selected for the saliency map study</italic>: <bold>B</bold>. <italic>breast mass region</italic>, <bold>C</bold>. <italic>peritumoral region (mass boundary), and</italic><bold>D</bold>. <italic>region below the breast mass</italic></p></figCaption><img xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="graphic/j_JoU.2022.0013_fig_003.jpg" src="https://sciendo-parsed.s3.eu-central-1.amazonaws.com/64722e5c215d2f6c89dbec5f/j_JoU.2022.0013_fig_003.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20240630T133622Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=18000&amp;X-Amz-Credential=AKIA6AP2G7AKP25APDM2%2F20240630%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=667372acd8ff21a7ea89bcb625fc7a21d6ddc9b4c47370679ee23b64324a71e2" class="mw-100"/></figure></sec><sec id="j_JoU.2022.0013_s_005"><div>Results</div><p>The classification performance of the implemented method is summarized in <a ref-type="table" href="#j_JoU.2022.0013_tab_001">Tab. 1</a>. The network achieved good performance, with AUC value and accuracy of 0.887, and 0.835, respectively. <a ref-type="table" href="#j_JoU.2022.0013_tab_002">Tab. 2</a> presents the pointing game scores determined for the correctly classified cases from the test set. Here, we can see that in 71% of the cases the extreme points of the saliency maps corresponded to at least one of the three pre-selected regions. The points of extreme activation were related to the breast mass region, the peritumoral region, and the region below the mass in 34%, 38%, and 30% of cases, respectively. Notice that in some cases the maximum and minimum of the saliency map could correspond to two different regions, actually resulting in two pointing game hits. Exemplary class activation maps calculated for several masses are presented in <a ref-type="fig" href="#j_JoU.2022.0013_fig_004">Fig. 4</a>.</p><table-wrap id="j_JoU.2022.0013_tab_001" position="float"><label>Tab. 1</label><caption><p>Breast mass classification performance of the deep learning model on the test set. AUC – area under the receiver-operating characteristic curve</p></caption><table frame="hsides" rules="all"><thead><tr><th align="center">AUC</th><th align="center">Accuracy</th><th align="center">Sensitivity</th><th align="center">Specificity</th></tr></thead><tbody><tr><td align="center">0.887 ± 0.015</td><td align="center">0.835 ± 0.018</td><td align="center">0.801 ± 0.025</td><td align="center">0.868 ± 0.023</td></tr></tbody></table></table-wrap><table-wrap id="j_JoU.2022.0013_tab_002" position="float"><label>Tab. 2</label><caption><p>Pointing game scores obtained for the network’s saliency maps and the three pre-defined regions. The results were calculated for the correctly classified cases from the test set</p></caption><table frame="hsides" rules="all"><thead><tr><th align="center">Region</th><th align="center">Percentage of accurate hits</th></tr></thead><tbody><tr><td align="center">Breast mass region</td><td align="center">34%</td></tr><tr><td align="center">Peritumoral region (boundary region)</td><td align="center">38%</td></tr><tr><td align="center">Region below the breast mass</td><td align="center">30%</td></tr><tr><td align="center">At least one of the above three regions</td><td align="center">71%</td></tr></tbody></table></table-wrap><figure id="j_JoU.2022.0013_fig_004" position="float" fig-type="figure"><h2>Fig. 4</h2><figCaption><p>US images presenting benign and malignant breast masses and the corresponding saliency maps pointing out the three pre-determined regions in US images. The white cross indicates the extreme activation value of the saliency map responsible for the particular pointing game result</p></figCaption><img xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="graphic/j_JoU.2022.0013_fig_004.jpg" src="https://sciendo-parsed.s3.eu-central-1.amazonaws.com/64722e5c215d2f6c89dbec5f/j_JoU.2022.0013_fig_004.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20240630T133622Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=18000&amp;X-Amz-Credential=AKIA6AP2G7AKP25APDM2%2F20240630%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=7e011c7cc61cfacb15c8f27ec8cce84af87e9882c58ae87ad2258e61a66a5878" class="mw-100"/></figure></sec><sec id="j_JoU.2022.0013_s_006"><div>Discussion</div><p>In this study, we developed a deep learning model for breast mass classification in US imaging. Moreover, we showed that the decisions conducted by the network could be visually explained. Regions of extreme activation in the saliency maps could be associated with the appearance of certain tissue regions in breast mass US images. We quantitatively assessed the saliency maps with the pointing game metric to find that the network’s decisions could be related in 71% to the following three regions: the breast mass region, the peritumoral region, and the region below the breast mass. Our results revealed that the appearance of the breast mass boundary (peritumoral region) was the most frequently indicated by the network as important. This finding may be in an agreement with the review papers stating that the handcrafted mass boundary features are the better performing features for the breast mass classification<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_016">16</a>)</sup>. Nevertheless, the decisions of the network could also be associated with the appearance of the other two regions, depicting that the network could take into account the visual appearance of various US image regions.</p><p>There are several issues related to our study. First of all, we used only one technique to generate the saliency maps, but various other methods have been proposed recently, for example the GRAD-CAM or CAMERAS<sup>(<a ref-type="bibr" href="#j_JoU.2022.0013_ref_017">17</a>,<a ref-type="bibr" href="#j_JoU.2022.0013_ref_018">18</a>)</sup>. While saliency maps generated with different algorithms should highlight similar tissue structures, it would be interesting to evaluate the usefulness and limitations of each technique. For example, the resolution of the saliency map may have an impact on the results obtained with the pointing game metric, which focuses only on the extreme points of the saliency map. As presented in <a ref-type="fig" href="#j_JoU.2022.0013_fig_004">Fig. 4</a>, the activation area may be sometimes large enough to overlap with several tissue regions. For such cases, the conclusions drawn from the pointing game results may not be accurate. Second, saliency maps were calculated based on the ResNet CNN fine-tuned on a relatively small dataset of breast mass US images. It would be interesting to investigate whether the obtained saliency maps are universal in the case of the breast mass diagnosis or whether they perhaps depend on the utilized deep learning model. Moreover, we did not assess the relationship between the performance of the classifier and the pointing game results. Presumably, a network achieving low performance would probably provide noisy and meaningless saliency maps. Third, in the future it would be interesting to utilize saliency maps to improve classification performance. For example, it might be beneficial to suspend classification decisions if the saliency map does not highlight relevant tissue structures in the breast mass US image.</p></sec><sec id="j_JoU.2022.0013_s_007"><div>Conclusions</div><p>Our study is an important preliminary step toward a better understanding of deep learning models developed for breast mass diagnosis. We demonstrated that the decisions conducted by the deep network could be associated with the appearance of certain tissue regions in breast mass ultrasound images. In the future, we plan to perform additional experiments to further enhance the interpretability and explainability of deep learning models.</p></sec></div>

Explaining a deep learning based breast ultrasound image classifier with saliency maps

Journal of Ultrasonography

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Aim of the studyDeep neural networks have achieved good performance in breast mass classification in ultrasound imaging. However, their usage in clinical...

Explaining a deep learning based breast ultrasound image classifier with saliency maps

Article Category: Research paper

Published Online: Apr 27, 2022

Page range: 70 - 75

Received: Nov 14, 2021

Accepted: Jan 24, 2022

DOI: https://doi.org/10.15557/JoU.2022.0013

Keywords
deep learning, breast mass diagnosis, attention maps, explainability

© 2022 Michał Byra, Katarzyna Dobruch-Sobczak, Hanna Piotrzkowska-Wroblewska, Ziemowit Klimonda, Jerzy Litniewski, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Explaining a deep learning based breast ultrasound image classifier with saliency maps

Article Category: Research paper

Published Online: Apr 27, 2022

Page range: 70 - 75

Received: Nov 14, 2021

Accepted: Jan 24, 2022

DOI: https://doi.org/10.15557/JoU.2022.0013

Keywordsdeep learning, breast mass diagnosis, attention maps, explainability

© 2022 Michał Byra, Katarzyna Dobruch-Sobczak, Hanna Piotrzkowska-Wroblewska, Ziemowit Klimonda, Jerzy Litniewski, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Keywords
deep learning, breast mass diagnosis, attention maps, explainability