Constrained deep networks for medical image segmentation

Kervadec, Hoel (2021). Constrained deep networks for medical image segmentation. Thèse de doctorat électronique, Montréal, École de technologie supérieure.

Prévisualisation

PDF Télécharger (19MB) | Prévisualisation

Résumé

Weakly supervised image segmentation, in the form of partially labeled images, is attracting significant research attention as it can mitigate the need for laborious pixel annotations required by deep learning models. Enforcing high-order, global inequality constraints on the network outputs can leverage unlabeled data by guiding the training with prior knowledge, restricting the search space during training to anatomically feasible solutions. A range of possible values (such as a lower/upper bounds on the size of a organ) can be very valuable to guide training. However, in the context of deep neural networks, standard Lagrangian optimization has been largely avoided, mainly due to the instability and computational complexity ensuing from alternating explicit dual updates and stochastic optimization. Interior point methods, despite their popularity in convex optimization, are not applicable neither, as they require a feasible starting point, which is itself a difficult constrained problem for deep neural networks. In this thesis, we investigate hard inequality constraints in the context of deep networks with both quadratic penalties and more principled log-barrier extensions. We also investigate methods to mitigate class-imbalance in segmentation problems, such as in brain lesions dataset, by constraining the boundary of the predicted segmentation to match the ground-truth boundary. This thesis produced five different publications as first author, and four papers as co-author. Our papers received several awards, and we were invited to publish extended versions of our works in two special issues of Medical Image Analysis (MedIA).

In our first contribution, we propose to introduce a differentiable penalty, which enforces inequality constraints directly in the loss function, avoiding expensive Lagrangian dual iterates and proposal generation. From constrained-optimization perspective, our simple penalty-based approach is not optimal as there is no guarantee that the constraints are satisfied. However, surprisingly, it yields substantially better results than Lagrangian-based constrained convolutional neural networks, while reducing the computational demand for training. By annotating only a small fraction of the pixels, our approach reaches performances comparable to full supervision, on three separate tasks. While our experiments focused on basic linear constraints such as the target-region size and image tags, our framework can be easily extended to other non-linear constraints, e.g., invariant shape moments and other region statistics.

In our second contribution, we propose log-barrier extensions, which approximate Lagrangian optimization of constrained-CNN problems with a sequence of unconstrained losses. Unlike standard interior-point and log-barrier methods, our formulation does not need an initial feasible solution. We report comprehensive weakly supervised segmentation experiments, with various constraints, showing that our formulation outperforms substantially the existing constrained-CNN methods, both in terms of accuracy, constraint satisfaction and training stability.

In our third contribution, we enforce constraints on the boundary of predicted segmentation. Widely used loss functions for CNN segmentation, such as Dice or cross entropy, are based on integrals over the segmentation regions. Unfortunately, for highly unbalanced segmentations, such regional summations have values that differ by several orders of magnitude across classes, which affects training performance and stability. We propose a boundary loss, which takes the form of a distance metric on the space of contours, not regions. This can mitigate the difficulties of highly unbalanced problems because it uses integrals over the interface between regions instead of unbalanced integrals over the regions. Furthermore, a boundary loss complements regional information. Inspired by graph-based optimization techniques for computing activecontour flows, we express a non-symmetric L2 distance on the space of contours as a regional integral, which avoids completely local differential computations involving contour points. This yields a boundary loss expressed with the regional softmax probability outputs of the network, which can be easily combined with standard regional losses and implemented with any existing deep network architecture for N-D segmentation. We report comprehensive evaluations on different unbalanced problems, showing that our boundary loss can yield significant increases in performances while improving training stability.

In a fourth contribution, we investigates a curriculum-style strategy for semi-supervised CNN segmentation, which devises a regression network to learn image-level information such as the size of the target region. These regressions are used to effectively regularize the segmentation network, constraining the softmax predictions of the unlabeled images to match the inferred label distributions. Our framework is based on inequality constraints, which tolerate uncertainties in the inferred knowledge, e.g., regressed region size. It can be used for a large variety of region attributes. We evaluated our approach for left ventricle segmentation in magnetic resonance images (MRI), and compared it to standard proposal-based semi-supervision strategies. Our method achieves competitive results, leveraging unlabeled data in a more efficient manner and approaching full-supervision performance.

In our fifth and last contribution, we propose a novel weakly supervised framework based on several global constraints derived from box annotations. Particularly, we leverage a classical tightness prior to a deep learning setting via imposing a set of constraints on the network outputs. Such a powerful topological prior prevents solutions from excessive shrinking by enforcing any horizontal or vertical line within the bounding box to contain, at least, one pixel of the target region. Furthermore, we integrate our deep tightness prior with a global background emptiness constraint, guiding training with information outside the bounding box. We demonstrate experimentally that such a global constraint is much more powerful than standard cross-entropy for the background class. The ensuing optimization problem is challenging as it takes the form of a large set of inequality constraints on the network outputs. We solve it with a sequence of unconstrained losses based on our log-barrier extensions. This accommodates standard stochastic gradient descent, while avoiding computationally expensive and unstable Lagrangian dual steps and projections. Extensive experiments over two different public data sets and applications (prostate and brain lesions) demonstrate that the synergy between our global tightness and emptiness priors yield competitive performances, approaching full-supervision performances.

All the codes ensuing from this thesis are publicly available, and free to reuse and modify. The functional programming style used makes it easy to integrate new loss functions and constraints, with little-to-no additional coding efforts.

Titre traduit

Réseau profonds contraints appliqué à la segmentation d’imagerie médicale

Résumé traduit

La segmentation sémantique faiblement supervisée, prenant la forme d’images partiellement annotées, fait l’object d’une grande attention académie, puisqu’elle peut limiter le besoin d’annotations (chère à produire) requises par les modèles de réseaux profonds. Imposer des contraintes globales et non linéaires (sous la forme d’inégalités) aux prédictions d’un réseau de neurone peut ainsi guider l’entrainement vers des solutions atanomiquement possibles, et ainsi permettre d’utiliser des informations à priori sur la tâche. Les inégalités sont très flexibles, puisqu’elles ne requiert pas une information précise et parfaite. L’optimisation Lagrangienne standard a très peu été utilisée dans le cadre des réseaux de neurone, principalement à cause du coût de calcul très élevé dû à l’alternance des mises-à-jour explicites entre paramètres et multiplicateurs. Au cours de cette thèse, nous avons testé différents méthodes – pénalités naïves et extension de log-barrier plus formelles – afin de contourner les limitations de l’optimisation Lagrangienne. Les deux méthodes ont produit des résultats significativement meilleurs que les quelques méthodes existantes (limitées quant à elles à de simples contraintes linéaires), ainsi qu’un entraineemnt plus stable avec une meilleure convergence. L’extension des log-barrier, plus puissante, a permis l’utilisation de fonctions plus complexes, et plus compétitives entre elles. Nous présentons des expériences robustes et variées, sur une multitude de tâches de segmentation sémantique ; démontrant à la fois l’efficacité de nos méthodes et la pertinence de l’entrainement sous contrainte dans le contexte de l’imagerie médicale. Tout le code produit par cette thèse est disponible en ligne, et peut être réutilisé et modifié librement.

Type de document:	Mémoire ou thèse (Thèse de doctorat électronique)
Renseignements supplémentaires:	"Manuscript-based thesis presented to École de technologie supérieure in partial fulfillment for the degree of doctor of philosophy". Comprend des références bibliographiques (pages 171-181).
Mots-clés libres:	optimisation sous constraintes, apprentissage profond, imagerie médicale, faible supervision
Directeur de mémoire/thèse:	Directeur de mémoire/thèse Ben Ayed, Ismail
Codirecteur:	Codirecteur Dolz, Jose Granger, Éric
Programme:	Doctorat en génie > Génie
Date de dépôt:	06 oct. 2021 18:18
Dernière modification:	03 nov. 2021 18:49
URI:	https://espace.etsmtl.ca/id/eprint/2734

Gestion Actions (Identification requise)

Dernière vérification avant le dépôt