Mumford and Shah Model and Its Applications to Image Segmentation and Image Restoration

We present in this chapter an overview of the Mumford and Shah model for image segmentation. We discuss its various formulations, some of its properties, the mathematical framework, and several approximations. We also present numerical algorithms and segmentation results using the Ambrosio–Tortorelli phase-field approximations on one hand, and using the level set formulations on the other hand. Several applications of the Mumford–Shah problem to image restoration are also presented. . Introduction: Description of theMumford and Shah Model An important problem in image analysis and computer vision is the segmentation one, that aims to partition a given image into its constituent objects, or to find boundaries of such objects. This chapter is devoted to the description, analysis, approximations, and applications of the classical Mumford and Shah functional proposed for image segmentation. In [–], David Mumford and Jayant Shah have formulated an energy minimization problem that allows to compute optimal piecewise-smooth or piecewise-constant approximations u of a given initial image g. Since then, their model has been analyzed and considered in depth by many authors, by studying properties of minimizers, approximations, and applications to image segmentation, image partition, image restoration, and more generally to image analysis and computer vision. We denote by Ω ⊂ Rd the image domain (an interval if d = , or a rectangle in the plane if d = ). More generally, we assume that Ω is open, bounded, and connected. Let g : Ω → R be a given gray-scale image (a signal in one dimension, a planar image in two dimensions, or a volumetric image in three dimensions). It is natural and without losing any generality to assume that g is a bounded function in Ω, g ∈ L(Ω). As formulated byMumford and Shah [], the segmentation problem in image analysis and computer vision consists in computing a decomposition Ω = Ω ∪Ω ∪ . . . ∪ Ωn ∪ K of the domain of the image g such that (a) The image g varies smoothly and/or slowly within each Ω i . (b) The image g varies discontinuously and/or rapidly across most of the boundary K between different Ω i . From the point of view of approximation theory, the segmentation problem may be restated as seeking ways to define and compute optimal approximations of a general function g(x) by piecewise-smooth functions u(x), i.e., functions u whose restrictions ui to the pieces Ω i of a decomposition of the domain Ω are continuous or differentiable.   Mumford and ShahModel and its Applications to Image Segmentation and Image Restoration In what follows, Ω i will be disjoint connected open subsets of a domain Ω, each one with a piecewise-smooth boundary, and K will be a closed set, as the union of boundaries of Ω i inside Ω, thus Ω = Ω ∪Ω ∪ . . . ∪ Ωn ∪ K, K = Ω ∩ (∂Ω ∪ . . . ∪ ∂Ωn). The functional E to be minimized for image segmentation is defined by [–], E(u,K) = μ ∫ Ω (u − g)dx + ∫ Ω/K ∣∇u∣dx + ∣K∣, (.) where u : Ω → R is continuous or even differentiable inside each Ω i (or u ∈ H(Ω i)) and may be discontinuous across K. Here, ∣K∣ stands for the total surface measure of the hypersurface K (the counting measure if d = , the length measure if d = , the area measure if d = ). Later, we will define ∣K∣ by Hd−(K), the d −  dimensional Hausdorff measure in Rd . As explained by Mumford and Shah, dropping any of these three terms in (> .), inf E = : without the first, take u = , K = /; without the second, take u = g, K = /; without the third, take for example, in the discrete case K to be the boundary of all pixels of the image g, each Ω i be a pixel and u to be the average (value) of g over each pixel. The presence of all three terms leads to nontrivial solutions u, and an optimal pair (u,K) can be seen as a cartoon of the actual image g, providing a simplification of g. An important particular case is obtained when we restrict E to piecewise-constant functions u, i.e., u = constant ci on each open set Ω i . Multiplying E by μ−, we have μ−E(u,K) = ∑ i ∫ Ω i (g − ci)dx + ∣K∣, where  = /μ. It is easy to verify that this is minimized in the variables ci by setting ci = meanΩ i (g) = ∫Ω i g(x)dx ∣Ω i ∣ , where ∣Ω i ∣ denotes here the Lebesgue measure of Ω i (e.g., area if d = , volume if d = ), so it is sufficient to minimize E(K) = ∑ i ∫ Ω i (g −meanΩ i g) dx + ∣K∣. It is possible to interpret E as the limit functional of E as μ →  []. Finally, the Mumford and Shah model can also be seen as a deterministic refinement of Geman and Geman’s image restoration model []. . Background: The First Variation In order to better understand, analyze, and use the minimization problem (> .), it is useful to compute its first variation with respect to each of the unknowns. Mumford and Shah Model and its Applications to Image Segmentation and Image Restoration   We first recall the definition of Sobolev functions u ∈ W ,(U) [], necessary to properly define a minimizer u when K is fixed. Definition  LetU ⊂ Rd be an open set. We denote byW ,(U) (or by H(U)) the set of functions u ∈ L(Ω), whose first-order distributional partial derivatives belong to L(U). This means that there are functions u, . . . ,ud ∈ L(U) such that ∫ U u(x) ∂φ ∂xi (x)dx = − ∫ U ui(x)φ(x)dx for  ≤ i ≤ d and for all functions φ ∈ C∞c (U). We may denote by ∂u ∂xi the distributional derivative ui of u and by∇u = ( ∂u ∂x , . . . , ∂u ∂xd ) its distributional gradient. In what follows, we denote by ∣∇u∣(x) the Euclidean norm of the gradient vector at x. H(U) = W ,(U) becomes a Banach space endowed with the norm ∥u∥W ,(U) = ∫ U udx + d ∑ i= ∫ U ( ∂u ∂xi )  dx] / . .. Minimizing in uwith K Fixed Let us assume first that K is fixed, as a closed subset of the open and bounded set Ω ⊂ Rd , and denote by E(u) = μ ∫ Ω/K (u − g)dx + ∫ Ω/K ∣∇u∣dx, for u ∈ W ,(Ω/K), where Ω/K is open and bounded, and g ∈ L(Ω/K). We have the following classical results obtained as a consequence of the standard method of calculus of variations. Proposition  There is a unique minimizer of the problem inf u∈W ,(Ω/K) E(u). (.) Proof [] First, we note that  ≤ inf E < +∞, since we can choose u ≡  and E(u) = μ ∫Ω/K g  (x)dx < +∞. Thus, we can denote by m = inf u E(u) and let {uj} j≥ ∈ W ,(Ω/K) be a minimizing sequence such that lim j→∞ E(uj) = m. Recall that for u, v ∈ L,