Image Inpainting Based on Structural Tensor Edge Intensity Model

Jing Wang Yan-Hong Zhou Hai-Feng Sima Zhan-Qiang Huo Ai-Zhong Mi

Jing Wang, Yan-Hong Zhou, Hai-Feng Sima, Zhan-Qiang Huo, Ai-Zhong Mi. Image Inpainting Based on Structural Tensor Edge Intensity Model. International Journal of Automation and Computing. doi: 10.1007/s11633-020-1256-x
Citation: Jing Wang, Yan-Hong Zhou, Hai-Feng Sima, Zhan-Qiang Huo, Ai-Zhong Mi. Image Inpainting Based on Structural Tensor Edge Intensity Model. International Journal of Automation and Computing. doi: 10.1007/s11633-020-1256-x

doi: 10.1007/s11633-020-1256-x

Image Inpainting Based on Structural Tensor Edge Intensity Model

More Information
    Author Bio:

    Jing Wang received the B. Sc. degree in computer science and technology from Henan University of Science and Technology, China in 2006, and the Ph. D. degree in computer application technology from College of Computing and Communication Engineering, Graduate University of Chinese Academy of Science, China in 2012. Currently, she is an associate professor in College of Computer Science and Technology, Henan Polytechnic University, China.Her research interests include image processing computer vision and machine learning.E-mail: wjasmine@hpu.edu.cnORCID iD: 0000-0002-3288-2111

    Yan-Hong Zhou received the B. Eng. degree in computer science and technology from Fuyang Teachers College, China in 2018. Currently, she is a master student in software engineering at College of Computer Science and Technology, Henan Polytechnic University, China.Her research interests include image processing and computer vision.E-mail: zhouyanhong_zyh123@163.com

    Hai-Feng Sima received the B. Eng. and M. Eng. degrees in computer science from Zhengzhou University, China in 2004 and 2007, respectively, and the Ph. D. degree in software and theory from Beijing Institute of Technology, China in 2015. Since 2007, he has been with Faculty of Henan Polytechnic University, China, and is currently a lecturer with College of Computer Science and Technology, Henan Polytechnic University, China.His current research interests include pattern recognition, image processing, image segmentation and image classification.E-mail: smhf@hpu.edu.cn (Corresponding author)ORCID iD: 0000-0002-2049-3637

    Zhan-Qiang Huo received the B. Sc. degree in mathematics and applied mathematics from the Hebei Normal University of Science and Technology, China in 2003. He received the M. Sc. degree in computer software and theory and the Ph. D. degreD. degree in circuit and system from Yanshan University, China in 2006 and 2009. Currently, he is an associate professor in the College of Computer Science and Technology, Henan Polytechnic University, China.His research interests include computer vision and machine learning.E-mail: hzq@hpu.edu.cn

    Ai-Zhong Mi received the M. Sc. degree in computer and application from Guangxi University, China in 2005, and the Ph. D. degree in computer application technology from University of Science and Technology Beijing, China in 2009. He is currently an associate professor in College of Computer Science and Technology, Henan Polytechnic University, China.His research interests include pattern recognition and network security.E-mail: miaizhong@163.com

图(8) / 表(3)
计量
  • 文章访问数:  9
  • HTML全文浏览量:  26
  • PDF下载量:  7
  • 被引次数: 0
出版历程
  • 收稿日期:  2020-06-13
  • 录用日期:  2020-09-16
  • 网络出版日期:  2020-11-28

Image Inpainting Based on Structural Tensor Edge Intensity Model

doi: 10.1007/s11633-020-1256-x
    作者简介:

    Jing Wang received the B. Sc. degree in computer science and technology from Henan University of Science and Technology, China in 2006, and the Ph. D. degree in computer application technology from College of Computing and Communication Engineering, Graduate University of Chinese Academy of Science, China in 2012. Currently, she is an associate professor in College of Computer Science and Technology, Henan Polytechnic University, China.Her research interests include image processing computer vision and machine learning.E-mail: wjasmine@hpu.edu.cnORCID iD: 0000-0002-3288-2111

    Yan-Hong Zhou received the B. Eng. degree in computer science and technology from Fuyang Teachers College, China in 2018. Currently, she is a master student in software engineering at College of Computer Science and Technology, Henan Polytechnic University, China.Her research interests include image processing and computer vision.E-mail: zhouyanhong_zyh123@163.com

    Hai-Feng Sima received the B. Eng. and M. Eng. degrees in computer science from Zhengzhou University, China in 2004 and 2007, respectively, and the Ph. D. degree in software and theory from Beijing Institute of Technology, China in 2015. Since 2007, he has been with Faculty of Henan Polytechnic University, China, and is currently a lecturer with College of Computer Science and Technology, Henan Polytechnic University, China.His current research interests include pattern recognition, image processing, image segmentation and image classification.E-mail: smhf@hpu.edu.cn (Corresponding author)ORCID iD: 0000-0002-2049-3637

    Zhan-Qiang Huo received the B. Sc. degree in mathematics and applied mathematics from the Hebei Normal University of Science and Technology, China in 2003. He received the M. Sc. degree in computer software and theory and the Ph. D. degreD. degree in circuit and system from Yanshan University, China in 2006 and 2009. Currently, he is an associate professor in the College of Computer Science and Technology, Henan Polytechnic University, China.His research interests include computer vision and machine learning.E-mail: hzq@hpu.edu.cn

    Ai-Zhong Mi received the M. Sc. degree in computer and application from Guangxi University, China in 2005, and the Ph. D. degree in computer application technology from University of Science and Technology Beijing, China in 2009. He is currently an associate professor in College of Computer Science and Technology, Henan Polytechnic University, China.His research interests include pattern recognition and network security.E-mail: miaizhong@163.com

English Abstract

Jing Wang, Yan-Hong Zhou, Hai-Feng Sima, Zhan-Qiang Huo, Ai-Zhong Mi. Image Inpainting Based on Structural Tensor Edge Intensity Model. International Journal of Automation and Computing. doi: 10.1007/s11633-020-1256-x
Citation: Jing Wang, Yan-Hong Zhou, Hai-Feng Sima, Zhan-Qiang Huo, Ai-Zhong Mi. Image Inpainting Based on Structural Tensor Edge Intensity Model. International Journal of Automation and Computing. doi: 10.1007/s11633-020-1256-x
    • Image restoration (i.e., image de-noising, non-blind de-blurring and image inpainting)[1] can help recover different kinds of images. Among them, image inpainting refers to using the reliable information remaining in the image to recover the target area (missing or damaged parts). The inpainting technique relies on the color and structural information in the image, ensuring the reconstructed image is visually reasonable. Now that digital images have become a part of our lives, people have more requirements for images, i.e., restoring the images that have been damaged due to improper storage and removing unnecessary parts. This technique also plays an important role in the post-processing of videos and movies, i.e., removing watermarks and recovering damaged vintage films[2, 3]. Because of the wide application of image inpainting, this technology has attracted the attention of a large number of researchers. It is also an important task in the field of computer vision.

      In recent years, image inpainting has made great progress, using methods based on partial differential equations (PDEs), exemplar-based techniques and methods based on sparse representations. The partial differential equation image inpainting algorithm[4] is a method based on thermal diffusion, of which the curvature-driven diffusions (CDD) model[5] and the total variation (TV) model[6] are two typical ones. Li et al.[7] proposed an improved TV algorithm in which the calculation of the diffusion coefficient is determined by the distance and direction between the pixels in the target area and the pixels in the domain. However, diffusion-based methods are more suitable for inpainting non-textured areas, because the texture information will be blurred as the information diffuses. Yang et al.[8] applied a structural tensor to the PDE algorithm to improve this defect. The advantage of the algorithm based on partial differential equations[4] is that it deals with small damaged areas (i.e., scratches). For areas with large damage or images with rich structural information, the algorithm often does not get the expected results. Therefore, an exemplar-based image inpainting method is proposed. In the exemplar-based image inpainting algorithm, determining the repair order of image edges and the patch lookup strategy are general steps. This inpainting method is good at dealing with images with large area damage. But, the exemplar-based image inpainting algorithm has two major problems: the failure of the priority function and the incomplete matching patch search strategy. Many inpainting algorithms for these problems have been proposed and the details of these algorithms have been introduced in [9]. Li et al.[10] subtracted a constant from the confidence term to avoid the priority function failing. Wang et al.[11] improved the robustness of the algorithm by introducing regularization factors to adjust the priority function, and using a method which combines a modified sum of squared differences (SSD) and normalized cross-correlation (NCC) to adjust the search strategy for the matching patch. Liu et al.[12] used the structural tensor to construct a local measurement function to optimize the priority and proposed a matching patch search strategy. The method in [12] contributes to the structural fidelity of the image inpainting results. In [13], the size of the matching patch is calculated by using the structure tensor, which is more suitable for the inpainting of various texture structure images. Zhang et al.[14] proposed a nonlocal patch tensor-based visual data completion algorithm, and the experimental results on real-world datasets showed the superiority of the method. In addition, tensor theory is widely used. It was used in object recognition in 1988[15]. The semi-tensor product of matrices is used to show whether a hyper-networked evolutionary games (HNEG) is potential and how to calculate the potential[16]. In order to consider the overall structure contour of the image, some of the literature[17] used image pyramid and PatchMatch algorithms[18] to solve the problem of correct reconstruction of texture and structure in image. Liu et al.[19] combined the image pyramid with the Criminisi algorithm[20] in order to use the inpainting of the upper layer of the pyramid to guide the inpainting of the lower layer. This method uses the results of the upper layer to improve the priority function to obtain a more reliable inpainting order. In addition, research found that the same sparsity exists in the target area and the known area of the image. So, there is also an inpainting method based on sparse representation[21, 22]. In recent years, deep learning is becoming popular. It is widely applied in many domains and has obtained many encouraging results. Yao et al.[23] introduced the application of deep learning in healthcare extensively. Ha et al.[1] gave an overview of recent advances in deep learning-based models and methods that have been applied to single image super-resolution tasks. Wang et al.[24] proposed an image inpainting method for large-scale irregular masks.

      In exemplar-based image inpainting algorithms, the quality of the inpainting is affected by both the edge filling priority and the patch lookup strategy. While the filling progresses, both the data term and the confidence term will gradually approach zero, causing the priority function to fail. Moreover, the patch lookup strategy relies on the single color feature to search for the patch. For images with complex texture and structure, it is easy to match the wrong patches in inpainting. Therefore, for the two problems mentioned above, no matter which of these two parts has a slight error, it will directly affect the subsequent filling effect. The inpainting of this algorithm depends on the previously filled information. When an error occurs in the previous inpainting, the subsequent inpainting information also changes. It will cause a chain reaction in the later inpainting, which leads to inconsistent results. In addition, this algorithm is also susceptible to the shape of the target area. When the selected target area changes a little bit, the pixels at the edges will change greatly. Therefore, this paper proposes an image inpainting technique based on the structural tensor edge intensity model. It uses a progressive scan inpainting method to avoid calculating the priority of edge filling patches. When it finds the target area, it starts to search for matching patches. In order to accurately identify the structural information in the image, the constructed edge strength model is used to search for matching patches to ensure patches are found accurately. Framework of the proposed algorithm is shown in Fig. 1.

      Figure 1.  Framework of the proposed algorithm. Colored figures are available in the online version.

    • As shown in Fig. 2, Ω is the source area, Φ is the target area, Ψx (a square patch centered on the pixel x, the size is $ 9\times 9 $ pixels) is the scan patch, and Ψp (the size is $ 6\times 6 $ pixels) is the patch to be filled. The area Ψz is a source area adjacent to the patch Ψp. As shown in Fig. 2(a), the area Ψz is divided into two parts: A (the blue box on the left of patch Ψp) and B (the green box above patch Ψp). When the scan patch Ψx is in the state of Fig. 2(b), the patch Ψx is the patch to be matched, the patch Ψy is the best matching patch, and the patch Ψq is the best filling patch. In Fig. 2(c), ${{h}}$ is the length of the entire target area Φ, w is the width of the entire target area Φ, $\widehat{{{h}}}$ is the distance from the bottom of the patch ${{\varPsi }}_{{x}}$ to the lower boundary of the target area Φ, and $\widehat{{{w}}}$ is the distance from the right side of the patch Ψx to the right boundary of the target area Φ. Let $\widehat{{{h}}}/{{h}}$ be the value Lh and $\widehat{{{w}}}/{{w}}$ be the value Lw. The adjacent square patches to the left and above of the patch to be filled Ψp are denoted as ${{\varPsi }}_{{p}}^{{A}}$ and ${{\varPsi }}_{{p}}^{{B}}$ (the size is $ 6\times 6 $ pixels).

      Figure 2.  Symbol definition: (a) Source area $\Omega $, target area $\varPhi$ and scan patch $\varPsi_x $; (b) Patch to be matched $\varPsi_x$ and the best matching patch $\varPsi_y$; (c) Schematic diagram of the degree of image to be inpainted.

    • To define the structure tensor, let the gradient of the image be

      $$ {\bf{\nabla}} {{I}}={\left[{I}_{x},{I}_{y}\right]}^{\rm{T}} $$ (1)

      where Ix and Iy are partial deviations of pixels in the x direction and y direction. The structural tensor E of the image can be defined as

      $$ {{E}}={\bf{\nabla}} {{I}}{{\bf{\nabla}} {{I}}}^{\rm{T}}=\left[ {\begin{array}{*{20}{c}} {I_x^2}&{{I_x}{I_y}}\\ {{I_x}{I_y}}&{I_y^2} \end{array}} \right]. $$ (2)

      E is a symmetric and semi-positive two-dimensional matrix, it has two non-negative eigenvalues λ1 and λ2. The edge intensity is using λ1 and λ2 to define the coherence of the image data. The edge intensity at pixel $ \left(i,j\right) $ is

      $$ { H\left(i,j\right)=\left({{\lambda }}_{1}-{{\lambda }}_{2}\right)}^{2} .$$ (3)

      The eigenvalues λ1 and λ2 can be used to analyze the local image structure. When both eigenvalues are approaching zero, it means that the gray value has little changes in all directions and they are located in a flat area. When one eigenvalue is much larger than the other eigenvalue (toward zero), it means that the image has a significant edge structure. When both eigenvalues are greater than zero and one eigenvalue is much larger than the other eigenvalue, it means that a corner area exists.

      In the traditional exemplar-based image inpainting algorithm, only the color features are used to search for matching patches. As a result, the structural information around the target area cannot be correctly identified. As shown in Fig. 3, the source area of the patch to be matched Ψx already has a straight line structure and a part of the corner structure. If only a single color feature is used to find the best matching patch, it is easy to miss the corner information near the patch Ψx to be matched. So, the wrong matching patch Ψy2 is found, and the correct matching patch Ψy1 is missed. As a result, the straight line structure is inpainted, but the information of the corner points is lost. This causes the image inpainting to fail.

      Figure 3.  Incorrect reconstruction of patch $\varPsi_x $ local structure

      The structure tensor can identify the characteristics of different texture structures in the image. By calculating the edge intensities of the patches Ψx and Ψy (represented by the edge intensities at the central pixels ${{x}}\left(i,j\right)$ and ${{y}}\left(i,j\right)$ of patches Ψx and Ψy), the accuracy of identifying the structural information in the target image is improved. The two edge intensities are denoted as $ H({i}_{x}, {j}_{x}) $ and $ H({i}_{y}, {j}_{y}) $. In order to judge the coherence between the patch to be matched ${{\varPsi }}_{{x}}$ and the matching patch Ψy, an edge intensity model of the image is constructed:

      $$ \overline{H}\left(x,y\right)=\dfrac{{\left({{H}}\left({i}_{x}, {j}_{{x}}\right)+{{H}}\left({i}_{{{y}}}, {j}_{{{y}}}\right)\right)}^{2}}{\rm{\omega }} $$ (4)

      where $ {\rm{\omega }} $ is a constant ($ {\rm{\omega }}=0.001 $). From (3) and the characteristics of the structural tensor eigenvalues, it can be seen that when the value of $ \overline{H}\left(x,y\right) $ tends to zero or equals zero, it means that Ψx and Ψy have low coherence and the pixels are located in the flat region. If the value of $ \overline{H}\left(x,y\right) $ is much larger than zero, it means that Ψx and Ψy are highly coherent and there are structure information around pixels.

    • There are usually two major problems for the exemplar-based image inpainting approach: the priority function is prone to failure and the patch lookup strategy is not perfect. This paper proposes an image inpainting technique based on the structural tensor edge intensity model method. The implementation steps are as follows.

    • In exemplar-based image inpainting algorithms, the priority is determined by the product of the data term and the confidence term. However, as the filling progresses, both the data term and the confidence term will gradually approach zero, making the priority function invalid. As a result, the order of edge filling cannot be determined, and it can cause filling errors easily.

      In order to avoid being affected by the priority function, the algorithm uses a progressive scan method in inpainting. The algorithm scans the image progressively with scan patch Ψx, and moves one pixel length at a time. When the scan patch Ψx finds the target area Φ, it continues to move the scan patch Ψx until Ψx and Φ have the largest coincident area.

      When it satisfies (5), then it starts searching for matching patches.

      $$ {{\varPsi }}_{\rm{z}}\cap {\varPhi }={\varnothing } .$$ (5)
    • The single color feature is considered in the traditional patch lookup strategy. It ignores the structural information of the image. In this paper, the edge intensity model is introduced to identify the local image structural features. The patches similarity function is defined as

      $$ \overline{D}\left({\varPsi }_{x},{\varPsi }_{y}\right)=D\left({\varPsi }_{x},{\varPsi }_{y}\right)+\alpha \times \overline{H}\left(x,y\right) $$ (6)

      where $ {\rm{\alpha }} $ is the intensity adjustment factor (empirical value α = 10), $ \overline{H}\left(x,y\right) $ is the edge intensity between the patch to be matched Ψx and the matching patch Ψy, $D\left({\varPsi }_{x},{\varPsi }_{y}\right)$ is expressed as

      $$\begin{aligned} &D\left({\varPsi }_{x},{\varPsi }_{y}\right)=\\ &\;\;\;\;\sum \sqrt{{\left({R}_{{{\varPsi }}_{{x}}}-{R}_{{{\varPsi }}_{{{y}}}}\right)}^{2}+{\left({G}_{{{\varPsi }}_{{x}}}-{G}_{{{\varPsi }}_{{{y}}}}\right)}^{2}+{\left({B}_{{{\varPsi }}_{{x}}}-{B}_{{{\varPsi }}_{{{y}}}}\right)}^{2}}. \end{aligned}$$ (7)

      The SSD distance $ D\left({\varPsi }_{x},{\varPsi }_{y}\right) $ is the similarity between each patch Ψy in the source area and patch Ψx, where R, G and B represent the pixel values corresponding to each channel.

      In this paper, the scan patch ${{\varPsi }}_{{x}}$ also provides the basis for the calculation of the edge intensity $ H({i}_{x}, {j}_{x}) $. In Fig. 2(b), the area Ψz contains more reliable information, which can accurately identify the texture and structure around the patch to be filled Ψp, then search for the best matching patch Ψy according to (6) all over the image. In order to improve the accuracy of matching patches, a filtering strategy is defined as

      $$ {\varPsi _{\hat y}} < {\rm{arg}}\mathop {{\rm{min}}}_{{\varPsi _{{y}}} \in \Omega } \overline D\left( {{\varPsi _x},{\varPsi _y}} \right) \times err$$ (8)

      where $ \overline{D}\left({\varPsi }_{x},{\varPsi }_{y}\right) $ is the similarity between the patch to be matched Ψx and the matching patch Ψy, $ err $ is the error adjustment factor: $ err={\rm{\lambda }}+{\rm{\beta }}\times H({i}_{x}, {j}_{x}) $, experience values $ {\rm{\lambda }}=1.3 $, $ {\rm{\beta }}=10 $, $ H({i}_{x}, {j}_{x}) $ is the edge intensity of the patch to be matched Ψx.

      After filtering, there may be three cases of the matching patch Ψy that exist. The area Ψz is divided into two areas A and B (as shown in Fig. 2(a)).

      1) The matching patch Ψy matches the image information of the corresponding area A and area B, then it is the best matching patch with the best filling performance.

      2) The matching patch Ψy matches the image information of the corresponding area A or area B, then some of them have reliable filling information.

      3) The matching patch Ψy matches part of the image information of the corresponding area A and area B, but not all of them completely match.

      If the filtered matching patch Ψy does not exist in 1), the best matching patch needs further matching from 2) and 3). Through analysis, if there is a matching difference between the two matching patches, there will be structural information in one of the two areas. This indicates that the image information of areas A and B (as shown in Fig. 2(a)) should have a large difference. Structure information is the skeleton of the image information, so a direction with structure information is chosen in these two areas to inpainting first.

    • If structural information is allowed to spread excessively, the inpainting will fail. So we define a balance factor to limit the excessive expansion of the structure. This balance factor is based on the image gradient and the degree of image inpainting. First determine the degree of inpainting of the current target area Φ. The degree of inpainting is defined as the ratio of the width of the remaining inpainting area to the width of the whole target area. The smaller the ratio is, the greater the degree of inpainting in a certain direction is. The higher the degree of repair, the less difficult it is for inpainting. Therefore, the direction with a large degree of inpainting is inpainted first. As shown in Fig. 2(c), calculate the inpainting degree of the patch to be matched Ψx in both vertical and horizontal directions. Let Lh and Lw denote the inpainting degree of the patch to be matched Ψx in these two directions. Note the modulus values of the gradients of ${{\varPsi }}_{{p}}^{{A}}$ and ${{\varPsi }}_{{p}}^{{B}}$ (adjacent patches of the patch Ψp to be filled, as shown in Fig. 2(c)) are ${\left|{{I}}\right|}_{{A}}$ and ${\left|{{I}}\right|}_{{B}}$. The larger the modulus value is, the stronger the structural information contained in the image is. Therefore, the area with strong structural information is preferentially inpainted.

      The balance factor determines the direction in which the patch to be matched Ψx is preferentially matched. As shown in Fig. 2(a), it performs priority matching according to the matching information in the area A or the area B. The balance factor $ V $ can be expressed as

      $$ V=\frac{\left|{{I}}\right|}{{L}} $$ (9)

      where $ \left|{{I}}\right| $ is ${\left|{{I}}\right|}_{{A}}$ or ${\left|{{I}}\right|}_{{B}}$, and L is ${{L}}_{{{h}}}$ or ${{L}}_{{{w}}}$.

      Finally, use (7) to calculate the SSD distance ${{D}}\left({{\varPsi }}_{{p}}^{{r}},{{\varPsi }}_{{q}}\right)$ of ${{\varPsi }}_{{p}}^{{A}}$ or ${{\varPsi }}_{{p}}^{{B}}$ (as shown in Fig. 2(c)) and the filled patch Ψq (r is A or B). Take the average of these distances as Ave:

      $$ Ave=\frac{\sum _{i=1}^{n}{{D}}\left({{\varPsi }}_{{p}}^{{r}},{{\varPsi }}_{{q}}\right)}{{n}} $$ (10)

      where ${{\varPsi }}_{{p}}^{{r}}$ is ${{\varPsi }}_{{p}}^{{A}}$ or ${{\varPsi }}_{{p}}^{{B}}$, n is the total number of matching patches in Ψŷ (matching patches filtered by (8)). The larger the $ {\rm{S}}{\rm{S}}{\rm{D}} $ distance between the two patches is, the smaller the similarity between the two patches is. The smaller the $ {\rm{S}}{\rm{S}}{\rm{D}} $ distance is, the higher the similarity between the two patches is. Therefore, among the candidate patches whose distance $ D\left({\varPsi }_{p}^{r},{\varPsi }_{q}\right) $ is smaller than Ave, we select the matching patch which is most similar to the patch Ψp as the best filling patch Ψq, and the pixels in the patch Ψq are copied to the corresponding position in the patch Ψp.

      The pseudo code of the algorithm in this paper is shown in Algorithm 1. Fig. 4 presents the process of four groups searching for matching patch in the image inpainting of Sill. In Fig. 4(a) State 1, the white patches represent the matching patches left after being filtered by (8). In Fig. 4(b) State 2, the blue patches represent further filtering of white matching patches using the balance factor V. In Fig. 4(c) State 3, the red patch is the best matching patch. It can be seen from the four pairs of matching patches search process diagrams that the matching patches search is correct whether it is a flat area or a structural area. It shows that our search strategy is effective in finding the best matching patch. The complete image inpainting process of Sill is shown in the animation (Online Resource 1).

      Figure 4.  Process of searching for matching patch in the image inpainting of Sill: (a) State 1; (b) State 2; (c) State 3.

      Algorithm 1. Image inpainting based on structural tensor edge intensity model

      Input: Target image, marker color

      Output: Inpainted image

      1) Use the 9 × 9 scan patch Ψx as the basic patch for scanning. Scan the target image progressively from top to bottom and from left to right, when the scan patch Ψx finds the target area Φ, it continues to move the scan patch Ψx;

      2) while ${{\varPsi }}_{{z}}\cap {\varPhi }={\varnothing }$ do

      i) Use patch similarity (6) to perform a global search on the image and search for matching patches for the patch to be matched Ψx;

      ii) Use (8) to filter the matching patches in Step i);

      iii) Determine the priority matching direction by calculating the balance factor V of the patch to be matched Ψx;

      iv) Use (7) to calculation the SSD distance $ D\left({\varPsi }_{p}^{r},{\varPsi }_{q}\right) $ between the patch ${{\varPsi }}_{{p}}^{{A}}$ or ${{\varPsi }}_{{p}}^{{B}}$ and the filling patch Ψq, and average these distances using (10);

      v) Among the matching patches whose distance $ D\left({\varPsi }_{p}^{r},{\varPsi }_{q}\right) $ is smaller than the average Ave, the matching patch most similar to the patch to be filled Ψp is the best filling patch Ψq. Then, the pixels in the best filling patch Ψq are correspondingly copied to the corresponding positions in the patch to be filled Ψp.

      3) end while

      4) Repeat the above steps until the image is inpainted.

    • Visual perception is generally used to evaluate the results of image inpainting. In addition, it can be illustrated by using the structural similarity index measure (SSIM), the peak signal to noise ratio (PSNR) value and the time required for inpainting. The algorithm demonstrates the effectiveness of the experiments in this paper by comparing with the methods in [20], [11], and [10].

      To test our method, we check its inpainting ability in various scene images as shown Figs. 5-8. Among them, the column (a) is the target image, the columns (b), (c) and (d) are the inpainting results of the methods in [20], [11] and [10], and the column (e) is the inpainting results of our method.

      Figure 5.  Inpainting results of fence, window and stair: (a) Target image; (b) Method in [20]; (c) Method in [11]; (d) Method in [10]; (e) Our method.

      Figure 8.  Object removal for images people, tourist and boat: (a) Target image; (b) Method in [20]; (c) Method in [11]; (d) Method in[10]; (e) Our method.

    • Figs. 5 and 6 show the inpainting of damaged images. They are the fence, window, stair, island, mountain and highway images, respectively.

      Figure 6.  Inpainting results of island, mountain and highway: (a) Target image; (b) Method in [20]; (c) Method in [11]; (d) Method in [10]; (e) Our method.

      In Fig. 5, there are some images with complex texture and structures. In Figs. 5(b)-5(d), we can see that the results of methods in [20], [11] and [10] show incorrect information. This is because failure to consider the surrounding features of the image structure leads to repair failure. For example, in the inpainting of the images fence and window, the methods of [20] and [11] have broken the linear structure and cross structure. The method of [10] cannot recover the linear structure of fence and stair, but it reconstructs the cross structure of window. In the inpainting of stair, the methods in [20] and [10] are incorrect in rebuilding the linear structure of the step. However, our method restores the incorrect structure of the images of fence, window and stair.

      There are three scene images in Fig. 6. In the inpainting of images island and mountain, the method in [20] copies the wrong information into the sky, and the methods in [11] and [10] are wrong in rebuilding trees and mountains. In this case, our method can rebuild the trees and sky. And our method and methods in [20], [11] and [10] have achieved good results in the inpainting of highway.

      Tables 1 and 2 are the experimental evaluation of the above six groups using the indicators PSNR and SSIM. These two indicators can reflect the inpainting effect of different methods. The results show that our method is superior to the others in [20], [11] and [10].

      Table 1.  PSNR of different algorithms to inpainting images (dB)

      Image (size)Method in [20]Method in [11]Method in [10]Our method
      Fence (227×227)26.456727.423827.966833.6871
      Window (202×256)32.422331.853234.822038.1820
      Stair (392×258)30.877832.319429.982332.9061
      Island (128×128)23.355425.693026.408030.1055
      Mountain (300×200)22.167721.977825.621827.9696
      Highway (240×212)30.855330.699029.894531.6898

      Table 2.  SSIM of different algorithms to inpainting images

      Image (size)Method in [20]Method in [11]Method in [10]Our method
      Fence (227×227)0.88430.84260.87390.9026
      Window (202×256)0.92010.90590.93190.9468
      Stair (392×258)0.91390.92210.90660.9280
      Island (128×128)0.84440.85680.87880.8879
      Mountain (300×200)0.81020.78750.85470.8592
      Highway (240×212)0.84460.84010.82900.8545
    • The inpainting images of the removed object are shown in Figs. 7 and 8. These are images inpainted with highly textured irregular patterns. They are the girl, woman, fox, people, tourist and boat images, respectively. The object removal means to remove the extra objects in the image, i.e., the people in the images girl and woman, the fox lying on wood, the people and car in the images people and tourist, and the sailing ship in the image boat.

      Figure 7.  Object removal for images girl, woman and fox: (a) Target image; (b) Method in [20]; (c) Method in [11]; (d) Method in [10]; (e) Our method.

      Table 3 lists a time comparison of several methods. From Table 3, we can see that compared with the three methods in [20], [11] and [10], our method can not only save inpainting time but also ensure that inpainting results meet the visual requirements.

      Table 3.  Image restoration time by different methods (s)

      Image (size)Method in [20]Method in [11]Method in [10]Our method
      Fence (227×227)41.1230695.431943.568340.5685
      Window (202×256)12.3578139.361613.039812.2736
      Stair (392×258)15.1306272.137517.850014.1943
      Island (128×128)15.2993112.654415.844114.0476
      Mountain (300×200)38.9698540.881150.818536.0710
      Highway (240×212)35.1901758.211845.167334.0085
      Girl (384×256)54.50841861.666.005543.5231
      Woman (128×128)8.621271.90.859.27838.3218
      Fox (128×128)4.841142.82574.79804.7057
      People (128×128)4.840534.44884.91184.6854
      Tourist (240×180)21.7440371.810622.071120.6946
      Boat (128×128)10.838362.356911.717510.1854
    • This paper proposes an image inpainting algorithm based on the structure tensor edge intensity model. It uses a progressive scan method to avoid the edge filling order being affected by the priority function. An edge intensity model is introduced to the patch lookup strategy. Meanwhile, the structural information and color information of the image are considered to ensure the correctness of the matching patch. The effectiveness of the algorithm is proved by multiple simulation experiments in Matlab. Our experiments are carried out on an Intel core i5CPU@1.6GHz. For the reconstruction of damaged images and the removal of objects, the results of inpainting are visually reasonable. In addition, the PSNR and SSIM indicators are used to compare with different algorithms. Experiments show that the algorithm we proposed in this paper is satisfactory for image inpainting of different types of structures. Moreover, by comparing the inpainting time of several algorithms, it also illustrates the high efficiency of our algorithm.

    • This work was supported by National Science Foundation of China (Nos. 61401150, 61602157 and 61872311), Key Science and Technology Program of Henan Province (Nos.182102210053 and 202102210167), Excellent Young Teachers Program of Henan Polytechnic University (No. 2019XQG-02).

参考文献 (24)

目录

    /

    返回文章
    返回