Guided Intra-Patch Smoothing Graph Filtering for Single-Image Denoising

Yibin Tang,Ying Chen,Aimin Jiang,Jian Li,Yan Zhou,＊and Hon Keung Kwan

1College of Internet of Things Engineering,Hohai University,Changzhou,213022,China

2School of Microelectronics and Control Engineering,Changzhou University,Changzhou,213022,China

3Department of Electrical Engineering,University of Windsor,Ontario,N9B 3P4,Canada

Abstract:Graph filtering is an important part of graph signal processing and a useful tool for image denoising.Existing graph filtering methods,such as adaptive weighted graph filtering(AWGF),focus on coefficient shrinkage strategies in a graph-frequency domain.However,they seldom consider the image attributes in their graph-filtering procedure.Consequently,the denoising performance of graph filtering is barely comparable with that of other state-of-the-art denoising methods.To fully exploit the image attributes,we propose a guided intra-patch smoothing AWGF(AWGF-GPS)method for single-image denoising.Unlike AWGF,which employs graph topology on patches,AWGF-GPS learns the topology of superpixels by introducing the pixel smoothing attribute of a patch.This operation forces the restored pixels to smoothly evolve in local areas,where both intra- and inter-patch relationships of the image are utilized during patch restoration.Meanwhile,a guided-patch regularizer is incorporated into AWGF-GPS.The guided patch is obtained in advance using a maximum-a-posteriori probability estimator.Because the guided patch is considered as a sketch of a denoised patch,AWGF-GPS can effectively supervise patch restoration during graph filtering to increase the reliability of the denoised patch.Experiments demonstrate that the AWGF-GPS method suitably rebuilds denoising images.It outperforms most state-of-the-art single-image denoising methods and is competitive with certain deep-learning methods.In particular,it has the advantage of managing images with significant noise.

Keywords:Graph filtering;image denoising;MAP estimation;superpixel

1 Introduction

Image denoising aims to remove noise from images,thereby benefitting the subsequent analysis and processing of images and videos.Current image denoising methods are divided into two categories:Model- and deep-learning-based methods.Prior to the success of deep learning methods,model-based methods have been used for decades to exploit the intrinsic attributes of images[1–3].Most of them resolve the denoising problem using the internal prior of the target noisy image without considering other images.Therefore,they are often considered as single-image denoising methods.In contrast,deep-learning-based methods employ the external prior from other image databases to recover denoised images.Thus,they can be considered as data-driven methods that require numerous image data[4–6].Flexible feature learning strategies are adopted in these methods via various deep networks.Because of the feature analysis of external images instead of the target image,deep-learning-based methods have a higher denoising performance than the model-based methods.However,in this study,we focus on the model-based denoising method assuming that only a single noisy image is provided.

In general,model-based denoising methods involve a filtering process,where the input and output data are noisy and denoised images,respectively.Thus,various filtering methods are used in different domains such as spatial,transform,and learned domains.In earlier studies,spatial-domain methods,such as bilateral filtering[7]and non-local means filtering[8],were used for the direct operation on pixels or patches.A spatial smoothing filter was used to remove noise-like components from the target image.However,since this smoothing procedure is only performed in local areas or patch groups,spatial filtering has a limited ability to exploit the statistical information of the entire image.To overcome this problem,transform-domain filters are presented by considering the directional structures in the images.Wavelet and Curvelet filters have been successfully employed for their adequate structural description of basis[9–11].A popular denoising method,named block matching and three-dimensional(BM3D)filtering,was proposed using a collaborative spatial-wavelet filter[12,13].It adopts a spatial filtering result to guide sequential wavelet filtering on similar patches.Unfortunately,none of the aforementioned filters are contented-based due to their fixed filter coefficients.Therefore,more advanced models,such as sparse representation and low-rank representation models[3,14–16],have been deployed in different learned domains in recent years.For example,a trilateral weighted sparse coding(TWSC)method is used to estimate data fidelity based on the sparse representation theory[17].A low-rank approximation approach with adaptive regularizer learning(ARLLR)[18]is presented to shrink the eigenvalues of patches and is highly successful in image denoising.However,it is not yet considered as a filter-based model.The current study proves that the low-rank model is equivalent to a subspace graph filter[19].The filtering procedure takes place in a graph subspace supported by the eigenvectors of the patch group.

Graph filtering is an essential component of graph signal processing.Its basic idea is to filter the input signal on the network nodes[20].Once pixels or patches are chosen according to the nodes,graph filtering can adequately fit image denoising.Certain graph polynomial filtering methods are presented to employ various Laplacian matrix regularizers in the existing denoising model[21–23].Several adaptive graph filtering methods are also proposed,by applying different coefficient shrinkage strategies in the graph-frequency domain.In a pioneer work,an idea lowpass graph filter was designed using a full shrinkage approach in the high graph-frequency band,achieving denoising performance comparable with that of BM3D[24–26].Given a patch group,the study proves that the eigenvectors of the Laplacian matrix are a set of graph Fourier bases[27].Recently,an adaptive weighted graph filtering(AWGF)method introduced an effective shrinkage approach in the entire band[19].Although it theoretically builds a bridge from the existing low-rank model to graph filtering,its denoising performance is barely comparable with that of low-rank denoising methods.In the traditional AWGF method,the graph filter is prioritized,whereas patch attributes are seldom considered for image denoising.

Motivated by the recent progress,we propose a guided intra-patch smoothing AWGF(AWGFGPS)method for single-image denoising.Our contributions are twofold.(1)Unlike AWGF that uses graph topology on patches,AWGF-GPS learns the superpixel graph to exploit the pixel smoothing attribute in patches.This operation forces the restored pixels to smoothly evolve in the local area,where both intra- and inter-patch relationships are utilized during patch restoration.(2)A guided-patch regularizer is incorporated into AWGF-GPS.The guided patch is obtained in advance using a maximum-a-posteriori(MAP)probability estimator.By considering the guided patch as a sketch of a denoised patch,AWGF-GPS effectively supervises the patch restoration procedure.Consequently,the reliability of the denoised patch is increased.Experiments demonstrate that our AWGF-GPS method suitably rebuilds denoising images.It outperforms most state-of-the-art model-based methods and is competitive with certain deep-learning methods.

2 Related Work

3 Proposed Method

We propose the AWGF-GPS method to employ both the self-similarity of patches and the local similarity of intra-patch pixels.Fig.1 depicts the filtering flowchart.The superpixel is defined as a pixel set sharing the same location on the patches.A graph is learned using these superpixels to strengthen the smooth attributes of the neighboring pixels.Sequentially,the corresponding graph Fourier bases are obtained from the Laplacian matrix during the graph analysis.The guided patches are evaluated based on their corresponding noisy patches via a MAP estimator.Combined with the graph Fourier bases and the guided patches,the AWGF-GPS model is finally implemented to restore the denoised patches.

3.1 Graph Learning on Superpixels

We obtain the superpixels from the noisy patch group,Y,where each row vector ofYrepresents a superpixel node.Graph learning is then performed on these nodes.Various graph learning methods have been proposed to form graphs with different strategies.However,we prefer a log-model graph learning approach[30],which builds a connected graph without any isolated nodes.This graph learning model is described as

whereWis an adjacent matrix to measure the weighted edges among the superpixel nodes,Zis a pairwise distance matrix for the superpixels,αandβare two weighted coefficients,1 represents a vector of ones,and symbol ° denotes the Hadamard product.

Figure 1:Flowchart of AWGF-GPS denoising

The optimization problem in Eq.(3)can be conveniently solved using the GSP Toolbox[31].Then,the graph Laplacian matrix is given asL=D?W,where the diagonal entry of the degree matrix,D,is the corresponding row sum ofW.Finally,the graph Fourier bases,Ug,are obtained from the eigenvectors ofL.

3.2 Guided Patch Estimation

3.3 AWGF-GPS Model

3.4 AWGF-GPS Denoising Framework

We present the iterative AWGF-GPS denoising framework in Algorithm 1.Given an intermediate noisy image,Y(k),in thek-th iteration,we deal with itsj-th noisy patch group,Yj,in four steps.

Algorithm Input:Noi 1:Image denoising by AWGF-GPS sy image Y,noise standard variance σn.1:Initialize intermediate noisy and denoised images with Y(0)=X(0)=Y.2:For k=1: K do 3:Set intermediate noisy image Y(k)=αX(k?1)+(1 ?α)Y.4:For each patch yj of Y(k) do 5:Find similar patches of yj to form patch group Yj.6:Learn the graph from superpixels using Eq.(3).7:Calculate eigenvectors Ug from adjacent matrix W.8:Define noise standard variance σn,j using Eq.(20).9:Achieve guided patches Xj using Eq.(8)with its SNR Sj and mean patch group ˉXj.10:Obtain shrinkage coefficients ■?σg,i■using Eq.(19).11:Obtain restored patch group Xj using Eq.(14).12:end for 13:Aggregate all■Xj■to form intermediate denoised image X(k).14:end for Output:Optimal denoised image ?X=X(K).

4 Experimental Results

We compare the AWGF-GPS method with several state-of-the-art model-based denoising methods,including BM3D[12],TWSC[17],ARLLR[18],and AWGF[19].A deep-learning image denoising method,named the fast and flexible denoising convolutional neural network(FFDNET)[5],is adopted to show the gap between the model-and deep-learning-based denoising methods.Moreover,because our method is derived from AWGF,the denoising parameters are also inherited from those in AWGF.However,for the weighted coefficients in Eq.(13),parameters

Figure 2:Clean images.From left to right,the images on the top line are named C.Man,House,Peppers,Starfish,Man,Monarch,and Lena,and those on the bottom line are named Airplane,Boats,Parrot,Barbara,Couple,Montage,and Hills

A denoised image comparison of the noise standard deviation ofσn=40 is shown in Fig.3.As a benchmark,BM3D achieves an acceptable denoising performance through collaborative spatial-wavelet filtering.TWSC is better than BM3D with its efficiently learned features of noisy images via sparse representation.ARLLR outperforms the other methods.This indicates that the eigenvalue shrinkage approach is powerful for managing noise.The noise is sufficiently removed in the high “rank-bands” corresponding to the small eigenvalues of the patch group.AWGF emphasizes graph filtering but has less consideration for the image attributes.This leads to a denoising performance that is barely comparable with that of ARLLR.Our method is proven to be the best.The denoised images derived from AWG-GPS are smoothed with a few artificial textures by employing the intra-patch smoothing attribute among the superpixels and the supervision from guided patches.However,FFDNET achieves an outstanding performance in restoring images more naturally.Unlike model-based methods,it benefits from learning the prior from the external images.

The denoised image comparison of the noise standard deviation ofσn= 70 is shown in Fig.4.BM3D obtains good results.TWSC and ARLLR are better than BM3D due to their self-learned features from noisy images.We observe that ARLLR suffers from stronger artificial textures(House).ARLLR is an eigenvalue shrinkage method that uses the statistical information of patch groups via singular value decomposition(SVD).However,it seldom focuses on the pixel smoothing attribute in local areas.AWGF outperforms the aforementioned methods.This proves that graph filtering on the learned graph is useful to alleviate the artificial textures,since the restored patches are required to smoothly evolve on the graph.The proposed AWGF-GPS further smoothens the denoised images,which is owing to the superpixel-based graph,where the similarities between the patches and intra-patch pixels are fully exploited.The shrinkage approach is also efficiently conducted using guided patches.Note that the denoising performance of FFDNET is unsatisfactory.For example,the eaves area of the house is not sufficiently restored.This phenomenon is caused by feature mismatch.In this case,the external prior from the image databases cannot precisely guide the patch reconstruction for denoised images because the features of clean images are significantly contaminated by noise and are difficult to recognize.

Figure 3:Denoised image comparison of the noise deviation of σn=40.(a)House(b)Monarch

Figure 4:Denoised image comparison of the noise deviation of σn=70.(a)House(b)Monarch

Table 1:PSNR(dB)results of different denoising methods

The statistical results of the PSNR and SSIM are listed in Tabs.1 and 2.We use BM3D as a baseline,for it achieves an acceptable performance for all the images and noise levels.TWSC outperforms BM3D because it is a learned domain method.In the sparse representation model,dictionary learning aims to catch patch features as atoms,whereas sparse coding focuses on patch restoration.However,as the noise level increases,the performance of the TWSC degenerates significantly.In this case,the atoms are distorted by direct feature learning from the noisy image.ARLLR is better than the former methods.The shrinkage approach on the patch group is effective in dealing with noise because the relationship among patches is exploited.Our AWGF-GPS method performs the best among these model-based denoising methods.As previously indicated,its coefficient shrinkage strategy is presented on the superpixel graph,where both attributes,within and among the patches,are considered.The guided patch regularizer further enhances the denoising performance.We also note that our method may suffer from an over-smoothing problem.It is inappropriate to deal with images(e.g.,Barbara)that contain strong textures.The AWGF-GPS model attempts to design a lowpass graph filter.Because the components of the texture are centralized in high graph-frequency bands,the AWGF-GPS filter is unsatisfactory.FFDNET is best in terms of the noise standard deviation ofσn≤70 but the worst forσn=100.The use of an external prior is effective in improving denoising performance.However,once the feature mismatch phenomenon occurs,the denoised result significantly deteriorates and becomes inferior to the results of the model-based methods.Compared with FFDNET,our method has the advantage in coping with large noise.

Table 2:SSIM results of different denoising methods

5 Conclusion

We have proposed a guided intra-patch smoothing graph filtering method for single-image denoising.Unlike the traditional AWGF,which only focuses on coefficient shrinkage,the proposed AWGF-GPS method considers more image attributes for denoising.The similarities between patches and intra-patch pixels are exploited by introducing the superpixel operation.Moreover,the guided patches from the MAP estimator provide a reliable optimal direction for the AWGFGPS model.This provides an additional way to supervise patch restoration during graph filtering.Experiments have demonstrated that the AWGF-GPS method outperforms several state-of-the-art model-based denoising methods and is comparable with certain deep-learning methods.

Funding Statement:This work is supported by Natural Science Foundation of Jiangsu Province,China[BK20170306]and National Key R&D Program,China[2017YFC0306100].The initials of authors who received these grants are YZ and JL,respectively.It is also supported by Fundamental Research Funds for Central Universities,China[B200202217]and Changzhou Science and Technology Program,China[CJ20200065].The initials of author who received these grants are YT.

Conflicts of Interest:The authors declare that they have no conflicts of interest to report regarding the present study.