# Comparison of PCA, NNMF, and ICA over image de-noising

## Introduction

In a previous blog post, [1], I compared Principal Component Analysis (PCA) / Singular Value Decomposition (SVD) and Non-Negative Matrix Factorization (NNMF) over a collection of noised images of digit handwriting from the MNIST data set, [3], which is available in Mathematica.

This blog post adds to that comparison the use of Independent Component Analysis (ICA) proclaimed in my previous blog post, [1].

## Computations

The ICA related additional computations to those in [1] follow.

First we load the package IndependentComponentAnalysis.m :

``````Import["https://raw.githubusercontent.com/antononcube/\
MathematicaForPrediction/master/IndependentComponentAnalysis.m"]``````

From that package we can use the function `IndependentComponentAnalysis` to find components:

``````{S, A} = IndependentComponentAnalysis[Transpose[noisyTestImagesMat], 9, PrecisionGoal -> 4.5];
Norm[noisyTestImagesMat - Transpose[S.A]]/Norm[noisyTestImagesMat]
(* 0.592739 *)``````

Let us visualize the norms of the mixing matrix A :

``````norms = Norm /@ A;
ListPlot[norms, PlotRange -> All, PlotLabel -> "Norms of A rows",
PlotTheme -> "Detailed"] //
ColorPlotOutliers[TopOutliers@*HampelIdentifierParameters]
pos = OutlierPosition[norms, TopOutliers@*HampelIdentifierParameters]``````

Next we can visualize the found "source" images:

``````ncol = 2;
Grid[Partition[Join[
MapIndexed[{#2[[1]], Norm[#],
Transpose[S]] /. (# -> Style[#, Red] & /@ pos),
Table["", {ncol - QuotientRemainder[Dimensions[S][[2]], ncol][[2]]}]
], ncol], Dividers -> All]``````

After selecting several of source images we zero the rest by modifying the matrix A:

``````pos = {6, 7, 9};
norms = Norm /@ A;
dA = DiagonalMatrix[
ReplacePart[ConstantArray[0, Length[norms]], Map[List, pos] -> 1]];
newMatICA =
Transpose[Map[# + Mean[Transpose[noisyTestImagesMat]] &, S.dA.A]];
denoisedImagesICA = Map[Image[Partition[#, dims[[2]]]] &, newMatICA];``````

## Visual comparison of de-noised images

Next we visualize the ICA de-noised images together with the original images and the SVD and NNMF de-noised ones.

There are several ways to present that combination of images.

## Comparison using classifiers

We can further compare the de-noising results by building digit classifiers and running them over the de-noised images.

``````icaCM = ClassifierMeasurements[digitCF,
denoisedImagesICA) -> testImageLabels]]``````

We can see that with ICA we get better results than with PCA/SVD, probably not as good as NNMF, but very close.

## References

[1] A. Antonov, "Comparison of PCA and NNMF over image de-noising", (2016), MathematicaForPrediction at WordPress.

[2] A. Antonov, "Independent Component Analysis for multidimensional signals", (2016), MathematicaForPrediction at WordPress.

[3] Wikipedia entry, MNIST database.