## Introduction

In a previous blog post, [1], I compared Principal Component Analysis (PCA) / Singular Value Decomposition (SVD) and Non-Negative Matrix Factorization (NNMF) over a collection of noised images of digit handwriting from the MNIST data set, [3], which is available in *Mathematica*.

This blog post adds to that comparison the use of Independent Component Analysis (ICA) proclaimed in my previous blog post, [1].

## Computations

The ICA related additional computations to those in [1] follow.

First we load the package IndependentComponentAnalysis.m :

```
Import["https://raw.githubusercontent.com/antononcube/\
MathematicaForPrediction/master/IndependentComponentAnalysis.m"]
```

From that package we can use the function `IndependentComponentAnalysis`

to find components:

```
{S, A} = IndependentComponentAnalysis[Transpose[noisyTestImagesMat], 9, PrecisionGoal -> 4.5];
Norm[noisyTestImagesMat - Transpose[S.A]]/Norm[noisyTestImagesMat]
(* 0.592739 *)
```

Let us visualize the norms of the mixing matrix *A* :

```
norms = Norm /@ A;
ListPlot[norms, PlotRange -> All, PlotLabel -> "Norms of A rows",
PlotTheme -> "Detailed"] //
ColorPlotOutliers[TopOutliers@*HampelIdentifierParameters]
pos = OutlierPosition[norms, TopOutliers@*HampelIdentifierParameters]
```

Next we can visualize the found "source" images:

```
ncol = 2;
Grid[Partition[Join[
MapIndexed[{#2[[1]], Norm[#],
ImageAdjust@Image[Partition[#, dims[[1]]]]} &,
Transpose[S]] /. (# -> Style[#, Red] & /@ pos),
Table["", {ncol - QuotientRemainder[Dimensions[S][[2]], ncol][[2]]}]
], ncol], Dividers -> All]
```

After selecting several of source images we zero the rest by modifying the matrix *A*:

```
pos = {6, 7, 9};
norms = Norm /@ A;
dA = DiagonalMatrix[
ReplacePart[ConstantArray[0, Length[norms]], Map[List, pos] -> 1]];
newMatICA =
Transpose[Map[# + Mean[Transpose[noisyTestImagesMat]] &, S.dA.A]];
denoisedImagesICA = Map[Image[Partition[#, dims[[2]]]] &, newMatICA];
```

## Visual comparison of de-noised images

Next we visualize the ICA de-noised images together with the original images and the SVD and NNMF de-noised ones.

There are several ways to present that combination of images.

## Comparison using classifiers

We can further compare the de-noising results by building digit classifiers and running them over the de-noised images.

```
icaCM = ClassifierMeasurements[digitCF,
Thread[(Binarize[#, 0.55] &@*ImageAdjust@*ColorNegate /@
denoisedImagesICA) -> testImageLabels]]
```

We can see that with ICA we get better results than with PCA/SVD, probably not as good as NNMF, but very close.

## All images of this blog post

## References

[1] A. Antonov, "Comparison of PCA and NNMF over image de-noising", (2016), MathematicaForPrediction at WordPress.

[2] A. Antonov, "Independent Component Analysis for multidimensional signals", (2016), MathematicaForPrediction at WordPress.

[3] Wikipedia entry, MNIST database.