Introduction
In a previous blog post, [1], I compared Principal Component Analysis (PCA) / Singular Value Decomposition (SVD) and Non-Negative Matrix Factorization (NNMF) over a collection of noised images of digit handwriting from the MNIST data set, [3], which is available in Mathematica.
This blog post adds to that comparison the use of Independent Component Analysis (ICA) proclaimed in my previous blog post, [1].
Computations
The ICA related additional computations to those in [1] follow.
First we load the package IndependentComponentAnalysis.m :
Import["https://raw.githubusercontent.com/antononcube/\
MathematicaForPrediction/master/IndependentComponentAnalysis.m"]
From that package we can use the function IndependentComponentAnalysis
to find components:
{S, A} = IndependentComponentAnalysis[Transpose[noisyTestImagesMat], 9, PrecisionGoal -> 4.5];
Norm[noisyTestImagesMat - Transpose[S.A]]/Norm[noisyTestImagesMat]
(* 0.592739 *)
Let us visualize the norms of the mixing matrix A :
norms = Norm /@ A;
ListPlot[norms, PlotRange -> All, PlotLabel -> "Norms of A rows",
PlotTheme -> "Detailed"] //
ColorPlotOutliers[TopOutliers@*HampelIdentifierParameters]
pos = OutlierPosition[norms, TopOutliers@*HampelIdentifierParameters]
Next we can visualize the found "source" images:
ncol = 2;
Grid[Partition[Join[
MapIndexed[{#2[[1]], Norm[#],
ImageAdjust@Image[Partition[#, dims[[1]]]]} &,
Transpose[S]] /. (# -> Style[#, Red] & /@ pos),
Table["", {ncol - QuotientRemainder[Dimensions[S][[2]], ncol][[2]]}]
], ncol], Dividers -> All]
After selecting several of source images we zero the rest by modifying the matrix A:
pos = {6, 7, 9};
norms = Norm /@ A;
dA = DiagonalMatrix[
ReplacePart[ConstantArray[0, Length[norms]], Map[List, pos] -> 1]];
newMatICA =
Transpose[Map[# + Mean[Transpose[noisyTestImagesMat]] &, S.dA.A]];
denoisedImagesICA = Map[Image[Partition[#, dims[[2]]]] &, newMatICA];
Visual comparison of de-noised images
Next we visualize the ICA de-noised images together with the original images and the SVD and NNMF de-noised ones.
There are several ways to present that combination of images.
Comparison using classifiers
We can further compare the de-noising results by building digit classifiers and running them over the de-noised images.
icaCM = ClassifierMeasurements[digitCF,
Thread[(Binarize[#, 0.55] &@*ImageAdjust@*ColorNegate /@
denoisedImagesICA) -> testImageLabels]]
We can see that with ICA we get better results than with PCA/SVD, probably not as good as NNMF, but very close.
All images of this blog post
References
[1] A. Antonov, "Comparison of PCA and NNMF over image de-noising", (2016), MathematicaForPrediction at WordPress.
[2] A. Antonov, "Independent Component Analysis for multidimensional signals", (2016), MathematicaForPrediction at WordPress.
[3] Wikipedia entry, MNIST database.
Pingback: Mathematica vs. R at GitHub | Mathematica for prediction algorithms
Pingback: A monad for Latent Semantic Analysis workflows | Mathematica for prediction algorithms