In this document we describe the design and implementation of a (software programming) monad, [Wk1], for Latent Semantic Analysis workflows specification and execution. The design and implementation are done with Mathematica / Wolfram Language (WL).
What is Latent Semantic Analysis (LSA)? : A statistical method (or a technique) for finding relationships in natural language texts that is based on the so called Distributional hypothesis, [Wk2, Wk3]. (The Distributional hypothesis can be simply stated as “linguistic items with similar distributions have similar meanings”; for an insightful philosophical and scientific discussion see [MS1].) LSA can be seen as the application of Dimensionality reduction techniques over matrices derived with the Vector space model.
The goal of the monad design is to make the specification of LSA workflows (relatively) easy and straightforward by following a certain main scenario and specifying variations over that scenario.
The monad is named LSAMon
and it is based on the State monad package “StateMonadCodeGenerator.m”, [AAp1, AA1], the document-term matrix making package “DocumentTermMatrixConstruction.m”, [AAp4, AA2], the Non-Negative Matrix Factorization (NNMF) package “NonNegativeMatrixFactorization.m”, [AAp5, AA2], and the package “SSparseMatrix.m”, [AAp2, AA5], that provides matrix objects with named rows and columns.
The data for this document is obtained from WL’s repository and it is manipulated into a certain ready-to-utilize form (and uploaded to GitHub.)
The monadic programming design is used as a Software Design Pattern. The LSAMon
monad can be also seen as a Domain Specific Language (DSL) for the specification and programming of machine learning classification workflows.
Here is an example of using the LSAMon
monad over a collection of documents that consists of 233 US state of union speeches.
The table above is produced with the package “MonadicTracing.m”, [AAp2, AA1], and some of the explanations below also utilize that package.
As it was mentioned above the monad LSAMon
can be seen as a DSL. Because of this the monad pipelines made with LSAMon
are sometimes called “specifications”.
Remark: In this document with “term” we mean “a word, a word stem, or other type of token.”
Remark: LSA and Latent Semantic Indexing (LSI) are considered more or less to be synonyms. I think that “latent semantic analysis” sounds more universal and that “latent semantic indexing” as a name refers to a specific Information Retrieval technique. Below we refer to “LSI functions” like “IDF” and “TF-IDF” that are applied within the generic LSA workflow.
The document has the following structure.
The sections “LSAMon overview”, “Monad elements”, and “The utilization of SSparseMatrix objects” provide technical descriptions needed to utilize the LSAMon
monad .
LSAMon
monad.
LSAMon
’s development process and this document follow the ones of the classifications workflows monad ClCon, [AA6].
Remark: One can read only the sections “Introduction”, “Design consideration”, “Monad design”, and “LSAMon overview”. That set of sections provide a fairly good, programming language agnostic exposition of the substance and novel ideas of this document.
The following commands load the packages [AAp1–AAp7]:
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicLatentSemanticAnalysis.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicTracing.m"]
In this section we load data that is used in the rest of the document. The text data was obtained through WL’s repository, transformed in a certain more convenient form, and uploaded to GitHub.
The text summarization and plots are done through LSAMon
, which in turn uses the function RecordsSummary from the package “MathematicaForPredictionUtilities.m”, [AAp7].
textHamlet =
ToString /@
Flatten[Import["https://raw.githubusercontent.com/antononcube/MathematicaVsR/master/Data/MathematicaVsR-Data-Hamlet.csv"]];
TakeLargestBy[
Tally[DeleteStopwords[ToLowerCase[Flatten[TextWords /@ textHamlet]]]], #[[2]] &, 20]
(* {{"ham", 358}, {"lord", 225}, {"king", 196}, {"o", 124}, {"queen", 120},
{"shall", 114}, {"good", 109}, {"hor", 109}, {"come", 107}, {"hamlet", 107},
{"thou", 105}, {"let", 96}, {"thy", 86}, {"pol", 86}, {"like", 81}, {"sir", 75},
{"'t", 75}, {"know", 74}, {"enter", 73}, {"th", 72}} *)
LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix⟹LSAMonEchoDocumentTermMatrixStatistics;
url = "https://github.com/antononcube/MathematicaVsR/blob/master/Data/MathematicaVsR-Data-StateOfUnionSpeeches.JSON.zip?raw=true";
str = Import[url, "String"];
filename = First@Import[StringToStream[str], "ZIP"];
aStateOfUnionSpeeches = Association@ImportString[Import[StringToStream[str], {"ZIP", filename, "String"}], "JSON"];
lsaObj =
LSAMonUnit[aStateOfUnionSpeeches]⟹
LSAMonMakeDocumentTermMatrix⟹
LSAMonEchoDocumentTermMatrixStatistics["LogBase" -> 10];
TakeLargest[ColumnSumsAssociation[lsaObj⟹LSAMonTakeDocumentTermMatrix], 12]
(* <|"government" -> 7106, "states" -> 6502, "congress" -> 5023,
"united" -> 4847, "people" -> 4103, "year" -> 4022,
"country" -> 3469, "great" -> 3276, "public" -> 3094, "new" -> 3022,
"000" -> 2960, "time" -> 2922|> *)
In some of the examples below we want to explicitly specify the stop words. Here are stop words derived using the built-in functions DictionaryLookup
and DeleteStopwords
.
stopWords = Complement[DictionaryLookup["*"], DeleteStopwords[DictionaryLookup["*"]]];
Short[stopWords]
(* {"a", "about", "above", "across", "add-on", "after", "again", <<290>>,
"you'll", "your", "you're", "yours", "yourself", "yourselves", "you've" } *)
The steps of the main LSA workflow addressed in this document follow.
Create a document-term matrix.
That vector space axes correspond to the unique words found in the whole document collection.
Consider the application of stemming rules.
Consider the removal of stop words.
Apply matrix-entries weighting functions.
Functions like “IDF”, “TF-IDF”, “GFIDF”.
Extract topics.
We consider using Singular Value Decomposition (SVD) and Non-Negative Matrix Factorization (NNMF).
Make and display the topics table.
Extract and display a statistical thesaurus of selected words.
Map search queries or unseen documents over the extracted topics.
Find the most important documents in the document collection. (Optional.)
The following flow-chart corresponds to the list of steps above.
In order to address:
workflows elements variability, and
workflows iterative changes and refining,
it is beneficial to have a DSL for LSA workflows. We choose to make such a DSL through a functional programming monad, [Wk1, AA1].
Here is a quote from [Wk1] that fairly well describes why we choose to make a classification workflow monad and hints on the desired properties of such a monad.
[…] The monad represents computations with a sequential structure: a monad defines what it means to chain operations together. This enables the programmer to build pipelines that process data in a series of steps (i.e. a series of actions applied to the data), in which each action is decorated with the additional processing rules provided by the monad. […]
Monads allow a programming style where programs are written by putting together highly composable parts, combining in flexible ways the possible actions that can work on a particular type of data. […]
Remark: Note that quote from [Wk1] refers to chained monadic operations as “pipelines”. We use the terms “monad pipeline” and “pipeline” below.
The monad we consider is designed to speed-up the programming of LSA workflows outlined in the previous section. The monad is named LSAMon
for “Latent Semantic Analysis** Mon**ad”.
We want to be able to construct monad pipelines of the general form:
LSAMon
is based on the State monad, [Wk1, AA1], so the monad pipeline form (1) has the following more specific form:
This means that some monad operations will not just change the pipeline value but they will also change the pipeline context.
In the monad pipelines of LSAMon
we store different objects in the contexts for at least one of the following two reasons.
The object is (relatively) hard to compute.
Such objects are document-term matrix, Dimensionality reduction factors and the related topics.
Let us list the desired properties of the monad.
The monad works with associations with string values, list of strings.
The monad use the Linear vector spaces model .
The document-term frequency matrix is can be created after removing stop words and/or word stemming.
It is easy to specify and apply different LSI weight functions. (Like “IDF” or “GFIDF”.)
The monad can do dimension reduction with SVD and NNMF and corresponding matrix factors are retrievable with monad functions.
Documents (or query strings) external to the monad a easily mapped into monad’s Linear vector space of terms and the Linear vector space of topics.
The monad allows of cursory examination and summarization of the data.
The pipeline values can be of different types. Most monad functions modify the pipeline value; some modify the context; some just echo results.
It is easy to obtain the pipeline value, context, and different context objects for manipulation outside of the monad.
It is easy to tabulate extracted topics and related statistical thesauri.
It is easy to specify and apply re-weighting functions for the entries of the document-term contingency matrices.
The LSAMon
components and their interactions are fairly simple.
The main LSAMon
operations implicitly put in the context or utilize from the context the following objects:
the factors obtained by matrix factorization algorithms,
extracted topics.
Note the that the monadic set of types of LSAMon
pipeline values is fairly heterogenous and certain awareness of “the current pipeline value” is assumed when composing LSAMon
pipelines.
Obviously, we can put in the context any object through the generic operations of the State monad of the package “StateMonadGenerator.m”, [AAp1].
When using a monad we lift certain data into the “monad space”, using monad’s operations we navigate computations in that space, and at some point we take results from it.
With the approach taken in this document the “lifting” into the LSAMon
monad is done with the function LSAMonUnit. Results from the monad can be obtained with the functions LSAMonTakeValue
, LSAMonContext
, or with the other LSAMon
functions with the prefix “LSAMonTake” (see below.)
Here is a corresponding diagram of a generic computation with the LSAMon
monad:
Remark: It is a good idea to compare the diagram with formulas (1) and (2).
Let us examine a concrete LSAMon
pipeline that corresponds to the diagram above. In the following table each pipeline operation is combined together with a short explanation and the context keys after its execution.
Here is the output of the pipeline:
The LSAMon
functions are separated into four groups:
setters and droppers,
takers,
State Monad generic functions.
An overview of the those functions is given in the tables in next two sub-sections. The next section, “Monad elements”, gives details and examples for the usage of the LSAMon
operations.
Here are the LSAMon
State Monad functions (generated using the prefix “LSAMon”, [AAp1, AA1].)
Here are the usage descriptions of the main (not monad-supportive) LSAMon
functions, which are explained in detail in the next section.
In this section we show that LSAMon
has all of the properties listed in the previous section.
The monad head is LSAMon
. Anything wrapped in LSAMon
can serve as monad’s pipeline value. It is better though to use the constructor LSAMonUnit
. (Which adheres to the definition in [Wk1].)
LSAMon[textHamlet, <||>]⟹LSAMonMakeDocumentTermMatrix[Automatic, Automatic]⟹LSAMonEchoFunctionContext[Short];
The function lifting the data into the monad QRMon is QRMonUnit.
The lifting to the monad marks the beginning of the monadic pipeline. It can be done with data or without data. Examples follow.
LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix⟹LSAMonTakeDocumentTermMatrix
LSAMonUnit[]⟹LSAMonSetDocuments[textHamlet]⟹LSAMonMakeDocumentTermMatrix⟹LSAMonTakeDocumentTermMatrix
(See the sub-section “Setters, droppers, and takers” for more details of setting and taking values in LSAMon
contexts.)
Currently the monad can deal with data in the following forms:
associations with string values.
Generally, WL makes it easy to extract columns datasets order to obtain vectors or matrices, so datasets are not currently supported in LSAMon
.
As it was mentioned above with “term” we mean “a word or a stemmed word”. Here is are examples of stemmed words.
WordData[#, "PorterStem"] & /@ {"consequential", "constitution", "forcing", ""}
The fundamental model of LSAMon
is the so called Vector space model (or the closely related Bag-of-words model.) The document-term matrix is a linear vector space representation of the documents collection. That representation is further used in LSAMon
to find topics and statistical thesauri.
Here is an example of ad hoc construction of a document-term matrix using a couple of paragraphs from “Hamlet”.
inds = {10, 19};
aTempText = AssociationThread[inds, textHamlet[[inds]]]
MatrixForm @ CrossTabulate[Flatten[KeyValueMap[Thread[{#1, #2}] &, TextWords /@ ToLowerCase[aTempText]], 1]]
When we construct the document-term matrix we (often) want to stem the words and (almost always) want to remove stop words. LSAMon
’s function LSAMonMakeDocumentTermMatrix makes the document-term matrix and takes specifications for stemming and stop words.
lsaObj =
LSAMonUnit[textHamlet]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> Automatic, "StopWords" -> Automatic]⟹
LSAMonEchoFunctionContext[ MatrixPlot[#documentTermMatrix] &]⟹
LSAMonEchoFunctionContext[TakeLargest[ColumnSumsAssociation[#documentTermMatrix], 12] &];
We can retrieve the stop words used in a monad with the function LSAMonTakeStopWords.
Short[lsaObj⟹LSAMonTakeStopWords]
We can retrieve the stemming rules used in a monad with the function LSAMonTakeStemmingRules.
Short[lsaObj⟹LSAMonTakeStemmingRules]
The specification Automatic
for stemming rules uses WordData[#,"PorterStem"]&
.
Instead of the options style signature we can use positional signature.
LSAMonMakeDocumentTermMatrix["StemmingRules" -> {}, "StopWords" -> Automatic]
.
Positional style: LSAMonMakeDocumentTermMatrix[{}, Automatic]
.
After making the document-term matrix we will most likely apply LSI weight functions, [Wk2], like “GFIDF” and “TF-IDF”. (This follows the “standard” approach used in search engines for calculating weights for document-term matrices; see [MB1].)
We use the following definition of the frequency document-term matrix F.
Each entry f_{ij} of the matrix F is the number of occurrences of the term j in the document i.
Each entry of the weighted document-term matrix M derived from the frequency document-term matrix F is expressed with the formula
where g_{j} – global term weight; l_{ij} – local term weight; d_{i} – normalization weight.
Various formulas exist for these weights and one of the challenges is to find the right combination of them when using different document collections.
Here is a table of weight functions formulas.
LSAMon
function LSAMonApplyTermWeightFunctions delegates the LSI weight functions application to the package “DocumentTermMatrixConstruction.m”, [AAp4].
Here is an example.
lsaHamlet = LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix;
wmat =
lsaHamlet⟹
LSAMonApplyTermWeightFunctions["IDF", "TermFrequency", "Cosine"]⟹
LSAMonTakeWeightedDocumentTermMatrix;
TakeLargest[ColumnSumsAssociation[wmat], 6]
Instead of using the positional signature of LSAMonApplyTermWeightFunctions we can specify the LSI functions using options.
wmat2 =
lsaHamlet⟹
LSAMonApplyTermWeightFunctions["GlobalWeightFunction" -> "IDF", "LocalWeightFunction" -> "TermFrequency", "NormalizerFunction" -> "Cosine"]⟹
LSAMonTakeWeightedDocumentTermMatrix;
TakeLargest[ColumnSumsAssociation[wmat2], 6]
Here we are summaries of the non-zero values of the weighted document-term matrix derived with different combinations of global, local, and normalization weight functions.
Magnify[#, 0.8] &@Multicolumn[Framed /@ #, 6] &@Flatten@
Table[
(wmat =
lsaHamlet⟹
LSAMonApplyTermWeightFunctions[gw, lw, nf]⟹
LSAMonTakeWeightedDocumentTermMatrix;
RecordsSummary[SparseArray[wmat]["NonzeroValues"],
List@StringRiffle[{gw, lw, nf}, ", "]]),
{gw, {"IDF", "GFIDF", "Binary", "None", "ColumnStochastic"}},
{lw, {"Binary", "Log", "None"}},
{nf, {"Cosine", "None", "RowStochastic"}}]
AutoCollapse[]
Streamlining topic extraction is one of the main reasons LSAMon
was implemented. The topic extraction correspond to the so called “syntagmatic” relationships between the terms, [MS1].
The original weighed document-term matrix M is decomposed into the matrix factors W and H.
M ≈ W.H, W ∈ ℝ^{m × k}, H ∈ ℝ^{k × n}.
The i-th row of M is expressed with the i-th row of W multiplied by H.
The rows of H are the topics. SVD produces orthogonal topics; NNMF does not.
The i-the document of the collection corresponds to the i-th row W. Finding the Nearest Neighbors (NN’s) of the i-th document using the rows similarity of the matrix W gives document NN’s through topic similarity.
The terms correspond to the columns of H. Finding NN’s based on similarities of H’s columns produces statistical thesaurus entries.
The term groups provided by H’s rows correspond to “syntagmatic” relationships. Using similarities of H’s columns we can produce term clusters that correspond to “paradigmatic” relationships.
Here is an example using the play “Hamlet” in which we specify additional stop words.
stopWords2 = {"enter", "exit", "[exit", "ham", "hor", "laer", "pol", "oph", "thy", "thee", "act", "scene"};
SeedRandom[2381]
lsaHamlet =
LSAMonUnit[textHamlet]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> Automatic, "StopWords" -> Join[stopWords, stopWords2]]⟹
LSAMonApplyTermWeightFunctions["GlobalWeightFunction" -> "IDF", "LocalWeightFunction" -> "None", "NormalizerFunction" -> "Cosine"]⟹
LSAMonExtractTopics["NumberOfTopics" -> 12, "MinNumberOfDocumentsPerTerm" -> 10, Method -> "NNMF", "MaxSteps" -> 20]⟹
LSAMonEchoTopicsTable["NumberOfTableColumns" -> 6, "NumberOfTerms" -> 10];
Here is an example using the USA presidents “state of union” speeches.
SeedRandom[7681]
lsaSpeeches =
LSAMonUnit[aStateOfUnionSpeeches]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> Automatic, "StopWords" -> Automatic]⟹
LSAMonApplyTermWeightFunctions["GlobalWeightFunction" -> "IDF", "LocalWeightFunction" -> "None", "NormalizerFunction" -> "Cosine"]⟹
LSAMonExtractTopics["NumberOfTopics" -> 36, "MinNumberOfDocumentsPerTerm" -> 40, Method -> "NNMF", "MaxSteps" -> 12]⟹
LSAMonEchoTopicsTable["NumberOfTableColumns" -> 6, "NumberOfTerms" -> 10];
Note that in both examples:
the default LSI re-weighting functions are used: “IDF”, “None”, “Cosine”,
the dimension reduction algorithm NNMF is used.
Things to keep in mind.
NNMF is prone to get stuck into local minima, so several topic extractions (and corresponding evaluations) have to be done.
We would get different results with different NNMF runs using the same parameters. (NNMF uses random numbers initialization.)
The NNMF topic vectors are not orthogonal.
SVD is much faster than NNMF, but it topic vectors are hard to interpret.
Generally, the topics derived with SVD are stable, they do not change with different runs with the same parameters.
The SVD topics vectors are orthogonal, which provides for quick to find representations of documents not in the monad’s document collection.
The document-topic matrix W has column names that are automatically derived from the top three terms in each topic.
ColumnNames[lsaHamlet⟹LSAMonTakeW]
(* {"player-plai-welcom", "ro-lord-sir", "laert-king-attend",
"end-inde-make", "state-room-castl", "daughter-pass-love",
"hamlet-ghost-father", "father-thou-king",
"rosencrantz-guildenstern-king", "ophelia-queen-poloniu",
"answer-sir-mother", "horatio-attend-gentleman"} *)
Of course the row names of H have the same names.
RowNames[lsaHamlet⟹LSAMonTakeH]
(* {"player-plai-welcom", "ro-lord-sir", "laert-king-attend",
"end-inde-make", "state-room-castl", "daughter-pass-love",
"hamlet-ghost-father", "father-thou-king",
"rosencrantz-guildenstern-king", "ophelia-queen-poloniu",
"answer-sir-mother", "horatio-attend-gentleman"} *)
The statistical thesaurus extraction corresponds to the “paradigmatic” relationships between the terms, [MS1].
Here is an example over the State of Union speeches.
entryWords = {"bank", "war", "economy", "school", "port", "health", "enemy", "nuclear"};
lsaSpeeches⟹
LSAMonExtractStatisticalThesaurus["Words" -> Map[WordData[#, "PorterStem"] &, entryWords], "NumberOfNearestNeighbors" -> 12]⟹
LSAMonEchoStatisticalThesaurus;
In the code above: (i) the options signature style is used, (ii) the statistical thesaurus entry words are stemmed first.
We can also call LSAMonEchoStatisticalThesaurus
directly without calling LSAMonExtractStatisticalThesaurus
first.
lsaSpeeches⟹
LSAMonEchoStatisticalThesaurus["Words" -> Map[WordData[#, "PorterStem"] &, entryWords], "NumberOfNearestNeighbors" -> 12];
One of the most natural operations is to find the representation of an arbitrary document (or sentence or a list of words) in monad’s Linear vector space of terms. This is done with the function LSAMonRepresentByTerms.
Here is an example in which a sentence is represented as a one-row matrix (in that space.)
obj =
lsaHamlet⟹
LSAMonRepresentByTerms["Hamlet, Prince of Denmark killed the king."]⟹
LSAMonEchoValue;
Here we display only the non-zero columns of that matrix.
obj⟹
LSAMonEchoFunctionValue[MatrixForm[Part[#, All, Keys[Select[SSparseMatrix`ColumnSumsAssociation[#], # > 0& ]]]]& ];
Assume that LSAMonRepresentByTerms is given a list of sentences. Then that function performs the following steps.
1. The sentence is split into a list of words.
2. If monad’s document-term matrix was made by removing stop words the same stop words are removed from the list of words.
3. If monad’s document-term matrix was made by stemming the same stemming rules are applied to the list of words.
4. The LSI global weights and the LSI local weight and normalizer functions are applied to sentence’s contingency matrix.
Let us look convince ourselves that documents used in the monad to built the weighted document-term matrix have the same representation as the corresponding rows of that matrix.
Here is an association of documents from monad’s document collection.
inds = {6, 10};
queries = Part[lsaHamlet⟹LSAMonTakeDocuments, inds];
queries
(* <|"id.0006" -> "Getrude, Queen of Denmark, mother to Hamlet. Ophelia, daughter to Polonius.",
"id.0010" -> "ACT I. Scene I. Elsinore. A platform before the Castle."|> *)
lsaHamlet⟹
LSAMonRepresentByTerms[queries]⟹
LSAMonEchoFunctionValue[MatrixForm[Part[#, All, Keys[Select[SSparseMatrix`ColumnSumsAssociation[#], # > 0& ]]]]& ];
lsaHamlet⟹
LSAMonEchoFunctionContext[MatrixForm[Part[Slot["weightedDocumentTermMatrix"], inds, Keys[Select[SSparseMatrix`ColumnSumsAssociation[Part[Slot["weightedDocumentTermMatrix"], inds, All]], # > 0& ]]]]& ];
Another natural operation is to find the representation of an arbitrary document (or a list of words) in monad’s Linear vector space of topics. This is done with the function LSAMonRepresentByTopics.
Here is an example.
inds = {6, 10};
queries = Part[lsaHamlet⟹LSAMonTakeDocuments, inds];
Short /@ queries
(* <|"id.0006" -> "Getrude, Queen of Denmark, mother to Hamlet. Ophelia, daughter to Polonius.",
"id.0010" -> "ACT I. Scene I. Elsinore. A platform before the Castle."|> *)
lsaHamlet⟹
LSAMonRepresentByTopics[queries]⟹
LSAMonEchoFunctionValue[MatrixForm[Part[#, All, Keys[Select[SSparseMatrix`ColumnSumsAssociation[#], # > 0& ]]]]& ];
lsaHamlet⟹
LSAMonEchoFunctionContext[MatrixForm[Part[Slot["W"], inds, Keys[Select[SSparseMatrix`ColumnSumsAssociation[Part[Slot["W"], inds, All]], # > 0& ]]]]& ];
In order to clarify what the function LSAMonRepresentByTopics is doing let us go through the formulas it is based on.
The original weighed document-term matrix M is decomposed into the matrix factors W and H.
M ≈ W.H, W ∈ ℝ^{m × k}, H ∈ ℝ^{k × n}
The i-th row of M is expressed with the i-th row of W multiplied by H.
m_{i} ≈ w_{i}.H.
For a query vector q_{0} ∈ ℝ^{m} we want to find its topics representation vector x ∈ ℝ^{k}:
q_{0} ≈ x.H.
Denote with H^{( − 1)} the inverse or pseudo-inverse matrix of H. We have:
q_{0}.H^{( − 1)} ≈ (x.H).H^{( − 1)} = x.(H.H^{( − 1)}) = x.I,
x ∈ ℝ^{k}, H^{( − 1)} ∈ ℝ^{n × k}, I ∈ ℝ^{k × k}.
In LSAMon
for SVD H^{( − 1)} = H^{T}; for NNMF H^{( − 1)} is the pseudo-inverse of H.
The vector x obtained with LSAMonRepresentByTopics.
Sometimes we want to find the topics representation of tags associated with monad’s documents and the tag-document associations are one-to-many. See [AA3].
Let us consider a concrete example – we want to find what topics correspond to the different presidents in the collection of State of Union speeches.
Here we find the document tags (president names in this case.)
tags = StringReplace[
RowNames[
lsaSpeeches⟹LSAMonTakeDocumentTermMatrix],
RegularExpression[".\\d\\d\\d\\d-\\d\\d-\\d\\d"] -> ""];
Short[tags]
Here is the number of unique tags (president names.)
Length[Union[tags]]
(* 42 *)
Here we compute the tag-topics representation matrix using the function LSAMonRepresentDocumentTagsByTopics
.
tagTopicsMat =
lsaSpeeches⟹
LSAMonRepresentDocumentTagsByTopics[tags]⟹
LSAMonTakeValue
Here is a heatmap plot of the tag-topics matrix made with the package “HeatmapPlot.m”, [AAp11].
HeatmapPlot[tagTopicsMat[[All, Ordering@ColumnSums[tagTopicsMat]]], DistanceFunction -> None, ImageSize -> Large]
There are several algorithms we can apply for finding the most important documents in the collection. LSAMon
utilizes two types algorithms: (1) graph centrality measures based, and (2) matrix factorization based. With certain graph centrality measures the two algorithms are equivalent. In this sub-section we demonstrate the matrix factorization algorithm (that uses SVD.)
Definition: The most important sentences have the most important words and the most important words are in the most important sentences.
That definition can be used to derive an iterations-based model that can be expressed with SVD or eigenvector finding algorithms, [LE1].
Here we pick an important part of the play “Hamlet”.
focusText =
First@Pick[textHamlet, StringMatchQ[textHamlet, ___ ~~ "to be" ~~ __ ~~ "or not to be" ~~ ___, IgnoreCase -> True]];
Short[focusText]
(* "Ham. To be, or not to be- that is the question: Whether 'tis ....y.
O, woe is me T' have seen what I have seen, see what I see!" *)
LSAMonUnit[StringSplit[ToLowerCase[focusText], {",", ".", ";", "!", "?"}]]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> {}, "StopWords" -> Automatic]⟹
LSAMonApplyTermWeightFunctions⟹
LSAMonFindMostImportantDocuments[3]⟹
LSAMonEchoFunctionValue[GridTableForm];
The values from the monad context can be set, obtained, or dropped with the corresponding “setter”, “dropper”, and “taker” functions as summarized in a previous section.
For example:
p = LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix[Automatic, Automatic];
p⟹LSAMonTakeMatrix
If other values are put in the context they can be obtained through the (generic) function LSAMonTakeContext, [AAp1]:
Short@(p⟹QRMonTakeContext)["documents"]
(* <|"id.0001" -> "1604", "id.0002" -> "THE TRAGEDY OF HAMLET, PRINCE OF DENMARK", <<220>>, "id.0223" -> "THE END"|> *)
Another generic function from [AAp1] is LSAMonTakeValue (used many times above.)
Here is an example of the “data dropper” LSAMonDropDocuments:
Keys[p⟹LSAMonDropDocuments⟹QRMonTakeContext]
(* {"documentTermMatrix", "terms", "stopWords", "stemmingRules"} *)
(The “droppers” simply use the state monad function LSAMonDropFromContext, [AAp1]. For example, LSAMonDropDocuments is equivalent to LSAMonDropFromContext[“documents”].)
The LSAMon monad heavily relies on SSparseMatrix
objects, [AAp6, AA5], for internal representation of data and computation results.
A SSparseMatrix
object is a matrix with named rows and columns.
Here is an example.
n = 6;
rmat = ToSSparseMatrix[
SparseArray[{{1, 2} -> 1, {4, 5} -> 1}, {n, n}],
"RowNames" -> RandomSample[CharacterRange["A", "Z"], n],
"ColumnNames" -> RandomSample[CharacterRange["a", "z"], n]];
MatrixForm[rmat]
In this section we look into some useful SSparseMatrix idioms applied within LSAMon.
In some situations it is beneficial to sort rows and columns of the (weighted) document-term matrix.
docTermMat =
lsaSpeeches⟹LSAMonTakeDocumentTermMatrix;
MatrixPlot[docTermMat[[Ordering[RowSums[docTermMat]], Ordering[ColumnSums[docTermMat]]]], MaxPlotPoints -> 300, ImageSize -> Large]
The most popular terms in the document collection can be found through the association of the column sums of the document-term matrix.
TakeLargest[ColumnSumsAssociation[lsaSpeeches⟹LSAMonTakeDocumentTermMatrix], 10]
(* <|"state" -> 8852, "govern" -> 8147, "year" -> 6362, "nation" -> 6182,
"congress" -> 5040, "unit" -> 5040, "countri" -> 4504,
"peopl" -> 4306, "american" -> 3648, "law" -> 3496|> *)
Similarly for the lest popular terms.
TakeSmallest[
ColumnSumsAssociation[
lsaSpeeches⟹LSAMonTakeDocumentTermMatrix], 10]
(* <|"036" -> 1, "027" -> 1, "_____________________" -> 1, "0111" -> 1,
"006" -> 1, "0000" -> 1, "0001" -> 1, "______________________" -> 1,
"____" -> 1, "____________________" -> 1|> *)
In some cases we want to show only columns of the data or computation results matrices that have non-zero elements.
Here is an example (similar to other examples in the previous section.)
lsaHamlet⟹
LSAMonRepresentByTerms[{"this country is rotten",
"where is my sword my lord",
"poison in the ear should be in the play"}]⟹
LSAMonEchoFunctionValue[ MatrixForm[#1[[All, Keys[Select[ColumnSumsAssociation[#1], #1 > 0 &]]]]] &];
In the pipeline code above: (i) from the list of queries a representation matrix is made, (ii) that matrix is assigned to the pipeline value, (iii) in the pipeline echo value function the non-zero columns are selected with by using the keys of the non-zero elements of the association obtained with ColumnSumsAssociation.
Here is way to compute the similarity matrix of different sets of documents that are not required to be in monad’s document collection.
sMat1 =
lsaSpeeches⟹
LSAMonRepresentByTerms[ aStateOfUnionSpeeches[[ Range[-5, -2] ]] ]⟹
LSAMonTakeValue
sMat2 =
lsaSpeeches⟹
LSAMonRepresentByTerms[ aStateOfUnionSpeeches[[ Range[-7, -3] ]] ]⟹
LSAMonTakeValue
MatrixForm[sMat1.Transpose[sMat2]]
Similarly to weighted Boolean similarities matrix computation above we can compute a similarity matrix using the topics representations. Note that an additional normalization steps is required.
sMat1 =
lsaSpeeches⟹
LSAMonRepresentByTopics[ aStateOfUnionSpeeches[[ Range[-5, -2] ]] ]⟹
LSAMonTakeValue;
sMat1 = WeightTermsOfSSparseMatrix[sMat1, "None", "None", "Cosine"]
sMat2 =
lsaSpeeches⟹
LSAMonRepresentByTopics[ aStateOfUnionSpeeches[[ Range[-7, -3] ]] ]⟹
LSAMonTakeValue;
sMat2 = WeightTermsOfSSparseMatrix[sMat2, "None", "None", "Cosine"]
MatrixForm[sMat1.Transpose[sMat2]]
Note the differences with the weighted Boolean similarity matrix in the previous sub-section – the similarities that are less than 1 are noticeably larger.
The development of LSAMon
was done with two types of unit tests: (i) directly specified tests, [AAp7], and (ii) tests based on randomly generated pipelines, [AA8].
The unit test package should be further extended in order to provide better coverage of the functionalities and illustrate – and postulate – pipeline behavior.
Here we run the unit tests file “MonadicLatentSemanticAnalysis-Unit-Tests.wlt”, [AAp8].
AbsoluteTiming[
testObject = TestReport["~/MathematicaForPrediction/UnitTests/MonadicLatentSemanticAnalysis-Unit-Tests.wlt"]
]
The natural language derived test ID’s should give a fairly good idea of the functionalities covered in [AAp3].
Values[Map[#["TestID"] &, testObject["TestResults"]]]
(* {"LoadPackage", "USASpeechesData", "HamletData", "StopWords",
"Make-document-term-matrix-1", "Make-document-term-matrix-2",
"Apply-term-weights-1", "Apply-term-weights-2", "Topic-extraction-1",
"Topic-extraction-2", "Topic-extraction-3", "Topic-extraction-4",
"Statistical-thesaurus-1", "Topics-representation-1",
"Take-document-term-matrix-1", "Take-weighted-document-term-matrix-1",
"Take-document-term-matrix-2", "Take-weighted-document-term-matrix-2",
"Take-terms-1", "Take-Factors-1", "Take-Factors-2", "Take-Factors-3",
"Take-Factors-4", "Take-StopWords-1", "Take-StemmingRules-1"} *)
Since the monad LSAMon
is a DSL it is natural to test it with a large number of randomly generated “sentences” of that DSL. For the LSAMon
DSL the sentences are LSAMon
pipelines. The package “MonadicLatentSemanticAnalysisRandomPipelinesUnitTests.m”, [AAp9], has functions for generation of LSAMon
random pipelines and running them as verification tests. A short example follows.
Generate pipelines:
SeedRandom[234]
pipelines = MakeLSAMonRandomPipelines[100];
Length[pipelines]
(* 100 *)
Here is a sample of the generated pipelines:
Here we run the pipelines as unit tests:
AbsoluteTiming[
res = TestRunLSAMonPipelines[pipelines, "Echo" -> False];
]
From the test report results we see that a dozen tests failed with messages, all of the rest passed.
rpTRObj = TestReport[res]
(The message failures, of course, have to be examined – some bugs were found in that way. Currently the actual test messages are expected.)
It would be nice to extend the Dimension reduction functionalities of LSAMon
to include other algorithms like Independent Component Analysis (ICA), [Wk5]. Ideally with LSAMon
we can do comparisons between SVD, NNMF, and ICA like the image de-nosing based comparison explained in [AA8].
Another direction is to utilize Neural Networks for the topic extraction and making of statistical thesauri.
Since LSAMon
is a DSL it can be relatively easily interfaced with a natural language interface.
Here is an example of natural language commands parsed into LSA code using the package [AAp13].
The implementation methodology of the LSAMon
monad packages [AAp3, AAp9] followed the methodology created for the ClCon
monad package [AAp10, AA6]. Similarly, this document closely follows the structure and exposition of the `ClCon monad document “A monad for classification workflows”, [AA6].
A lot of the functionalities and signatures of LSAMon
were designed and programed through considerations of natural language commands specifications given to a specialized conversational agent.
[AAp1] Anton Antonov, State monad code generator Mathematica package, (2017), MathematicaForPrediction at GitHub*.
[AAp2] Anton Antonov, Monadic tracing Mathematica package, (2017), MathematicaForPrediction at GitHub*.
[AAp3] Anton Antonov, Monadic Latent Semantic Analysis Mathematica package, (2017), MathematicaForPrediction at GitHub.
[AAp4] Anton Antonov, Implementation of document-term matrix construction and re-weighting functions in Mathematica, (2013), MathematicaForPrediction at GitHub.
[AAp5] Anton Antonov, Non-Negative Matrix Factorization algorithm implementation in Mathematica, (2013), MathematicaForPrediction at GitHub.
[AAp6] Anton Antonov, SSparseMatrix Mathematica package, (2018), MathematicaForPrediction at GitHub.
[AAp7] Anton Antonov, MathematicaForPrediction utilities, (2014), MathematicaForPrediction at GitHub.
[AAp8] Anton Antonov, Monadic Latent Semantic Analysis unit tests, (2019), MathematicaVsR at GitHub.
[AAp9] Anton Antonov, Monadic Latent Semantic Analysis random pipelines Mathematica unit tests, (2019), MathematicaForPrediction at GitHub.
[AAp10] Anton Antonov, Monadic contextual classification Mathematica package, (2017), MathematicaForPrediction at GitHub.
[AAp11] Anton Antonov, Heatmap plot Mathematica package, (2017), MathematicaForPrediction at GitHub.
[AAp12] Anton Antonov,
Independent Component Analysis Mathematica package, MathematicaForPrediction at GitHub.
[AAp13] Anton Antonov, Latent semantic analysis workflows grammar in EBNF, (2018), ConverasationalAgents at GitHub.
[AA1] Anton Antonov, “Monad code generation and extension”, (2017), MathematicaForPrediction at GitHub.
[AA2] Anton Antonov, “Topic and thesaurus extraction from a document collection”, (2013), MathematicaForPrediction at GitHub.
[AA3] Anton Antonov, “The Great conversation in USA presidential speeches”, (2017), MathematicaForPrediction at WordPress.
[AA4] Anton Antonov, “Contingency tables creation examples”, (2016), MathematicaForPrediction at WordPress.
[AA5] Anton Antonov, “RSparseMatrix for sparse matrices with named rows and columns”, (2015), MathematicaForPrediction at WordPress.
[AA6] Anton Antonov, “A monad for classification workflows”, (2018), MathematicaForPrediction at WordPress.
[AA7] Anton Antonov, “Independent component analysis for multidimensional signals”, (2016), MathematicaForPrediction at WordPress.
[AA8] Anton Antonov, “Comparison of PCA, NNMF, and ICA over image de-noising”, (2016), MathematicaForPrediction at WordPress.
[Wk1] Wikipedia entry, Monad,
[Wk2] Wikipedia entry, Latent semantic analysis,
[Wk3] Wikipedia entry, Distributional semantics,
[Wk4] Wikipedia entry, Non-negative matrix factorization,
[LE1] Lars Elden, Matrix Methods in Data Mining and Pattern Recognition, 2007, SIAM. ISBN-13: 978-0898716269.
[MB1] Michael W. Berry & Murray Browne, Understanding Search Engines: Mathematical Modeling and Text Retrieval, 2nd. ed., 2005, SIAM. ISBN-13: 978-0898715811.
[MS1] Magnus Sahlgren, “The Distributional Hypothesis”, (2008), Rivista di Linguistica. 20 (1): 33[Dash]53.
[PS1] Patrick Scheibe, Mathematica (Wolfram Language) support for IntelliJ IDEA, (2013-2018), Mathematica-IntelliJ-Plugin at GitHub.
]]>In this document we show how to find the so called “structural breaks”, [Wk1], in a given time series. The algorithm is based in on a systematic application of Chow Test, [Wk2], combined with an algorithm for local extrema finding in noisy time series, [AA1].
The algorithm implementation is based on the packages “MonadicQuantileRegression.m”, [AAp1], and “MonadicStructuralBreaksFinder.m”, [AAp2]. The package [AAp1] provides the software monad QRMon that allows rapid and concise specification of Quantile Regression workflows. The package [AAp2] extends QRMon with functionalities related to structural breaks finding.
It looks like at least one type of “structural breaks” are defined through regression models, [Wk1]. Roughly speaking a structural break point of time series is a regressor point that splits the time series in such way that the obtained two parts have very different regression parameters.
One way to test such a point is to use Chow test, [Wk2]. From [Wk2] we have the definition:
The Chow test, proposed by econometrician Gregory Chow in 1960, is a test of whether the true coefficients in two linear regressions on different data sets are equal. In econometrics, it is most commonly used in time series analysis to test for the presence of a structural break at a period which can be assumed to be known a priori (for instance, a major historical event such as a war).
Here is an example of the described algorithm application to the data from [Wk2].
QRMonUnit[data]⟹QRMonPlotStructuralBreakSplits[ImageSize -> Small];
Here we load the packages [AAp1] and [AAp2].
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicQuantileRegression.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicStructuralBreaksFinder.m"]
In this section we assign the data used in this document.
Here is the data used in the Wikipedia article “Chow test”, [Wk2].
data = {{0.08, 0.34}, {0.16, 0.55}, {0.24, 0.54}, {0.32, 0.77}, {0.4,
0.77}, {0.48, 1.2}, {0.56, 0.57}, {0.64, 1.3}, {0.72, 1.}, {0.8,
1.3}, {0.88, 1.2}, {0.96, 0.88}, {1., 1.2}, {1.1, 1.3}, {1.2,
1.3}, {1.3, 1.4}, {1.4, 1.5}, {1.4, 1.5}, {1.5, 1.5}, {1.6,
1.6}, {1.7, 1.1}, {1.8, 0.98}, {1.8, 1.1}, {1.9, 1.4}, {2.,
1.3}, {2.1, 1.5}, {2.2, 1.3}, {2.2, 1.3}, {2.3, 1.2}, {2.4,
1.1}, {2.5, 1.1}, {2.6, 1.2}, {2.6, 1.4}, {2.7, 1.3}, {2.8,
1.6}, {2.9, 1.5}, {3., 1.4}, {3., 1.8}, {3.1, 1.4}, {3.2,
1.4}, {3.3, 1.4}, {3.4, 2.}, {3.4, 2.}, {3.5, 1.5}, {3.6,
1.8}, {3.7, 2.1}, {3.8, 1.6}, {3.8, 1.8}, {3.9, 1.9}, {4., 2.1}};
ListPlot[data]
Here we get the time series corresponding to S&P 500 Index.
tsSP500 = FinancialData[Entity["Financial", "^SPX"], {{2015, 1, 1}, Date[]}]
DateListPlot[tsSP500, ImageSize -> Medium]
The Chow Test statistic is implemented in [AAp1]. In this document we rely on the relative comparison of the Chow Test statistic values: the larger the value of the Chow test statistic, the more likely we have a structural break.
Here is how we can apply the Chow Test with a QRMon pipeline to the [Wk2] data given above.
chowStats =
QRMonUnit[data]⟹
QRMonChowTestStatistic[Range[1, 3, 0.05], {1, x}]⟹
QRMonTakeValue;
We see that the regressor points $Failed and 1.7 have the largest Chow Test statistic values.
Block[{chPoint = TakeLargestBy[chowStats, Part[#, 2]& , 1]},
ListPlot[{chowStats, chPoint}, Filling -> Axis, PlotLabel -> Row[{"Point with largest Chow Test statistic:",
Spacer[8], chPoint}]]]
The first argument of QRMonChowTestStatistic is a list of regressor points or Automatic. The second argument is a list of functions to be used for the regressions.
Here is an example of an automatic values call.
chowStats2 = QRMonUnit[data]⟹QRMonChowTestStatistic⟹QRMonTakeValue;
ListPlot[chowStats2, GridLines -> {
Part[
Part[chowStats2, All, 1],
OutlierIdentifiers`OutlierPosition[
Part[chowStats2, All, 2], OutlierIdentifiers`SPLUSQuartileIdentifierParameters]], None}, GridLinesStyle -> Directive[{Orange, Dashed}], Filling -> Axis]
For the set of values displayed above we can apply simple 1D outlier identification methods, [AAp3], to automatically find the structural break point.
chowStats2[[All, 1]][[OutlierPosition[chowStats2[[All, 2]], SPLUSQuartileIdentifierParameters]]]
(* {1.7} *)
OutlierPosition[chowStats2[[All, 2]], SPLUSQuartileIdentifierParameters]
(* {20} *)
We cannot use that approach for finding all structural breaks in the general time series cases though as exemplified with the following code using the time series S&P 500 Index.
chowStats3 = QRMonUnit[tsSP500]⟹QRMonChowTestStatistic⟹QRMonTakeValue;
DateListPlot[chowStats3, Joined -> False, Filling -> Axis]
OutlierPosition[chowStats3[[All, 2]], SPLUSQuartileIdentifierParameters]
(* {} *)
OutlierPosition[chowStats3[[All, 2]], HampelIdentifierParameters]
(* {} *)
In the rest of the document we provide an algorithm that works for general time series.
Consider the problem of finding of all structural breaks in a given time series. That can be done (reasonably well) with the following procedure.
QRMon has a function, QRMonFindLocalExtrema, for finding local extrema; see [AAp1, AA1]. For the goal of finding all structural breaks, that semi-symbolic algorithm is the crucial part in the steps above.
fitFuncs = {1, x};
The computation below combines steps 2,3, and 4.
qrObj =
QRMonUnit[tsSP500]⟹
QRMonFindChowTestLocalMaxima["Knots" -> 20,
"NearestWithOutliers" -> True,
"NumberOfProximityPoints" -> 5, "EchoPlots" -> True,
"DateListPlot" -> True,
ImageSize -> Medium]⟹
QRMonEchoValue;
splitPoint = TakeLargestBy[qrObj⟹QRMonTakeValue, #[[2]] &, 1][[1, 1]]
Here we just make the plots without showing them.
sbPlots =
QRMonUnit[tsSP500]⟹
QRMonPlotStructuralBreakSplits[(qrObj⟹ QRMonTakeValue)[[All, 1]],
"LeftPartColor" -> Gray, "DateListPlot" -> True,
"Echo" -> False,
ImageSize -> Medium]⟹
QRMonTakeValue;
The function QRMonPlotStructuralBreakSplits returns an association that has as keys paired split points and Chow Test statistics; the plots are association’s values.
Here we tabulate the plots with plots with most significant breaks shown first.
Multicolumn[
KeyValueMap[
Show[#2, PlotLabel ->
Grid[{{"Point:", #1[[1]]}, {"Chow Test statistic:", #1[[2]]}}, Alignment -> Left]] &, KeySortBy[sbPlots, -#[[2]] &]], 2]
We can further apply the algorithm explained above to identifying time series states or components. The structural break points are used as knots in appropriate Quantile Regression fitting. Here is an example.
The plan is to develop such an identifier of time series states in the near future. (And present it at WTC-2019.)
[Wk1] Wikipedia entry, Structural breaks.
[Wk2] Wikipedia entry, Chow test.
[AA1] Anton Antonov, “Finding local extrema in noisy data using Quantile Regression”, (2019), MathematicaForPrediction at WordPress.
[AA2] Anton Antonov, “A monad for Quantile Regression workflows”, (2018), MathematicaForPrediction at GitHub.
[AAp1] Anton Antonov, Monadic Quantile Regression Mathematica package, (2018), MathematicaForPrediction at GitHub.
[AAp2] Anton Antonov, Monadic Structural Breaks Finder Mathematica package, (2019), MathematicaForPrediction at GitHub.
[AAp3] Anton Antonov, Implementation of one dimensional outlier identifying algorithms in Mathematica, (2013), MathematicaForPrediction at GitHub.
[AAv1] Anton Antonov, Structural Breaks with QRMon, (2019), YouTube.
]]>This document describes the package CallGraph.m for making call graphs between the functions that belong to specified contexts.
The main function is CallGraph
that gives a graph with vertices that are functions names and edges that show which functions call which other functions. With the default option values the graph vertices are labeled and have tooltips with function usage messages.
The call graphs produced by the main package function CallGraph
are assumed to be used for studying or refactoring of large code bases written with Mathematica / Wolfram Language.
The argument of CallGraph
is a context string or a list of context strings.
With the default values of its options CallGraph
produces a graph with labeled nodes and the labels have tooltips that show the usage messages of the functions from the specified contexts. It is assumed that this would be the most useful call graph type for studying the codes of different sets of packages.
We can make simple, non-label, non-tooltip call graph using CallGraph[ ... , "UsageTooltips" -> False ]
.
The simple call graph can be modified with the functions:
CallGraphAddUsageMessages, CallGraphAddPrintDefinitionsButtons, CallGraphBiColorCircularEmbedding
Each of those functions is decorating the simple graph in a particular way.
This loads the package CallGraph.m :
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/Misc/CallGraph.m"]
The following packages are used in the examples below.
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicQuantileRegression.m"]
Get["https://raw.githubusercontent.com/szhorvat/IGraphM/master/IGInstaller.m"];
Needs["IGraphM`"]
CallGraph["IGraphM`", GraphLayout -> "SpringElectricalEmbedding", ImageSize -> Large]
gr = CallGraph["IGraphM`", Exclusions -> Map[ToExpression, Names["IG*Q"]], ImageSize -> 900]
gr0 = CallGraph["IGraphM`", "UsageTooltips" -> False];
gr1 = CallGraphAddPrintDefinitionsButtons[gr0, GraphLayout -> "StarEmbedding", ImageSize -> 900]
cols = RandomSample[ ColorData["Rainbow"] /@ Rescale[Range[VertexCount[gr1]]]];
CallGraphBiColorCircularEmbedding[ gr1, "VertexColors" -> cols, ImageSize -> 900 ]
(The core functions used for the implementation of CallGraphBiColorCircularEmbedding
were taken from kglr’s Mathematica Stack Exchange answer: https://mathematica.stackexchange.com/a/188390/34008 . Those functions were modified to take additional arguments.)
The package functions "CallGraph*" take all of the options of the function Graph. Below are described the additional options of CallGraph.
“SelfReferencing”
Should the self referencing edges be excluded or not.
“AtomicSymbols”
Should atomic symbols be included in the call graph.
Exclusions
Symbols to be excluded from the call graph.
“UsageTooltips”
Should vertex labels with the usage tooltips be added.
“UsageTooltipsStyle”
The style of the usage tooltips.
With large context (e.g. “System`”) the call graph generation might take long time. (See the TODOs below.)
With “PrivateContexts”->False the call graph will be empty if the public functions do not depend on each other.
For certain packages the scanning of the down values would produce (multiple) error messages or warnings.
The following is my TODO list for this project.
Use the symbols up-values to make the call graph.
Consider/implement call graph making with specified patterns and list of symbols.
Provide special functions for “call sequence” tracing for a specified symbol.
In this document we describe transformations of events records data in order to make that data more amenable for the application of Machine Learning (ML) algorithms.
Consider the following problem formulation (done with the next five bullet points.)
The events are observations of the values of a certain set of variables for a certain set of entities. Not all entities have events for all variables.
The observation times do not form a regular time grid.
Each contingency matrix has rows corresponding to the entities in the data and has columns corresponding to time.
The software component providing the functionality should allow parametrization and repeated execution. (As in ML classifier training and testing scenarios.)
The phrase “event records data” is used instead of “time series” in order to emphasize that (i) some variables have categorical values, and (ii) the data can be given in some general database form, like transactions long-form.
The required transformations of the event records in the problem formulation above are done through the monad ERTMon
, [AAp3]. (The name “ERTMon” comes from “Event Records Transformations Monad”.)
The monad code generation and utilization is explained in [AA1] and implemented with [AAp1].
It is assumed that the event records data is put in a form that makes it (relatively) easy to extract time series for the set of entity-variable pairs present in that data.
In brief ERTMon
performs the following sequence of transformations.
The event records for each entity-variable pair are aggregated and normalized with specified functions over a specified regular grid,
Entity vs. time interval contingency matrices are made for each combination of variable and aggregation function.
The transformations are specified with a “computation specification” dataset.
Here is an example of an ERTMon
pipeline over event records:
The rest of the document describes in detail:
the transformations of time series aligning, aggregation, and normalization,
the software pattern design – a monad – that allows sequential specifications of desired transformations.
Concrete examples are given using weather data. See [AAp9].
The following commands load the packages [AAp1-AAp9].
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicEventRecordsTransformations.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicTracing.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/Misc/WeatherEventRecords.m"]
The data we use is weather data from meteorological stations close to certain major cities. We retrieve the data with the function WeatherEventRecords
from the package [AAp9].
?WeatherEventRecords
WeatherEventRecords[ citiesSpec_: {{_String, _String}..}, dateRange:{{_Integer, _Integer, _Integer}, {_Integer, _Integer, _Integer}}, wProps:{_String..} : {“Temperature”}, nStations_Integer : 1 ] gives an association with event records data.
citiesSpec = {{"Miami", "USA"}, {"Chicago", "USA"}, {"London", "UK"}};
dateRange = {{2017, 7, 1}, {2018, 6, 31}};
wProps = {"Temperature", "MaxTemperature", "Pressure", "Humidity", "WindSpeed"};
res1 = WeatherEventRecords[citiesSpec, dateRange, wProps, 1];
citiesSpec = {{"Jacksonville", "USA"}, {"Peoria", "USA"}, {"Melbourne", "Australia"}};
dateRange = {{2016, 12, 1}, {2017, 12, 31}};
res2 = WeatherEventRecords[citiesSpec, dateRange, wProps, 1];
Here we assign the obtained datasets to variables we use below:
eventRecords = Join[res1["eventRecords"], res2["eventRecords"]];
entityAttributes = Join[res1["entityAttributes"], res2["entityAttributes"]];
Here are the summaries of the datasets eventRecords
and entityAttributes
:
RecordsSummary[eventRecords]
RecordsSummary[entityAttributes]
The steps of the main event records transformations workflow addressed in this document follow.
Ingest a computation specification.
Optionally, for each variable find outlier boundaries.
Align each group of records to start or finish at some specified point.
For each time series do prescribed aggregation and normalization.
The following flow-chart corresponds to the list of steps above.
A corresponding monadic pipeline is given in the section “Larger example pipeline”.
The workflow above describes a way to do feature engineering over a collection of event records data. For a given entity ID and a variable we derive several different time series.
Couple of examples follow.
Another possible derived feature (time series) is for each entity-variable pair to make a time series with the number of outliers in the each half-hour interval, considering the most recent 20 half-hour intervals. The outliers are found by using outlier boundaries derived by analyzing all values of the corresponding variable, across all entities.
From the examples above – and some others – we conclude that for each feature we want to be able to specify:
aggregation interval length,
aggregation function (to be applied in each interval),
normalization function (per entity, per cohort of entities, per variable),
conversion of categorical values into numerical ones.
We want to be able to do repeated executions of the specified workflow steps.
Consider the following scenario. After the event records data is converted to a entity-vs-feature contingency matrix, we use that matrix to train and test a classifier. We want to find the combination of features that gives the best classifier results. For that reason we want to be able to easily and systematically change the computation specifications (interval size, aggregation and normalization functions, etc.) With different computation specifications we obtain different entity-vs-feature contingency matrices, that would have different performance with different classifiers.
Using the classifier training and testing scenario we see that there is another repeated execution perspective: after the feature engineering is done over the training data, we want to be able to execute exactly the same steps over the test data. Note that with the training data we find certain global or cohort normalization values and outlier boundaries that have to be used over the test data. (Not derived from the test data.)
The following diagram further describes the repeated execution workflow.
Further discussion of making and using ML classification workflows through the monad software design pattern can be found in [AA2].
The data is structured to follow the style of Star schema. We have event records dataset (table) and entity attributes dataset (table).
The structure datasets (tables) proposed satisfy a wide range of modeling data requirements. (Medical and financial modeling included.)
The entity event data has the columns “EntityID”, “LocationID”, “ObservationTime”, “Variable”, “Value”.
RandomSample[eventRecords, 6]
Most events can be described through “Entity event data”. The entities can be anything that produces a set of event data: financial transactions, vital sign monitors, wind speed sensors, chemical concentrations sensors.
The locations can be anything that gives the events certain “spatial” attributes: medical units in hospitals, sensors geo-locations, tiers of financial transactions.
The entity attributes dataset (table) has attributes (immutable properties) of the entities. (Like, gender and race for people, longitude and latitude for wind speed sensors.)
entityAttributes[[1 ;; 6]]
For example, here we take all weather stations in USA:
ws = Normal[entityAttributes[Select[#Attribute == "Country" && #Value == "USA" &], "EntityID"]]
(* {"KMFL", "KMDW", "KNIP", "KGEU"} *)
Here we take all temperature event records for those weather stations:
srecs = eventRecords[Select[#Variable == "Temperature" && MemberQ[ws, #EntityID] &]];
And here plot the corresponding time series obtained by grouping the records by station (entity ID’s) and taking the columns “ObservationTime” and “Value”:
grecs = Normal @ GroupBy[srecs, #EntityID &][All, All, {"ObservationTime", "Value"}];
DateListPlot[grecs, ImageSize -> Large, PlotTheme -> "Detailed"]
This section goes through the steps of the general ERTMon
workflow. For didactic purposes each sub-section changes the pipeline assigned to the variable p
. Of course all functions can be chained into one big pipeline as shown in the section “Larger example pipeline”.
The monad is initialized with ERTMonUnit.
ERTMonUnit[]
(* ERTMon[None, <||>] *)
The event records dataset (table) and entity attributes dataset (table) are set with corresponding setter functions. Alternatively, they can be read from files in a specified directory.
p =
ERTMonUnit[]⟹
ERTMonSetEventRecords[eventRecords]⟹
ERTMonSetEntityAttributes[entityAttributes]⟹
ERTMonEchoDataSummary;
Using the package [AAp3] we can create computation specification dataset. Below is given an example of constructing a fairly complicated computation specification.
The package function EmptyComputationSpecificationRow
can be used to construct the rows of the specification.
EmptyComputationSpecificationRow[]
(* <|"Variable" -> Missing[], "Explanation" -> "",
"MaxHistoryLength" -> 3600, "AggregationIntervalLength" -> 60,
"AggregationFunction" -> "Mean", "NormalizationScope" -> "Entity",
"NormalizationFunction" -> "None"|> *)
compSpecRows =
Join[EmptyComputationSpecificationRow[], <|"Variable" -> #,
"MaxHistoryLength" -> 60*24*3600,
"AggregationIntervalLength" -> 2*24*3600,
"AggregationFunction" -> "Mean",
"NormalizationScope" -> "Entity",
"NormalizationFunction" -> "Mean"|>] & /@
Union[Normal[eventRecords[All, "Variable"]]];
compSpecRows =
Join[
compSpecRows,
Join[EmptyComputationSpecificationRow[], <|"Variable" -> #,
"MaxHistoryLength" -> 60*24*3600,
"AggregationIntervalLength" -> 2*24*3600,
"AggregationFunction" -> "Range",
"NormalizationScope" -> "Country",
"NormalizationFunction" -> "Mean"|>] & /@
Union[Normal[eventRecords[All, "Variable"]]],
Join[EmptyComputationSpecificationRow[], <|"Variable" -> #,
"MaxHistoryLength" -> 60*24*3600,
"AggregationIntervalLength" -> 2*24*3600,
"AggregationFunction" -> "OutliersCount",
"NormalizationScope" -> "Variable"|>] & /@
Union[Normal[eventRecords[All, "Variable"]]]
];
The constructed rows are assembled into a dataset (with Dataset
). The function ProcessComputationSpecification
is used to convert a user-made specification dataset into a form used by ERTMon
.
wCompSpec =
ProcessComputationSpecification[Dataset[compSpecRows]][SortBy[#Variable &]]
The computation specification is set to the monad with the function ERTMonSetComputationSpecification
.
Alternatively, a computation specification can be created and filled-in as a CSV file and read into the monad. (Not described here.)
With the function ERTMonGroupEntityVariableRecords
we group the event records by the found unique entity-variable pairs. Note that in the pipeline below we set the computation specification first.
p =
p⟹
ERTMonSetComputationSpecification[wCompSpec]⟹
ERTMonGroupEntityVariableRecords;
After the data is ingested into the monad and the event records are grouped per entity-variable pairs we can find certain descriptive statistics for the data. This is done with the general function ERTMonComputeVariableStatistic and the specialized function ERTMonFindVariableOutlierBoundaries.
p⟹ERTMonComputeVariableStatistic[RecordsSummary]⟹ERTMonEchoValue;
p⟹ERTMonComputeVariableStatistic⟹ERTMonEchoValue;
p⟹ERTMonComputeVariableStatistic[TakeLargest[#, 3] &]⟹ERTMonEchoValue;
(* value: <|Humidity->{1.,1.,0.993}, MaxTemperature->{48,48,48},
Pressure->{1043.1,1042.8,1041.1}, Temperature->{42.28,41.94,41.89},
WindSpeed->{54.82,44.63,44.08}|> *)
The finding of outliers counts and fractions can be specified in the computation specification. Because of this there is a specialized function for outlier finding ERTMonFindVariableOutlierBoundaries
. That function makes the association of the found variable outlier boundaries (i) to be the pipeline value and (ii) to be the value of context key “variableOutlierBoundaries”. The outlier boundaries are found using the functions of the package [AAp6].
If no argument is specified ERTMonFindVariableOutlierBoundaries uses the Hampel identifier (HampelIdentifierParameters
).
p⟹ERTMonFindVariableOutlierBoundaries⟹ERTMonEchoValue;
(* value: <|Humidity->{0.522536,0.869464}, MaxTemperature->{14.2106,31.3494},
Pressure->{1012.36,1022.44}, Temperature->{9.88823,28.3318},
WindSpeed->{5.96141,19.4086}|> *)
Keys[p⟹ERTMonFindVariableOutlierBoundaries⟹ERTMonTakeContext]
(* {"eventRecords", "entityAttributes", "computationSpecification",
"entityVariableRecordGroups", "variableOutlierBoundaries"} *)
In the rest of document we use the outlier boundaries found with the more conservative identifier SPLUSQuartileIdentifierParameters
.
p =
p⟹
ERTMonFindVariableOutlierBoundaries[SPLUSQuartileIdentifierParameters]⟹
ERTMonEchoValue;
(* value: <|Humidity->{0.176,1.168}, MaxTemperature->{-1.67,45.45},
Pressure->{1003.75,1031.35}, Temperature->{-5.805,43.755},
WindSpeed->{-5.005,30.555}|> *)
The grouped event records are converted into time series with the function ERTMonEntityVariableGroupsToTimeSeries
. The time series are aligned to a time point specification given as an argument. The argument can be: a date object, “MinTime”, “MaxTime”, or “None”. (“MaxTime” is the default.)
p⟹
ERTMonEntityVariableGroupsToTimeSeries["MinTime"]⟹
ERTMonEchoFunctionContext[#timeSeries[[{1, 3, 5}]] &];
Compare the last output with the output of the following command.
p =
p⟹
ERTMonEntityVariableGroupsToTimeSeries["MaxTime"]⟹
ERTMonEchoFunctionContext[#timeSeries[[{1, 3, 5}]] &];
The main goal of ERTMon
is to convert a diverse, general collection of event records into a collection of aligned time series over specified regular time grids.
The regular time grids are specified with the columns “MaxHistoryLength” and “AggregationIntervalLength” of the computation specification. The time series of the variables in the computation specification are restricted to the corresponding maximum history lengths and are aggregated using the corresponding aggregation lengths and functions.
p =
p⟹
ERTMonAggregateTimeSeries⟹
ERTMonEchoFunctionContext[DateListPlot /@ #timeSeries[[{1, 3, 5}]] &];
At this point we can apply time series modifying functions. An often used such function is moving average.
p⟹
ERTMonApplyTimeSeriesFunction[MovingAverage[#, 6] &]⟹
ERTMonEchoFunctionValue[DateListPlot /@ #[[{1, 3, 5}]] &];
Note that the result is given as a pipeline value, the value of the context key “timeSeries” is not changed.
(In the future, the computation specification and its handling might be extended to handle moving average or other time series function specifications.)
With “normalization” we mean that the values of a given time series values are divided (normalized) with a descriptive statistic derived from a specified set of values. The specified set of values is given with the parameter “NormalizationScope” in computation specification.
At the normalization stage each time series is associated with an entity ID and a variable.
Normalization is done at three different scopes: “entity”, “attribute”, and “variable”.
Given a time series corresponding to entity ID and a variable we define the normalization values for the different scopes in the following ways.
Normalization with scope attribute means that
next we find all entity ID’s that are associated with the same attribute value,
next we find value of normalization descriptive statistic using the time series that correspond to the variable and the entity ID’s found in the previous step.
Normalization with scope “variable” means that the descriptive statistic is derived from the values of all time series corresponding to .
Note that the scope “entity” is the most granular, and the scope “variable” is the coarsest.
The following command demonstrates the normalization effect – compare the -axes scales of the time series corresponding to the same entity-variable pair.
p =
p⟹
ERTMonEchoFunctionContext[DateListPlot /@ #timeSeries[[{1, 3, 5}]] &]⟹
ERTMonNormalize⟹
ERTMonEchoFunctionContext[DateListPlot /@ #timeSeries[[{1, 3, 5}]] &];
Here are the normalization values that should be used when normalizing “unseen data.”
p⟹ERTMonTakeNormalizationValues
(* <|{"Humidity.Range", "Country", "USA"} -> 0.0864597,
{"Humidity.Range", "Country", "UK"} -> 0.066,
{"Humidity.Range", "Country", "Australia"} -> 0.145968,
{"MaxTemperature.Range", "Country", "USA"} -> 2.85468,
{"MaxTemperature.Range", "Country", "UK"} -> 78/31,
{"MaxTemperature.Range", "Country", "Australia"} -> 3.28871,
{"Pressure.Range", "Country", "USA"} -> 2.08222,
{"Pressure.Range", "Country", "Australia"} -> 3.33871,
{"Temperature.Range", "Country", "USA"} -> 2.14411,
{"Temperature.Range", "Country", "UK"} -> 1.25806,
{"Temperature.Range", "Country", "Australia"} -> 2.73032,
{"WindSpeed.Range", "Country", "USA"} -> 4.13532,
{"WindSpeed.Range", "Country", "UK"} -> 3.62097,
{"WindSpeed.Range", "Country", "Australia"} -> 3.17226|> *)
One of the main goals of ERTMon
is to produce contingency matrices corresponding to the event records data.
The contingency matrices are created and stored as SSparseMatrix
objects, [AAp7].
p =
p⟹ERTMonMakeContingencyMatrices;
We can obtain an association of the contingency matrices for each variable-and-aggregation-function pair, or obtain the overall contingency matrix.
p⟹ERTMonTakeContingencyMatrices
Dimensions /@ %
smat = p⟹ERTMonTakeContingencyMatrix;
MatrixPlot[smat, ImageSize -> 700]
RowNames[smat]
(* {"EGLC", "KGEU", "KMDW", "KMFL", "KNIP", "WMO95866"} *)
The pipeline shown in this section utilizes all main workflow functions of ERTMon
. The used weather data and computation specification are described above.
[AAp1] Anton Antonov, State monad code generator Mathematica package, (2017), MathematicaForPrediction at GitHub*. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/StateMonadCodeGenerator.m .
[AAp2] Anton Antonov, Monadic tracing Mathematica package, (2017), MathematicaForPrediction at GitHub*. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicTracing.m .
[AAp3] Anton Antonov, Monadic Event Records Transformations Mathematica package, (2018), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicEventRecordsTransformations.m .
[AAp4] Anton Antonov, MathematicaForPrediction utilities, (2014), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MathematicaForPredictionUtilities.m .
[AAp5] Anton Antonov, Cross tabulation implementation in Mathematica, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/CrossTabulate.m .
[AAp6] Anton Antonov, Implementation of one dimensional outlier identifying algorithms in Mathematica, (2013), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/OutlierIdentifiers.m.
[AAp7] Anton Antonov, SSparseMatrix Mathematica package, (2018), MathematicaForPrediction at GitHub*. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/SSparseMatrix.m .
[AAp8] Anton Antonov, Monadic contextual classification Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicContextualClassification.m .
[AAp9] Anton Antonov, Weather event records data Mathematica package, (2018), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/Misc/WeatherEventRecords.m .
[AA1] Anton Antonov, Monad code generation and extension, (2017), MathematicaForPrediction at WordPress. URL: https://mathematicaforprediction.wordpress.com/2017/06/23/monad-code-generation-and-extension .
[AA1a] Anton Antonov, Monad code generation and extension, (2017), MathematicaForPrediction at GitHub.
[AA2] Anton Antonov, A monad for classification workflows, (2018), MathematicaForPrediction at WordPress. URL: https://mathematicaforprediction.wordpress.com/2018/05/15/a-monad-for-classification-workflows .
]]>In this notebook/document we apply the monad QRMon
[3] over data of the article [1]. In order to get the data we use extraction procedure described in [2].
(I saw the article [1] while browsing LinkedIn today. I met one of the authors during the event "Data Science Salon Miami Feb 2018".)
I extracted the data from the image using "Recovering data points from an image".
img = Import["https://www.spglobal.com/_assets/images/marketintelligence/blog-images/demonstration-of-model-fit-comparison-visualization.png"]
extractedData
(* {{-1., 0.284894}, {-0.987395, 0.340483}, {-0.966387,
0.215408}, {-0.941176, 0.416918}, {-0.894958, 0.222356}, {-0.890756,
0.215408}, {-0.878151, 0.0903323}, {-0.848739,
0.132024}, {-0.844538, 0.10423}, {-0.831933, 0.333535}, {-0.819328,
0.180665}, {-0.781513, 0.423867}, {-0.756303, 0.40997}, {-0.752101,
0.528097}, {-0.747899, 0.416918}, {-0.731092, 0.375227}, {-0.714286,
0.194562}, {-0.710084, 0.340483}, {-0.651261,
0.555891}, {-0.647059, 0.333535}, {-0.605042, 0.496828}, {-0.57563,
0.}, {-0.512605, 0.354381}, {-0.491597, 0.368278}, {-0.487395,
0.472508}, {-0.478992, 0.479456}, {-0.453782, 0.437764}, {-0.357143,
0.15287}, {-0.344538, 0.340483}, {-0.331933, 0.333535}, {-0.315126,
0.500302}, {-0.285714, 0.396073}, {-0.247899,
0.618429}, {-0.201681, 0.541994}, {-0.159664, 0.680967}, {-0.10084,
1.06314}, {-0.0966387, 0.993656}, {0., 1.36193}, {0.0210084,
1.44532}, {0.0420168, 1.5148}, {0.0504202, 1.5148}, {0.0882353,
1.41405}, {0.130252, 1.70937}, {0.172269, 2.029}, {0.176471,
1.7858}, {0.222689, 2.20272}, {0.226891, 2.23746}, {0.231092,
2.23746}, {0.239496, 1.96647}, {0.268908, 1.94562}, {0.273109,
1.91088}, {0.277311, 1.91088}, {0.281513, 1.94562}, {0.294118,
2.2861}, {0.319328, 2.26526}, {0.327731, 2.3}, {0.432773,
1.68157}, {0.462185, 1.86918}, {0.5, 2.00121}} *)
ListPlot[extractedData, PlotRange -> All, PlotTheme -> "Detailed"]
QRMon
Load packages. (For more details see [4].)
Import["https://raw.githubusercontent.com/antononcube/\
MathematicaForPrediction/master/MonadicProgramming/\
MonadicQuantileRegression.m"]
Import["https://raw.githubusercontent.com/antononcube/\
MathematicaForPrediction/master/MonadicProgramming/MonadicTracing.m"]
Apply the QRMon workflow within the TraceMonad:
TraceMonadUnit[QRMonUnit[extractedData]]⟹"lift data to the monad"⟹
QRMonEchoDataSummary⟹"echo data summary"⟹
QRMonQuantileRegression[12, 0.5]⟹"do Quantile Regression with\nB-spline basis with 12 knots"⟹
QRMonPlot⟹"plot the data and regression curve"⟹
QRMonEcho[Style["Tabulate QRMon steps and explanations:", Purple, Bold]]⟹"echo an explanation message"⟹
TraceMonadEchoGrid;
[1] Moody Hadi and Danny Haydon, "A Perspective On Machine Learning In Credit Risk", (2018), S&P Global Market Intelligence.
[2] Andy Ross, answer of "Recovering data points from an image", (2012).
[3] Anton Antonov, "A monad for Quantile Regression workflows", (2018), MathematicaForPrediction at WordPress.
[4] Anton Antonov, "Monad code generation and extension", (2017), MathematicaForPrediction at GitHub*,https://github.com/antononcube/MathematicaForPrediction.
]]>In this document we describe the design and implementation of a (software programming) monad for Quantile Regression workflows specification and execution. The design and implementation are done with Mathematica / Wolfram Language (WL).
What is Quantile Regression? : Assume we have a set of two dimensional points each point being a pair of an independent variable value and a dependent variable value. We want to find a curve that is a function of the independent variable that splits the points in such a way that, say, 30% of the points are above that curve. This is done with Quantile Regression, see [Wk2, CN1, AA2, AA3]. Quantile Regression is a method to estimate the variable relations for all parts of the distribution. (Not just, say, the mean of the relationships found with Least Squares Regression.)
The goal of the monad design is to make the specification of Quantile Regression workflows (relatively) easy, straightforward, by following a certain main scenario and specifying variations over that scenario. Since Quantile Regression is often compared with Least Squares Regression and some type of filtering (like, Moving Average) those functionalities should be included in the monad design scenarios.
The monad is named QRMon and it is based on the State monad package "StateMonadCodeGenerator.m", [AAp1, AA1] and the Quantile Regression package "QuantileRegression.m", [AAp4, AA2].
The data for this document is read from WL’s repository or created ad-hoc.
The monadic programming design is used as a Software Design Pattern. The QRMon monad can be also seen as a Domain Specific Language (DSL) for the specification and programming of machine learning classification workflows.
Here is an example of using the QRMon monad over heteroscedastic data::
The table above is produced with the package "MonadicTracing.m", [AAp2, AA1], and some of the explanations below also utilize that package.
As it was mentioned above the monad QRMon can be seen as a DSL. Because of this the monad pipelines made with QRMon are sometimes called "specifications".
Remark: With "regression quantile" we mean "a curve or function that is computed with Quantile Regression".
The document has the following structure.
The sections "QRMon overview" and "Monad elements" provide technical description of the QRMon monad needed to utilize it.
ClCon
, [AA6].
Remark: One can read only the sections "Introduction", "Design consideration", "Monad design", and "QRMon overview". That set of sections provide a fairly good, programming language agnostic exposition of the substance and novel ideas of this document.
The table above is produced with the package "MonadicTracing.m", [AAp2, AA1], and some of the explanations below also utilize that package.
As it was mentioned above the monad QRMon can be seen as a DSL. Because of this the monad pipelines made with QRMon are sometimes called "specifications".
Remark: With "regression quantile" we mean "a curve or function that is computed with Quantile Regression".
The following commands load the packages [AAp1–AAp6]:
Import["https://raw.githubusercontent.com/antononcube/\
MathematicaForPrediction/master/MonadicProgramming/\
MonadicQuantileRegression.m"]
Import["https://raw.githubusercontent.com/antononcube/\
MathematicaForPrediction/master/MonadicProgramming/MonadicTracing.m"]
In this section we load data that is used in the rest of the document. The time series data is obtained through WL’s repository.
The data summarization and plots are done through QRMon, which in turn uses the function RecordsSummary from the package "MathematicaForPredictionUtilities.m", [AAp6].
The following data is generated to have [heteroscedasticity(https://en.wikipedia.org/wiki/Heteroscedasticity).
distData =
Table[{x,
Exp[-x^2] +
RandomVariate[
NormalDistribution[0, .15 Sqrt[Abs[1.5 - x]/1.5]]]}, {x, -3,
3, .01}];
Length[distData]
(* 601 *)
QRMonUnit[distData]⟹QRMonEchoDataSummary⟹QRMonPlot;
tsData = WeatherData[{"Orlando", "USA"}, "Temperature", {{2015, 1, 1}, {2018, 1, 1}, "Day"}]
QRMonUnit[tsData]⟹QRMonEchoDataSummary⟹QRMonDateListPlot;
The following data is typical for financial time series. (Note the differences with the temperature time series.)
finData = TimeSeries[FinancialData["NYSE:GE", {{2014, 1, 1}, {2018, 1, 1}, "Day"}]];
QRMonUnit[finData]⟹QRMonEchoDataSummary⟹QRMonDateListPlot;
The steps of the main regression workflow addressed in this document follow.
Optionally, transform the data.
Rescale data along one or both of the axes.
Apply moving average (or median, or map.)
Verify assumptions of the data.
Run a regression algorithm with a certain basis of functions using:
Least Squares Regression.
Visualize the data and regression functions.
If the regression functions fit is not satisfactory go to step 4.
Utilize the found regression functions to compute:
local extrema,
approximation or fitting errors,
conditional density distributions,
time series simulations.
The following flow-chart corresponds to the list of steps above.
In order to address:
workflows elements variability, and
workflows iterative changes and refining,
it is beneficial to have a DSL for regression workflows. We choose to make such a DSL through a functional programming monad, [Wk1, AA1].
Here is a quote from [Wk1] that fairly well describes why we choose to make a classification workflow monad and hints on the desired properties of such a monad.
[…] The monad represents computations with a sequential structure: a monad defines what it means to chain operations together. This enables the programmer to build pipelines that process data in a series of steps (i.e. a series of actions applied to the data), in which each action is decorated with the additional processing rules provided by the monad. […] Monads allow a programming style where programs are written by putting together highly composable parts, combining in flexible ways the possible actions that can work on a particular type of data. […]
Remark: Note that quote from [Wk1] refers to chained monadic operations as "pipelines". We use the terms "monad pipeline" and "pipeline" below.
The monad we consider is designed to speed-up the programming of quantile regression workflows outlined in the previous section. The monad is named QRMon for "Quantile Regression Monad".
We want to be able to construct monad pipelines of the general form:
QRMon is based on the State monad, [Wk1, AA1], so the monad pipeline form (1) has the following more specific form:
This means that some monad operations will not just change the pipeline value but they will also change the pipeline context.
In the monad pipelines of QRMon we store different objects in the contexts for at least one of the following two reasons.
The object is (relatively) hard to compute.
Such objects are transformed data, regression functions, and outliers.
Let us list the desired properties of the monad.
The monad works with time series, numerical matrices, and numerical vectors.
The pipeline values can be of different types. Most monad functions modify the pipeline value; some modify the context; some just echo results.
The monad can do quantile regression with B-Splines bases, quantile regression fit and least squares fit with specified bases of functions.
The monad allows of cursory examination and summarization of the data.
It is easy to obtain the pipeline value, context, and different context objects for manipulation outside of the monad.
It is easy to plot different combinations of data, regression functions, outliers, approximation errors, etc.
The QRMon components and their interactions are fairly simple.
The main QRMon operations implicitly put in the context or utilize from the context the following objects:
regression functions,
outliers and outlier regression functions.
Note the that the monadic set of types of QRMon pipeline values is fairly heterogenous and certain awareness of "the current pipeline value" is assumed when composing QRMon pipelines.
Obviously, we can put in the context any object through the generic operations of the State monad of the package "StateMonadGenerator.m", [AAp1].
When using a monad we lift certain data into the "monad space", using monad’s operations we navigate computations in that space, and at some point we take results from it.
With the approach taken in this document the "lifting" into the QRMon monad is done with the function QRMonUnit. Results from the monad can be obtained with the functions QRMonTakeValue, QRMonContext, or with the other QRMon functions with the prefix "QRMonTake" (see below.)
Here is a corresponding diagram of a generic computation with the QRMon monad:
Remark: It is a good idea to compare the diagram with formulas (1) and (2).
Let us examine a concrete QRMon pipeline that corresponds to the diagram above. In the following table each pipeline operation is combined together with a short explanation and the context keys after its execution.
Here is the output of the pipeline:
The QRMon functions are separated into four groups:
setters and droppers,
takers,
State Monad generic functions.
An overview of the those functions is given in the tables in next two sub-sections. The next section, "Monad elements", gives details and examples for the usage of the QRMon operations.
The following table gives an overview the interaction of the QRMon monad functions with the pipeline value and context.
The following table shows the functions that are function synonyms or short-cuts.
Here are the QRMon State Monad functions (generated using the prefix "QRMon", [AAp1, AA1]):
In this section we show that QRMon has all of the properties listed in the previous section.
The monad head is QRMon. Anything wrapped in QRMon can serve as monad’s pipeline value. It is better though to use the constructor QRMonUnit. (Which adheres to the definition in [Wk1].)
QRMon[{{1, 223}, {2, 323}}, <||>]⟹QRMonEchoDataSummary;
The function lifting the data into the monad QRMon is QRMonUnit.
The lifting to the monad marks the beginning of the monadic pipeline. It can be done with data or without data. Examples follow.
QRMonUnit[distData]⟹QRMonEchoDataSummary;
QRMonUnit[]⟹QRMonSetData[distData]⟹QRMonEchoDataSummary;
(See the sub-section "Setters, droppers, and takers" for more details of setting and taking values in QRMon contexts.)
Currently the monad can deal with data in the following forms:
numerical vectors,
numerical matrices of rank two.
When the data lifted to the monad is a numerical vector vec it is assumed that vec has to become the second column of a "time series" matrix; the first column is derived with Range[Length[vec]] .
Generally, WL makes it easy to extract columns datasets order to obtain numerical matrices, so datasets are not currently supported in QRMon.
This computes quantile regression with B-spline basis over 12 regularly spaced knots. (Using Linear Programming algorithms; see [AA2] for details.)
QRMonUnit[distData]⟹
QRMonQuantileRegression[12]⟹
QRMonPlot;
The monad function QRMonQuantileRegression has the same options as QuantileRegression. (The default value for option Method is different, since using "CLP" is generally faster.)
Options[QRMonQuantileRegression]
(* {InterpolationOrder -> 3, Method -> {LinearProgramming, Method -> "CLP"}} *)
Let us compute regression using a list of particular knots, specified quantiles, and the method "InteriorPoint" (instead of the Linear Programming library CLP):
p =
QRMonUnit[distData]⟹
QRMonQuantileRegression[{-3, -2, 1, 0, 1, 1.5, 2.5, 3}, Range[0.1, 0.9, 0.2], Method -> {LinearProgramming, Method -> "InteriorPoint"}]⟹
QRMonPlot;
Remark: As it was mentioned above the function QRMonRegression is a synonym of QRMonQuantileRegression.
The fit functions can be extracted from the monad with QRMonTakeRegressionFunctions, which gives an association of quantiles and pure functions.
ListPlot[# /@ distData[[All, 1]]] & /@ (p⟹QRMonTakeRegressionFunctions)
Instead of using a B-spline basis of functions we can compute a fit with our own basis of functions.
Here is a basis functions:
bFuncs = Table[PDF[NormalDistribution[m, 1], x], {m, Min[distData[[All, 1]]], Max[distData[[All, 1]]], 1}];
Plot[bFuncs, {x, Min[distData[[All, 1]]], Max[distData[[All, 1]]]},
PlotRange -> All, PlotTheme -> "Scientific"]
Here we do a Quantile Regression fit, a Least Squares fit, and plot the results:
p =
QRMonUnit[distData]⟹
QRMonQuantileRegressionFit[bFuncs]⟹
QRMonLeastSquaresFit[bFuncs]⟹
QRMonPlot;
Remark: The functions "QRMon*Fit" should generally have a second argument for the symbol of the basis functions independent variable. Often that symbol can be omitted and implied. (Which can be seen in the pipeline above.)
Remark: As it was mentioned above the function QRMonRegressionFit is a synonym of QRMonQuantileRegressionFit and QRMonFit is a synonym of QRMonLeastSquaresFit.
As it was pointed out in the previous sub-section, the fit functions can be extracted from the monad with QRMonTakeRegressionFunctions. Here the keys of the returned/taken association consist of quantiles and "mean" since we applied both Quantile Regression and Least Squares Regression.
ListPlot[# /@ distData[[All, 1]]] & /@ (p⟹QRMonTakeRegressionFunctions)
One of the main advantages of using the function QuanileRegression of the package [AAp4] is that the functions used to do the regression with are specified with a few numeric parameters. (Most often only the number of knots is sufficient.) This is achieved by using a basis of B-spline functions of a certain interpolation order.
We want similar behaviour for Quantile Regression fitting we need to select a certain well known basis with certain desirable properties. Such basis is given by Chebyshev polynomials of first kind [Wk3]. Chebyshev polynomials bases can be easily generated in Mathematica with the functions ChebyshevT or ChebyshevU.
Here is an application of fitting with a basis of 12 Chebyshev polynomials of first kind:
QRMonUnit[distData]⟹
QRMonQuantileRegressionFit[12]⟹
QRMonLeastSquaresFit[12]⟹
QRMonPlot;
The code above is equivalent to the following code:
bfuncs = Table[ChebyshevT[i, Rescale[x, MinMax[distData[[All, 1]]], {-0.95, 0.95}]], {i, 0, 12}];
p =
QRMonUnit[distData]⟹
QRMonQuantileRegressionFit[bfuncs]⟹
QRMonLeastSquaresFit[bfuncs]⟹
QRMonPlot;
The shrinking of the ChebyshevT domain seen in the definitions of bfuncs is done in order to prevent approximation error effects at the ends of the data domain. The following code uses the ChebyshevT domain { − 1, 1} instead of the domain { − 0.95, 0.95} used above.
QRMonUnit[distData]⟹
QRMonQuantileRegressionFit[{4, {-1, 1}}]⟹
QRMonPlot;
The computed quantile and least squares regression functions can be evaluated with the monad function QRMonEvaluate.
Evaluation for a given value of the independent variable:
p⟹QRMonEvaluate[0.12]⟹QRMonTakeValue
(* <|0.25 -> 0.930402, 0.5 -> 1.01411, 0.75 -> 1.08075, "mean" -> 0.996963|> *)
Evaluation for a vector of values:
p⟹QRMonEvaluate[Range[-1, 1, 0.5]]⟹QRMonTakeValue
(* <|0.25 -> {0.258241, 0.677461, 0.943299, 0.703812, 0.293741},
0.5 -> {0.350025, 0.768617, 1.02311, 0.807879, 0.374545},
0.75 -> {0.499338, 0.912183, 1.10325, 0.856729, 0.431227},
"mean" -> {0.355353, 0.776006, 1.01118, 0.783304, 0.363172}|> *)
Evaluation for complicated lists of numbers:
p⟹QRMonEvaluate[{0, 1, {1.5, 1.4}}]⟹QRMonTakeValue
(* <|0.25 -> {0.943299, 0.293741, {0.0762883, 0.10759}},
0.5 -> {1.02311, 0.374545, {0.103386, 0.139142}},
0.75 -> {1.10325, 0.431227, {0.133755, 0.177161}},
"mean" -> {1.01118, 0.363172, {0.107989, 0.142021}}|> *)
The obtained values can be used to compute estimates of the distributions of the dependent variable. See the sub-sections "Estimating conditional distributions" and "Dependent variable simulation".
Here with "errors" we mean the differences between data’s dependent variable values and the corresponding values calculated with the fitted regression curves.
In the pipeline below we compute couple of regression quantiles, plot them together with the data, we plot the errors, compute the errors, and summarize them.
QRMonUnit[finData]⟹
QRMonQuantileRegression[10, {0.5, 0.1}]⟹
QRMonDateListPlot[Joined -> False]⟹
QRMonErrorPlots["DateListPlot" -> True, Joined -> False]⟹
QRMonErrors⟹
QRMonEchoFunctionValue["Errors summary:", RecordsSummary[#[[All, 2]]] & /@ # &];
Each of the functions QRMonErrors and QRMonErrorPlots computes the errors. (That computation is considered cheap.)
Finding outliers can be done with the function QRMonOultiers. The outliers found by QRMonOutliers are simply points that below or above certain regression quantile curves, for example, the ones corresponding to 0.02 and 0.98.
Here is an example:
p =
QRMonUnit[distData]⟹
QRMonQuantileRegression[6, {0.02, 0.98}]⟹
QRMonOutliers⟹
QRMonEchoValue⟹
QRMonOutliersPlot;
The function QRMonOutliers puts in the context values for the keys "outliers" and "outlierRegressionFunctions". The former is for the found outliers, the latter is for the functions corresponding to the used regression quantiles.
Keys[p⟹QRMonTakeContext]
(* {"data", "regressionFunctions", "outliers", "outlierRegressionFunctions"} *)
Here are the corresponding quantiles of the plot above:
Keys[p⟹QRMonTakeOutlierRegressionFunctions]
(* {0.02, 0.98} *)
The control of the outliers computation is done though the arguments and options of QRMonQuantileRegression (or the rest of the regression calculation functions.)
If only one regression quantile is found in the context and the corresponding quantile is less than 0.5 then QRMonOutliers finds only bottom outliers. If only one regression quantile is found in the context and the corresponding quantile is greater than 0.5 then QRMonOutliers finds only top outliers.
Here is an example for finding only the top outliers:
QRMonUnit[finData]⟹
QRMonQuantileRegression[5, 0.8]⟹
QRMonOutliers⟹
QRMonEchoFunctionContext["outlier quantiles:", Keys[#outlierRegressionFunctions] &]⟹
QRMonOutliersPlot["DateListPlot" -> True];
The function QRMonOutliersPlot makes an outliers plot. If the outliers are not in the context then QRMonOutliersPlot calls QRMonOutliers first.
Here are the options of QRMonOutliersPlot:
Options[QRMonOutliersPlot]
(* {"Echo" -> True, "DateListPlot" -> False, ListPlot -> {Joined -> False}, Plot -> {}} *)
The default behavior is to echo the plot. That can be suppressed with the option "Echo".
QRMonOutliersPlot utilizes combines with Show two plots:
the other with Plot for the regression quantiles used to find the outliers.
That is why separate lists of options can be given to manipulate those two plots. The option DateListPlot can be used make plots with date or time axes.
QRMonUnit[tsData]⟹
QRMonQuantileRegression[12, {0.01, 0.99}]⟹
QRMonOutliersPlot[
"Echo" -> False,
"DateListPlot" -> True,
ListPlot -> {PlotStyle -> {Green, {PointSize[0.02],
Red}, {PointSize[0.02], Blue}}, Joined -> False,
PlotTheme -> "Grid"},
Plot -> {PlotStyle -> Orange}]⟹
QRMonTakeValue
Consider the following problem:
How to estimate the conditional density of the dependent variable given a value of the conditioning independent variable?
(In other words, find the distribution of the y-values for a given, fixed x-value.)
The solution of this problem using Quantile Regression is discussed in detail in [PG1] and [AA4].
Finding a solution for this problem can be seen as a primary motivation to develop Quantile Regression algorithms.
The following pipeline (i) computes and plots a set of five regression quantiles and (ii) then using the found regression quantiles computes and plots the conditional distributions for two focus points (−2 and 1.)
QRMonUnit[distData]⟹
QRMonQuantileRegression[6,
Range[0.1, 0.9, 0.2]]⟹
QRMonPlot[GridLines -> {{-2, 1}, None}]⟹
QRMonConditionalCDF[{-2, 1}]⟹
QRMonConditionalCDFPlot;
Fairly often it is a good idea for a given time series to apply filter functions like Moving Average or Moving Median. We might want to:
do regression over the transformed data,
compare with regression curves over the original data.
For these reasons QRMon has the functions QRMonMovingAverage, QRMonMovingMedian, and QRMonMovingMap that correspond to the built-in functions MovingAverage, MovingMedian, and MovingMap.
Here is an example:
QRMonUnit[tsData]⟹
QRMonDateListPlot[ImageSize -> Small]⟹
QRMonMovingAverage[20]⟹
QRMonEchoFunctionValue["Moving avg: ", DateListPlot[#, ImageSize -> Small] &]⟹
QRMonMovingMap[Mean, Quantity[20, "Days"]]⟹
QRMonEchoFunctionValue["Moving map: ", DateListPlot[#, ImageSize -> Small] &];
Consider the problem of making a time series that is a simulation of a process given with a known time series.
More formally,
we consider each grid node to correspond to a random variable,
we want to generate time series based on the empirical CDF’s of the random variables that correspond to the grid nodes.
The formulation of the problem hints to an (almost) straightforward implementation using Quantile Regression.
p = QRMonUnit[tsData]⟹QRMonQuantileRegression[30, Join[{0.01}, Range[0.1, 0.9, 0.1], {0.99}]];
tsNew =
p⟹
QRMonSimulate[1000]⟹
QRMonTakeValue;
opts = {ImageSize -> Medium, PlotTheme -> "Detailed"};
GraphicsGrid[{{DateListPlot[tsData, PlotLabel -> "Actual", opts],
DateListPlot[tsNew, PlotLabel -> "Simulated", opts]}}]
Using regression fitting — and Quantile Regression in particular — we can easily construct semi-symbolic algorithms for finding local extrema in noisy time series data; see [AA5]. The QRMon function with such an algorithm is QRMonLocalExtrema.
In brief, the algorithm steps are as follows. (For more details see [AA5].)
Find the local extrema of the fitted polynomial. (We will call them fit estimated extrema.)
Around each of the fit estimated extrema find the most extreme point in the data by a nearest neighbors search (by using Nearest).
The function QRMonLocalExtrema uses the regression quantiles previously found in the monad pipeline (and stored in the context.) The bottom regression quantile is used for finding local minima, the top regression quantile is used for finding the local maxima.
An example of finding local extrema follows.
QRMonUnit[TimeSeriesWindow[tsData, {{2015, 1, 1}, {2018, 12, 31}}]]⟹
QRMonQuantileRegression[10, {0.05, 0.95}]⟹
QRMonDateListPlot[Joined -> False, PlotTheme -> "Scientific"]⟹
QRMonLocalExtrema["NumberOfProximityPoints" -> 100]⟹
QRMonEchoValue⟹
QRMonAddToContext⟹
QRMonEchoFunctionContext[
DateListPlot[{#localMinima, #localMaxima, #data},
PlotStyle -> {PointSize[0.015], PointSize[0.015], Gray},
Joined -> False,
PlotLegends -> {"localMinima", "localMaxima", "data"},
PlotTheme -> "Scientific"] &];
Note that in the pipeline above in order to plot the data and local extrema together some additional steps are needed. The result of QRMonLocalExtrema becomes the pipeline value; that pipeline value is displayed with QRMonEchoValue, and stored in the context with QRMonAddToContext. If the pipeline value is an association — which is the case here — the monad function QRMonAddToContext joins that association with the context association. In this case this means that we will have key-value elements in the context for "localMinima" and "localMaxima". The date list plot at the end of the pipeline uses values of those context keys (together with the value for "data".)
The values from the monad context can be set, obtained, or dropped with the corresponding "setter", "dropper", and "taker" functions as summarized in a previous section.
For example:
p = QRMonUnit[distData]⟹QRMonQuantileRegressionFit[2];
p⟹QRMonTakeRegressionFunctions
(* <|0.25 -> (0.0191185 + 0.00669159 #1 + 3.05509*10^-14 #1^2 &),
0.5 -> (0.191408 + 9.4728*10^-14 #1 + 3.02272*10^-14 #1^2 &),
0.75 -> (0.563422 + 3.8079*10^-11 #1 + 7.63637*10^-14 #1^2 &)|> *)
If other values are put in the context they can be obtained through the (generic) function QRMonTakeContext, [AAp1]:
p = QRMonUnit[RandomReal[1, {2, 2}]]⟹QRMonAddToContext["data"];
(p⟹QRMonTakeContext)["data"]
(* {{0.608789, 0.741599}, {0.877074, 0.861554}} *)
Another generic function from [AAp1] is QRMonTakeValue (used many times above.)
Here is an example of the "data dropper" QRMonDropData:
p⟹QRMonDropData⟹QRMonTakeContext
(* <||> *)
(The "droppers" simply use the state monad function QRMonDropFromContext, [AAp1]. For example, QRMonDropData is equivalent to QRMonDropFromContext["data"].)
The development of QRMon was done with two types of unit tests: (i) directly specified tests, [AAp7], and (ii) tests based on randomly generated pipelines, [AA8].
The unit test package should be further extended in order to provide better coverage of the functionalities and illustrate — and postulate — pipeline behavior.
Here we run the unit tests file "MonadicQuantileRegression-Unit-Tests.wlt", [AAp7]:
AbsoluteTiming[
testObject = TestReport["~/MathematicaForPrediction/UnitTests/MonadicQuantileRegression-Unit-Tests.wlt"]
]
The natural language derived test ID’s should give a fairly good idea of the functionalities covered in [AAp3].
Values[Map[#["TestID"] &, testObject["TestResults"]]]
(* {"LoadPackage", "GenerateData", "QuantileRegression-1", \
"QuantileRegression-2", "QuantileRegression-3", \
"QuantileRegression-and-Fit-1", "Fit-and-QuantileRegression-1", \
"QuantileRegressionFit-and-Fit-1", "Fit-and-QuantileRegressionFit-1", \
"Outliers-1", "Outliers-2", "GridSequence-1", "BandsSequence-1", \
"ConditionalCDF-1", "Evaluate-1", "Evaluate-2", "Evaluate-3", \
"Simulate-1", "Simulate-2", "Simulate-3"} *)
Since the monad QRMon is a DSL it is natural to test it with a large number of randomly generated "sentences" of that DSL. For the QRMon DSL the sentences are QRMon pipelines. The package "MonadicQuantileRegressionRandomPipelinesUnitTests.m", [AAp8], has functions for generation of QRMon random pipelines and running them as verification tests. A short example follows.
Generate pipelines:
SeedRandom[234]
pipelines = MakeQRMonRandomPipelines[100];
Length[pipelines]
(* 100 *)
Here is a sample of the generated pipelines:
(*
Block[{DoubleLongRightArrow, pipelines = RandomSample[pipelines, 6]},
Clear[DoubleLongRightArrow];
pipelines = pipelines /. {_TemporalData -> "tsData", _?MatrixQ -> "distData"};
GridTableForm[Map[List@ToString[DoubleLongRightArrow @@ #, FormatType -> StandardForm] &, pipelines], TableHeadings -> {"pipeline"}]
]
AutoCollapse[] *)
Here we run the pipelines as unit tests:
AbsoluteTiming[
res = TestRunQRMonPipelines[pipelines, "Echo" -> False];
]
From the test report results we see that a dozen tests failed with messages, all of the rest passed.
rpTRObj = TestReport[res]
(The message failures, of course, have to be examined — some bugs were found in that way. Currently the actual test messages are expected.)
A list of possible, additional workflow operations and improvements follows.
It will be useful to develop a function for automatic finding of over-fitting parameters.
The time series simulation should be done by aggregation of similar time intervals.
Using the packages [AAp10, AAp11] we can generate QRMon pipelines with natural commands. The plan is to develop and document those functionalities further.
Here is an example of a pipeline constructed with natural language commands:
QRMonUnit[distData]⟹
ToQRMonPipelineFunction["show data summary"]⟹
ToQRMonPipelineFunction["calculate quantile regression for quantiles 0.2, 0.8 and with 40 knots"]⟹
ToQRMonPipelineFunction["plot"];
The implementation methodology of the QRMon monad packages [AAp3, AAp8] followed the methodology created for the ClCon monad package [AAp9, AA6]. Similarly, this document closely follows the structure and exposition of the ClCon monad document "A monad for classification workflows", [AA6].
A lot of the functionalities and signatures of QRMon were designed and programed through considerations of natural language commands specifications given to a specialized conversational agent. (As discussed in the previous section.)
[AAp1] Anton Antonov, State monad code generator Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/StateMonadCodeGenerator.m .
[AAp2] Anton Antonov, Monadic tracing Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicTracing.m .
[AAp3] Anton Antonov, Monadic Quantile Regression Mathematica package, (2018), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicQuantileRegression.m.
[AAp4] Anton Antonov, Quantile regression Mathematica package, (2014), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/QuantileRegression.m .
[AAp5] Anton Antonov, Monadic contextual classification Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicContextualClassification.m .
[AAp6] Anton Antonov, MathematicaForPrediction utilities, (2014), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MathematicaForPredictionUtilities.m .
[AAp7] Anton Antonov, Monadic Quantile Regression unit tests, (2018), MathematicaVsR at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/UnitTests/MonadicQuantileRegression-Unit-Tests.wlt .
[AAp8] Anton Antonov, Monadic Quantile Regression random pipelines Mathematica unit tests, (2018), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/UnitTests/MonadicQuantileRegressionRandomPipelinesUnitTests.m .
[AAp9] Anton Antonov, Monadic contextual classification Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicContextualClassification.m .
[AAp10] Anton Antonov, Time series workflows grammar in EBNF, (2018), ConversationalAgents at GitHub, https://github.com/antononcube/ConversationalAgents.
[AAp11] Anton Antonov, QRMon translator Mathematica package,(2018), ConversationalAgents at GitHub, https://github.com/antononcube/ConversationalAgents.
[AA1] Anton Antonov, "Monad code generation and extension", (2017), MathematicaForPrediction at GitHub, https://github.com/antononcube/MathematicaForPrediction.
[AA2] Anton Antonov, "Quantile regression through linear programming", (2013), MathematicaForPrediction at WordPress. URL: https://mathematicaforprediction.wordpress.com/2013/12/16/quantile-regression-through-linear-programming/ .
[AA3] Anton Antonov, "Quantile regression with B-splines", (2014), MathematicaForPrediction at WordPress. URL: https://mathematicaforprediction.wordpress.com/2014/01/01/quantile-regression-with-b-splines/ .
[AA4] Anton Antonov, "Estimation of conditional density distributions", (2014), MathematicaForPrediction at WordPress. URL: https://mathematicaforprediction.wordpress.com/2014/01/13/estimation-of-conditional-density-distributions/ .
[AA5] Anton Antonov, "Finding local extrema in noisy data using Quantile Regression", (2015), MathematicaForPrediction at WordPress. URL: https://mathematicaforprediction.wordpress.com/2015/09/27/finding-local-extrema-in-noisy-data-using-quantile-regression/ .
[AA6] Anton Antonov, "A monad for classification workflows", (2018), MathematicaForPrediction at WordPress. URL: https://mathematicaforprediction.wordpress.com/2018/05/15/a-monad-for-classification-workflows/ .
[Wk1] Wikipedia entry, Monad, URL: https://en.wikipedia.org/wiki/Monad_(functional_programming) .
[Wk2] Wikipedia entry, Quantile Regression, URL: https://en.wikipedia.org/wiki/Quantile_regression .
[Wk3] Wikipedia entry, Chebyshev polynomials, URL: https://en.wikipedia.org/wiki/Chebyshev_polynomials .
[CN1] Brian S. Code and Barry R. Noon, "A gentle introduction to quantile regression for ecologists", (2003). Frontiers in Ecology and the Environment. 1 (8): 412[Dash]420. doi:10.2307/3868138. URL: http://www.econ.uiuc.edu/~roger/research/rq/QReco.pdf .
[PS1] Patrick Scheibe, Mathematica (Wolfram Language) support for IntelliJ IDEA, (2013-2018), Mathematica-IntelliJ-Plugin at GitHub. URL: https://github.com/halirutan/Mathematica-IntelliJ-Plugin .
[RK1] Roger Koenker, Quantile Regression, Cambridge University Press, 2005.
]]>This MathematicaVsR at GitHub project is for the comparison of the Deep Learning functionalities in R/RStudio and Mathematica/Wolfram Language (WL).
The project is aimed to mirror and aid the talk "Deep Learning series (session 2)" of the meetup Orlando Machine Learning and Data Science.
The focus of the talk is R and Keras, so the project structure is strongly influenced by the content of the book Deep learning with R, [1], and the corresponding Rmd notebooks, [2].
Some of Mathematica’s notebooks repeat the material in [2]. Some are original versions.
WL’s Neural Nets framework and abilities are fairly well described in the reference page "Neural Networks in the Wolfram Language overview", [4], and the webinar talks [5].
The corresponding documentation pages [3] (R) and [6] (WL) can be used for a very fruitful comparison of features and abilities.
Remark: With "deep learning with R" here we mean "Keras with R".
Remark: An alternative to R/Keras and Mathematica/MXNet is the library H2O (that has interfaces to Java, Python, R, Scala.) See project’s directory R.H2O for examples.
The slideshow is part of Sebastian Bodenstein’s presentation at Wolfram U.
(It was separated/extracted for clarity and convenience during the meetup presentation.)
Recording of the presentation at YouTube:
ORLMLDS Deep learning series (2): "Using Keras with R (… and MXNet with WL)".
The info-chart "Classification of handwritten digits by matrix factorization" (used in the presentation.)
Deep learning can be used for both supervised and unsupervised learning. In this project we concentrate on supervised learning.
The following diagram outlines the general, simple classification workflow we have in mind.
Here is a corresponding classification monadic pipeline in Mathematica:
R-Keras uses monadic pipelines through the library magrittr
. For example:
model <- keras_model_sequential()
model %>%
layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>%
layer_dropout(rate = 0.4) %>%
layer_dense(units = 128, activation = 'relu') %>%
layer_dropout(rate = 0.3) %>%
layer_dense(units = 10, activation = 'softmax')
The corresponding Mathematica command is:
model =
NetChain[{
LinearLayer[256, "Input" -> 784],
ElementwiseLayer[Ramp],
DropoutLayer[0.4],
LinearLayer[128],
ElementwiseLayer[Ramp],
DropoutLayer[0.3],
LinearLayer[10]
}]
The neural networks framework comes with Mathematica. (No additional installation required.)
R
Pretty straightforward using the directions in [3]. (A short list.)
Some additional Python installation is required.
Mathematica: Simple-neural-network-classifier-over-MNIST-data.pdf
R-Keras: Keras-with-R-talk-introduction.nb.html, Keras-with-R-talk-introduction.Rmd.
TBD…
TBD…
R-Keras: 3.6-predicting-house-prices.nb.html, 3.6-predicting-house-prices.Rmd.
(Those are links to notebooks in [2].)
The Mathematica encoders (for neural networks and generally for machine learning tasks) are very well designed and with a very advanced development.
The encoders in R-Keras are fairly useful but not was advanced as those in Mathematica.
[TBD: Encoder correspondence…]
R-Keras: Training-Neural-Networks-with-Regularization.nb.html, Training-Neural-Networks-with-Regularization.Rmd.
Mathematica: Wolfram Research repository of neural networks; can import externally trained networks in MXNet format.
R-Keras: has commands loading for pre-trained models, [3].
Mathematica: "Neural Networks guide".
R-Keras: "Keras reference", cheatsheet.
[1] F. Chollet, J. J. Allaire, Deep learning with R, (2018).
[2] J. J. Allaire, Deep Learing with R notebooks, (2018), GitHub.
[3] RStudio, Keras reference.
[4] Wolfram Research, "Neural Networks in the Wolfram Language overview".
[5] Wolfram Research, "Machine Learning Webinar Series".
[6] Wolfram Research, "Neural Networks guide".
]]>In this document we describe the design and implementation of a (software programming) monad for classification workflows specification and execution. The design and implementation are done with Mathematica / Wolfram Language (WL).
The goal of the monad design is to make the specification of classification workflows (relatively) easy, straightforward, by following a certain main scenario and specifying variations over that scenario.
The monad is named ClCon
and it is based on the State monad package "StateMonadCodeGenerator.m", [AAp1, AA1], the classifier ensembles package "ClassifierEnsembles.m", [AAp4, AA2], and the package for Receiver Operating Characteristic (ROC) functions calculation and plotting "ROCFunctions.m", [AAp5, AA2, Wk2].
The data for this document is read from WL’s repository using the package "GetMachineLearningDataset.m", [AAp10].
The monadic programming design is used as a Software Design Pattern. The ClCon
monad can be also seen as a Domain Specific Language (DSL) for the specification and programming of machine learning classification workflows.
Here is an example of using the ClCon
monad over the Titanic data:
The table above is produced with the package "MonadicTracing.m", [AAp2, AA1], and some of the explanations below also utilize that package.
As it was mentioned above the monad ClCon
can be seen as a DSL. Because of this the monad pipelines made with ClCon
are sometimes called "specifications".
The document has the following structure.
The sections "Design consideration" and "Monad design" provide motivation and design decisions rationale.
The sections "ClCon overview" and "Monad elements" provide technical description of the ClCon
monad needed to utilize it.
(Using a fair amount of examples.)
The section "Example use cases" gives several more elaborated examples of ClCon
that have "real life" flavor.
(But still didactic and concise enough.)
The section "Unit test" describes the tests used in the development of the ClCon
monad.
(The random pipelines unit tests are especially interesting.)
The section "Future plans" outlines future directions of development.
(The most interesting and important one is the "conversational agent" direction.)
The section "Implementation notes" has (i) a diagram outlining the ClCon
development process, and (ii) a list of observations and morals.
(Some fairly obvious, but deemed fairly significant and hence stated explicitly.)
Remark: One can read only the sections "Introduction", "Design consideration", "Monad design", and "ClCon overview". That set of sections provide a fairly good, programming language agnostic exposition of the substance and novel ideas of this document.
The following commands load the packages [AAp1–AAp10, AAp12]:
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicContextualClassification.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicTracing.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaVsR/master/Projects/ProgressiveMachineLearning/Mathematica/GetMachineLearningDataset.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/UnitTests/MonadicContextualClassificationRandomPipelinesUnitTests.m"]
(*
Importing from GitHub: MathematicaForPredictionUtilities.m
Importing from GitHub: MosaicPlot.m
Importing from GitHub: CrossTabulate.m
Importing from GitHub: StateMonadCodeGenerator.m
Importing from GitHub: ClassifierEnsembles.m
Importing from GitHub: ROCFunctions.m
Importing from GitHub: VariableImportanceByClassifiers.m
Importing from GitHub: SSparseMatrix.m
Importing from GitHub: OutlierIdentifiers.m
*)
In this section we load data that is used in the rest of the document. The "quick" data is created in order to specify quick, illustrative computations.
Remark: In all datasets the classification labels are in the last column.
The summarization of the data is done through ClCon
, which in turn uses the function RecordsSummary
from the package "MathematicaForPredictionUtilities.m", [AAp7].
The following commands produce datasets using the package [AAp10] (that utilizes ExampleData
):
dsTitanic = GetMachineLearningDataset["Titanic"];
dsMushroom = GetMachineLearningDataset["Mushroom"];
dsWineQuality = GetMachineLearningDataset["WineQuality"];
Here is are the dimensions of the datasets:
Dataset[Dataset[Map[Prepend[Dimensions[ToExpression[#]], #] &, {"dsTitanic", "dsMushroom", "dsWineQuality"}]][All, AssociationThread[{"name", "rows", "columns"}, #] &]]
Here is the summary of dsTitanic
:
ClConUnit[dsTitanic]⟹ClConSummarizeData["MaxTallies" -> 12];
Here is the summary of dsMushroom
in long form:
ClConUnit[dsMushroom]⟹ClConSummarizeDataLongForm["MaxTallies" -> 12];
Here is the summary of dsWineQuality
in long form:
ClConUnit[dsWineQuality]⟹ClConSummarizeDataLongForm["MaxTallies" -> 12];
In this subsection we make up some data that is used for illustrative purposes.
SeedRandom[212]
dsData = RandomInteger[{0, 1000}, {100}];
dsData = Dataset[
Transpose[{dsData, Mod[dsData, 3], Last@*IntegerDigits /@ dsData, ToString[Mod[#, 3]] & /@ dsData}]];
dsData = Dataset[dsData[All, AssociationThread[{"number", "feature1", "feature2", "label"}, #] &]];
Dimensions[dsData]
(* {100, 4} *)
Here is a sample of the data:
RandomSample[dsData, 6]
Here is a summary of the data:
ClConUnit[dsData]⟹ClConSummarizeData;
Here we convert the data into a list of record-label rules (and show the summary):
mlrData = ClConToNormalClassifierData[dsData];
ClConUnit[mlrData]⟹ClConSummarizeData;
Finally, we make the array version of the dataset:
arrData = Normal[dsData[All, Values]];
The steps of the main classification workflow addressed in this document follow.
Optionally, transform the data.
Split data into training and test parts.
Test the classifier over the test data.
The following diagram shows the steps.
Very often the workflow above is too simple in real situations. Often when making "real world" classifiers we have to experiment with different transformations, different classifier algorithms, and parameters for both transformations and classifiers. Examine the following mind-map that outlines the activities in making competition classifiers.
In view of the mind-map above we can come up with the following flow-chart that is an elaboration on the main, simple workflow flow-chart.
In order to address:
workflows elements variability, and
workflows iterative changes and refining,
it is beneficial to have a DSL for classification workflows. We choose to make such a DSL through a functional programming monad, [Wk1, AA1].
Here is a quote from [Wk1] that fairly well describes why we choose to make a classification workflow monad and hints on the desired properties of such a monad.
[…] The monad represents computations with a sequential structure: a monad defines what it means to chain operations together. This enables the programmer to build pipelines that process data in a series of steps (i.e. a series of actions applied to the data), in which each action is decorated with the additional processing rules provided by the monad. […]
Monads allow a programming style where programs are written by putting together highly composable parts, combining in flexible ways the possible actions that can work on a particular type of data. […]
Remark: Note that quote from [Wk1] refers to chained monadic operations as "pipelines". We use the terms "monad pipeline" and "pipeline" below.
The monad we consider is designed to speed-up the programming of classification workflows outlined in the previous section. The monad is named ClCon for "Classification with Context".
We want to be able to construct monad pipelines of the general form:
ClCon is based on the State monad, [Wk1, AA1], so the monad pipeline form (1) has the following more specific form:
This means that some monad operations will not just change the pipeline value but they will also change the pipeline context.
In the monad pipelines of ClCon
we store different objects in the contexts for at least one of the following two reasons.
The object is hard to compute.
Such objects are training data, ROC data, and classifiers.
Let us list the desired properties of the monad.
The monad works with different data types: Dataset, lists of machine learning rules, full arrays.
The pipeline values can be of different types. Most monad functions modify the pipeline value; some modify the context; some just echo results.
The monad works with single classifier objects and with classifier ensembles.
We can easily obtain the pipeline value, context, and different context objects for manipulation outside of the monad.
We can calculate classification measures using a specified ROC parameter and a class label.
We can easily plot different combinations of ROC functions.
The ClCon
components and their interaction are given in the following diagram. (The components correspond to the main workflow given in the previous section.)
In the diagram above the operations are given in rectangles. Data objects are given in round corner rectangles and classifier objects are given in round corner squares.
The main ClCon operations implicitly put in the context or utilize from the context the following objects:
test data,
validation data,
classifier (a classifier function or an association of classifier functions),
ROC data,
variable names list.
Note the that the monadic set of types of ClCon
pipeline values is fairly heterogenous and certain awareness of "the current pipeline value" is assumed when composing ClCon pipelines.
Obviously, we can put in the context any object through the generic operations of the State monad of the package "StateMonadGenerator.m", [AAp1].
When using a monad we lift certain data into the "monad space", using monad’s operations we navigate computations in that space, and at some point we take results from it.
With the approach taken in this document the "lifting" into the ClCon
monad is done with the function ClConUnit
. Results from the monad can be obtained with the functions ClConTakeValue
, ClConContext
, or with the other ClCon
functions with the prefix "ClConTake" (see below.)
Here is a corresponding diagram of a generic computation with the ClCon
monad:
Remark: It is a good idea to compare the diagram with formulas (1) and (2).
Let us examine a concrete ClCon
pipeline that corresponds to the diagram above. In the following table each pipeline operation is combined together with a short explanation and the context keys after its execution.
Here is the output of the pipeline:
In the specified pipeline computation the last column of the dataset is assumed to be the one with the class labels.
The ClCon functions are separated into four groups:
setters,
takers,
State Monad generic functions.
An overview of the those functions is given in the tables in next two sub-sections. The next section, "Monad elements", gives details and examples for the usage of the ClCon
operations.
The following table gives an overview the interaction of the ClCon monad functions with the pipeline value and context.
Several functions that use ROC data have two rows in the table because they calculate the needed ROC data if it is not available in the monad context.
Here are the ClCon
State Monad functions (generated using the prefix "ClCon", [AAp1, AA1]):
In this section we show that ClCon
has all of the properties listed in the previous section.
The monad head is ClCon
. Anything wrapped in ClCon
can serve as monad’s pipeline value. It is better though to use the constructor ClConUnit
. (Which adheres to the definition in [Wk1].)
ClCon[{{1, "a"}, {2, "b"}}, <||>]⟹ClConSummarizeData;
The function lifting the data into the monad ClCon
is ClConUnit
.
The lifting to the monad marks the beginning of the monadic pipeline. It can be done with data or without data. Examples follow.
ClConUnit[dsData]⟹ClConSummarizeData;
ClConUnit[]⟹ClConSetTrainingData[dsData]⟹ClConSummarizeData;
(See the sub-section "Setters and takers" for more details of setting and taking values in ClCon
contexts.)
Currently the monad can deal with data in the following forms:
matrices,
lists of example->label rules.
The ClCon monad also has the non-monadic function ClConToNormalClassifierData
which can be used to convert datasets and matrices to lists of example->label rules. Here is an example:
Short[ClConToNormalClassifierData[dsData], 3]
(*
{{639, 0, 9} -> "0", {121, 1, 1} -> "1", {309, 0, 9} -> "0", {648, 0, 8} -> "0", {995, 2, 5} -> "2", {127, 1, 7} -> "1", {908, 2, 8} -> "2", {564, 0, 4} -> "0", {380, 2, 0} -> "2", {860, 2, 0} -> "2",
<<80>>,
{464, 2, 4} -> "2", {449, 2, 9} -> "2", {522, 0, 2} -> "0", {288, 0, 8} -> "0", {51, 0, 1} -> "0", {108, 0, 8} -> "0", {76, 1, 6} -> "1", {706, 1, 6} -> "1", {765, 0, 5} -> "0", {195, 0, 5} -> "0"}
*)
When the data lifted to the monad is a dataset or a matrix it is assumed that the last column has the class labels. WL makes it easy to rearrange columns in such a way the any column of dataset or a matrix to be the last.
The splitting is made with ClConSplitData
, which takes up to two arguments and options. The first argument specifies the fraction of training data. The second argument — if given — specifies the fraction of the validation part of the training data. If the value of option Method
is "LabelsProportional", then the splitting is done in correspondence of the class labels tallies. ("LabelsProportional" is the default value.) Data splitting demonstration examples follow.
Here are the dimensions of the dataset dsData:
Dimensions[dsData]
(* {100, 4} *)
Here we split the data into 70% for training and 30% for testing and then we verify that the corresponding number of rows add to the number of rows of dsData:
val = ClConUnit[dsData]⟹ClConSplitData[0.7]⟹ClConTakeValue;
Map[Dimensions, val]
Total[First /@ %]
(*
<|"trainingData" -> {69, 4}, "testData" -> {31, 4}|>
100
*)
Note that if Method is not "LabelsProportional" we get slightly different results.
val = ClConUnit[dsData]⟹ClConSplitData[0.7, Method -> "Random"]⟹ClConTakeValue;
Map[Dimensions, val]
Total[First /@ %]
(*
<|"trainingData" -> {70, 4}, "testData" -> {30, 4}|>
100
*)
In the following code we split the data into 70% for training and 30% for testing, then the training data is further split into 90% for training and 10% for classifier training validation; then we verify that the number of rows add up.
val = ClConUnit[dsData]⟹ClConSplitData[0.7, 0.1]⟹ClConTakeValue;
Map[Dimensions, val]
Total[First /@ %]
(*
<|"trainingData" -> {61, 4}, "testData" -> {31, 4}, "validationData" -> {8, 4}|>
100
*)
The monad ClCon supports both single classifiers obtained with Classify and classifier ensembles obtained with Classify and managed with the package "ClassifierEnsembles.m", [AAp4].
With the following pipeline we take the Titanic data, split it into 75/25 % parts, train a Logistic Regression classifier, and finally take that classifier from the monad.
cf =
ClConUnit[dsTitanic]⟹
ClConSplitData[0.75]⟹
ClConMakeClassifier["LogisticRegression"]⟹
ClConTakeClassifier;
Here is information about the obtained classifier:
ClassifierInformation[cf, "TrainingTime"]
(* Quantity[3.84008, "Seconds"] *)
If we want to pass parameters to the classifier training we can use the Method
option. Here we train a Random Forest classifier with 400 trees:
cf =
ClConUnit[dsTitanic]⟹
ClConSplitData[0.75]⟹
ClConMakeClassifier[Method -> {"RandomForest", "TreeNumber" -> 400}]⟹
ClConTakeClassifier;
ClassifierInformation[cf, "TreeNumber"]
(* 400 *)
With the following pipeline we take the Titanic data, split it into 75/25 % parts, train a classifier ensemble of three Logistic Regression classifiers and two Nearest Neighbors classifiers using random sampling of 90% of the training data, and finally take that classifier ensemble from the monad.
ensemble =
ClConUnit[dsTitanic]⟹
ClConSplitData[0.75]⟹
ClConMakeClassifier[{{"LogisticRegression", 0.9, 3}, {"NearestNeighbors", 0.9, 2}}]⟹
ClConTakeClassifier;
The classifier ensemble is simply an association with keys that are automatically assigned names and corresponding values that are classifiers.
ensemble
Here are the training times of the classifiers in the obtained ensemble:
ClassifierInformation[#, "TrainingTime"] & /@ ensemble
(*
<|"LogisticRegression[1,0.9]" -> Quantity[3.47836, "Seconds"],
"LogisticRegression[2,0.9]" -> Quantity[3.47681, "Seconds"],
"LogisticRegression[3,0.9]" -> Quantity[3.4808, "Seconds"],
"NearestNeighbors[1,0.9]" -> Quantity[1.82454, "Seconds"],
"NearestNeighbors[2,0.9]" -> Quantity[1.83804, "Seconds"]|>
*)
A more precise specification can be given using associations. The specification
<|"method" -> "LogisticRegression", "sampleFraction" -> 0.9, "numberOfClassifiers" -> 3, "samplingFunction" -> RandomChoice|>
says "make three Logistic Regression classifiers, for each taking 90% of the training data using the function RandomChoice
."
Here is a pipeline specification equivalent to the pipeline specification above:
ensemble2 =
ClConUnit[dsTitanic]⟹
ClConSplitData[0.75]⟹
ClConMakeClassifier[{
<|"method" -> "LogisticRegression",
"sampleFraction" -> 0.9,
"numberOfClassifiers" -> 3,
"samplingFunction" -> RandomSample|>,
<|"method" -> "NearestNeighbors",
"sampleFraction" -> 0.9,
"numberOfClassifiers" -> 2,
"samplingFunction" -> RandomSample|>}]⟹
ClConTakeClassifier;
ensemble2
Classifier testing is done with the testing data in the context.
Here is a pipeline that takes the Titanic data, splits it, and trains a classifier:
p =
ClConUnit[dsTitanic]⟹
ClConSplitData[0.75]⟹
ClConMakeClassifier["DecisionTree"];
Here is how we compute selected classifier measures:
p⟹
ClConClassifierMeasurements[{"Accuracy", "Precision", "Recall", "FalsePositiveRate"}]⟹
ClConTakeValue
(*
<|"Accuracy" -> 0.792683,
"Precision" -> <|"died" -> 0.802691, "survived" -> 0.771429|>,
"Recall" -> <|"died" -> 0.881773, "survived" -> 0.648|>,
"FalsePositiveRate" -> <|"died" -> 0.352, "survived" -> 0.118227|>|>
*)
(The measures are listed in the function page of ClassifierMeasurements.)
Here we show the confusion matrix plot:
p⟹ClConClassifierMeasurements["ConfusionMatrixPlot"]⟹ClConEchoValue;
Here is how we plot ROC curves by specifying the ROC parameter range and the image size:
p⟹ClConROCPlot["FPR", "TPR", "ROCRange" -> Range[0, 1, 0.1], ImageSize -> 200];
Remark: ClCon uses the package ROCFunctions.m, [AAp5], which implements all functions defined in [Wk2].
Here we plot ROC functions values (y-axis) over the ROC parameter (x-axis):
p⟹ClConROCListLinePlot[{"ACC", "TPR", "FPR", "SPC"}];
Note of the "ClConROC*Plot" functions automatically echo the plots. The plots are also made to be the pipeline value. Using the option specification "Echo"->False the automatic echoing of plots can be suppressed. With the option "ClassLabels" we can focus on specific class labels.
p⟹
ClConROCListLinePlot[{"ACC", "TPR", "FPR", "SPC"}, "Echo" -> False, "ClassLabels" -> "survived", ImageSize -> Medium]⟹
ClConEchoValue;
Using the pipeline constructed above let us find the most decisive variables using systematic random shuffling (as explained in [AA3]):
p⟹
ClConAccuracyByVariableShuffling⟹
ClConTakeValue
(*
<|None -> 0.792683, "id" -> 0.664634, "passengerClass" -> 0.75, "passengerAge" -> 0.777439, "passengerSex" -> 0.612805|>
*)
We deduce that "passengerSex" is the most decisive variable because its corresponding classification success rate is the smallest. (See [AA3] for more details.)
Using the option "ClassLabels" we can focus on specific class labels:
p⟹ClConAccuracyByVariableShuffling["ClassLabels" -> "survived"]⟹ClConTakeValue
(*
<|None -> {0.771429}, "id" -> {0.595506}, "passengerClass" -> {0.731959}, "passengerAge" -> {0.71028}, "passengerSex" -> {0.414414}|>
*)
The values from the monad context can be set or obtained with the corresponding "setters" and "takers" functions as summarized in previous section.
For example:
p⟹ClConTakeClassifier
(* ClassifierFunction[__] *)
Short[Normal[p⟹ClConTakeTrainingData]]
(*
{<|"id" -> 858, "passengerClass" -> "3rd", "passengerAge" -> 30, "passengerSex" -> "male", "passengerSurvival" -> "survived"|>, <<979>> }
*)
Short[Normal[p⟹ClConTakeTestData]]
(* {<|"id" -> 285, "passengerClass" -> "1st", "passengerAge" -> 60, "passengerSex" -> "female", "passengerSurvival" -> "survived"|> , <<327>> }
*)
p⟹ClConTakeVariableNames
(* {"id", "passengerClass", "passengerAge", "passengerSex", "passengerSurvival"} *)
If other values are put in the context they can be obtained through the (generic) function ClConTakeContext
, [AAp1]:
p = ClConUnit[RandomReal[1, {2, 2}]]⟹ClConAddToContext["data"];
(p⟹ClConTakeContext)["data"]
(* {{0.815836, 0.191562}, {0.396868, 0.284587}} *)
Another generic function from [AAp1] is ClConTakeValue
(used many times above.)
Here we show an example of using ClCon with the reasonably large dataset of images MNIST, [YL1].
mnistData = ExampleData[{"MachineLearning", "MNIST"}, "Data"];
SeedRandom[3423]
p =
ClConUnit[RandomSample[mnistData, 20000]]⟹
ClConSplitData[0.7]⟹
ClConSummarizeData⟹
ClConMakeClassifier["NearestNeighbors"]⟹
ClConClassifierMeasurements[{"Accuracy", "ConfusionMatrixPlot"}]⟹
ClConEchoValue;
Here we plot the ROC curve for a specified digit:
p⟹ClConROCPlot["ClassLabels" -> 5];
In this sub-section we show how the computations in a ClCon pipeline can be stopped or continued based on a certain condition.
The pipeline below makes a simple classifier ("LogisticRegression") for the WineQuality data, and if the recall for the important label ("high") is not large enough makes a more complicated classifier ("RandomForest"). The pipeline marks intermediate steps by echoing outcomes and messages.
SeedRandom[267]
res =
ClConUnit[dsWineQuality[All, Join[#, <|"wineQuality" -> If[#wineQuality >= 7, "high", "low"]|>] &]]⟹
ClConSplitData[0.75, 0.2]⟹
ClConSummarizeData(* summarize the data *)⟹
ClConMakeClassifier[Method -> "LogisticRegression"](* training a simple classifier *)⟹
ClConROCPlot["FPR", "TPR", "ROCPointCallouts" -> False]⟹
ClConClassifierMeasurements[{"Accuracy", "Precision", "Recall", "FalsePositiveRate"}]⟹
ClConEchoValue⟹
ClConIfElse[#["Recall", "high"] > 0.70 & (* criteria based on the recall for "high" *),
ClConEcho["Good recall for \"high\"!", "Success:"],
ClConUnit[##]⟹
ClConEcho[Style["Recall for \"high\" not good enough... making a large random forest.", Darker[Red]], "Info:"]⟹
ClConMakeClassifier[Method -> {"RandomForest", "TreeNumber" -> 400}](* training a complicated classifier *)⟹
ClConROCPlot["FPR", "TPR", "ROCPointCallouts" -> False]⟹
ClConClassifierMeasurements[{"Accuracy", "Precision", "Recall", "FalsePositiveRate"}]⟹
ClConEchoValue &];
We can see that the recall with the more complicated is classifier is higher. Also the ROC plots of the second classifier are visibly closer to the ideal one. Still, the recall is not good enough, we have to find a threshold that is better that the default one. (See the next sub-section.)
(In this sub-section we use the monad from the previous sub-section.)
Here we compute classification measures using the threshold 0.3 for the important class label ("high"):
res⟹
ClConClassifierMeasurementsByThreshold[{"Accuracy", "Precision", "Recall", "FalsePositiveRate"}, "high" -> 0.3]⟹
ClConTakeValue
(* <|"Accuracy" -> 0.782857, "Precision" -> <|"high" -> 0.498871, "low" -> 0.943734|>,
"Recall" -> <|"high" -> 0.833962, "low" -> 0.76875|>,
"FalsePositiveRate" -> <|"high" -> 0.23125, "low" -> 0.166038|>|> *)
We can see that the recall for "high" is fairly large and the rest of the measures have satisfactory values. (The accuracy did not drop that much, and the false positive rate is not that large.)
Here we compute suggestions for the best thresholds:
res (* start with a previous monad *)⟹
ClConROCPlot[ImageSize -> 300] (* make ROC plots *)⟹
ClConSuggestROCThresholds[3] (* find the best 3 thresholds per class label *)⟹
ClConEchoValue (* echo the result *);
The suggestions are the ROC points that closest to the point {0, 1} (which corresponds to the ideal classifier.)
Here is a way to use threshold suggestions within the monad pipeline:
res⟹
ClConSuggestROCThresholds⟹
ClConEchoValue⟹
(ClConUnit[##]⟹
ClConClassifierMeasurementsByThreshold[{"Accuracy", "Precision", "Recall"}, "high" -> First[#1["high"]]] &)⟹
ClConEchoValue;
(*
value: <|high->{0.35},low->{0.65}|>
value: <|Accuracy->0.825306,Precision-><|high->0.571831,low->0.928736|>,Recall-><|high->0.766038,low->0.841667|>|>
*)
The development of ClCon
was done with two types of unit tests: (1) directly specified tests, [AAp11], and (2) tests based on randomly generated pipelines, [AAp12].
Both unit test packages should be further extended in order to provide better coverage of the functionalities and illustrate — and postulate — pipeline behavior.
Here we run the unit tests file "MonadicContextualClassification-Unit-Tests.wlt", [AAp11]:
AbsoluteTiming[
testObject = TestReport["~/MathematicaForPrediction/UnitTests/MonadicContextualClassification-Unit-Tests.wlt"]
]
The natural language derived test ID’s should give a fairly good idea of the functionalities covered in [AAp11].
Values[Map[#["TestID"] &, testObject["TestResults"]]]
(* {"LoadPackage", "EvenOddDataset", "EvenOddDataMLRules", \
"DataToContext-no-[]", "DataToContext-with-[]", \
"ClassifierMaking-with-Dataset-1", "ClassifierMaking-with-MLRules-1", \
"AccuracyByVariableShuffling-1", "ROCData-1", \
"ClassifierEnsemble-different-methods-1", \
"ClassifierEnsemble-different-methods-2-cont", \
"ClassifierEnsemble-different-methods-3-cont", \
"ClassifierEnsemble-one-method-1", "ClassifierEnsemble-one-method-2", \
"ClassifierEnsemble-one-method-3-cont", \
"ClassifierEnsemble-one-method-4-cont", "AssignVariableNames-1", \
"AssignVariableNames-2", "AssignVariableNames-3", "SplitData-1", \
"Set-and-take-training-data", "Set-and-take-test-data", \
"Set-and-take-validation-data", "Partial-data-summaries-1", \
"Assign-variable-names-1", "Split-data-100-pct", \
"MakeClassifier-with-empty-unit-1", \
"No-rocData-after-second-MakeClassifier-1"} *)
Since the monad ClCon
is a DSL it is natural to test it with a large number of randomly generated "sentences" of that DSL. For the ClCon
DSL the sentences are ClCon
pipelines. The package "MonadicContextualClassificationRandomPipelinesUnitTests.m", [AAp12], has functions for generation of ClCon random pipelines and running them as verification tests. A short example follows.
Generate pipelines:
SeedRandom[234]
pipelines = MakeClConRandomPipelines[300];
Length[pipelines]
(* 300 *)
Here is a sample of the generated pipelines:
Block[{DoubleLongRightArrow, pipelines = RandomSample[pipelines, 6]},
Clear[DoubleLongRightArrow];
pipelines = pipelines /. {_Dataset -> "ds", _?DataRulesForClassifyQ -> "mlrData"};
GridTableForm[
Map[List@ToString[DoubleLongRightArrow @@ #, FormatType -> StandardForm] &, pipelines],
TableHeadings -> {"pipeline"}]
]
AutoCollapse[]
Here we run the pipelines as unit tests:
AbsoluteTiming[
res = TestRunClConPipelines[pipelines, "Echo" -> True];
]
(* {350.083, Null} *)
From the test report results we see that a dozen tests failed with messages, all of the rest passed.
rpTRObj = TestReport[res]
(The message failures, of course, have to be examined — some bugs were found in that way. Currently the actual test messages are expected.)
Better outliers finding and manipulation incorporation in ClCon
. Currently only outlier finding is surfaced in [AAp3]. (The package internally has other related functions.)
ClConUnit[dsTitanic[Select[#passengerSex == "female" &]]]⟹
ClConOutlierPosition⟹
ClConTakeValue
(* {4, 17, 21, 22, 25, 29, 38, 39, 41, 59} *)
Support of dimension reduction application — quick construction of pipelines that allow the applying different dimension reduction methods.
Currently with ClCon
dimension reduction is applied only to data the non-label parts of which can be easily converted into numerical matrices.
ClConUnit[dsWineQuality]⟹
ClConSplitData[0.7]⟹
ClConReduceDimension[2, "Echo" -> True]⟹
ClConRetrieveFromContext["svdRes"]⟹
ClConEchoFunctionValue["SVD dimensions:", Dimensions /@ # &]⟹
ClConSummarizeData;
Using the packages [AAp13, AAp15] we can generate ClCon
pipelines with natural commands. The plan is to develop and document those functionalities further.
The ClCon package, MonadicContextualClassification.m, [AAp3], is based on the packages [AAp1, AAp4-AAp9]. It was developed using Mathematica and the Mathematica plug-in for IntelliJ IDEA, by Patrick Scheibe , [PS1]. The following diagram shows the development workflow.
Some observations and morals follow.
"Eat your own dog food", or in this case: "use ClCon functionalities to implement ClCon functionalities."
Again, in retrospect that is obvious. Also probably should be seen as a consequence of practicing a certain code refactoring discipline.
The reason to list that moral is that often it is somewhat "easier" to implement functionalities thinking locally, ad-hoc, forgetting or not reviewing other, already implemented functions.
In order come be better design and find inconsistencies: write many pipelines and discuss with co-workers.
[AAp1] Anton Antonov, State monad code generator Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/StateMonadCodeGenerator.m .
[AAp2] Anton Antonov, Monadic tracing Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicTracing.m .
[AAp3] Anton Antonov, Monadic contextual classification Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicContextualClassification.m .
[AAp4] Anton Antonov, Classifier ensembles functions Mathematica package, (2016), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/ClassifierEnsembles.m .
[AAp5] Anton Antonov, Receiver operating characteristic functions Mathematica package, (2016), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/ROCFunctions.m .
[AAp6] Anton Antonov, Variable importance determination by classifiers implementation in Mathematica,(2015), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/VariableImportanceByClassifiers.m .
[AAp7] Anton Antonov, MathematicaForPrediction utilities, (2014), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MathematicaForPredictionUtilities.m .
[AAp8] Anton Antonov, Cross tabulation implementation in Mathematica, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/CrossTabulate.m .
[AAp9] Anton Antonov, SSparseMatrix Mathematica package, (2018), MathematicaForPrediction at GitHub.
[AAp10] Anton Antonov, Obtain and transform Mathematica machine learning data-sets, (2018), MathematicaVsR at GitHub.
[AAp11] Anton Antonov, Monadic contextual classification Mathematica unit tests, (2018), MathematicaVsR at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/UnitTests/MonadicContextualClassification-Unit-Tests.wlt .
[AAp12] Anton Antonov, Monadic contextual classification random pipelines Mathematica unit tests, (2018), MathematicaVsR at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/UnitTests/MonadicContextualClassificationRandomPipelinesUnitTests.m .
[AAp13] Anton Antonov, Classifier workflows grammar in EBNF, (2018), ConversationalAgents at GitHub, https://github.com/antononcube/ConversationalAgents.
[AAp14] Anton Antonov, Classifier workflows grammar Mathematica unit tests, (2018), ConversationalAgents at GitHub, https://github.com/antononcube/ConversationalAgents.
[AAp15] Anton Antonov, ClCon translator Mathematica package, (2018), ConversationalAgents at GitHub, https://github.com/antononcube/ConversationalAgents.
[AA1] Anton Antonov, Monad code generation and extension, (2017), MathematicaForPrediction at GitHub, https://github.com/antononcube/MathematicaForPrediction.
[AA2] Anton Antonov, "ROC for classifier ensembles, bootstrapping, damaging, and interpolation", (2016), MathematicaForPrediction at WordPress. URL: https://mathematicaforprediction.wordpress.com/2016/10/15/roc-for-classifier-ensembles-bootstrapping-damaging-and-interpolation/ .
[AA3] Anton Antonov, "Importance of variables investigation guide", (2016), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MarkdownDocuments/Importance-of-variables-investigation-guide.md .
[Wk1] Wikipedia entry, Monad, URL: https://en.wikipedia.org/wiki/Monad_(functional_programming) .
[Wk2] Wikipedia entry, Receiver operating characteristic, URL: https://en.wikipedia.org/wiki/Receiver_operating_characteristic .
[YL1] Yann LeCun et al., MNIST database site. URL: http://yann.lecun.com/exdb/mnist/ .
[PS1] Patrick Scheibe, Mathematica (Wolfram Language) support for IntelliJ IDEA, (2013-2018), Mathematica-IntelliJ-Plugin at GitHub. URL: https://github.com/halirutan/Mathematica-IntelliJ-Plugin .
]]>In this MathematicaVsR at GitHub project we show how to do Progressive machine learning using two types of classifiers based on:
Progressive learning is a type of Online machine learning. For more details see [Wk1]. The Progressive learning problem is defined as follows.
Problem definition:
In view of classification, it is assumed that at a given time not all class labels are presented in the data already obtained.
Let us call this a data stream.
Make a machine learning algorithm that updates its model continuously or sequentially in time over a given data stream.
In comparison, the typical (classical) machine learning algorithms assume that representative training data is available and after training that data is no longer needed to make predictions. Progressive machine learning has more general assumptions about the data and its problem formulation is closer to how humans learn to classify objects.
Below we are shown the applications of two types of classifiers as PLA’s. One is based on Tries with Frequencies (TF), [AAp2, AAp3, AA1], the other on an Item-item Recommender (IIR) framework [AAp4, AA2].
Remark: Note that both TF and IIR come from tackling Unsupervised machine learning tasks, but here they are applied in the context of Supervised machine learning.
The Mathematica and R notebooks follow the steps in the following flow chart.
For detailed explanations see any of the notebooks.
(For details see Progressive-machine-learning-examples.md.)
Here is an example run with Tries with Frequencies, [AAp2, AA1]:
Here are the obtained ROC curves:
We can see that with the Progressive learning process does improve its success rates in time.
Here is an example run with an Item-item recommender system, [AAp4, AA2]:
Here are the obtained ROC curves:
[AAp1] Anton Antonov, Obtain and transform Mathematica machine learning data-sets, GetMachineLearningDataset.m, (2018), MathematicaVsR at GitHub.
[AAp2] Anton Antonov, Java tries with frequencies Mathematica package, JavaTriesWithFrequencies.m, (2017), MathematicaForPrediction at GitHub.
[AAp3] Anton Antonov, Tries with frequencies R package, TriesWithFrequencies.R, (2014), MathematicaForPrediction at GitHub.
[AAp4] Anton Antonov, Sparse matrix recommender framework in Mathematica, SparseMatrixRecommenderFramework.m, (2014), MathematicaForPrediction at GitHub.
[Wk1] Wikipedia entry, Online machine learning.
[AA1] Anton Antonov, "Tries with frequencies in Java", (2017), MathematicaForPrediction at WordPress.
[AA2] Anton Antonov, "A Fast and Agile Item-Item Recommender: Design and Implementation", (2011), Wolfram Technology Conference 2011.
]]>In this blog post I try to provide some further context for the panel discussion:
"Artificial Intelligence and Machine Learning application in finance and technology",
which was part of the conference "Data Science Salon Miami 2018" held in Miami on February 8 and 9.
The blog post can be read independently, but my intent is to provide a brief review of the discussion and further context to (some of) the answers. (It is probably better to see discussion’s recording first.)
Also, my comments and remarks are saturated with links for reference and further reading.
You can also read the post by panel’s discussion host, Irma Becerra.
This blog post is not written by someone particularly enamored by Artificial Intelligence (AI), Data Science (DS), or Machine Learning (ML). I do like working in those disciplines, but I do think there are more interesting and fascinating mathematical disciplines than them. Also, I think AI, DS, and ML should be seen through the lens of Operations Research, which gives the most fruitful perspectives of their utilization.
The conference "Data Science Salon Miami 2018" included approximately 300 attendants. (I was told that there were ~60% data scientist and data analysts, and ~40% managers.)
Here is a list of the people participating in the panel (Irma was the host):
Here are the main questions asked during the panel session:
What are some misconceptions about AI? What do you see as being the future of AI?
How is artificial intelligence helping us engage with customers?
What techniques can we expect to see in terms of informing our marketing strategy so we can build predictive models to support our businesses? (backup question)
What can we do make advancements in Data Science more accessible to the public?
Here are some of the questions from the audience:
What is going to happen if an adopted AI technology fails?
All of the people in the panel became data scientists later in their career. Each of us at first was studying or doing something else.
Of course, that is to be expected since the term "data science" and the related occupation came into fashion relatively recently.
(Although apparently the term itself is fairly old.)
This is of course a long and fruitful topic.
Here — and during the panel session — I want to concentrate of what kind of thinking processes generate those misconceptions. (Obviously this is a less laborious task than enumerating and discussing the different concrete misconceptions.)
I side with the so called "Weak AI". Here are some summarizing points.
All algorithms are based on 0s and 1s, so generally, I find "AI" misleading and "Strong AI" an interesting thing to think about, but far from practical use.
Classifiers are just complicated if-statements.
Also, classification problems belong to the conceptually simplest ML branch (Supervised learning.)
Generally, my personal opinions on "AI" come from living through at least one AI winter, and being a subject in the AI effect.
Study the following mind-map, "AI misconceptions reasons".
I think the major ingredients of getting the wrong AI ideas are two.
My favorite analogy to point 1 above is the following.
There are two principal ways to fish: (i) with bait and a fishing rod, and (ii) by draining the lake and picking the fish you want. Any other way of fishing is something in between. (For example, using a net or a bomb.)
So, the high performance of draining the lake can be confused with the ability of bait selection and fishing-spot picking. (And yes, not knowing how the AI algorithms work is a big misconception maker too.)
As an illustration of "draining the lake" approach to interesting puzzles humans play with, review (skim over) this article: "Solving Sudoku as an integer programming problem".
Note, that this is not even considered AI now. Granted, we do need a symbolic computations system like Mathematica, which is a very advanced system. (And yes, a good symbolic manipulation system can be seen as AI, especially 4-5 decades ago.)
I specialize in making recommenders and conversational agents.
For items that require significant investment in time (movies, books) or money (houses, automobiles, freelancers) recommenders have to make very good explanations of the recommendations. This is usually based on similarities to items consumed in the past or stated preferences.
For items that can be easily tried on and discarded (songs) we are better off making prediction algorithms that predict the "survival" of the item in customer’s mind. (How long a song is going to be in a playlist?)
The recommenders I developed in the past tend to be very agile : fast, easy to tune, with clear "recommendation proofs." Very often the agility of the recommender system is more important that giving good or precise recommendations "out of the box."
Customer engagement through conversational agents is much more about envisioning of the right work-flow and mind-flow of the customer. Although clarity and simplicity are important, I like the idea of using grammar rules and agent names that are challenging or intriguing. (Like, "Will they kill me?" or "I wanna eat!".)
In many ways the application of AI to Finance is somewhat easier, since financial data tends to be well-curated.
For example,
health insurance financial data, though, is well-curated (since people need to get payed) and fairly simple.
To clarify, if we have records with four columns:
claim ID, claim total amount, transaction amount, timestamp
then we are already in a position to do a fair amount of modeling and prediction.
My answer was two-fold:
from the other hand, some of the underlying algorithms require mastery of multiple computer science and mathematical disciplines.
In other words:
from the other hand, "art is for the educated."
I would like to point out that technology itself has already democratized and popularized the AI and ML advancements. Everyone is familiar with, say, content recommendations, search queries completion, and optical character recognition. Few decades ago these functionalities would have been considered science fiction or part of Strong AI.
Descriptions of how to analyze the pre-election tweets of Donald Trump or apply the Great Conversation idea to interesting collections of text do help the communication of Data Science ideas to the curious.
I would be much more interested in developing agriculture recommendation systems than recommenders for digital content consumption (movies, books, songs) or dating.
Also, I am a big advocate of utilizing recommenders in Operations Research frameworks.
It seems that AI came recently into fashion somewhat coinciding with the advancements in Neural Networks through the so called "deep learning" on 2011-2012.