# A monad for Latent Semantic Analysis workflows

## Introduction

In this document we describe the design and implementation of a (software programming) monad, [Wk1], for Latent Semantic Analysis workflows specification and execution. The design and implementation are done with Mathematica / Wolfram Language (WL).

What is Latent Semantic Analysis (LSA)? : A statistical method (or a technique) for finding relationships in natural language texts that is based on the so called Distributional hypothesis, [Wk2, Wk3]. (The Distributional hypothesis can be simply stated as “linguistic items with similar distributions have similar meanings”; for an insightful philosophical and scientific discussion see [MS1].) LSA can be seen as the application of Dimensionality reduction techniques over matrices derived with the Vector space model.

The goal of the monad design is to make the specification of LSA workflows (relatively) easy and straightforward by following a certain main scenario and specifying variations over that scenario.

The monad is named `LSAMon` and it is based on the State monad package “StateMonadCodeGenerator.m”, [AAp1, AA1], the document-term matrix making package “DocumentTermMatrixConstruction.m”, [AAp4, AA2], the Non-Negative Matrix Factorization (NNMF) package “NonNegativeMatrixFactorization.m”, [AAp5, AA2], and the package “SSparseMatrix.m”, [AAp2, AA5], that provides matrix objects with named rows and columns.

The data for this document is obtained from WL’s repository and it is manipulated into a certain ready-to-utilize form (and uploaded to GitHub.)

The monadic programming design is used as a Software Design Pattern. The `LSAMon` monad can be also seen as a Domain Specific Language (DSL) for the specification and programming of machine learning classification workflows.

Here is an example of using the `LSAMon` monad over a collection of documents that consists of 233 US state of union speeches.

The table above is produced with the package “MonadicTracing.m”, [AAp2, AA1], and some of the explanations below also utilize that package.

As it was mentioned above the monad `LSAMon` can be seen as a DSL. Because of this the monad pipelines made with `LSAMon` are sometimes called “specifications”.

Remark: In this document with “term” we mean “a word, a word stem, or other type of token.”

Remark: LSA and Latent Semantic Indexing (LSI) are considered more or less to be synonyms. I think that “latent semantic analysis” sounds more universal and that “latent semantic indexing” as a name refers to a specific Information Retrieval technique. Below we refer to “LSI functions” like “IDF” and “TF-IDF” that are applied within the generic LSA workflow.

### Contents description

The document has the following structure.

• The sections “Package load” and “Data load” obtain the needed code and data.
• The sections “Design consideration” and “Monad design” provide motivation and design decisions rationale.
• The sections “LSAMon overview”, “Monad elements”, and “The utilization of SSparseMatrix objects” provide technical descriptions needed to utilize the `LSAMon` monad .
• (Using a fair amount of examples.)
• The section “Unit tests” describes the tests used in the development of the `LSAMon` monad.
• (The random pipelines unit tests are especially interesting.)
• The section “Future plans” outlines future directions of development.
• The section “Implementation notes” just says that `LSAMon`’s development process and this document follow the ones of the classifications workflows monad ClCon, [AA6].

Remark: One can read only the sections “Introduction”, “Design consideration”, “Monad design”, and “LSAMon overview”. That set of sections provide a fairly good, programming language agnostic exposition of the substance and novel ideas of this document.

The following commands load the packages [AAp1–AAp7]:

``````Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicLatentSemanticAnalysis.m"]

In this section we load data that is used in the rest of the document. The text data was obtained through WL’s repository, transformed in a certain more convenient form, and uploaded to GitHub.

The text summarization and plots are done through `LSAMon`, which in turn uses the function RecordsSummary from the package “MathematicaForPredictionUtilities.m”, [AAp7].

### Hamlet

``````textHamlet =
ToString /@
Flatten[Import["https://raw.githubusercontent.com/antononcube/MathematicaVsR/master/Data/MathematicaVsR-Data-Hamlet.csv"]];

TakeLargestBy[
Tally[DeleteStopwords[ToLowerCase[Flatten[TextWords /@ textHamlet]]]], #[[2]] &, 20]

(* {{"ham", 358}, {"lord", 225}, {"king", 196}, {"o", 124}, {"queen", 120},
{"shall", 114}, {"good", 109}, {"hor", 109}, {"come",  107}, {"hamlet", 107},
{"thou", 105}, {"let", 96}, {"thy", 86}, {"pol", 86}, {"like", 81}, {"sir", 75},
{"'t", 75}, {"know", 74}, {"enter", 73}, {"th", 72}} *)

LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix⟹LSAMonEchoDocumentTermMatrixStatistics;``````

### USA state of union speeches

``````url = "https://github.com/antononcube/MathematicaVsR/blob/master/Data/MathematicaVsR-Data-StateOfUnionSpeeches.JSON.zip?raw=true";
str = Import[url, "String"];
filename = First@Import[StringToStream[str], "ZIP"];
aStateOfUnionSpeeches = Association@ImportString[Import[StringToStream[str], {"ZIP", filename, "String"}], "JSON"];

lsaObj =
LSAMonUnit[aStateOfUnionSpeeches]⟹
LSAMonMakeDocumentTermMatrix⟹
LSAMonEchoDocumentTermMatrixStatistics["LogBase" -> 10];``````
``````TakeLargest[ColumnSumsAssociation[lsaObj⟹LSAMonTakeDocumentTermMatrix], 12]

(* <|"government" -> 7106, "states" -> 6502, "congress" -> 5023,
"united" -> 4847, "people" -> 4103, "year" -> 4022,
"country" -> 3469, "great" -> 3276, "public" -> 3094, "new" -> 3022,
"000" -> 2960, "time" -> 2922|> *)``````

### Stop words

In some of the examples below we want to explicitly specify the stop words. Here are stop words derived using the built-in functions `DictionaryLookup` and `DeleteStopwords`.

``````stopWords = Complement[DictionaryLookup["*"], DeleteStopwords[DictionaryLookup["*"]]];

Short[stopWords]

"you'll", "your", "you're", "yours", "yourself", "yourselves", "you've" } *)
``````

## Design considerations

The steps of the main LSA workflow addressed in this document follow.

1. Get a collection of documents with associated ID’s.
2. Create a document-term matrix.
1. Here we apply the Bag-or-words model and Vector space model.
1. The sequential order of the words is ignored and each document is represented as a point in a multi-dimensional vector space.
2. That vector space axes correspond to the unique words found in the whole document collection.
2. Consider the application of stemming rules.
3. Consider the removal of stop words.
3. Apply matrix-entries weighting functions.
1. Those functions come from LSI.
2. Functions like “IDF”, “TF-IDF”, “GFIDF”.
4. Extract topics.
1. One possible statistical way of doing this is with Dimensionality reduction.
2. We consider using Singular Value Decomposition (SVD) and Non-Negative Matrix Factorization (NNMF).
5. Make and display the topics table.
6. Extract and display a statistical thesaurus of selected words.
7. Map search queries or unseen documents over the extracted topics.
8. Find the most important documents in the document collection. (Optional.)

The following flow-chart corresponds to the list of steps above.

• the introduction of new elements in LSA workflows,
• workflows elements variability, and
• workflows iterative changes and refining,

it is beneficial to have a DSL for LSA workflows. We choose to make such a DSL through a functional programming monad, [Wk1, AA1].

Here is a quote from [Wk1] that fairly well describes why we choose to make a classification workflow monad and hints on the desired properties of such a monad.

[…] The monad represents computations with a sequential structure: a monad defines what it means to chain operations together. This enables the programmer to build pipelines that process data in a series of steps (i.e. a series of actions applied to the data), in which each action is decorated with the additional processing rules provided by the monad. […]

Monads allow a programming style where programs are written by putting together highly composable parts, combining in flexible ways the possible actions that can work on a particular type of data. […]

Remark: Note that quote from [Wk1] refers to chained monadic operations as “pipelines”. We use the terms “monad pipeline” and “pipeline” below.

The monad we consider is designed to speed-up the programming of LSA workflows outlined in the previous section. The monad is named `LSAMon` for “Latent Semantic Analysis** Mon**ad”.

We want to be able to construct monad pipelines of the general form:

`LSAMon` is based on the State monad, [Wk1, AA1], so the monad pipeline form (1) has the following more specific form:

This means that some monad operations will not just change the pipeline value but they will also change the pipeline context.

In the monad pipelines of `LSAMon` we store different objects in the contexts for at least one of the following two reasons.

1. The object will be needed later on in the pipeline, or
2. The object is (relatively) hard to compute.

Such objects are document-term matrix, Dimensionality reduction factors and the related topics.

Let us list the desired properties of the monad.

• Rapid specification of non-trivial LSA workflows.
• The text data supplied to the monad can be: (i) a list of strings, or (ii) an association with string values.
• The monad uses the Linear vector space model .
• The document-term frequency matrix can be created after removing stop words and/or word stemming.
• It is easy to specify and apply different LSI weight functions. (Like “IDF” or “GFIDF”.)
• The monad can do dimension reduction with SVD and NNMF and corresponding matrix factors are retrievable with monad functions.
• Documents (or query strings) external to the monad are easily mapped into monad’s Linear vector space of terms and Linear vector space of topics.
• The monad allows of cursory examination and summarization of the data.
• The pipeline values can be of different types. (Most monad functions modify the pipeline value; some modify the context; some just echo results.)
• It is easy to obtain the pipeline value, context, and different context objects for manipulation outside of the monad.
• It is easy to tabulate extracted topics and related statistical thesauri.

The `LSAMon` components and their interactions are fairly simple.

The main `LSAMon` operations implicitly put in the context or utilize from the context the following objects:

• document-term matrix,
• the factors obtained by matrix factorization algorithms,
• LSI weight functions specifications,
• extracted topics.

Note the that the monadic set of types of `LSAMon` pipeline values is fairly heterogenous and certain awareness of “the current pipeline value” is assumed when composing `LSAMon` pipelines.

Obviously, we can put in the context any object through the generic operations of the State monad of the package “StateMonadGenerator.m”, [AAp1].

## LSAMon overview

When using a monad we lift certain data into the “monad space”, using monad’s operations we navigate computations in that space, and at some point we take results from it.

With the approach taken in this document the “lifting” into the `LSAMon` monad is done with the function LSAMonUnit. Results from the monad can be obtained with the functions `LSAMonTakeValue`, `LSAMonContext`, or with the other `LSAMon` functions with the prefix “LSAMonTake” (see below.)

Here is a corresponding diagram of a generic computation with the `LSAMon` monad:

Remark: It is a good idea to compare the diagram with formulas (1) and (2).

Let us examine a concrete `LSAMon` pipeline that corresponds to the diagram above. In the following table each pipeline operation is combined together with a short explanation and the context keys after its execution.

Here is the output of the pipeline:

The `LSAMon` functions are separated into four groups:

• operations,
• setters and droppers,
• takers,

### Monad functions interaction with the pipeline value and context

An overview of the those functions is given in the tables in next two sub-sections. The next section, “Monad elements”, gives details and examples for the usage of the `LSAMon` operations.

Here are the `LSAMon` State Monad functions (generated using the prefix “LSAMon”, [AAp1, AA1].)

Here are the usage descriptions of the main (not monad-supportive) `LSAMon` functions, which are explained in detail in the next section.

In this section we show that `LSAMon` has all of the properties listed in the previous section.

The monad head is `LSAMon`. Anything wrapped in `LSAMon` can serve as monad’s pipeline value. It is better though to use the constructor `LSAMonUnit`. (Which adheres to the definition in [Wk1].)

``LSAMon[textHamlet, <||>]⟹LSAMonMakeDocumentTermMatrix[Automatic, Automatic]⟹LSAMonEchoFunctionContext[Short];``

### Lifting data to the monad

The function lifting the data into the monad `LSAMon` is `LSAMonUnit`.

The lifting to the monad marks the beginning of the monadic pipeline. It can be done with data or without data. Examples follow.

``````LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix⟹LSAMonTakeDocumentTermMatrix

LSAMonUnit[]⟹LSAMonSetDocuments[textHamlet]⟹LSAMonMakeDocumentTermMatrix⟹LSAMonTakeDocumentTermMatrix``````

(See the sub-section “Setters, droppers, and takers” for more details of setting and taking values in `LSAMon` contexts.)

Currently the monad can deal with data in the following forms:

• vectors of strings,
• associations with string values.

Generally, WL makes it easy to extract columns datasets order to obtain vectors or matrices, so datasets are not currently supported in `LSAMon`.

### Making of the document-term matrix

As it was mentioned above with “term” we mean “a word or a stemmed word”. Here is are examples of stemmed words.

``WordData[#, "PorterStem"] & /@ {"consequential", "constitution", "forcing", ""}``

The fundamental model of `LSAMon` is the so called Vector space model (or the closely related Bag-of-words model.) The document-term matrix is a linear vector space representation of the documents collection. That representation is further used in `LSAMon` to find topics and statistical thesauri.

Here is an example of ad hoc construction of a document-term matrix using a couple of paragraphs from “Hamlet”.

``````inds = {10, 19};

MatrixForm @ CrossTabulate[Flatten[KeyValueMap[Thread[{#1, #2}] &, TextWords /@ ToLowerCase[aTempText]], 1]]``````

When we construct the document-term matrix we (often) want to stem the words and (almost always) want to remove stop words. `LSAMon`’s function LSAMonMakeDocumentTermMatrix makes the document-term matrix and takes specifications for stemming and stop words.

``````lsaObj =
LSAMonUnit[textHamlet]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> Automatic, "StopWords" -> Automatic]⟹
LSAMonEchoFunctionContext[ MatrixPlot[#documentTermMatrix] &]⟹
LSAMonEchoFunctionContext[TakeLargest[ColumnSumsAssociation[#documentTermMatrix], 12] &];``````

We can retrieve the stop words used in a monad with the function LSAMonTakeStopWords.

``Short[lsaObj⟹LSAMonTakeStopWords]``

We can retrieve the stemming rules used in a monad with the function LSAMonTakeStemmingRules.

``Short[lsaObj⟹LSAMonTakeStemmingRules]``

The specification `Automatic` for stemming rules uses `WordData[#,"PorterStem"]&`.

Instead of the options style signature we can use positional signature.

• Options style: `LSAMonMakeDocumentTermMatrix["StemmingRules" -> {}, "StopWords" -> Automatic]` .
• Positional style: `LSAMonMakeDocumentTermMatrix[{}, Automatic]` .

### LSI weight functions

After making the document-term matrix we will most likely apply LSI weight functions, [Wk2], like “GFIDF” and “TF-IDF”. (This follows the “standard” approach used in search engines for calculating weights for document-term matrices; see [MB1].)

#### Frequency matrix

We use the following definition of the frequency document-term matrix F.

Each entry fij of the matrix F is the number of occurrences of the term j in the document i.

#### Weights

Each entry of the weighted document-term matrix M derived from the frequency document-term matrix F is expressed with the formula

where gj – global term weight; lij – local term weight; di – normalization weight.

Various formulas exist for these weights and one of the challenges is to find the right combination of them when using different document collections.

Here is a table of weight functions formulas.

#### Computation specifications

`LSAMon` function LSAMonApplyTermWeightFunctions delegates the LSI weight functions application to the package “DocumentTermMatrixConstruction.m”, [AAp4].

Here is an example.

``````lsaHamlet = LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix;
wmat =
lsaHamlet⟹
LSAMonApplyTermWeightFunctions["IDF", "TermFrequency", "Cosine"]⟹
LSAMonTakeWeightedDocumentTermMatrix;

TakeLargest[ColumnSumsAssociation[wmat], 6]``````

Instead of using the positional signature of LSAMonApplyTermWeightFunctions we can specify the LSI functions using options.

``````wmat2 =
lsaHamlet⟹
LSAMonApplyTermWeightFunctions["GlobalWeightFunction" -> "IDF", "LocalWeightFunction" -> "TermFrequency", "NormalizerFunction" -> "Cosine"]⟹
LSAMonTakeWeightedDocumentTermMatrix;

TakeLargest[ColumnSumsAssociation[wmat2], 6]``````

Here we are summaries of the non-zero values of the weighted document-term matrix derived with different combinations of global, local, and normalization weight functions.

``````Magnify[#, 0.8] &@Multicolumn[Framed /@ #, 6] &@Flatten@
Table[
(wmat =
lsaHamlet⟹
LSAMonApplyTermWeightFunctions[gw, lw, nf]⟹
LSAMonTakeWeightedDocumentTermMatrix;
RecordsSummary[SparseArray[wmat]["NonzeroValues"],
List@StringRiffle[{gw, lw, nf}, ", "]]),
{gw, {"IDF", "GFIDF", "Binary", "None", "ColumnStochastic"}},
{lw, {"Binary", "Log", "None"}},
{nf, {"Cosine", "None", "RowStochastic"}}]
AutoCollapse[]``````

### Extracting topics

Streamlining topic extraction is one of the main reasons `LSAMon` was implemented. The topic extraction correspond to the so called “syntagmatic” relationships between the terms, [MS1].

#### Theoretical outline

The original weighed document-term matrix M is decomposed into the matrix factors W and H.

M ≈ W.H, W ∈ ℝm × k, H ∈ ℝk × n.

The i-th row of M is expressed with the i-th row of W multiplied by H.

The rows of H are the topics. SVD produces orthogonal topics; NNMF does not.

The i-the document of the collection corresponds to the i-th row W. Finding the Nearest Neighbors (NN’s) of the i-th document using the rows similarity of the matrix W gives document NN’s through topic similarity.

The terms correspond to the columns of H. Finding NN’s based on similarities of H’s columns produces statistical thesaurus entries.

The term groups provided by H’s rows correspond to “syntagmatic” relationships. Using similarities of H’s columns we can produce term clusters that correspond to “paradigmatic” relationships.

#### Computation specifications

Here is an example using the play “Hamlet” in which we specify additional stop words.

``````stopWords2 = {"enter", "exit", "[exit", "ham", "hor", "laer", "pol", "oph", "thy", "thee", "act", "scene"};

SeedRandom[2381]
lsaHamlet =
LSAMonUnit[textHamlet]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> Automatic, "StopWords" -> Join[stopWords, stopWords2]]⟹
LSAMonApplyTermWeightFunctions["GlobalWeightFunction" -> "IDF", "LocalWeightFunction" -> "None", "NormalizerFunction" -> "Cosine"]⟹
LSAMonExtractTopics["NumberOfTopics" -> 12, "MinNumberOfDocumentsPerTerm" -> 10, Method -> "NNMF", "MaxSteps" -> 20]⟹
LSAMonEchoTopicsTable["NumberOfTableColumns" -> 6, "NumberOfTerms" -> 10];``````

Here is an example using the USA presidents “state of union” speeches.

``````SeedRandom[7681]
lsaSpeeches =
LSAMonUnit[aStateOfUnionSpeeches]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> Automatic,  "StopWords" -> Automatic]⟹
LSAMonApplyTermWeightFunctions["GlobalWeightFunction" -> "IDF", "LocalWeightFunction" -> "None", "NormalizerFunction" -> "Cosine"]⟹
LSAMonExtractTopics["NumberOfTopics" -> 36, "MinNumberOfDocumentsPerTerm" -> 40, Method -> "NNMF", "MaxSteps" -> 12]⟹
LSAMonEchoTopicsTable["NumberOfTableColumns" -> 6, "NumberOfTerms" -> 10];``````

Note that in both examples:

1. stemming is used when creating the document-term matrix,
2. the default LSI re-weighting functions are used: “IDF”, “None”, “Cosine”,
3. the dimension reduction algorithm NNMF is used.

Things to keep in mind.

1. The interpretability provided by NNMF comes at a price.
2. NNMF is prone to get stuck into local minima, so several topic extractions (and corresponding evaluations) have to be done.
3. We would get different results with different NNMF runs using the same parameters. (NNMF uses random numbers initialization.)
4. The NNMF topic vectors are not orthogonal.
5. SVD is much faster than NNMF, but it topic vectors are hard to interpret.
6. Generally, the topics derived with SVD are stable, they do not change with different runs with the same parameters.
7. The SVD topics vectors are orthogonal, which provides for quick to find representations of documents not in the monad’s document collection.

The document-topic matrix W has column names that are automatically derived from the top three terms in each topic.

``````ColumnNames[lsaHamlet⟹LSAMonTakeW]

(* {"player-plai-welcom", "ro-lord-sir", "laert-king-attend",
"end-inde-make", "state-room-castl", "daughter-pass-love",
"hamlet-ghost-father", "father-thou-king",
"rosencrantz-guildenstern-king", "ophelia-queen-poloniu",

Of course the row names of H have the same names.

``````RowNames[lsaHamlet⟹LSAMonTakeH]

(* {"player-plai-welcom", "ro-lord-sir", "laert-king-attend",
"end-inde-make", "state-room-castl", "daughter-pass-love",
"hamlet-ghost-father", "father-thou-king",
"rosencrantz-guildenstern-king", "ophelia-queen-poloniu",

### Extracting statistical thesauri

The statistical thesaurus extraction corresponds to the “paradigmatic” relationships between the terms, [MS1].

Here is an example over the State of Union speeches.

``````entryWords = {"bank", "war", "economy", "school", "port", "health", "enemy", "nuclear"};

lsaSpeeches⟹
LSAMonExtractStatisticalThesaurus["Words" -> Map[WordData[#, "PorterStem"] &, entryWords], "NumberOfNearestNeighbors" -> 12]⟹
LSAMonEchoStatisticalThesaurus;``````

In the code above: (i) the options signature style is used, (ii) the statistical thesaurus entry words are stemmed first.

We can also call `LSAMonEchoStatisticalThesaurus` directly without calling `LSAMonExtractStatisticalThesaurus` first.

`````` lsaSpeeches⟹
LSAMonEchoStatisticalThesaurus["Words" -> Map[WordData[#, "PorterStem"] &, entryWords], "NumberOfNearestNeighbors" -> 12];``````

### Mapping queries and documents to terms

One of the most natural operations is to find the representation of an arbitrary document (or sentence or a list of words) in monad’s Linear vector space of terms. This is done with the function `LSAMonRepresentByTerms`.

Here is an example in which a sentence is represented as a one-row matrix (in that space.)

``````obj =
lsaHamlet⟹
LSAMonRepresentByTerms["Hamlet, Prince of Denmark killed the king."]⟹
LSAMonEchoValue;``````

Here we display only the non-zero columns of that matrix.

``````obj⟹
LSAMonEchoFunctionValue[MatrixForm[Part[#, All, Keys[Select[SSparseMatrix`ColumnSumsAssociation[#], # > 0& ]]]]& ];``````

#### Transformation steps

Assume that `LSAMonRepresentByTerms` is given a list of sentences. Then that function performs the following steps.

1. The sentence is split into a list of words.

2. If monad’s document-term matrix was made by removing stop words the same stop words are removed from the list of words.

3. If monad’s document-term matrix was made by stemming the same stemming rules are applied to the list of words.

4. The LSI global weights and the LSI local weight and normalizer functions are applied to sentence’s contingency matrix.

#### Equivalent representation

Let us look convince ourselves that documents used in the monad to built the weighted document-term matrix have the same representation as the corresponding rows of that matrix.

Here is an association of documents from monad’s document collection.

``````inds = {6, 10};
queries = Part[lsaHamlet⟹LSAMonTakeDocuments, inds];
queries

(* <|"id.0006" -> "Getrude, Queen of Denmark, mother to Hamlet. Ophelia, daughter to Polonius.",
"id.0010" -> "ACT I. Scene I. Elsinore. A platform before the Castle."|> *)

lsaHamlet⟹
LSAMonRepresentByTerms[queries]⟹
LSAMonEchoFunctionValue[MatrixForm[Part[#, All, Keys[Select[SSparseMatrix`ColumnSumsAssociation[#], # > 0& ]]]]& ];``````
``````lsaHamlet⟹
LSAMonEchoFunctionContext[MatrixForm[Part[Slot["weightedDocumentTermMatrix"], inds, Keys[Select[SSparseMatrix`ColumnSumsAssociation[Part[Slot["weightedDocumentTermMatrix"], inds, All]], # > 0& ]]]]& ];``````

### Mapping queries and documents to topics

Another natural operation is to find the representation of an arbitrary document (or a list of words) in monad’s Linear vector space of topics. This is done with the function `LSAMonRepresentByTopics`.

Here is an example.

``````inds = {6, 10};
queries = Part[lsaHamlet⟹LSAMonTakeDocuments, inds];
Short /@ queries

(* <|"id.0006" -> "Getrude, Queen of Denmark, mother to Hamlet. Ophelia, daughter to Polonius.",
"id.0010" -> "ACT I. Scene I. Elsinore. A platform before the Castle."|> *)

lsaHamlet⟹
LSAMonRepresentByTopics[queries]⟹
LSAMonEchoFunctionValue[MatrixForm[Part[#, All, Keys[Select[SSparseMatrix`ColumnSumsAssociation[#], # > 0& ]]]]& ];``````
``````lsaHamlet⟹
LSAMonEchoFunctionContext[MatrixForm[Part[Slot["W"], inds, Keys[Select[SSparseMatrix`ColumnSumsAssociation[Part[Slot["W"], inds, All]], # > 0& ]]]]& ];``````

#### Theory

In order to clarify what the function LSAMonRepresentByTopics is doing let us go through the formulas it is based on.

The original weighed document-term matrix M is decomposed into the matrix factors W and H.

M ≈ W.H, W ∈ ℝm × k, H ∈ ℝk × n

The i-th row of M is expressed with the i-th row of W multiplied by H.

mi ≈ wi.H.

For a query vector q0 ∈ ℝm we want to find its topics representation vector x ∈ ℝk:

q0 ≈ x.H.

Denote with H( − 1) the inverse or pseudo-inverse matrix of H. We have:

q0.H( − 1) ≈ (x.H).H( − 1) = x.(H.H( − 1)) = x.I,

x ∈ ℝk, H( − 1) ∈ ℝn × k, I ∈ ℝk × k.

In `LSAMon` for SVD H( − 1) = HT; for NNMF H( − 1) is the pseudo-inverse of H.

The vector x obtained with `LSAMonRepresentByTopics`.

### Tags representation

Sometimes we want to find the topics representation of tags associated with monad’s documents and the tag-document associations are one-to-many. See [AA3].

Let us consider a concrete example – we want to find what topics correspond to the different presidents in the collection of State of Union speeches.

Here we find the document tags (president names in this case.)

``````tags = StringReplace[
RowNames[
lsaSpeeches⟹LSAMonTakeDocumentTermMatrix],
RegularExpression[".\\d\\d\\d\\d-\\d\\d-\\d\\d"] -> ""];
Short[tags]``````

Here is the number of unique tags (president names.)

``````Length[Union[tags]]
(* 42 *)``````

Here we compute the tag-topics representation matrix using the function `LSAMonRepresentDocumentTagsByTopics`.

``````tagTopicsMat =
lsaSpeeches⟹
LSAMonRepresentDocumentTagsByTopics[tags]⟹
LSAMonTakeValue``````

Here is a heatmap plot of the tag-topics matrix made with the package “HeatmapPlot.m”, [AAp11].

``HeatmapPlot[tagTopicsMat[[All, Ordering@ColumnSums[tagTopicsMat]]], DistanceFunction -> None, ImageSize -> Large]``

### Finding the most important documents

There are several algorithms we can apply for finding the most important documents in the collection. `LSAMon` utilizes two types algorithms: (1) graph centrality measures based, and (2) matrix factorization based. With certain graph centrality measures the two algorithms are equivalent. In this sub-section we demonstrate the matrix factorization algorithm (that uses SVD.)

Definition: The most important sentences have the most important words and the most important words are in the most important sentences.

That definition can be used to derive an iterations-based model that can be expressed with SVD or eigenvector finding algorithms, [LE1].

Here we pick an important part of the play “Hamlet”.

``````focusText =
First@Pick[textHamlet, StringMatchQ[textHamlet, ___ ~~ "to be" ~~ __ ~~ "or not to be" ~~ ___, IgnoreCase -> True]];
Short[focusText]

(* "Ham. To be, or not to be- that is the question: Whether 'tis ....y.
O, woe is me T' have seen what I have seen, see what I see!" *)

LSAMonUnit[StringSplit[ToLowerCase[focusText], {",", ".", ";", "!", "?"}]]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> {}, "StopWords" -> Automatic]⟹
LSAMonApplyTermWeightFunctions⟹
LSAMonFindMostImportantDocuments[3]⟹
LSAMonEchoFunctionValue[GridTableForm];``````

### Setters, droppers, and takers

The values from the monad context can be set, obtained, or dropped with the corresponding “setter”, “dropper”, and “taker” functions as summarized in a previous section.

For example:

``````p = LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix[Automatic, Automatic];

p⟹LSAMonTakeMatrix``````

If other values are put in the context they can be obtained through the (generic) function LSAMonTakeContext, [AAp1]:

``````Short@(p⟹QRMonTakeContext)["documents"]

(* <|"id.0001" -> "1604", "id.0002" -> "THE TRAGEDY OF HAMLET, PRINCE OF DENMARK", <<220>>, "id.0223" -> "THE END"|> *) ``````

Another generic function from [AAp1] is LSAMonTakeValue (used many times above.)

Here is an example of the “data dropper” LSAMonDropDocuments:

``````Keys[p⟹LSAMonDropDocuments⟹QRMonTakeContext]

(* {"documentTermMatrix", "terms", "stopWords", "stemmingRules"} *)``````

(The “droppers” simply use the state monad function LSAMonDropFromContext, [AAp1]. For example, LSAMonDropDocuments is equivalent to LSAMonDropFromContext[“documents”].)

## The utilization of SSparseMatrix objects

The LSAMon monad heavily relies on `SSparseMatrix` objects, [AAp6, AA5], for internal representation of data and computation results.

A `SSparseMatrix` object is a matrix with named rows and columns.

Here is an example.

``````n = 6;
rmat = ToSSparseMatrix[
SparseArray[{{1, 2} -> 1, {4, 5} -> 1}, {n, n}],
"RowNames" -> RandomSample[CharacterRange["A", "Z"], n],
"ColumnNames" -> RandomSample[CharacterRange["a", "z"], n]];
MatrixForm[rmat]``````

In this section we look into some useful SSparseMatrix idioms applied within `LSAMon`.

### Visualize with sorted rows and columns

In some situations it is beneficial to sort rows and columns of the (weighted) document-term matrix.

``````docTermMat =
lsaSpeeches⟹LSAMonTakeDocumentTermMatrix;
MatrixPlot[docTermMat[[Ordering[RowSums[docTermMat]],  Ordering[ColumnSums[docTermMat]]]], MaxPlotPoints -> 300, ImageSize -> Large]``````

The most popular terms in the document collection can be found through the association of the column sums of the document-term matrix.

``````TakeLargest[ColumnSumsAssociation[lsaSpeeches⟹LSAMonTakeDocumentTermMatrix], 10]

(* <|"state" -> 8852, "govern" -> 8147, "year" -> 6362, "nation" -> 6182,
"congress" -> 5040, "unit" -> 5040, "countri" -> 4504,
"peopl" -> 4306, "american" -> 3648, "law" -> 3496|> *)
``````

Similarly for the lest popular terms.

``````TakeSmallest[
ColumnSumsAssociation[
lsaSpeeches⟹LSAMonTakeDocumentTermMatrix], 10]

(* <|"036" -> 1, "027" -> 1, "_____________________" -> 1, "0111" -> 1,
"006" -> 1, "0000" -> 1, "0001" -> 1, "______________________" -> 1,
"____" -> 1, "____________________" -> 1|> *)``````

### Showing only non-zero columns

In some cases we want to show only columns of the data or computation results matrices that have non-zero elements.

Here is an example (similar to other examples in the previous section.)

``````lsaHamlet⟹
LSAMonRepresentByTerms[{"this country is rotten",
"where is my sword my lord",
"poison in the ear should be in the play"}]⟹
LSAMonEchoFunctionValue[ MatrixForm[#1[[All, Keys[Select[ColumnSumsAssociation[#1], #1 > 0 &]]]]] &];``````

In the pipeline code above: (i) from the list of queries a representation matrix is made, (ii) that matrix is assigned to the pipeline value, (iii) in the pipeline echo value function the non-zero columns are selected with by using the keys of the non-zero elements of the association obtained with ColumnSumsAssociation.

### Similarities based on representation by terms

Here is way to compute the similarity matrix of different sets of documents that are not required to be in monad’s document collection.

``````sMat1 =
lsaSpeeches⟹
LSAMonRepresentByTerms[ aStateOfUnionSpeeches[[ Range[-5, -2] ]] ]⟹
LSAMonTakeValue

sMat2 =
lsaSpeeches⟹
LSAMonRepresentByTerms[ aStateOfUnionSpeeches[[ Range[-7, -3] ]] ]⟹
LSAMonTakeValue

MatrixForm[sMat1.Transpose[sMat2]]``````

### Similarities based on representation by topics

Similarly to weighted Boolean similarities matrix computation above we can compute a similarity matrix using the topics representations. Note that an additional normalization steps is required.

``````sMat1 =
lsaSpeeches⟹
LSAMonRepresentByTopics[ aStateOfUnionSpeeches[[ Range[-5, -2] ]] ]⟹
LSAMonTakeValue;
sMat1 = WeightTermsOfSSparseMatrix[sMat1, "None", "None", "Cosine"]

sMat2 =
lsaSpeeches⟹
LSAMonRepresentByTopics[ aStateOfUnionSpeeches[[ Range[-7, -3] ]] ]⟹
LSAMonTakeValue;
sMat2 = WeightTermsOfSSparseMatrix[sMat2, "None", "None", "Cosine"]

MatrixForm[sMat1.Transpose[sMat2]]``````

Note the differences with the weighted Boolean similarity matrix in the previous sub-section – the similarities that are less than 1 are noticeably larger.

## Unit tests

The development of `LSAMon` was done with two types of unit tests: (i) directly specified tests, [AAp7], and (ii) tests based on randomly generated pipelines, [AA8].

The unit test package should be further extended in order to provide better coverage of the functionalities and illustrate – and postulate – pipeline behavior.

### Directly specified tests

Here we run the unit tests file “MonadicLatentSemanticAnalysis-Unit-Tests.wlt”, [AAp8].

``````AbsoluteTiming[
]``````

The natural language derived test ID’s should give a fairly good idea of the functionalities covered in [AAp3].

``````Values[Map[#["TestID"] &, testObject["TestResults"]]]

"Make-document-term-matrix-1", "Make-document-term-matrix-2",
"Apply-term-weights-1", "Apply-term-weights-2", "Topic-extraction-1",
"Topic-extraction-2", "Topic-extraction-3", "Topic-extraction-4",
"Statistical-thesaurus-1", "Topics-representation-1",
"Take-document-term-matrix-1", "Take-weighted-document-term-matrix-1",
"Take-document-term-matrix-2", "Take-weighted-document-term-matrix-2",
"Take-terms-1", "Take-Factors-1", "Take-Factors-2", "Take-Factors-3",
"Take-Factors-4", "Take-StopWords-1", "Take-StemmingRules-1"} *)``````

### Random pipelines tests

Since the monad `LSAMon` is a DSL it is natural to test it with a large number of randomly generated “sentences” of that DSL. For the `LSAMon` DSL the sentences are `LSAMon` pipelines. The package “MonadicLatentSemanticAnalysisRandomPipelinesUnitTests.m”, [AAp9], has functions for generation of `LSAMon` random pipelines and running them as verification tests. A short example follows.

Generate pipelines:

``````SeedRandom[234]
pipelines = MakeLSAMonRandomPipelines[100];
Length[pipelines]

(* 100 *)``````

Here is a sample of the generated pipelines:

Here we run the pipelines as unit tests:

``````AbsoluteTiming[
res = TestRunLSAMonPipelines[pipelines, "Echo" -> False];
]``````

From the test report results we see that a dozen tests failed with messages, all of the rest passed.

``rpTRObj = TestReport[res]``

(The message failures, of course, have to be examined – some bugs were found in that way. Currently the actual test messages are expected.)

## Future plans

### Dimension reduction extensions

It would be nice to extend the Dimension reduction functionalities of `LSAMon` to include other algorithms like Independent Component Analysis (ICA), [Wk5]. Ideally with `LSAMon` we can do comparisons between SVD, NNMF, and ICA like the image de-nosing based comparison explained in [AA8].

Another direction is to utilize Neural Networks for the topic extraction and making of statistical thesauri.

### Conversational agent

Since `LSAMon` is a DSL it can be relatively easily interfaced with a natural language interface.

Here is an example of natural language commands parsed into LSA code using the package [AAp13].

## Implementation notes

The implementation methodology of the `LSAMon` monad packages [AAp3, AAp9] followed the methodology created for the `ClCon` monad package [AAp10, AA6]. Similarly, this document closely follows the structure and exposition of the `ClCon monad document “A monad for classification workflows”, [AA6].

A lot of the functionalities and signatures of `LSAMon` were designed and programed through considerations of natural language commands specifications given to a specialized conversational agent.

## References

### Packages

[AAp1] Anton Antonov, State monad code generator Mathematica package, (2017), MathematicaForPrediction at GitHub*.

[AAp2] Anton Antonov, Monadic tracing Mathematica package, (2017), MathematicaForPrediction at GitHub*.

[AAp3] Anton Antonov, Monadic Latent Semantic Analysis Mathematica package, (2017), MathematicaForPrediction at GitHub.

[AAp5] Anton Antonov, Non-Negative Matrix Factorization algorithm implementation in Mathematica, (2013), MathematicaForPrediction at GitHub.

[AAp6] Anton Antonov, SSparseMatrix Mathematica package, (2018), MathematicaForPrediction at GitHub.

[AAp7] Anton Antonov, MathematicaForPrediction utilities, (2014), MathematicaForPrediction at GitHub.

[AAp8] Anton Antonov, Monadic Latent Semantic Analysis unit tests, (2019), MathematicaVsR at GitHub.

[AAp9] Anton Antonov, Monadic Latent Semantic Analysis random pipelines Mathematica unit tests, (2019), MathematicaForPrediction at GitHub.

[AAp10] Anton Antonov, Monadic contextual classification Mathematica package, (2017), MathematicaForPrediction at GitHub.

[AAp11] Anton Antonov, Heatmap plot Mathematica package, (2017), MathematicaForPrediction at GitHub.

[AAp12] Anton Antonov,
Independent Component Analysis Mathematica package, MathematicaForPrediction at GitHub.

[AAp13] Anton Antonov, Latent semantic analysis workflows grammar in EBNF, (2018), ConverasationalAgents at GitHub.

### MathematicaForPrediction articles

[AA1] Anton Antonov, “Monad code generation and extension”, (2017), MathematicaForPrediction at GitHub.

[AA2] Anton Antonov, “Topic and thesaurus extraction from a document collection”, (2013), MathematicaForPrediction at GitHub.

[AA3] Anton Antonov, “The Great conversation in USA presidential speeches”, (2017), MathematicaForPrediction at WordPress.

[AA4] Anton Antonov, “Contingency tables creation examples”, (2016), MathematicaForPrediction at WordPress.

[AA5] Anton Antonov, “RSparseMatrix for sparse matrices with named rows and columns”, (2015), MathematicaForPrediction at WordPress.

[AA6] Anton Antonov, “A monad for classification workflows”, (2018), MathematicaForPrediction at WordPress.

[AA7] Anton Antonov, “Independent component analysis for multidimensional signals”, (2016), MathematicaForPrediction at WordPress.

[AA8] Anton Antonov, “Comparison of PCA, NNMF, and ICA over image de-noising”, (2016), MathematicaForPrediction at WordPress.

### Other

[Wk2] Wikipedia entry, Latent semantic analysis,

[Wk3] Wikipedia entry, Distributional semantics,

[Wk4] Wikipedia entry, Non-negative matrix factorization,

[LE1] Lars Elden, Matrix Methods in Data Mining and Pattern Recognition, 2007, SIAM. ISBN-13: 978-0898716269.

[MB1] Michael W. Berry & Murray Browne, Understanding Search Engines: Mathematical Modeling and Text Retrieval, 2nd. ed., 2005, SIAM. ISBN-13: 978-0898715811.

[MS1] Magnus Sahlgren, “The Distributional Hypothesis”, (2008), Rivista di Linguistica. 20 (1): 33[Dash]53.

[PS1] Patrick Scheibe, Mathematica (Wolfram Language) support for IntelliJ IDEA, (2013-2018), Mathematica-IntelliJ-Plugin at GitHub.

# Parametrized event records data transformations

## Introduction

In this document we describe transformations of events records data in order to make that data more amenable for the application of Machine Learning (ML) algorithms.

Consider the following problem formulation (done with the next five bullet points.)

• From data representing a (most likely very) diverse set of events we want to derive contingency matrices corresponding to each of the variables in that data.

• The events are observations of the values of a certain set of variables for a certain set of entities. Not all entities have events for all variables.

• The observation times do not form a regular time grid.

• Each contingency matrix has rows corresponding to the entities in the data and has columns corresponding to time.

• The software component providing the functionality should allow parametrization and repeated execution. (As in ML classifier training and testing scenarios.)

The phrase “event records data” is used instead of “time series” in order to emphasize that (i) some variables have categorical values, and (ii) the data can be given in some general database form, like transactions long-form.

The required transformations of the event records in the problem formulation above are done through the monad `ERTMon`, [AAp3]. (The name “ERTMon” comes from “Event Records Transformations Monad”.)

The monad code generation and utilization is explained in [AA1] and implemented with [AAp1].

It is assumed that the event records data is put in a form that makes it (relatively) easy to extract time series for the set of entity-variable pairs present in that data.

In brief `ERTMon` performs the following sequence of transformations.

1. The event records of each entity-variable pair are shifted to adhere to a specified start or end point,

2. The event records for each entity-variable pair are aggregated and normalized with specified functions over a specified regular grid,

3. Entity vs. time interval contingency matrices are made for each combination of variable and aggregation function.

The transformations are specified with a “computation specification” dataset.

Here is an example of an `ERTMon` pipeline over event records:

The rest of the document describes in detail:

• the structure, format, and interpretation of the event records data and computations specifications,

• the transformations of time series aligning, aggregation, and normalization,

• the software pattern design – a monad – that allows sequential specifications of desired transformations.

Concrete examples are given using weather data. See [AAp9].

The following commands load the packages [AAp1-AAp9].

``````Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicEventRecordsTransformations.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/Misc/WeatherEventRecords.m"]``````

The data we use is weather data from meteorological stations close to certain major cities. We retrieve the data with the function `WeatherEventRecords` from the package [AAp9].

``?WeatherEventRecords``

WeatherEventRecords[ citiesSpec_: {{_String, _String}..}, dateRange:{{_Integer, _Integer, _Integer}, {_Integer, _Integer, _Integer}}, wProps:{_String..} : {“Temperature”}, nStations_Integer : 1 ] gives an association with event records data.

``````citiesSpec = {{"Miami", "USA"}, {"Chicago", "USA"}, {"London",  "UK"}};
dateRange = {{2017, 7, 1}, {2018, 6, 31}};
wProps = {"Temperature", "MaxTemperature", "Pressure", "Humidity", "WindSpeed"};
res1 = WeatherEventRecords[citiesSpec, dateRange, wProps, 1];

citiesSpec = {{"Jacksonville", "USA"}, {"Peoria", "USA"}, {"Melbourne", "Australia"}};
dateRange = {{2016, 12, 1}, {2017, 12, 31}};
res2 = WeatherEventRecords[citiesSpec, dateRange, wProps, 1];``````

Here we assign the obtained datasets to variables we use below:

``````eventRecords = Join[res1["eventRecords"], res2["eventRecords"]];
entityAttributes = Join[res1["entityAttributes"], res2["entityAttributes"]];``````

Here are the summaries of the datasets `eventRecords` and `entityAttributes`:

``RecordsSummary[eventRecords]``
``RecordsSummary[entityAttributes]``

## Design considerations

### Workflow

The steps of the main event records transformations workflow addressed in this document follow.

1. Ingest event records and entity attributes given in the Star schema style.

2. Ingest a computation specification.

1. Specified are aggregation time intervals, aggregation functions, normalization types and functions.
3. Group event records based on unique entity ID and variable pairs.
1. Additional filtering can be applied using the entity attributes.
4. For each variable find descriptive statistics properties.
1. This is to facilitate normalization procedures.

2. Optionally, for each variable find outlier boundaries.

5. Align each group of records to start or finish at some specified point.

1. For each variable we want to impose a regular time grid.
6. From each group of records produce a time series.

7. For each time series do prescribed aggregation and normalization.

1. The variable that corresponds to each group of records has at least one (possibly several) computation specifications.
8. Make a contingency matrix for each time series obtained in the previous step.
1. The contingency matrices have entity ID’s as rows, and time intervals enumerating values of time intervals.

The following flow-chart corresponds to the list of steps above.

A corresponding monadic pipeline is given in the section “Larger example pipeline”.

### Feature engineering perspective

The workflow above describes a way to do feature engineering over a collection of event records data. For a given entity ID and a variable we derive several different time series.

Couple of examples follow.

• One possible derived feature (times series) is for each entity-variable pair we make time series of the hourly mean value in each of the eight most recent hours for that entity. The mean values are normalized by the average values of the records corresponding to that entity-variable pair.

• Another possible derived feature (time series) is for each entity-variable pair to make a time series with the number of outliers in the each half-hour interval, considering the most recent 20 half-hour intervals. The outliers are found by using outlier boundaries derived by analyzing all values of the corresponding variable, across all entities.

From the examples above – and some others – we conclude that for each feature we want to be able to specify:

• maximum history length (say from the most recent observation),

• aggregation interval length,

• aggregation function (to be applied in each interval),

• normalization function (per entity, per cohort of entities, per variable),

• conversion of categorical values into numerical ones.

### Repeated execution

We want to be able to do repeated executions of the specified workflow steps.

Consider the following scenario. After the event records data is converted to a entity-vs-feature contingency matrix, we use that matrix to train and test a classifier. We want to find the combination of features that gives the best classifier results. For that reason we want to be able to easily and systematically change the computation specifications (interval size, aggregation and normalization functions, etc.) With different computation specifications we obtain different entity-vs-feature contingency matrices, that would have different performance with different classifiers.

Using the classifier training and testing scenario we see that there is another repeated execution perspective: after the feature engineering is done over the training data, we want to be able to execute exactly the same steps over the test data. Note that with the training data we find certain global or cohort normalization values and outlier boundaries that have to be used over the test data. (Not derived from the test data.)

The following diagram further describes the repeated execution workflow.

Further discussion of making and using ML classification workflows through the monad software design pattern can be found in [AA2].

## Event records data design

The data is structured to follow the style of Star schema. We have event records dataset (table) and entity attributes dataset (table).

The structure datasets (tables) proposed satisfy a wide range of modeling data requirements. (Medical and financial modeling included.)

### Entity event data

The entity event data has the columns “EntityID”, “LocationID”, “ObservationTime”, “Variable”, “Value”.

``RandomSample[eventRecords, 6]``

Most events can be described through “Entity event data”. The entities can be anything that produces a set of event data: financial transactions, vital sign monitors, wind speed sensors, chemical concentrations sensors.

The locations can be anything that gives the events certain “spatial” attributes: medical units in hospitals, sensors geo-locations, tiers of financial transactions.

### Entity attributes data

The entity attributes dataset (table) has attributes (immutable properties) of the entities. (Like, gender and race for people, longitude and latitude for wind speed sensors.)

``entityAttributes[[1 ;; 6]]``

### Example

For example, here we take all weather stations in USA:

``````ws = Normal[entityAttributes[Select[#Attribute == "Country" && #Value == "USA" &], "EntityID"]]

(* {"KMFL", "KMDW", "KNIP", "KGEU"} *)``````

Here we take all temperature event records for those weather stations:

``srecs = eventRecords[Select[#Variable == "Temperature" && MemberQ[ws, #EntityID] &]];``

And here plot the corresponding time series obtained by grouping the records by station (entity ID’s) and taking the columns “ObservationTime” and “Value”:

``````grecs = Normal @ GroupBy[srecs, #EntityID &][All, All, {"ObservationTime", "Value"}];
DateListPlot[grecs, ImageSize -> Large, PlotTheme -> "Detailed"]``````

This section goes through the steps of the general `ERTMon` workflow. For didactic purposes each sub-section changes the pipeline assigned to the variable `p`. Of course all functions can be chained into one big pipeline as shown in the section “Larger example pipeline”.

The monad is initialized with ERTMonUnit.

``````ERTMonUnit[]

(* ERTMon[None, <||>] *)``````

### Ingesting event records and entity attributes

The event records dataset (table) and entity attributes dataset (table) are set with corresponding setter functions. Alternatively, they can be read from files in a specified directory.

``````p =
ERTMonUnit[]⟹
ERTMonSetEventRecords[eventRecords]⟹
ERTMonSetEntityAttributes[entityAttributes]⟹
ERTMonEchoDataSummary;
``````

### Computation specification

Using the package [AAp3] we can create computation specification dataset. Below is given an example of constructing a fairly complicated computation specification.

The package function `EmptyComputationSpecificationRow` can be used to construct the rows of the specification.

``````EmptyComputationSpecificationRow[]

(* <|"Variable" -> Missing[], "Explanation" -> "",
"MaxHistoryLength" -> 3600, "AggregationIntervalLength" -> 60,
"AggregationFunction" -> "Mean", "NormalizationScope" -> "Entity",
"NormalizationFunction" -> "None"|> *)

compSpecRows =
Join[EmptyComputationSpecificationRow[], <|"Variable" -> #,
"MaxHistoryLength" -> 60*24*3600,
"AggregationIntervalLength" -> 2*24*3600,
"AggregationFunction" -> "Mean",
"NormalizationScope" -> "Entity",
"NormalizationFunction" -> "Mean"|>] & /@
Union[Normal[eventRecords[All, "Variable"]]];
compSpecRows =
Join[
compSpecRows,
Join[EmptyComputationSpecificationRow[], <|"Variable" -> #,
"MaxHistoryLength" -> 60*24*3600,
"AggregationIntervalLength" -> 2*24*3600,
"AggregationFunction" -> "Range",
"NormalizationScope" -> "Country",
"NormalizationFunction" -> "Mean"|>] & /@
Union[Normal[eventRecords[All, "Variable"]]],
Join[EmptyComputationSpecificationRow[], <|"Variable" -> #,
"MaxHistoryLength" -> 60*24*3600,
"AggregationIntervalLength" -> 2*24*3600,
"AggregationFunction" -> "OutliersCount",
"NormalizationScope" -> "Variable"|>] & /@
Union[Normal[eventRecords[All, "Variable"]]]
];``````

The constructed rows are assembled into a dataset (with `Dataset`). The function `ProcessComputationSpecification` is used to convert a user-made specification dataset into a form used by `ERTMon`.

``````wCompSpec =
ProcessComputationSpecification[Dataset[compSpecRows]][SortBy[#Variable &]]
``````

The computation specification is set to the monad with the function `ERTMonSetComputationSpecification`.

Alternatively, a computation specification can be created and filled-in as a CSV file and read into the monad. (Not described here.)

### Grouping event records by entity-variable pairs

With the function `ERTMonGroupEntityVariableRecords` we group the event records by the found unique entity-variable pairs. Note that in the pipeline below we set the computation specification first.

``````p =
p⟹
ERTMonSetComputationSpecification[wCompSpec]⟹
ERTMonGroupEntityVariableRecords;``````

### Descriptive statistics (per variable)

After the data is ingested into the monad and the event records are grouped per entity-variable pairs we can find certain descriptive statistics for the data. This is done with the general function ERTMonComputeVariableStatistic and the specialized function ERTMonFindVariableOutlierBoundaries.

``p⟹ERTMonComputeVariableStatistic[RecordsSummary]⟹ERTMonEchoValue;``
``p⟹ERTMonComputeVariableStatistic⟹ERTMonEchoValue;``
``````p⟹ERTMonComputeVariableStatistic[TakeLargest[#, 3] &]⟹ERTMonEchoValue;

(* value: <|Humidity->{1.,1.,0.993}, MaxTemperature->{48,48,48},
Pressure->{1043.1,1042.8,1041.1}, Temperature->{42.28,41.94,41.89},
WindSpeed->{54.82,44.63,44.08}|> *)

``````

### Finding the variables outlier boundaries

The finding of outliers counts and fractions can be specified in the computation specification. Because of this there is a specialized function for outlier finding `ERTMonFindVariableOutlierBoundaries`. That function makes the association of the found variable outlier boundaries (i) to be the pipeline value and (ii) to be the value of context key “variableOutlierBoundaries”. The outlier boundaries are found using the functions of the package [AAp6].

If no argument is specified ERTMonFindVariableOutlierBoundaries uses the Hampel identifier (`HampelIdentifierParameters`).

``````p⟹ERTMonFindVariableOutlierBoundaries⟹ERTMonEchoValue;

(* value: <|Humidity->{0.522536,0.869464}, MaxTemperature->{14.2106,31.3494},
Pressure->{1012.36,1022.44}, Temperature->{9.88823,28.3318},
WindSpeed->{5.96141,19.4086}|> *)

Keys[p⟹ERTMonFindVariableOutlierBoundaries⟹ERTMonTakeContext]

(* {"eventRecords", "entityAttributes", "computationSpecification",
"entityVariableRecordGroups", "variableOutlierBoundaries"} *)
``````

In the rest of document we use the outlier boundaries found with the more conservative identifier `SPLUSQuartileIdentifierParameters`.

``````p =
p⟹
ERTMonFindVariableOutlierBoundaries[SPLUSQuartileIdentifierParameters]⟹
ERTMonEchoValue;

(* value: <|Humidity->{0.176,1.168}, MaxTemperature->{-1.67,45.45},
Pressure->{1003.75,1031.35}, Temperature->{-5.805,43.755},
WindSpeed->{-5.005,30.555}|> *)``````

### Conversion of event records to time series

The grouped event records are converted into time series with the function `ERTMonEntityVariableGroupsToTimeSeries`. The time series are aligned to a time point specification given as an argument. The argument can be: a date object, “MinTime”, “MaxTime”, or “None”. (“MaxTime” is the default.)

``````p⟹
ERTMonEntityVariableGroupsToTimeSeries["MinTime"]⟹
ERTMonEchoFunctionContext[#timeSeries[[{1, 3, 5}]] &];
``````

Compare the last output with the output of the following command.

``````p =
p⟹
ERTMonEntityVariableGroupsToTimeSeries["MaxTime"]⟹
ERTMonEchoFunctionContext[#timeSeries[[{1, 3, 5}]] &];``````

### Time series restriction and aggregation.

The main goal of `ERTMon` is to convert a diverse, general collection of event records into a collection of aligned time series over specified regular time grids.

The regular time grids are specified with the columns “MaxHistoryLength” and “AggregationIntervalLength” of the computation specification. The time series of the variables in the computation specification are restricted to the corresponding maximum history lengths and are aggregated using the corresponding aggregation lengths and functions.

``````p =
p⟹
ERTMonAggregateTimeSeries⟹
ERTMonEchoFunctionContext[DateListPlot /@ #timeSeries[[{1, 3, 5}]] &];
``````

### Application of time series functions

At this point we can apply time series modifying functions. An often used such function is moving average.

``````p⟹
ERTMonApplyTimeSeriesFunction[MovingAverage[#, 6] &]⟹
ERTMonEchoFunctionValue[DateListPlot /@ #[[{1, 3, 5}]] &];``````

Note that the result is given as a pipeline value, the value of the context key “timeSeries” is not changed.

(In the future, the computation specification and its handling might be extended to handle moving average or other time series function specifications.)

### Normalization

With “normalization” we mean that the values of a given time series values are divided (normalized) with a descriptive statistic derived from a specified set of values. The specified set of values is given with the parameter “NormalizationScope” in computation specification.

At the normalization stage each time series is associated with an entity ID and a variable.

Normalization is done at three different scopes: “entity”, “attribute”, and “variable”.

Given a time series $T(i,var)$ corresponding to entity ID $i$ and a variable $var$ we define the normalization values for the different scopes in the following ways.

• Normalization with scope “entity” means that the descriptive statistic is derived from the values of $T(i,var)$ only.

• Normalization with scope attribute means that

• from the entity attributes dataset we find attribute value that corresponds to $i$,

• next we find all entity ID’s that are associated with the same attribute value,

• next we find value of normalization descriptive statistic using the time series that correspond to the variable $var$ and the entity ID’s found in the previous step.

• Normalization with scope “variable” means that the descriptive statistic is derived from the values of all time series corresponding to $var$.

Note that the scope “entity” is the most granular, and the scope “variable” is the coarsest.

The following command demonstrates the normalization effect – compare the $y$-axes scales of the time series corresponding to the same entity-variable pair.

``````p =
p⟹
ERTMonEchoFunctionContext[DateListPlot /@ #timeSeries[[{1, 3, 5}]] &]⟹
ERTMonNormalize⟹
ERTMonEchoFunctionContext[DateListPlot /@ #timeSeries[[{1, 3, 5}]] &];
``````

Here are the normalization values that should be used when normalizing “unseen data.”

``````p⟹ERTMonTakeNormalizationValues

(* <|{"Humidity.Range", "Country", "USA"} -> 0.0864597,
{"Humidity.Range", "Country", "UK"} -> 0.066,
{"Humidity.Range", "Country", "Australia"} -> 0.145968,
{"MaxTemperature.Range", "Country", "USA"} -> 2.85468,
{"MaxTemperature.Range", "Country", "UK"} -> 78/31,
{"MaxTemperature.Range", "Country", "Australia"} -> 3.28871,
{"Pressure.Range", "Country", "USA"} -> 2.08222,
{"Pressure.Range", "Country", "Australia"} -> 3.33871,
{"Temperature.Range", "Country", "USA"} -> 2.14411,
{"Temperature.Range", "Country", "UK"} -> 1.25806,
{"Temperature.Range", "Country", "Australia"} -> 2.73032,
{"WindSpeed.Range", "Country", "USA"} -> 4.13532,
{"WindSpeed.Range", "Country", "UK"} -> 3.62097,
{"WindSpeed.Range", "Country", "Australia"} -> 3.17226|> *)``````

### Making contingency matrices

One of the main goals of `ERTMon` is to produce contingency matrices corresponding to the event records data.

The contingency matrices are created and stored as `SSparseMatrix` objects, [AAp7].

``````p =
p⟹ERTMonMakeContingencyMatrices;``````

We can obtain an association of the contingency matrices for each variable-and-aggregation-function pair, or obtain the overall contingency matrix.

``````p⟹ERTMonTakeContingencyMatrices
Dimensions /@ %``````
``````smat = p⟹ERTMonTakeContingencyMatrix;
MatrixPlot[smat, ImageSize -> 700]``````
``````RowNames[smat]

(* {"EGLC", "KGEU", "KMDW", "KMFL", "KNIP", "WMO95866"} *)``````

## Larger example pipeline

The pipeline shown in this section utilizes all main workflow functions of `ERTMon`. The used weather data and computation specification are described above.

## References

### Packages

[AAp7] Anton Antonov, SSparseMatrix Mathematica package, (2018), MathematicaForPrediction at GitHub*. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/SSparseMatrix.m .

### Documents

[AA1a] Anton Antonov, Monad code generation and extension, (2017), MathematicaForPrediction at GitHub.

# A monad for Quantile Regression workflows

## Introduction

In this document we describe the design and implementation of a (software programming) monad for Quantile Regression workflows specification and execution. The design and implementation are done with Mathematica / Wolfram Language (WL).

What is Quantile Regression? : Assume we have a set of two dimensional points each point being a pair of an independent variable value and a dependent variable value. We want to find a curve that is a function of the independent variable that splits the points in such a way that, say, 30% of the points are above that curve. This is done with Quantile Regression, see [Wk2, CN1, AA2, AA3]. Quantile Regression is a method to estimate the variable relations for all parts of the distribution. (Not just, say, the mean of the relationships found with Least Squares Regression.)

The goal of the monad design is to make the specification of Quantile Regression workflows (relatively) easy, straightforward, by following a certain main scenario and specifying variations over that scenario. Since Quantile Regression is often compared with Least Squares Regression and some type of filtering (like, Moving Average) those functionalities should be included in the monad design scenarios.

The monad is named QRMon and it is based on the State monad package "StateMonadCodeGenerator.m", [AAp1, AA1] and the Quantile Regression package "QuantileRegression.m", [AAp4, AA2].

The data for this document is read from WL’s repository or created ad-hoc.

The monadic programming design is used as a Software Design Pattern. The QRMon monad can be also seen as a Domain Specific Language (DSL) for the specification and programming of machine learning classification workflows.

Here is an example of using the QRMon monad over heteroscedastic data::

The table above is produced with the package "MonadicTracing.m", [AAp2, AA1], and some of the explanations below also utilize that package.

As it was mentioned above the monad QRMon can be seen as a DSL. Because of this the monad pipelines made with QRMon are sometimes called "specifications".

Remark: With "regression quantile" we mean "a curve or function that is computed with Quantile Regression".

### Contents description

The document has the following structure.

• The sections "Package load" and "Data load" obtain the needed code and data.
• The sections "Design consideration" and "Monad design" provide motivation and design decisions rationale.

• The sections "QRMon overview" and "Monad elements" provide technical description of the QRMon monad needed to utilize it.

• (Using a fair amount of examples.)
• The section "Unit tests" describes the tests used in the development of the QRMon monad.
• (The random pipelines unit tests are especially interesting.)
• The section "Future plans" outlines future directions of development.
• The section "Implementation notes" just says that QRMon’s development process and this document follow the ones of the classifications workflows monad `ClCon`, [AA6].

Remark: One can read only the sections "Introduction", "Design consideration", "Monad design", and "QRMon overview". That set of sections provide a fairly good, programming language agnostic exposition of the substance and novel ideas of this document.

The table above is produced with the package "MonadicTracing.m", [AAp2, AA1], and some of the explanations below also utilize that package.

As it was mentioned above the monad QRMon can be seen as a DSL. Because of this the monad pipelines made with QRMon are sometimes called "specifications".

Remark: With "regression quantile" we mean "a curve or function that is computed with Quantile Regression".

The following commands load the packages [AAp1–AAp6]:

``````Import["https://raw.githubusercontent.com/antononcube/\
Import["https://raw.githubusercontent.com/antononcube/\

In this section we load data that is used in the rest of the document. The time series data is obtained through WL’s repository.

The data summarization and plots are done through QRMon, which in turn uses the function RecordsSummary from the package "MathematicaForPredictionUtilities.m", [AAp6].

### Distribution data

The following data is generated to have [heteroscedasticity(https://en.wikipedia.org/wiki/Heteroscedasticity).

``````distData =
Table[{x,
Exp[-x^2] +
RandomVariate[
NormalDistribution[0, .15 Sqrt[Abs[1.5 - x]/1.5]]]}, {x, -3,
3, .01}];
Length[distData]

(* 601 *)

QRMonUnit[distData]⟹QRMonEchoDataSummary⟹QRMonPlot;``````

### Temperature time series

``````tsData = WeatherData[{"Orlando", "USA"}, "Temperature", {{2015, 1, 1}, {2018, 1, 1}, "Day"}]

QRMonUnit[tsData]⟹QRMonEchoDataSummary⟹QRMonDateListPlot;``````

### Financial time series

The following data is typical for financial time series. (Note the differences with the temperature time series.)

``````finData = TimeSeries[FinancialData["NYSE:GE", {{2014, 1, 1}, {2018, 1, 1}, "Day"}]];

QRMonUnit[finData]⟹QRMonEchoDataSummary⟹QRMonDateListPlot;``````

## Design considerations

The steps of the main regression workflow addressed in this document follow.

1. Retrieving data from a data repository.

2. Optionally, transform the data.

1. Delete rows with missing fields.

2. Rescale data along one or both of the axes.

3. Apply moving average (or median, or map.)

3. Verify assumptions of the data.

4. Run a regression algorithm with a certain basis of functions using:

1. Quantile Regression, or

2. Least Squares Regression.

5. Visualize the data and regression functions.

6. If the regression functions fit is not satisfactory go to step 4.

7. Utilize the found regression functions to compute:

1. outliers,

2. local extrema,

3. approximation or fitting errors,

4. conditional density distributions,

5. time series simulations.

The following flow-chart corresponds to the list of steps above.

• the introduction of new elements in regression workflows,

• workflows elements variability, and

• workflows iterative changes and refining,

it is beneficial to have a DSL for regression workflows. We choose to make such a DSL through a functional programming monad, [Wk1, AA1].

Here is a quote from [Wk1] that fairly well describes why we choose to make a classification workflow monad and hints on the desired properties of such a monad.

[…] The monad represents computations with a sequential structure: a monad defines what it means to chain operations together. This enables the programmer to build pipelines that process data in a series of steps (i.e. a series of actions applied to the data), in which each action is decorated with the additional processing rules provided by the monad. […] Monads allow a programming style where programs are written by putting together highly composable parts, combining in flexible ways the possible actions that can work on a particular type of data. […]

Remark: Note that quote from [Wk1] refers to chained monadic operations as "pipelines". We use the terms "monad pipeline" and "pipeline" below.

The monad we consider is designed to speed-up the programming of quantile regression workflows outlined in the previous section. The monad is named QRMon for "Quantile Regression Monad".

We want to be able to construct monad pipelines of the general form:

QRMon is based on the State monad, [Wk1, AA1], so the monad pipeline form (1) has the following more specific form:

This means that some monad operations will not just change the pipeline value but they will also change the pipeline context.

In the monad pipelines of QRMon we store different objects in the contexts for at least one of the following two reasons.

1. The object will be needed later on in the pipeline, or

2. The object is (relatively) hard to compute.

Such objects are transformed data, regression functions, and outliers.

Let us list the desired properties of the monad.

• Rapid specification of non-trivial quantile regression workflows.

• The monad works with time series, numerical matrices, and numerical vectors.

• The pipeline values can be of different types. Most monad functions modify the pipeline value; some modify the context; some just echo results.

• The monad can do quantile regression with B-Splines bases, quantile regression fit and least squares fit with specified bases of functions.

• The monad allows of cursory examination and summarization of the data.

• It is easy to obtain the pipeline value, context, and different context objects for manipulation outside of the monad.

• It is easy to plot different combinations of data, regression functions, outliers, approximation errors, etc.

The QRMon components and their interactions are fairly simple.

The main QRMon operations implicitly put in the context or utilize from the context the following objects:

• (time series) data,

• regression functions,

• outliers and outlier regression functions.

Note the that the monadic set of types of QRMon pipeline values is fairly heterogenous and certain awareness of "the current pipeline value" is assumed when composing QRMon pipelines.

Obviously, we can put in the context any object through the generic operations of the State monad of the package "StateMonadGenerator.m", [AAp1].

## QRMon overview

When using a monad we lift certain data into the "monad space", using monad’s operations we navigate computations in that space, and at some point we take results from it.

With the approach taken in this document the "lifting" into the QRMon monad is done with the function QRMonUnit. Results from the monad can be obtained with the functions QRMonTakeValue, QRMonContext, or with the other QRMon functions with the prefix "QRMonTake" (see below.)

Here is a corresponding diagram of a generic computation with the QRMon monad:

Remark: It is a good idea to compare the diagram with formulas (1) and (2).

Let us examine a concrete QRMon pipeline that corresponds to the diagram above. In the following table each pipeline operation is combined together with a short explanation and the context keys after its execution.

Here is the output of the pipeline:

The QRMon functions are separated into four groups:

• operations,

• setters and droppers,

• takers,

An overview of the those functions is given in the tables in next two sub-sections. The next section, "Monad elements", gives details and examples for the usage of the QRMon operations.

### Monad functions interaction with the pipeline value and context

The following table gives an overview the interaction of the QRMon monad functions with the pipeline value and context.

The following table shows the functions that are function synonyms or short-cuts.

Here are the QRMon State Monad functions (generated using the prefix "QRMon", [AAp1, AA1]):

In this section we show that QRMon has all of the properties listed in the previous section.

The monad head is QRMon. Anything wrapped in QRMon can serve as monad’s pipeline value. It is better though to use the constructor QRMonUnit. (Which adheres to the definition in [Wk1].)

``QRMon[{{1, 223}, {2, 323}}, <||>]⟹QRMonEchoDataSummary;``

### Lifting data to the monad

The function lifting the data into the monad QRMon is QRMonUnit.

The lifting to the monad marks the beginning of the monadic pipeline. It can be done with data or without data. Examples follow.

``QRMonUnit[distData]⟹QRMonEchoDataSummary;``
``QRMonUnit[]⟹QRMonSetData[distData]⟹QRMonEchoDataSummary;``

(See the sub-section "Setters, droppers, and takers" for more details of setting and taking values in QRMon contexts.)

Currently the monad can deal with data in the following forms:

• time series,

• numerical vectors,

• numerical matrices of rank two.

When the data lifted to the monad is a numerical vector vec it is assumed that vec has to become the second column of a "time series" matrix; the first column is derived with Range[Length[vec]] .

Generally, WL makes it easy to extract columns datasets order to obtain numerical matrices, so datasets are not currently supported in QRMon.

### Quantile regression with B-splines

This computes quantile regression with B-spline basis over 12 regularly spaced knots. (Using Linear Programming algorithms; see [AA2] for details.)

``````QRMonUnit[distData]⟹
QRMonQuantileRegression[12]⟹
QRMonPlot;``````

The monad function QRMonQuantileRegression has the same options as QuantileRegression. (The default value for option Method is different, since using "CLP" is generally faster.)

``````Options[QRMonQuantileRegression]

(* {InterpolationOrder -> 3, Method -> {LinearProgramming, Method -> "CLP"}} *)``````

Let us compute regression using a list of particular knots, specified quantiles, and the method "InteriorPoint" (instead of the Linear Programming library CLP):

``````p =
QRMonUnit[distData]⟹
QRMonQuantileRegression[{-3, -2, 1, 0, 1, 1.5, 2.5, 3}, Range[0.1, 0.9, 0.2], Method -> {LinearProgramming, Method -> "InteriorPoint"}]⟹
QRMonPlot;``````

Remark: As it was mentioned above the function QRMonRegression is a synonym of QRMonQuantileRegression.

The fit functions can be extracted from the monad with QRMonTakeRegressionFunctions, which gives an association of quantiles and pure functions.

``ListPlot[# /@ distData[[All, 1]]] & /@ (p⟹QRMonTakeRegressionFunctions)``

### Quantile regression fit and Least squares fit

Instead of using a B-spline basis of functions we can compute a fit with our own basis of functions.

Here is a basis functions:

``````bFuncs = Table[PDF[NormalDistribution[m, 1], x], {m, Min[distData[[All, 1]]], Max[distData[[All, 1]]], 1}];
Plot[bFuncs, {x, Min[distData[[All, 1]]], Max[distData[[All, 1]]]},
PlotRange -> All, PlotTheme -> "Scientific"]``````

Here we do a Quantile Regression fit, a Least Squares fit, and plot the results:

``````p =
QRMonUnit[distData]⟹
QRMonQuantileRegressionFit[bFuncs]⟹
QRMonLeastSquaresFit[bFuncs]⟹
QRMonPlot;
``````

Remark: The functions "QRMon*Fit" should generally have a second argument for the symbol of the basis functions independent variable. Often that symbol can be omitted and implied. (Which can be seen in the pipeline above.)

Remark: As it was mentioned above the function QRMonRegressionFit is a synonym of QRMonQuantileRegressionFit and QRMonFit is a synonym of QRMonLeastSquaresFit.

As it was pointed out in the previous sub-section, the fit functions can be extracted from the monad with QRMonTakeRegressionFunctions. Here the keys of the returned/taken association consist of quantiles and "mean" since we applied both Quantile Regression and Least Squares Regression.

``ListPlot[# /@ distData[[All, 1]]] & /@ (p⟹QRMonTakeRegressionFunctions)``

### Default basis to fit (using Chebyshev polynomials)

One of the main advantages of using the function QuanileRegression of the package [AAp4] is that the functions used to do the regression with are specified with a few numeric parameters. (Most often only the number of knots is sufficient.) This is achieved by using a basis of B-spline functions of a certain interpolation order.

We want similar behaviour for Quantile Regression fitting we need to select a certain well known basis with certain desirable properties. Such basis is given by Chebyshev polynomials of first kind [Wk3]. Chebyshev polynomials bases can be easily generated in Mathematica with the functions ChebyshevT or ChebyshevU.

Here is an application of fitting with a basis of 12 Chebyshev polynomials of first kind:

``````QRMonUnit[distData]⟹
QRMonQuantileRegressionFit[12]⟹
QRMonLeastSquaresFit[12]⟹
QRMonPlot;``````

The code above is equivalent to the following code:

``````bfuncs = Table[ChebyshevT[i, Rescale[x, MinMax[distData[[All, 1]]], {-0.95, 0.95}]], {i, 0, 12}];

p =
QRMonUnit[distData]⟹
QRMonQuantileRegressionFit[bfuncs]⟹
QRMonLeastSquaresFit[bfuncs]⟹
QRMonPlot;``````

The shrinking of the ChebyshevT domain seen in the definitions of bfuncs is done in order to prevent approximation error effects at the ends of the data domain. The following code uses the ChebyshevT domain { − 1, 1} instead of the domain { − 0.95, 0.95} used above.

``````QRMonUnit[distData]⟹
QRMonQuantileRegressionFit[{4, {-1, 1}}]⟹
QRMonPlot;``````

### Regression functions evaluation

The computed quantile and least squares regression functions can be evaluated with the monad function QRMonEvaluate.

Evaluation for a given value of the independent variable:

``````p⟹QRMonEvaluate[0.12]⟹QRMonTakeValue

(* <|0.25 -> 0.930402, 0.5 -> 1.01411, 0.75 -> 1.08075, "mean" -> 0.996963|> *)``````

Evaluation for a vector of values:

``````p⟹QRMonEvaluate[Range[-1, 1, 0.5]]⟹QRMonTakeValue

(* <|0.25 -> {0.258241, 0.677461, 0.943299, 0.703812, 0.293741},
0.5 -> {0.350025, 0.768617, 1.02311, 0.807879, 0.374545},
0.75 -> {0.499338, 0.912183, 1.10325, 0.856729, 0.431227},
"mean" -> {0.355353, 0.776006, 1.01118, 0.783304, 0.363172}|> *)``````

Evaluation for complicated lists of numbers:

``````p⟹QRMonEvaluate[{0, 1, {1.5, 1.4}}]⟹QRMonTakeValue

(* <|0.25 -> {0.943299, 0.293741, {0.0762883, 0.10759}},
0.5 -> {1.02311, 0.374545, {0.103386, 0.139142}},
0.75 -> {1.10325, 0.431227, {0.133755, 0.177161}},
"mean" -> {1.01118, 0.363172, {0.107989, 0.142021}}|> *)
``````

The obtained values can be used to compute estimates of the distributions of the dependent variable. See the sub-sections "Estimating conditional distributions" and "Dependent variable simulation".

### Errors and error plots

Here with "errors" we mean the differences between data’s dependent variable values and the corresponding values calculated with the fitted regression curves.

In the pipeline below we compute couple of regression quantiles, plot them together with the data, we plot the errors, compute the errors, and summarize them.

``````QRMonUnit[finData]⟹
QRMonQuantileRegression[10, {0.5, 0.1}]⟹
QRMonDateListPlot[Joined -> False]⟹
QRMonErrorPlots["DateListPlot" -> True, Joined -> False]⟹
QRMonErrors⟹
QRMonEchoFunctionValue["Errors summary:", RecordsSummary[#[[All, 2]]] & /@ # &];``````

Each of the functions QRMonErrors and QRMonErrorPlots computes the errors. (That computation is considered cheap.)

### Finding outliers

Finding outliers can be done with the function QRMonOultiers. The outliers found by QRMonOutliers are simply points that below or above certain regression quantile curves, for example, the ones corresponding to 0.02 and 0.98.

Here is an example:

``````p =
QRMonUnit[distData]⟹
QRMonQuantileRegression[6, {0.02, 0.98}]⟹
QRMonOutliers⟹
QRMonEchoValue⟹
QRMonOutliersPlot;``````

The function QRMonOutliers puts in the context values for the keys "outliers" and "outlierRegressionFunctions". The former is for the found outliers, the latter is for the functions corresponding to the used regression quantiles.

``````Keys[p⟹QRMonTakeContext]

(* {"data", "regressionFunctions", "outliers", "outlierRegressionFunctions"} *)``````

Here are the corresponding quantiles of the plot above:

``````Keys[p⟹QRMonTakeOutlierRegressionFunctions]

(* {0.02, 0.98} *)``````

The control of the outliers computation is done though the arguments and options of QRMonQuantileRegression (or the rest of the regression calculation functions.)

If only one regression quantile is found in the context and the corresponding quantile is less than 0.5 then QRMonOutliers finds only bottom outliers. If only one regression quantile is found in the context and the corresponding quantile is greater than 0.5 then QRMonOutliers finds only top outliers.

Here is an example for finding only the top outliers:

``````QRMonUnit[finData]⟹
QRMonQuantileRegression[5, 0.8]⟹
QRMonOutliers⟹
QRMonEchoFunctionContext["outlier quantiles:", Keys[#outlierRegressionFunctions] &]⟹
QRMonOutliersPlot["DateListPlot" -> True];
``````

### Plotting outliers

The function QRMonOutliersPlot makes an outliers plot. If the outliers are not in the context then QRMonOutliersPlot calls QRMonOutliers first.

Here are the options of QRMonOutliersPlot:

``````Options[QRMonOutliersPlot]

(* {"Echo" -> True, "DateListPlot" -> False, ListPlot -> {Joined -> False}, Plot -> {}} *)``````

The default behavior is to echo the plot. That can be suppressed with the option "Echo".

QRMonOutliersPlot utilizes combines with Show two plots:

• one with ListPlot (or DateListPlot) for the data and the outliers,

• the other with Plot for the regression quantiles used to find the outliers.

That is why separate lists of options can be given to manipulate those two plots. The option DateListPlot can be used make plots with date or time axes.

``````QRMonUnit[tsData]⟹
QRMonQuantileRegression[12, {0.01, 0.99}]⟹
QRMonOutliersPlot[
"Echo" -> False,
"DateListPlot" -> True,
ListPlot -> {PlotStyle -> {Green, {PointSize[0.02],
Red}, {PointSize[0.02], Blue}}, Joined -> False,
PlotTheme -> "Grid"},
Plot -> {PlotStyle -> Orange}]⟹
QRMonTakeValue
``````

### Estimating conditional distributions

Consider the following problem:

How to estimate the conditional density of the dependent variable given a value of the conditioning independent variable?

(In other words, find the distribution of the y-values for a given, fixed x-value.)

The solution of this problem using Quantile Regression is discussed in detail in [PG1] and [AA4].

Finding a solution for this problem can be seen as a primary motivation to develop Quantile Regression algorithms.

The following pipeline (i) computes and plots a set of five regression quantiles and (ii) then using the found regression quantiles computes and plots the conditional distributions for two focus points (−2 and 1.)

``````QRMonUnit[distData]⟹
QRMonQuantileRegression[6,
Range[0.1, 0.9, 0.2]]⟹
QRMonPlot[GridLines -> {{-2, 1}, None}]⟹
QRMonConditionalCDF[{-2, 1}]⟹
QRMonConditionalCDFPlot;``````

### Moving average, moving median, and moving map

Fairly often it is a good idea for a given time series to apply filter functions like Moving Average or Moving Median. We might want to:

• visualize the obtained transformed data,

• do regression over the transformed data,

• compare with regression curves over the original data.

For these reasons QRMon has the functions QRMonMovingAverage, QRMonMovingMedian, and QRMonMovingMap that correspond to the built-in functions MovingAverage, MovingMedian, and MovingMap.

Here is an example:

``````QRMonUnit[tsData]⟹
QRMonDateListPlot[ImageSize -> Small]⟹
QRMonMovingAverage[20]⟹
QRMonEchoFunctionValue["Moving avg: ", DateListPlot[#, ImageSize -> Small] &]⟹
QRMonMovingMap[Mean, Quantity[20, "Days"]]⟹
QRMonEchoFunctionValue["Moving map: ", DateListPlot[#, ImageSize -> Small] &];``````

### Dependent variable simulation

Consider the problem of making a time series that is a simulation of a process given with a known time series.

More formally,

• we are given a time-axis grid (regular or irregular),

• we consider each grid node to correspond to a random variable,

• we want to generate time series based on the empirical CDF’s of the random variables that correspond to the grid nodes.

The formulation of the problem hints to an (almost) straightforward implementation using Quantile Regression.

``````p = QRMonUnit[tsData]⟹QRMonQuantileRegression[30, Join[{0.01}, Range[0.1, 0.9, 0.1], {0.99}]];

tsNew =
p⟹
QRMonSimulate[1000]⟹
QRMonTakeValue;

opts = {ImageSize -> Medium, PlotTheme -> "Detailed"};
GraphicsGrid[{{DateListPlot[tsData, PlotLabel -> "Actual", opts],
DateListPlot[tsNew, PlotLabel -> "Simulated", opts]}}]``````

### Finding local extrema in noisy data

Using regression fitting — and Quantile Regression in particular — we can easily construct semi-symbolic algorithms for finding local extrema in noisy time series data; see [AA5]. The QRMon function with such an algorithm is QRMonLocalExtrema.

In brief, the algorithm steps are as follows. (For more details see [AA5].)

1. Fit a polynomial through the data.

2. Find the local extrema of the fitted polynomial. (We will call them fit estimated extrema.)

3. Around each of the fit estimated extrema find the most extreme point in the data by a nearest neighbors search (by using Nearest).

The function QRMonLocalExtrema uses the regression quantiles previously found in the monad pipeline (and stored in the context.) The bottom regression quantile is used for finding local minima, the top regression quantile is used for finding the local maxima.

An example of finding local extrema follows.

``````QRMonUnit[TimeSeriesWindow[tsData, {{2015, 1, 1}, {2018, 12, 31}}]]⟹
QRMonQuantileRegression[10, {0.05, 0.95}]⟹
QRMonDateListPlot[Joined -> False, PlotTheme -> "Scientific"]⟹
QRMonLocalExtrema["NumberOfProximityPoints" -> 100]⟹
QRMonEchoValue⟹
QRMonEchoFunctionContext[
DateListPlot[{#localMinima, #localMaxima, #data},
PlotStyle -> {PointSize[0.015], PointSize[0.015], Gray},
Joined -> False,
PlotLegends -> {"localMinima", "localMaxima", "data"},
PlotTheme -> "Scientific"] &];``````

Note that in the pipeline above in order to plot the data and local extrema together some additional steps are needed. The result of QRMonLocalExtrema becomes the pipeline value; that pipeline value is displayed with QRMonEchoValue, and stored in the context with QRMonAddToContext. If the pipeline value is an association — which is the case here — the monad function QRMonAddToContext joins that association with the context association. In this case this means that we will have key-value elements in the context for "localMinima" and "localMaxima". The date list plot at the end of the pipeline uses values of those context keys (together with the value for "data".)

### Setters, droppers, and takers

The values from the monad context can be set, obtained, or dropped with the corresponding "setter", "dropper", and "taker" functions as summarized in a previous section.

For example:

``````p = QRMonUnit[distData]⟹QRMonQuantileRegressionFit[2];

p⟹QRMonTakeRegressionFunctions

(* <|0.25 -> (0.0191185 + 0.00669159 #1 + 3.05509*10^-14 #1^2 &),
0.5 -> (0.191408 + 9.4728*10^-14 #1 + 3.02272*10^-14 #1^2 &),
0.75 -> (0.563422 + 3.8079*10^-11 #1 + 7.63637*10^-14 #1^2 &)|> *)
``````

If other values are put in the context they can be obtained through the (generic) function QRMonTakeContext, [AAp1]:

``````p = QRMonUnit[RandomReal[1, {2, 2}]]⟹QRMonAddToContext["data"];

(p⟹QRMonTakeContext)["data"]

(* {{0.608789, 0.741599}, {0.877074, 0.861554}} *)``````

Another generic function from [AAp1] is QRMonTakeValue (used many times above.)

Here is an example of the "data dropper" QRMonDropData:

``````p⟹QRMonDropData⟹QRMonTakeContext

(* <||> *)``````

(The "droppers" simply use the state monad function QRMonDropFromContext, [AAp1]. For example, QRMonDropData is equivalent to QRMonDropFromContext["data"].)

## Unit tests

The development of QRMon was done with two types of unit tests: (i) directly specified tests, [AAp7], and (ii) tests based on randomly generated pipelines, [AA8].

The unit test package should be further extended in order to provide better coverage of the functionalities and illustrate — and postulate — pipeline behavior.

### Directly specified tests

Here we run the unit tests file "MonadicQuantileRegression-Unit-Tests.wlt", [AAp7]:

``````AbsoluteTiming[
]``````

The natural language derived test ID’s should give a fairly good idea of the functionalities covered in [AAp3].

``````Values[Map[#["TestID"] &, testObject["TestResults"]]]

"QuantileRegression-2", "QuantileRegression-3", \
"QuantileRegression-and-Fit-1", "Fit-and-QuantileRegression-1", \
"QuantileRegressionFit-and-Fit-1", "Fit-and-QuantileRegressionFit-1", \
"Outliers-1", "Outliers-2", "GridSequence-1", "BandsSequence-1", \
"ConditionalCDF-1", "Evaluate-1", "Evaluate-2", "Evaluate-3", \
"Simulate-1", "Simulate-2", "Simulate-3"} *)``````

### Random pipelines tests

Since the monad QRMon is a DSL it is natural to test it with a large number of randomly generated "sentences" of that DSL. For the QRMon DSL the sentences are QRMon pipelines. The package "MonadicQuantileRegressionRandomPipelinesUnitTests.m", [AAp8], has functions for generation of QRMon random pipelines and running them as verification tests. A short example follows.

Generate pipelines:

``````SeedRandom[234]
pipelines = MakeQRMonRandomPipelines[100];
Length[pipelines]

(* 100 *)``````

Here is a sample of the generated pipelines:

``````(*
Block[{DoubleLongRightArrow, pipelines = RandomSample[pipelines, 6]},
Clear[DoubleLongRightArrow];
pipelines = pipelines /. {_TemporalData -> "tsData", _?MatrixQ -> "distData"};
GridTableForm[Map[List@ToString[DoubleLongRightArrow @@ #, FormatType -> StandardForm] &, pipelines], TableHeadings -> {"pipeline"}]
]
AutoCollapse[] *)``````

Here we run the pipelines as unit tests:

``````AbsoluteTiming[
res = TestRunQRMonPipelines[pipelines, "Echo" -> False];
]``````

From the test report results we see that a dozen tests failed with messages, all of the rest passed.

``rpTRObj = TestReport[res]``

(The message failures, of course, have to be examined — some bugs were found in that way. Currently the actual test messages are expected.)

## Future plans

### Workflow operations

A list of possible, additional workflow operations and improvements follows.

• Certain improvements can be done over the specification of the different plot options.

• It will be useful to develop a function for automatic finding of over-fitting parameters.

• The time series simulation should be done by aggregation of similar time intervals.

• For example, for time series with span several years, for each month name is made Quantile Regression simulation and the results are spliced to obtain a one year simulation.
• If the time series is represented as a sequence of categorical values, then the time series simulation can use Bayesian probabilities derived from sub-sequences.
• QRMon already has functions that facilitate that, QRMonGridSequence and QRMonBandsSequence.

### Conversational agent

Using the packages [AAp10, AAp11] we can generate QRMon pipelines with natural commands. The plan is to develop and document those functionalities further.

Here is an example of a pipeline constructed with natural language commands:

``````QRMonUnit[distData]⟹
ToQRMonPipelineFunction["show data summary"]⟹
ToQRMonPipelineFunction["calculate quantile regression for quantiles 0.2, 0.8 and with 40 knots"]⟹
ToQRMonPipelineFunction["plot"];``````

## Implementation notes

The implementation methodology of the QRMon monad packages [AAp3, AAp8] followed the methodology created for the ClCon monad package [AAp9, AA6]. Similarly, this document closely follows the structure and exposition of the ClCon monad document "A monad for classification workflows", [AA6].

A lot of the functionalities and signatures of QRMon were designed and programed through considerations of natural language commands specifications given to a specialized conversational agent. (As discussed in the previous section.)

## References

### ConversationalAgents Packages

[AAp10] Anton Antonov, Time series workflows grammar in EBNF, (2018), ConversationalAgents at GitHub, https://github.com/antononcube/ConversationalAgents.

[AAp11] Anton Antonov, QRMon translator Mathematica package,(2018), ConversationalAgents at GitHub, https://github.com/antononcube/ConversationalAgents.

### Other

[Wk2] Wikipedia entry, Quantile Regression, URL: https://en.wikipedia.org/wiki/Quantile_regression .

[Wk3] Wikipedia entry, Chebyshev polynomials, URL: https://en.wikipedia.org/wiki/Chebyshev_polynomials .

[CN1] Brian S. Code and Barry R. Noon, "A gentle introduction to quantile regression for ecologists", (2003). Frontiers in Ecology and the Environment. 1 (8): 412[Dash]420. doi:10.2307/3868138. URL: http://www.econ.uiuc.edu/~roger/research/rq/QReco.pdf .

[PS1] Patrick Scheibe, Mathematica (Wolfram Language) support for IntelliJ IDEA, (2013-2018), Mathematica-IntelliJ-Plugin at GitHub. URL: https://github.com/halirutan/Mathematica-IntelliJ-Plugin .

[RK1] Roger Koenker, Quantile Regression, ‪Cambridge University Press, 2005‬.

# Phone dialing conversational agent

## Introduction

This blog post proclaims the first committed project in the repository ConversationalAgents at GitHub. The project has designs and implementations of a phone calling conversational agent that aims at providing the following functionalities:

• contacts retrieval (querying, filtering, selection),
• contacts prioritization, and
• phone call (work flow) handling.
• The design is based on a Finite State Machine (FSM) and context free grammar(s) for commands that switch between the states of the FSM. The grammar is designed as a context free grammar rules of a Domain Specific Language (DSL) in Extended Backus-Naur Form (EBNF). (For more details on DSLs design and programming see [1].)

The (current) implementation is with Wolfram Language (WL) / Mathematica using the functional parsers package [2, 3].

This movie gives an overview from an end user perspective.

## General design

The design of the Phone Conversational Agent (PhCA) is derived in a straightforward manner from the typical work flow of calling a contact (using, say, a mobile phone.)

The main goals for the conversational agent are the following:

1. contacts retrieval — search, filtering, selection — using both natural language commands and manual interaction,
2. intuitive integration with the usual work flow of phone calling.

An additional goal is to facilitate contacts retrieval by determining the most appropriate contacts in query responses. For example, while driving to work by pressing the dial button we might prefer the contacts of an up-coming meeting to be placed on top of the prompting contacts list.

In this project we assume that the voice to text conversion is done with an external (reliable) component.

It is assumed that an user of PhCA can react to both visual and spoken query results.

The main algorithm is the following.

1) Parse and interpret a natural language command.

2) If the command is a contacts query that returns a single contact then call that contact.

3) If the command is a contacts query that returns multiple contacts then :

3.1) use natural language commands to refine and filter the query results,

3.2) until a single contact is obtained. Call that single contact.

4) If other type of command is given act accordingly.

PhCA has commands for system usage help and for canceling the current contact search and starting over.

The following FSM diagram gives the basic structure of PhCA:

This movie demonstrates how different natural language commands switch the FSM states.

## Grammar design

The derived grammar describes sentences that: 1. fit end user expectations, and 2. are used to switch between the FSM states.

Because of the simplicity of the FSM and the natural language commands only few iterations were done with the Parser-generation-by-grammars work flow.

The base grammar is given in the file "./Mathematica/PhoneCallingDialogsGrammarRules.m" in EBNF used by [2].

Here are parsing results of a set of test natural language commands:

using the WL command:

``````ParsingTestTable[ParseJust[pCALLCONTACT\[CirclePlus]pCALLFILTER], ToLowerCase /@ queries]
``````

(Note that according to PhCA’s FSM diagram the parsing of `pCALLCONTACT` is separated from `pCALLFILTER`, hence the need to combine the two parsers in the code line above.)

PhCA’s FSM implementation provides interpretation and context of the functional programming expressions obtained by the parser.

In the running script "./Mathematica/PhoneDialingAgentRunScript.m" the grammar parsers are modified to do successful parsing using data elements of the provided fake address book.

The base grammar can be extended with the "Time specifications grammar" in order to include queries based on temporal commands.

## Running

In order to experiment with the agent just run in Mathematica the command:

``Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/Projects/PhoneDialingDialogsAgent/Mathematica/PhoneDialingAgentRunScript.m"]``

The imported Wolfram Language file, "./Mathematica/PhoneDialingAgentRunScript.m", uses a fake address book based on movie creators metadata. The code structure of "./Mathematica/PhoneDialingAgentRunScript.m" allows easy experimentation and modification of the running steps.

Here are several screen-shots illustrating a particular usage path (scan left-to-right):

See this movie demonstrating a PhCA run.

## References

[1] Anton Antonov, "Creating and programming domain specific languages", (2016), MathematicaForPrediction at WordPress blog.

[2] Anton Antonov, Functional parsers, Mathematica package, MathematicaForPrediction at GitHub, 2014.

[3] Anton Antonov, "Natural language processing with functional parsers", (2014), MathematicaForPrediction at WordPress blog.

# Monad code generation and extension

## … in Mathematica / Wolfram Language

Anton Antonov

MathematicaForPrediction at GitHub

MathematicaVsR at GitHub

June 2017

## Introduction

This document aims to introduce monadic programming in Mathematica / Wolfram Language (WL) in a concise and code-direct manner. The core of the monad codes discussed is simple, derived from the fundamental principles of Mathematica / WL.

The usefulness of the monadic programming approach manifests in multiple ways. Here are a few we are interested in:

1. easy to construct, read, and modify sequences of commands (pipelines),
2. easy to program polymorphic behaviour,
3. easy to program context utilization.

Speaking informally,

• Monad programming provides an interface that allows interactive, dynamic creation and change of sequentially structured computations with polymorphic and context-aware behavior.

The theoretical background provided in this document is given in the Wikipedia article on Monadic programming, [Wk1], and the article “The essence of functional programming” by Philip Wadler, [H3]. The code in this document is based on the primary monad definition given in [Wk1,H3]. (Based on the “Kleisli triple” and used in Haskell.)

The general monad structure can be seen as:

1. a software design pattern;
2. a fundamental programming construct (similar to class in object-oriented programming);
3. an interface for software types to have implementations of.

In this document we treat the monad structure as a design pattern, [Wk3]. (After reading [H3] point 2 becomes more obvious. A similar in spirit, minimalistic approach to Object-oriented Design Patterns is given in [AA1].)

We do not deal with types for monads explicitly, we generate code for monads instead. One reason for this is the “monad design pattern” perspective; another one is that in Mathematica / WL the notion of algebraic data type is not needed — pattern matching comes from the core “book of replacement rules” principle.

The rest of the document is organized as follows.

1. Fundamental sections The section “What is a monad?” gives the necessary definitions. The section “The basic Maybe monad” shows how to program a monad from scratch in Mathematica / WL. The section “Extensions with polymorphic behavior” shows how extensions of the basic monad functions can be made. (These three sections form a complete read on monadic programming, the rest of the document can be skipped.)

2. Monadic programming in practice The section “Monad code generation” describes packages for generating monad code. The section “Flow control in monads” describes additional, control flow functionalities. The section “General work-flow of monad code generation utilization” gives a general perspective on the use of monad code generation. The section “Software design with monadic programming” discusses (small scale) software design with monadic programming.

3. Case study sections The case study sections “Contextual monad classification” and “Tracing monad pipelines” hopefully have interesting and engaging examples of monad code generation, extension, and utilization.

In this document a monad is any set of a symbol m and two operators unit and bind that adhere to the monad laws. (See the next sub-section.) The definition is taken from [Wk1] and [H3] and phrased in Mathematica / WL terms in this section. In order to be brief, we deliberately do not consider the equivalent monad definition based on unit, join, and map (also given in [H3].)

Here are operators for a monad associated with a certain symbol `M`:

1. monad unit function (“return” in Haskell notation) is `Unit[x_] := M[x]`;
2. monad bind function (“>>=” in Haskell notation) is a rule like `Bind[M[x_], f_] := f[x] with MatchQ[f[x],M[_]]` giving `True`.

Note that:

• the function `Bind` unwraps the content of `M[_]` and gives it to the function `f`;
• the functions fi are responsible to return results wrapped with the monad symbol `M`.

Here is an illustration formula showing a monad pipeline:

From the definition and formula it should be clear that if for the result of `Bind[_M,f[x]]` the test `MatchQ[f[x],_M]` is `True` then the result is ready to be fed to the next binding operation in monad’s pipeline. Also, it is clear that it is easy to program the pipeline functionality with `Fold`:

``````Fold[Bind, M[x], {f1, f2, f3}]

(* Bind[Bind[Bind[M[x], f1], f2], f3] *)``````

The monad laws definitions are taken from [H1] and [H3].In the monad laws given below the symbol “⟹” is for monad’s binding operation and ↦ is for a function in anonymous form.

Here is a table with the laws:

Remark: The monad laws are satisfied for every symbol in Mathematica / WL with List being the unit operation and Apply being the binding operation.

Looking at formula (1) — and having certain programming experiences — we can expect the following features when using monadic programming.

• Computations that can be expressed with monad pipelines are easy to construct and read.
• By programming the binding function we can tuck-in a variety of monad behaviours — this is the so called “programmable semicolon” feature of monads.
• Monad pipelines can be constructed with `Fold`, but with suitable definitions of infix operators like `DoubleLongRightArrow` (⟹) we can produce code that resembles the pipeline in formula (1).
• A monad pipeline can have polymorphic behaviour by overloading the signatures of fi (and if we have to, `Bind`.)

These points are clarified below. For more complete discussions see [Wk1] or [H3].

It is fairly easy to program the basic monad Maybe discussed in [Wk1].

The goal of the Maybe monad is to provide easy exception handling in a sequence of chained computational steps. If one of the computation steps fails then the whole pipeline returns a designated failure symbol, say `None` otherwise the result after the last step is wrapped in another designated symbol, say Maybe.

Here is the special version of the generic pipeline formula (1) for the Maybe monad:

Here is the minimal code to get a functional Maybe monad (for a more detailed exposition of code and explanations see [AA7]):

``````MaybeUnitQ[x_] := MatchQ[x, None] || MatchQ[x, Maybe[___]];

MaybeUnit[None] := None;
MaybeUnit[x_] := Maybe[x];

MaybeBind[None, f_] := None;
MaybeBind[Maybe[x_], f_] :=
Block[{res = f[x]}, If[FreeQ[res, None], res, None]];

MaybeEcho[x_] := Maybe@Echo[x];
MaybeEchoFunction[f___][x_] := Maybe@EchoFunction[f][x];

MaybeOption[f_][xs_] :=
Block[{res = f[xs]}, If[FreeQ[res, None], res, Maybe@xs]];``````

In order to make the pipeline form of the code we write let us give definitions to a suitable infix operator (like “⟹”) to use MaybeBind:

``````DoubleLongRightArrow[x_?MaybeUnitQ, f_] := MaybeBind[x, f];
DoubleLongRightArrow[x_, y_, z__] :=
DoubleLongRightArrow[DoubleLongRightArrow[x, y], z];``````

Here is an example of a Maybe monad pipeline using the definitions so far:

``````data = {0.61, 0.48, 0.92, 0.90, 0.32, 0.11};

MaybeUnit[data]⟹(* lift data into the monad *)
(Maybe@ Join[#, RandomInteger[8, 3]] &)⟹(* add more values *)
MaybeEcho⟹(* display current value *)
(Maybe @ Map[If[# < 0.4, None, #] &, #] &)(* map values that are too small to None *)

(* {0.61,0.48,0.92,0.9,0.32,0.11,4,4,0}
None *)``````

The result is `None` because:

1. the data has a number that is too small, and
2. the definition of `MaybeBind` stops the pipeline aggressively using a `FreeQ[_,None]` test.

Let us convince ourselves that the current definition of `MaybeBind` gives a monad.

The verification is straightforward to program and shows that the implemented Maybe monad adheres to the monad laws.

## Extensions with polymorphic behavior

We can see from formulas (1) and (2) that the monad codes can be easily extended through overloading the pipeline functions.

For example the extension of the Maybe monad to handle of `Dataset` objects is fairly easy and straightforward.

Here is the formula of the Maybe monad pipeline extended with `Dataset` objects:

Here is an example of a polymorphic function definition for the Maybe monad:

``````MaybeFilter[filterFunc_][xs_] := Maybe@Select[xs, filterFunc[#] &];

MaybeFilter[critFunc_][xs_Dataset] := Maybe@xs[Select[critFunc]];``````

See [AA7] for more detailed examples of polymorphism in monadic programming with Mathematica / WL.

A complete discussion can be found in [H3]. (The main message of [H3] is the poly-functional and polymorphic properties of monad implementations.)

### Polymorphic monads in R’s dplyr

The R package dplyr, [R1], has implementations centered around monadic polymorphic behavior. The command pipelines based on dplyrcan work on R data frames, SQL tables, and Spark data frames without changes.

Here is a diagram of a typical work-flow with dplyr:

The diagram shows how a pipeline made with dplyr can be re-run (or reused) for data stored in different data structures.

We can see monad code definitions like the ones for Maybe as some sort of initial templates for monads that can be extended in specific ways depending on their applications. Mathematica / WL can easily provide code generation for such templates; (see [WL1]). As it was mentioned in the introduction, we do not deal with types for monads explicitly, we generate code for monads instead.

In this section are given examples with packages that generate monad codes. The case study sections have examples of packages that utilize generated monad codes.

The package [AA2] provides a Maybe code generator that takes as an argument a prefix for the generated functions. (Monad code generation is discussed further in the section “General work-flow of monad code generation utilization”.)

Here is an example:

``````Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MaybeMonadCodeGenerator.m"]

data = {0.61, 0.48, 0.92, 0.90, 0.32, 0.11};

AnotherMaybeUnit[data]⟹(* lift data into the monad *)
(AnotherMaybe@Join[#, RandomInteger[8, 3]] &)⟹(* add more values *)
AnotherMaybeEcho⟹(* display current value *)
(AnotherMaybe @ Map[If[# < 0.4, None, #] &, #] &)(* map values that are too small to None *)

(* {0.61,0.48,0.92,0.9,0.32,0.11,8,7,6}
AnotherMaybeBind: Failure when applying: Function[AnotherMaybe[Map[Function[If[Less[Slot[1], 0.4], None, Slot[1]]], Slot[1]]]]
None *)``````

We see that we get the same result as above (`None`) and a message prompting failure.

The State monad is also basic and its programming in Mathematica / WL is not that difficult. (See [AA3].)

Here is the special version of the generic pipeline formula (1) for the State monad:

Note that since the State monad pipeline caries both a value and a state, it is a good idea to have functions that manipulate them separately. For example, we can have functions for context modification and context retrieval. (These are done in [AA3].)

``Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/StateMonadCodeGenerator.m"]``

This generates the State monad for the prefix “StMon”:

``GenerateStateMonadCode["StMon"]``

The following StMon pipeline code starts with a random matrix and then replaces numbers in the current pipeline value according to a threshold parameter kept in the context. Several times are invoked functions for context deposit and retrieval.

``````SeedRandom[34]
StMonUnit[RandomReal[{0, 1}, {3, 2}], <|"mark" -> "TooSmall", "threshold" -> 0.5|>]⟹
StMonEchoValue⟹
StMonEchoContext⟹
StMonEchoContext⟹
(StMon[#1 /. (x_ /; x < #2["threshold"] :> #2["mark"]), #2] &)⟹
StMonEchoValue⟹
StMonRetrieveFromContext["data"]⟹
StMonEchoValue⟹
StMonRetrieveFromContext["mark"]⟹
StMonEchoValue;

(* value: {{0.789884,0.831468},{0.421298,0.50537},{0.0375957,0.289442}}
context: <|mark->TooSmall,threshold->0.5|>
context: <|mark->TooSmall,threshold->0.5,data->{{0.789884,0.831468},{0.421298,0.50537},{0.0375957,0.289442}}|>
value: {{0.789884,0.831468},{TooSmall,0.50537},{TooSmall,TooSmall}}
value: {{0.789884,0.831468},{0.421298,0.50537},{0.0375957,0.289442}}
value: TooSmall *)``````

We can implement dedicated functions for governing the pipeline flow in a monad.

Let us look at a breakdown of these kind of functions using the State monad StMon generated above.

### Optional acceptance of a function result

A basic and simple pipeline control function is for optional acceptance of result — if failure is obtained applying f then we ignore its result (and keep the current pipeline value.)

Here is an example with `StMonOption` :

``````SeedRandom[34]
StMonUnit[RandomReal[{0, 1}, 5]]⟹
StMonEchoValue⟹
StMonOption[If[# < 0.3, None] & /@ # &]⟹
StMonEchoValue

(* value: {0.789884,0.831468,0.421298,0.50537,0.0375957}
value: {0.789884,0.831468,0.421298,0.50537,0.0375957}
StMon[{0.789884, 0.831468, 0.421298, 0.50537, 0.0375957}, <||>] *)``````

Without `StMonOption` we get failure:

``````SeedRandom[34]
StMonUnit[RandomReal[{0, 1}, 5]]⟹
StMonEchoValue⟹
(If[# < 0.3, None] & /@ # &)⟹
StMonEchoValue

(* value: {0.789884,0.831468,0.421298,0.50537,0.0375957}
StMonBind: Failure when applying: Function[Map[Function[If[Less[Slot[1], 0.3], None]], Slot[1]]]
None *)``````

### Conditional execution of functions

It is natural to want to have the ability to chose a pipeline function application based on a condition.

This can be done with the functions `StMonIfElse` and `StMonWhen`.

``````SeedRandom[34]
StMonUnit[RandomReal[{0, 1}, 5]]⟹
StMonEchoValue⟹
StMonIfElse[
Or @@ (# < 0.4 & /@ #) &,
(Echo["A too small value is present.", "warning:"];
StMon[Style[#1, Red], #2]) &,
StMon[Style[#1, Blue], #2] &]⟹
StMonEchoValue

(* value: {0.789884,0.831468,0.421298,0.50537,0.0375957}
warning: A too small value is present.
value: {0.789884,0.831468,0.421298,0.50537,0.0375957}
StMon[{0.789884,0.831468,0.421298,0.50537,0.0375957},<||>] *)``````

Remark: Using flow control functions like `StMonIfElse` and `StMonWhen` with appropriate messages is a better way of handling computations that might fail. The silent failures handling of the basic Maybe monad is convenient only in a small number of use cases.

### Iterative functions

The last group of pipeline flow control functions we consider comprises iterative functions that provide the functionalities of `Nest`, `NestWhile`, `FoldList`, etc.

In [AA3] these functionalities are provided through the function StMonIterate.

Here is a basic example using `Nest` that corresponds to `Nest[#+1&,1,3]`:

``````StMonUnit[1]⟹StMonIterate[Nest, (StMon[#1 + 1, #2]) &, 3]

(* StMon[4, <||>] *)``````

Consider this command that uses the full signature of `NestWhileList`:

``````NestWhileList[# + 1 &, 1, # < 10 &, 1, 4]

(* {1, 2, 3, 4, 5} *)``````

Here is the corresponding StMon iteration code:

``````StMonUnit[1]⟹StMonIterate[NestWhileList, (StMon[#1 + 1, #2]) &, (#[[1]] < 10) &, 1, 4]

(* StMon[{1, 2, 3, 4, 5}, <||>] *)``````

Here is another results accumulation example with `FixedPointList` :

``````StMonUnit[1.]⟹
StMonIterate[FixedPointList, (StMon[(#1 + 2/#1)/2, #2]) &]

(* StMon[{1., 1.5, 1.41667, 1.41422, 1.41421, 1.41421, 1.41421}, <||>] *)``````

When the functions `NestList`, `NestWhileList`, `FixedPointList` are used with `StMonIterate` their results can be stored in the context. Here is an example:

``````StMonUnit[1.]⟹
StMonIterate[FixedPointList, (StMon[(#1 + 2/#1)/2, #2]) &, "fpData"]

(* StMon[{1., 1.5, 1.41667, 1.41422, 1.41421, 1.41421, 1.41421}, <|"fpData" -> {StMon[1., <||>],
StMon[1.5, <||>], StMon[1.41667, <||>], StMon[1.41422, <||>], StMon[1.41421, <||>],
StMon[1.41421, <||>], StMon[1.41421, <||>]} |>] *)``````

More elaborate tests can be found in [AA8].

### Partial pipelines

Because of the associativity law we can design pipeline flows based on functions made of “sub-pipelines.”

``````fEcho = Function[{x, ct}, StMonUnit[x, ct]⟹StMonEchoValue⟹StMonEchoContext];

fDIter = Function[{x, ct},
StMonUnit[y^x, ct]⟹StMonIterate[FixedPointList, StMonUnit@D[#, y] &, 20]];

StMonUnit[7]⟹fEcho⟹fDIter⟹fEcho;

(*
value: 7
context: <||>
value: {y^7,7 y^6,42 y^5,210 y^4,840 y^3,2520 y^2,5040 y,5040,0,0}
context: <||> *)``````

## General work-flow of monad code generation utilization

With the abilities to generate and utilize monad codes it is natural to consider the following work flow. (Also shown in the diagram below.)

1. Come up with an idea that can be expressed with monadic programming.
2. Look for suitable monad implementation.
3. If there is no such implementation, make one (or two, or five.)
6. Start making pipelines for the problem domain of interest.
7. Are the pipelines are satisfactory? If not go to 5. (Or 2.)

The template nature of the general monads can be exemplified with the group of functions in the package StateMonadCodeGenerator.m, [4].

They are in five groups:

1. base monad functions (unit testing, binding),
2. display of the value and context,
3. context manipulation (deposit, retrieval, modification),
4. flow governing (optional new value, conditional function application, iteration),
5. other convenience functions.

We can say that all monad implementations will have their own versions of these groups of functions. The more specialized monads will have functions specific to their intended use. Such special monads are discussed in the case study sections.

## Software design with monadic programming

The application of monadic programming to a particular problem domain is very similar to designing a software framework or designing and implementing a Domain Specific Language (DSL).

The answers of the question “When to use monadic programming?” can form a large list. This section provides only a couple of general, personal viewpoints on monadic programming in software design and architecture. The principles of monadic programming can be used to build systems from scratch (like Haskell and Scala.) Here we discuss making specialized software with or within already existing systems.

#### Framework design

Software framework design is about architectural solutions that capture the commonality and variability in a problem domain in such a way that: 1) significant speed-up can be achieved when making new applications, and 2) a set of policies can be imposed on the new applications.

The rigidness of the framework provides and supports its flexibility — the framework has a backbone of rigid parts and a set of “hot spots” where new functionalities are plugged-in.

Usually Object-Oriented Programming (OOP) frameworks provide inversion of control — the general work-flow is already established, only parts of it are changed. (This is characterized with “leave the driving to us” and “don’t call us we will call you.”)

The point of utilizing monadic programming is to be able to easily create different new work-flows that share certain features. (The end user is the driver, on certain rail paths.)

In my opinion making a software framework of small to moderate size with monadic programming principles would produce a library of functions each with polymorphic behaviour that can be easily sequenced in monadic pipelines. This can be contrasted with OOP framework design in which we are more likely to end up with backbone structures that (i) are static and tree-like, and (ii) are extended or specialized by plugging-in relevant objects. (Those plugged-in objects themselves can be trees, but hopefully short ones.)

#### DSL development

Given a problem domain the general monad structure can be used to shape and guide the development of DSLs for that problem domain.

Generally, in order to make a DSL we have to choose the language syntax and grammar. Using monadic programming the syntax and grammar commands are clear. (The monad pipelines are the commands.) What is left is “just” the choice of particular functions and their implementations.

Another way to develop such a DSL is through a grammar of natural language commands. Generally speaking, just designing the grammar — without developing the corresponding interpreters — would be very helpful in figuring out the components at play. Monadic programming meshes very well with this approach and applying the two approaches together can be very fruitful.

## Contextual monad classification (case study)

In this section we show an extension of the State monad into a monad aimed at machine learning classification work-flows.

### Motivation

We want to provide a DSL for doing machine learning classification tasks that allows us:

1. to do basic summarization and visualization of the data,
2. to control splitting of the data into training and testing sets;
3. to apply the built-in classifiers;
4. to apply classifier ensembles (see [AA9] and [AA10]);
5. to evaluate classifier performances with standard measures and
6. ROC plots.

Also, we want the DSL design to provide clear directions how to add (hook-up or plug-in) new functionalities.

The package [AA4] discussed below provides such a DSL through monadic programming.

``Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicContextualClassification.m"]``

This gets some test data (the Titanic dataset):

``````dataName = "Titanic";
ds = Dataset[Flatten@*List @@@ ExampleData[{"MachineLearning", dataName}, "Data"]];
varNames = Flatten[List @@ ExampleData[{"MachineLearning", dataName}, "VariableDescriptions"]];
varNames = StringReplace[varNames, "passenger" ~~ (WhitespaceCharacter ..) -> ""];
If[dataName == "FisherIris", varNames = Most[varNames]];
ds = ds[All, AssociationThread[varNames -> #] &];``````

The package [AA4] provides functions for the monad ClCon — the functions implemented in [AA4] have the prefix “ClCon”.

The classifier contexts are Association objects. The pipeline values can have the form:

``ClCon[ val, context:(_String|_Association) ]``

The ClCon specific monad functions deposit or retrieve values from the context with the keys: “trainData”, “testData”, “classifier”. The general idea is that if the current value of the pipeline cannot provide all arguments for a ClCon function, then the needed arguments are taken from the context. If that fails, then an message is issued. This is illustrated with the following pipeline with comments example.

The pipeline and results above demonstrate polymorphic behaviour over the classifier variable in the context: different functions are used if that variable is a ClassifierFunction object or an association of named `ClassifierFunction` objects.

Note the demonstrated granularity and sequentiality of the operations coming from using a monad structure. With those kind of operations it would be easy to make interpreters for natural language DSLs.

### Another usage example

This monadic pipeline in this example goes through several stages: data summary, classifier training, evaluation, acceptance test, and if the results are rejected a new classifier is made with a different algorithm using the same data splitting. The context keeps track of the data and its splitting. That allows the conditional classifier switch to be concisely specified.

First let us define a function that takes a `Classify` method as an argument and makes a classifier and calculates performance measures.

``````ClSubPipe[method_String] :=
Function[{x, ct},
ClConUnit[x, ct]⟹
ClConMakeClassifier[method]⟹
ClConEchoFunctionContext["classifier:",
ClassifierInformation[#["classifier"], Method] &]⟹
ClConEchoFunctionContext["training time:", ClassifierInformation[#["classifier"], "TrainingTime"] &]⟹
ClConClassifierMeasurements[{"Accuracy", "Precision", "Recall"}]⟹
ClConEchoValue⟹
ClConEchoFunctionContext[
ClassifierMeasurements[#["classifier"],
ClConToNormalClassifierData[#["testData"]], "ROCCurve"] &]
];``````

Using the sub-pipeline function `ClSubPipe` we make the outlined pipeline.

``````SeedRandom[12]
res =
ClConUnit[ds]⟹
ClConSplitData[0.7]⟹
ClConEchoFunctionValue["summaries:", ColumnForm[Normal[RecordsSummary /@ #]] &]⟹
ClConEchoFunctionValue["xtabs:",
MatrixForm[CrossTensorate[Count == varNames[[1]] + varNames[[-1]], #]] & /@ # &]⟹
ClSubPipe["LogisticRegression"]⟹
(If[#1["Accuracy"] > 0.8,
Echo["Good accuracy!", "Success:"]; ClConFail,
Echo["Make a new classifier", "Inaccurate:"];
ClConUnit[#1, #2]] &)⟹
ClSubPipe["RandomForest"];``````

## Tracing monad pipelines (case study)

The monadic implementations in the package MonadicTracing.m, [AA5] allow tracking of the pipeline execution of functions within other monads.

The primary reason for developing the package was the desire to have the ability to print a tabulated trace of code and comments using the usual monad pipeline notation. (I.e. without conversion to strings etc.)

It turned out that by programming MonadicTracing.m I came up with a monad transformer; see [Wk2], [H2].

``Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicTracing.m"]``

### Usage example

This generates a Maybe monad to be used in the example (for the prefix “Perhaps”):

``````GenerateMaybeMonadCode["Perhaps"]

In following example we can see that pipeline functions of the Perhaps monad are interleaved with comment strings. Producing the grid of functions and comments happens “naturally” with the monad function `TraceMonadEchoGrid`.

``````data = RandomInteger[10, 15];

PerhapsFilter[# > 3 &]⟹"filter current value"⟹
PerhapsEcho⟹"display current value"⟹
PerhapsWhen[#[[3]] > 3 &,
PerhapsEchoFunction[Style[#, Red] &]]⟹
(Perhaps[#/4] &)⟹
PerhapsEcho⟹"display current value again"⟹

Note that :

1. the tracing is initiated by just using `TraceMonadUnit`;
2. pipeline functions (actual code) and comments are interleaved;
3. putting a comment string after a pipeline function is optional.

Another example is the ClCon pipeline in the sub-section “Monad design” in the previous section.

## Summary

This document presents a style of using monadic programming in Wolfram Language (Mathematica). The style has some shortcomings, but it definitely provides convenient features for day-to-day programming and in coming up with architectural designs.

The style is based on WL’s basic language features. As a consequence it is fairly concise and produces light overhead.

Ideally, the packages for the code generation of the basic Maybe and State monads would serve as starting points for other more general or more specialized monadic programs.

## References

[Wk3] Wikipedia entry: Software Design Pattern, URL: https://en.wikipedia.org/wiki/Software_design_pattern .

[H2] Sheng Liang, Paul Hudak, Mark Jones, “Monad transformers and modular interpreters”, (1995), Proceedings of the 22nd ACM SIGPLAN-SIGACT symposium on Principles of programming languages. New York, NY: ACM. pp. 333[Dash]343. doi:10.1145/199448.199528.

[H3] Philip Wadler, “The essence of functional programming”, (1992), 19’th Annual Symposium on Principles of Programming Languages, Albuquerque, New Mexico, January 1992.

### R

[R1] Hadley Wickham et al., dplyr: A Grammar of Data Manipulation, (2014), tidyverse at GitHub, URL: https://github.com/tidyverse/dplyr . (See also, http://dplyr.tidyverse.org .)

### Mathematica / Wolfram Language

[WL1] Leonid Shifrin, “Metaprogramming in Wolfram Language”, (2012), Mathematica StackExchange. (Also posted at Wolfram Community in 2017.) URL of the Mathematica StackExchange answer: https://mathematica.stackexchange.com/a/2352/34008 . URL of the Wolfram Community post: http://community.wolfram.com/groups/-/m/t/1121273 .

### MathematicaForPrediction

[AA7] Anton Antonov, Simple monadic programming, (2017), MathematicaForPrediction at GitHub. (Preliminary version, 40% done.) URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/Documentation/Simple-monadic-programming.pdf .

# Creating and programming domain specific languages

## Introduction

In this blog post I will provide links to documents, packages, blog posts, and discussions for creating and utilizing Domain Specific Languages (DSLs). I have discussed a few DSLs in previous blog posts (linked below). This blog post provides a more general, higher level view on the application and creation of DSLs. The concrete examples are with Mathematica, but the steps are general and can be done with any programming languages and tools.

## When to apply DSLs

Here are some situations for applying DSLs.

1. When designing conversational engines.
2.  When there are too many usage scenarios and tuning options for the developed algorithms.
• For example, we have a bunch of search, recommendation, and interaction algorithms for a dating site. A different, User Experience Department (UED) designs interactive user interfaces for these algorithms. We make a natural language DSL that invokes the different algorithms according to specified outcomes. With the DSL the different designs produced by UED are much easily prototyped, implemented, or fleshed out. The DSL also gives to UED easier to understand view on the functionalities provided by the algorithms.
3. When designing an API for a collection of algorithms.
• Just designing a DSL can bring clarity of what signatures should be in the API.
• `NIntegrate`‘s `Method` option was designed and implemented using a DSL. See this video between 25:00 and 27:30.

## Designing DSLs

1. Decide what kind of sentences the DSL is going to have.
• Are natural language sentences going to be used?
• Are the language words known beforehand or not?
2. Prepare, create, or accumulate a list of representative sentences.
• In some cases using Morphological Analysis can greatly help for coming up with use cases and the corresponding sentences.
3. Create a context free grammar that describes the sentences from the previous step. (Or a large subset of them.)
• At this stage I use exclusively Extended Backus-Naur Form (EBNF).
• In some cases the grammar terminals are not know at the design stage and have to retrieved in some way. (From a database or though natural language processing.)
• Some conversational engine systems allow or require to the grammar specification to be done in XML. I would still do BNF and then move to XML
•  It is not that hard to write a parser-and-interpreter that translates BNF into XML. See the end of this blog post for that kind of translation of BNF into OMPL.
4. Program parser(s) for the grammar.
• I use most of the time functional parsers.
• The package FunctionalParsers.m provides a Mathematica implementation of this kind of parsing.
• The package can automatically generate parsers from a grammar given in EBNF. (See the coding example below.)
• I have programmed versions of this package in R and Lua.
5. Program an interpreter for the parsed sentences.
• At this stage the parsed sentences are hooked to the algorithms of the problem domain.
• The package FunctionalParsers.m allows this to be done fairly easy.
6. Test the parsing and interpretation.

See the code example below illustrating steps 3-6.

## Introduction to using DSLs in Mathematica

1. This blog post “Natural language processing with functional parsers” gives an introduction to the DSL application in Mathematica.
2. This detailed slide-show presentation “Functional parsers for an integration requests language grammar” shows how to use the package FunctionalParsers.m over a small grammar.
3. The answer of the MSE question “How to parse a clojure expression?” gives a good introduction with a simple grammar and shows both direct parser programming and automatic generation from EBNF.

The blog post “Simple time series conversational engine” discusses the creation (design and programming) of a simple conversational engine for time series analysis (data loading, finding outliers and trends.)

Here is a movie demonstrating that conversational engine: http://youtu.be/wlZ5ANglVI4.

## Other discussions

1. A small part, from 17:30 to 21:00, of the WTC 2012 “Spatial Access Methods and Route Finding” presentation shows a DSL for points of interest queries.
2. The answer of the MSE question “CSS Selectors for Symbolic XML” uses FunctionalParsers.m .
3. This Quantile Regression presentation is aided by the  “Simple time series conversational engine” mentioned above.

## Coding example

This coding example demonstrates steps 3-6 discussed above.