Time series search engines over COVID-19 data


In this article we proclaim the preparation and availability of interactive interfaces to two Time Series Search Engines (TSSEs) over COVID-19 data. One TSSE is based on Apple Mobility Trends data, [APPL1]; the other on The New York Times COVID-19 data, [NYT1].

Here are links to interactive interfaces of the TSSEs hosted (and publicly available) at shinyapps.io by RStudio:

Motivation: The primary motivation for making the TSSEs and their interactive interfaces is to use them as exploratory tools. Combined with relevant data analysis (e.g. [AA1, AA2]) the TSSEs should help to form better intuition and feel of the spread of COVID-19 and related data aggregation, public reactions, and government polices.

The rest of the article is structured as follows:

  1. Brief descriptions the overall process, the data
  2. Brief descriptions the search engines structure and implementation
  3. Discussions of a few search examples and their (possible) interpretations

The overall process

For both search engines the overall process has the same steps:

  1. Ingest the data
  2. Do basic (and advanced) data analysis
  3. Make (and publish) reports detailing the data ingestion and transformation steps
  4. Enhance the data with transformed versions of it or with additional related data
  5. Make a Time Series Sparse Matrix Recommender (TSSMR)
  6. Make a Time Series Search Engine Interactive Interface (TSSEII)
  7. Make the interactive interface easily accessible over the World Wide Web

Here is a flow chart that corresponds to the steps listed above:



The Apple data

The Apple Mobility Trends data is taken from Apple’s site, see [APPL1]. The data ingestion, basic data analysis, time series seasonality demonstration, (graph) clusterings are given in [AA1]. (Here is a link to the corresponding R-notebook .)

The weather data was taken using the Mathematica function WeatherData, [WRI1].

(It was too much work to get the weather data using some of the well known weather data R packages.)

The New York Times data

The New York Times COVID-19 data is taken from GitHub, see [NYT1]. The data ingestion, basic data analysis, and visualizations are given in [AA2]. (Here is a link to the corresponding R-notebook .)

The search engines

The following sub-sections have screenshots of the TSSE interactive interfaces.

I did experiment with combining the data of the two engines, but did not turn out to be particularly useful. It seems that is more interesting and useful to enhance the Apple data engine with temperature data, and to enhance The New Your Times engine with the (consecutive) differences of the time series.


The interactive interfaces have three panels:

  • Nearest Neighbors
    • Gives the time series nearest neighbors for the time series of selected entity.
    • Has interactive controls for entity selection and filtering.
  • Trend Finding
    • Gives the time series that adhere to a specified named trend.
    • Has interactive controls for trend curves selection and entity filtering.
  • Notes
    • Gives references and data objects summary.


Both TSSEs are implemented using the R packages “SparseMatrixRecommender”, [AAp1], and “SparseMatrixRecommenderInterfaces”, [AAp2].

The package “SparseMatrixRecommender” provides functions to create and use Sparse Matrix Recommender (SMR) objects. Both TSSEs use underlying SMR objects.

The package “SparseMatrixRecommenderInterfaces” provides functions to generate the server and client functions for the Shiny framework by RStudio.

As it was mentioned above, both TSSEs are published at shinyapps.io. The corresponding source codes can be found in [AAr1].

The Apple data TSSE has four types of time series (“entities”). The first three are normalized volumes of Apple maps requests while driving, transit transport use, and walking. (See [AA1] for more details.) The fourth is daily mean temperature at different geo-locations.

Here are screenshots of the panels “Nearest Neighbors” and “Trend Finding” (at interface launch):



The New York Times COVID-19 Data Search Engine

The New York Times TSSE has four types of time series (aggregated) cases and deaths, and their corresponding time series differences.

Here are screenshots of the panels “Nearest Neighbors” and “Trend Finding” (at interface launch):




In this section we discuss in some detail several examples of using each of the TSSEs.

Apple data search engine examples

Here are a few observations from [AA1]:

  • The COVID-19 lockdowns are clearly reflected in the time series.
  • The time series from the Apple Mobility Trends data shows strong weekly seasonality. Roughly speaking, people go to places they are not familiar with on Fridays and Saturdays. Other work week days people are more familiar with their trips. Since much lesser number of requests are made on Sundays, we can conjecture that many people stay at home or visit very familiar locations.

Here are a few assumptions:

  • Where people frequently go (work, school, groceries shopping, etc.) they do not need directions that much.
  • People request directions when they have more free time and will for “leisure trips.”
  • During vacations people are more likely to be in places they are less familiar with.
  • People are more likely to take leisure trips when the weather is good. (Warm, not raining, etc.)

Nice, France vs Florida, USA

Consider the results of the Nearest Neighbors panel for Nice, France.

Since French tend to go on vacation in July and August ([SS1, INSEE1]) we can see that driving, transit, and walking in Nice have pronounced peaks during that time:

Of course, we also observe the lockdown period in that geographical area.

Compare those time series with the time series from driving in Florida, USA:

We can see that people in Florida, USA have driving patterns unrelated to the typical weather seasons and vacation periods.

(Further TSSE queries show that there is a negative correlation with the temperature in south Florida and the volumes of Apple Maps directions requests.)

Italy and Balkan countries driving

We can see that according to the data people who have access to both iPhones and cars in Italy and the Balkan countries Bulgaria, Greece, and Romania have similar directions requests patterns:

(The similarities can be explained with at least a few “obvious” facts, but we are going to restrain ourselves.)

The New York Times data search engine examples

In Broward county, Florida, USA and Cook county, Illinois, USA we can see two waves of infections in the difference time series:



[APPL1] Apple Inc., Mobility Trends Reports, (2020), apple.com.

[NYT1] The New York Times, Coronavirus (Covid-19) Data in the United States, (2020), GitHub.

[WRI1] Wolfram Research (2008), WeatherData, Wolfram Language function.


[AA1] Anton Antonov, “Apple mobility trends data visualization (for COVID-19)”, (2020), SystemModeling at GitHub/antononcube.

[AA2] Anton Antonov, “NY Times COVID-19 data visualization”, (2020), SystemModeling at GitHub/antononcube.

[INSEE1] Institut national de la statistique et des études économiques, “En 2010, les salariés ont pris en moyenne six semaines de congé”, (2012).

[SS1] Sam Schechner and Lee Harris, “What Happens When All of France Takes Vacation? 438 Miles of Traffic”, (2019), The Wall Street Journal

Packages, repositories

[AAp1] Anton Antonov, Sparse Matrix Recommender framework functions, (2019), R-packages at GitHub/antononcube.

[AAp2] Anton Antonov, Sparse Matrix Recommender framework interface functions, (2019), R-packages at GitHub/antononcube.

[AAr1] Anton Antonov, Coronavirus propagation dynamics, (2020), SystemModeling at GitHub/antononcube.

Basic experiments workflow for simple epidemiological models


The primary purpose of this document (notebook) is to give a “stencil workflow” for simulations using the packages in the project “Coronavirus simulation dynamics”, [AAr1].

The model in this notebook – SEI2R – differs from the classical SEIR model with the following elements:

  1. Two separate infected populations: one is “severely symptomatic”, the other is “normally symptomatic”
  2. The monetary equivalent of lost productivity due to infected or died people is tracked.

Remark: We consider the coronavirus propagation models as instances of the more general System Dynamics (SD) models.

Remark: The SEI2R model is a modification of the classic epidemic model SEIR, [Wk1].

Remark: The interactive interfaces in the notebook can be used for attempts to calibrate SEI2R with real data. (For example, data for the 2019–20 coronavirus outbreak, [WRI1].)


  1. Get one of the classical epidemiology models.
  2. Extend the equations of model if needed or desired.
  3. Set relevant initial conditions for the populations.
  4. Pick model parameters to be adjust and “play with.”
  5. Derive parametrized solutions of model’s system of equations (ODE’s or DAE’s.)
    1. Using the parameters of the previous step.
  6. Using an interactive interface experiment with different values of the parameters.
    1. In order to form “qualitative understanding.”
  7. Get real life data.
    1. Say, for the 2019-20 coronavirus outbreak.
  8. Attempt manual or automatic calibration of the model.
    1. This step will most likely require additional data transformations and programming.
    2. Only manual calibration is shown in this notebook.

Load packages of the framework

The epidemiological models framework used in this notebook is implemented in the packages [AAp1, AAp2]; the interactive plots functions are from the package [AAp3].

Getting the model code

Here we take the SEI2R model implemented in the package “EpidemiologyModels.m”, [AAp1]:

We can show a tabulated visualization of the model using the function ModelGridTableForm from [AAp1]:


Model extensions and new models

The framework implemented with the packages [AAp1, AAp2, AAp3] can be utilized using custom made data structures that follow the structure of the models in [AAp1].

Of course, we can also just extend the models from [AAp1]. In this section we show how SEI2R can be extended in two ways:

  1. By adding a birth rate added to the Susceptible Population equation (the birth rate is not included by default)
  2. By adding a new equation for the infected deceased population.

Adding births term

Here are the equations of SEI2R (from [AAp1]):


Here we find the position of the equation that corresponds to “Susceptible Population”:

Here we make the births term using a birth rate that is the same as the death rate:

Here we add the births term to the equations of new model

Here we display the equations of the new model:


Adding infected deceased population equation

Here we add new population, equation, and initial condition that allow for tracking the deaths because of infection:

Here is how the model looks like:


Parameters and actual simulation equations code

Here are the parameters we want to experiment with (or do calibration with):

Here we set custom rates and initial conditions:

Here is the system of ODE’s we use with to do parametrized simulations:



Straightforward simulation for one year with using ParametricNDSolve :


(The advantage having parametrized solutions is that we can quickly compute simulation results with new parameter values without solving model’s system of ODE’s; see the interfaces below.)

Interactive interface

opts = {PlotRange -> All, PlotLegends -> None, PlotTheme -> "Detailed", PerformanceGoal -> "Speed", ImageSize -> 300};
lsPopulationKeys = GetPopulationSymbols[modelSI2R, __ ~~ "Population"];
lsEconKeys = {MLP};
 DynamicModule[{lsPopulationPlots, lsEconPlots, lsRestPlots},
  lsPopulationPlots =
    KeyTake[aSol, lsPopulationKeys],
    {aincp, aip, spf, crisp, criap}, ndays,
    "LogPlot" -> popLogPlotQ, "Together" -> popTogetherQ, 
    "Derivatives" -> popDerivativesQ, 
    "DerivativePrefix" -> "\[CapitalDelta]", opts];
  lsEconPlots =
    KeyTake[aSol, lsEconKeys],
    {aincp, aip, spf, crisp, criap}, ndays,
    "LogPlot" -> econLogPlotQ, "Together" -> econTogetherQ, 
    "Derivatives" -> econDerivativesQ, 
    "DerivativePrefix" -> "\[CapitalDelta]", opts];
  lsRestPlots =
    KeyDrop[aSol, Join[lsPopulationKeys, lsEconKeys]],
    {aincp, aip, spf, crisp, criap}, ndays,
    "LogPlot" -> econLogPlotQ, "Together" -> econTogetherQ, 
    "Derivatives" -> econDerivativesQ, 
    "DerivativePrefix" -> "\[CapitalDelta]", opts];
  Multicolumn[Join[lsPopulationPlots, lsEconPlots, lsRestPlots], 
   nPlotColumns, Dividers -> All, 
   FrameStyle -> GrayLevel[0.8]]
 {{aincp, 12., "Average incubation period (days)"}, 1, 60., 1, Appearance -> {"Open"}},
 {{aip, 21., "Average infectious period (days)"}, 1, 100., 1, Appearance -> {"Open"}},
 {{spf, 0.2, "Severely symptomatic population fraction"}, 0, 1, 0.025, Appearance -> {"Open"}},
 {{crisp, 6, "Contact rate of the infected severely symptomatic population"}, 0, 30, 0.1, Appearance -> {"Open"}},
 {{criap, 3, "Contact rate of the infected normally symptomatic population"}, 0, 30, 0.1, Appearance -> {"Open"}},
 {{ndays, 90, "Number of days"}, 1, 365, 1, Appearance -> {"Open"}},
 {{popTogetherQ, True, "Plot populations together"}, {False, True}},
 {{popDerivativesQ, False, "Plot populations derivatives"}, {False, True}},
 {{popLogPlotQ, False, "LogPlot populations"}, {False, True}},
 {{econTogetherQ, False, "Plot economics functions together"}, {False, True}},
 {{econDerivativesQ, False, "Plot economics functions derivatives"}, {False, True}},
 {{econLogPlotQ, False, "LogPlot economics functions"}, {False, True}},
 {{nPlotColumns, 1, "Number of plot columns"}, Range[5]},
 ControlPlacement -> Left, ContinuousAction -> False]

Calibration over real data

It is important to calibrate these kind of models with real data, or at least to give a serious attempt to such a calibration. If the calibration is “too hard” or “impossible” that would indicate that the model is not that adequate. (If adequate at all.)

The calibration efforts can be (semi-)automated using special model-to-data goodness of fit functions and a minimization procedure. (See for example, [AA2].)

In this section we just attempt to calibrate SEI2R over real data taken from [WRI1] using a specialized interactive interface.

Real data

Here is COVID-19 data taken from [WRI1] for the Chinese province Hubei:

The total population in Hubei is


But we have to use a fraction of that population in order to produce good fits. That can be justified with the conjecture that the citizens of Hubei are spread out and it is mostly one city (Wuhan) where the outbreak is.

The real data have to be padded with a certain number of 0’s related to the infectious and incubation periods in order to make good fits. Such padding is easy to justify: if we observe recovered people that means that they passed through the incubation and infectious periods.

Calibration interactive interface

In this interface we put the Infected Severely Symptomatic Population (ISSP) to zero. That way it is easier to compare the real data with the simulated results (and pick parameter values that give close fits.) Also note that since SEI2R is simple in this interface the system is always solved.

opts = {PlotRange -> All, PlotLegends -> None, PlotTheme -> "Detailed", PerformanceGoal -> "Speed", ImageSize -> 300};
 DynamicModule[{modelSI2R = modelSI2R, lsActualEquations, aSol, 
   lsPopulationPlots, lsEconPlots, lsRestPlots},
  modelSI2R = SetRateRules[modelSI2R, <|TP[t] -> population|>];
  modelSI2R = 
    modelSI2R, <|SP[0] -> population - 1, ISSP[0] -> 0, 
     INSP[0] -> 1|>];
  lsActualEquations = 
   Join[modelSI2R["Equations"] //. 
     KeyDrop[modelSI2R["RateRules"], lsFocusParams], 
  aSol =
      lsActualEquations, {SP, EP, INSP, RP, IDP}, {t, 0, 365}, 
  lsPopulationPlots =
    KeyTake[aSol, GetPopulationSymbols[modelSI2R, __ ~~ "Population"]],
    {aincp, aip, 0, criap, criap}, ndays, "Together" -> True, 
    PadRealData[aRealData, Round[aincp + padOffset], 
     Round[aip + padOffset]], PlotStyle -> {Blue, Black, Red}]]
 {{population, 58160000/600, "Population"}, 58160000/1000, 58160000, 10000, Appearance -> {"Open"}},
 {{padOffset, 0, "real data padding offset"}, -100, 100, 1, Appearance -> {"Open"}},
 {{aincp, 6, "Average incubation period (days)"}, 1, 60, 1, Appearance -> {"Open"}},
 {{aip, 32, "Average infectious period (days)"}, 1, 100, 1, Appearance -> {"Open"}},
 {{criap, 0.8, "Contact rate of the infected normally symptomatic population"}, 0, 30, 0.1, Appearance -> {"Open"}},
 {{ndays, 90, "Number of days"}, 1, 365, 1, Appearance -> {"Open"}},
 ControlPlacement -> Left, ContinuousAction -> False]

Maybe good enough parameters


Basic reproduction number:


Basic reproduction number:




[Wk1] Wikipedia entry, “Compartmental models in epidemiology”.

[HH1] Herbert W. Hethcote (2000). “The Mathematics of Infectious Diseases”. SIAM Review. 42 (4): 599–653. Bibcode:2000SIAMR..42..599H. doi:10.1137/s0036144500371907.

[AA1] Anton Antonov, “Coronavirus propagation modeling considerations”, (2020), SystemModeling at GitHub.

[AA2] Anton Antonov, Answer of “Model calibration with phase space data”, (2019), Mathematica StackExchage.

Repositories & packages

[WRI1] Wolfram Research, Inc., “Epidemic Data for Novel Coronavirus COVID-19”, WolframCloud.

[AAr1] Anton Antonov, Coronavirus propagation dynamics project, (2020), SystemModeling at GitHub.

[AAp1] Anton Antonov, “Epidemiology models Mathematica package”, (2020), SystemsModeling at GitHub.

[AAp2] Anton Antonov, “Epidemiology models modifications Mathematica package”, (2020), SystemsModeling at GitHub.

[AAp3] Anton Antonov, “System dynamics interactive interfaces functions Mathematica package”, (2020), SystemsModeling at GitHub.

Phone dialing conversational agent


This blog post proclaims the first committed project in the repository ConversationalAgents at GitHub. The project has designs and implementations of a phone calling conversational agent that aims at providing the following functionalities:

  • contacts retrieval (querying, filtering, selection),
  • contacts prioritization, and
  • phone call (work flow) handling.
  • The design is based on a Finite State Machine (FSM) and context free grammar(s) for commands that switch between the states of the FSM. The grammar is designed as a context free grammar rules of a Domain Specific Language (DSL) in Extended Backus-Naur Form (EBNF). (For more details on DSLs design and programming see [1].)

    The (current) implementation is with Wolfram Language (WL) / Mathematica using the functional parsers package [2, 3].

    This movie gives an overview from an end user perspective.

    General design

    The design of the Phone Conversational Agent (PhCA) is derived in a straightforward manner from the typical work flow of calling a contact (using, say, a mobile phone.)

    The main goals for the conversational agent are the following:

    1. contacts retrieval — search, filtering, selection — using both natural language commands and manual interaction,
    2. intuitive integration with the usual work flow of phone calling.

    An additional goal is to facilitate contacts retrieval by determining the most appropriate contacts in query responses. For example, while driving to work by pressing the dial button we might prefer the contacts of an up-coming meeting to be placed on top of the prompting contacts list.

    In this project we assume that the voice to text conversion is done with an external (reliable) component.

    It is assumed that an user of PhCA can react to both visual and spoken query results.

    The main algorithm is the following.

    1) Parse and interpret a natural language command.

    2) If the command is a contacts query that returns a single contact then call that contact.

    3) If the command is a contacts query that returns multiple contacts then :

    3.1) use natural language commands to refine and filter the query results,

    3.2) until a single contact is obtained. Call that single contact.

    4) If other type of command is given act accordingly.

    PhCA has commands for system usage help and for canceling the current contact search and starting over.

    The following FSM diagram gives the basic structure of PhCA:


    This movie demonstrates how different natural language commands switch the FSM states.

    Grammar design

    The derived grammar describes sentences that: 1. fit end user expectations, and 2. are used to switch between the FSM states.

    Because of the simplicity of the FSM and the natural language commands only few iterations were done with the Parser-generation-by-grammars work flow.

    The base grammar is given in the file "./Mathematica/PhoneCallingDialogsGrammarRules.m" in EBNF used by [2].

    Here are parsing results of a set of test natural language commands:


    using the WL command:

    ParsingTestTable[ParseJust[pCALLCONTACT\[CirclePlus]pCALLFILTER], ToLowerCase /@ queries]

    (Note that according to PhCA’s FSM diagram the parsing of pCALLCONTACT is separated from pCALLFILTER, hence the need to combine the two parsers in the code line above.)

    PhCA’s FSM implementation provides interpretation and context of the functional programming expressions obtained by the parser.

    In the running script "./Mathematica/PhoneDialingAgentRunScript.m" the grammar parsers are modified to do successful parsing using data elements of the provided fake address book.

    The base grammar can be extended with the "Time specifications grammar" in order to include queries based on temporal commands.


    In order to experiment with the agent just run in Mathematica the command:


    The imported Wolfram Language file, "./Mathematica/PhoneDialingAgentRunScript.m", uses a fake address book based on movie creators metadata. The code structure of "./Mathematica/PhoneDialingAgentRunScript.m" allows easy experimentation and modification of the running steps.

    Here are several screen-shots illustrating a particular usage path (scan left-to-right):

    "PhCA-1-call-someone-from-x-men"" "PhCA-2-a-producer" "PhCA-3-the-third-one

    See this movie demonstrating a PhCA run.


    [1] Anton Antonov, "Creating and programming domain specific languages", (2016), MathematicaForPrediction at WordPress blog.

    [2] Anton Antonov, Functional parsers, Mathematica package, MathematicaForPrediction at GitHub, 2014.

    [3] Anton Antonov, "Natural language processing with functional parsers", (2014), MathematicaForPrediction at WordPress blog.

    Monad code generation and extension

    … in Mathematica / Wolfram Language

    Anton Antonov

    MathematicaForPrediction at GitHub

    MathematicaVsR at GitHub

    June 2017


    This document aims to introduce monadic programming in Mathematica / Wolfram Language (WL) in a concise and code-direct manner. The core of the monad codes discussed is simple, derived from the fundamental principles of Mathematica / WL.

    The usefulness of the monadic programming approach manifests in multiple ways. Here are a few we are interested in:

    1. easy to construct, read, and modify sequences of commands (pipelines),
    2. easy to program polymorphic behaviour,
    3. easy to program context utilization.

    Speaking informally,

    • Monad programming provides an interface that allows interactive, dynamic creation and change of sequentially structured computations with polymorphic and context-aware behavior.

    The theoretical background provided in this document is given in the Wikipedia article on Monadic programming, [Wk1], and the article “The essence of functional programming” by Philip Wadler, [H3]. The code in this document is based on the primary monad definition given in [Wk1,H3]. (Based on the “Kleisli triple” and used in Haskell.)

    The general monad structure can be seen as:

    1. a software design pattern;
    2. a fundamental programming construct (similar to class in object-oriented programming);
    3. an interface for software types to have implementations of.

    In this document we treat the monad structure as a design pattern, [Wk3]. (After reading [H3] point 2 becomes more obvious. A similar in spirit, minimalistic approach to Object-oriented Design Patterns is given in [AA1].)

    We do not deal with types for monads explicitly, we generate code for monads instead. One reason for this is the “monad design pattern” perspective; another one is that in Mathematica / WL the notion of algebraic data type is not needed — pattern matching comes from the core “book of replacement rules” principle.

    The rest of the document is organized as follows.

    1. Fundamental sections The section “What is a monad?” gives the necessary definitions. The section “The basic Maybe monad” shows how to program a monad from scratch in Mathematica / WL. The section “Extensions with polymorphic behavior” shows how extensions of the basic monad functions can be made. (These three sections form a complete read on monadic programming, the rest of the document can be skipped.)

    2. Monadic programming in practice The section “Monad code generation” describes packages for generating monad code. The section “Flow control in monads” describes additional, control flow functionalities. The section “General work-flow of monad code generation utilization” gives a general perspective on the use of monad code generation. The section “Software design with monadic programming” discusses (small scale) software design with monadic programming.

    3. Case study sections The case study sections “Contextual monad classification” and “Tracing monad pipelines” hopefully have interesting and engaging examples of monad code generation, extension, and utilization.

    What is a monad?

    The monad definition

    In this document a monad is any set of a symbol m and two operators unit and bind that adhere to the monad laws. (See the next sub-section.) The definition is taken from [Wk1] and [H3] and phrased in Mathematica / WL terms in this section. In order to be brief, we deliberately do not consider the equivalent monad definition based on unit, join, and map (also given in [H3].)

    Here are operators for a monad associated with a certain symbol M:

    1. monad unit function (“return” in Haskell notation) is Unit[x_] := M[x];
    2. monad bind function (“>>=” in Haskell notation) is a rule like Bind[M[x_], f_] := f[x] with MatchQ[f[x],M[_]] giving True.

    Note that:

    • the function Bind unwraps the content of M[_] and gives it to the function f;
    • the functions fi are responsible to return results wrapped with the monad symbol M.

    Here is an illustration formula showing a monad pipeline:



    From the definition and formula it should be clear that if for the result of Bind[_M,f[x]] the test MatchQ[f[x],_M] is True then the result is ready to be fed to the next binding operation in monad’s pipeline. Also, it is clear that it is easy to program the pipeline functionality with Fold:

    Fold[Bind, M[x], {f1, f2, f3}]
    (* Bind[Bind[Bind[M[x], f1], f2], f3] *)

    The monad laws

    The monad laws definitions are taken from [H1] and [H3].In the monad laws given below the symbol “⟹” is for monad’s binding operation and ↦ is for a function in anonymous form.

    Here is a table with the laws:

    Remark: The monad laws are satisfied for every symbol in Mathematica / WL with List being the unit operation and Apply being the binding operation.

    Expected monadic programming features

    Looking at formula (1) — and having certain programming experiences — we can expect the following features when using monadic programming.

    • Computations that can be expressed with monad pipelines are easy to construct and read.
    • By programming the binding function we can tuck-in a variety of monad behaviours — this is the so called “programmable semicolon” feature of monads.
    • Monad pipelines can be constructed with Fold, but with suitable definitions of infix operators like DoubleLongRightArrow (⟹) we can produce code that resembles the pipeline in formula (1).
    • A monad pipeline can have polymorphic behaviour by overloading the signatures of fi (and if we have to, Bind.)

    These points are clarified below. For more complete discussions see [Wk1] or [H3].

    The basic Maybe monad

    It is fairly easy to program the basic monad Maybe discussed in [Wk1].

    The goal of the Maybe monad is to provide easy exception handling in a sequence of chained computational steps. If one of the computation steps fails then the whole pipeline returns a designated failure symbol, say None otherwise the result after the last step is wrapped in another designated symbol, say Maybe.

    Here is the special version of the generic pipeline formula (1) for the Maybe monad:



    Here is the minimal code to get a functional Maybe monad (for a more detailed exposition of code and explanations see [AA7]):

    MaybeUnitQ[x_] := MatchQ[x, None] || MatchQ[x, Maybe[___]];
    MaybeUnit[None] := None;
    MaybeUnit[x_] := Maybe[x];
    MaybeBind[None, f_] := None;
    MaybeBind[Maybe[x_], f_] := 
      Block[{res = f[x]}, If[FreeQ[res, None], res, None]];
    MaybeEcho[x_] := Maybe@Echo[x];
    MaybeEchoFunction[f___][x_] := Maybe@EchoFunction[f][x];
    MaybeOption[f_][xs_] := 
      Block[{res = f[xs]}, If[FreeQ[res, None], res, Maybe@xs]];

    In order to make the pipeline form of the code we write let us give definitions to a suitable infix operator (like “⟹”) to use MaybeBind:

    DoubleLongRightArrow[x_?MaybeUnitQ, f_] := MaybeBind[x, f];
    DoubleLongRightArrow[x_, y_, z__] := 
      DoubleLongRightArrow[DoubleLongRightArrow[x, y], z];

    Here is an example of a Maybe monad pipeline using the definitions so far:

    data = {0.61, 0.48, 0.92, 0.90, 0.32, 0.11};
    MaybeUnit[data]⟹(* lift data into the monad *)
     (Maybe@ Join[#, RandomInteger[8, 3]] &)⟹(* add more values *)
     MaybeEcho⟹(* display current value *)
     (Maybe @ Map[If[# < 0.4, None, #] &, #] &)(* map values that are too small to None *)
    (* {0.61,0.48,0.92,0.9,0.32,0.11,4,4,0}
     None *)

    The result is None because:

    1. the data has a number that is too small, and
    2. the definition of MaybeBind stops the pipeline aggressively using a FreeQ[_,None] test.

    Monad laws verification

    Let us convince ourselves that the current definition of MaybeBind gives a monad.

    The verification is straightforward to program and shows that the implemented Maybe monad adheres to the monad laws.



    Extensions with polymorphic behavior

    We can see from formulas (1) and (2) that the monad codes can be easily extended through overloading the pipeline functions.

    For example the extension of the Maybe monad to handle of Dataset objects is fairly easy and straightforward.

    Here is the formula of the Maybe monad pipeline extended with Dataset objects:

    Here is an example of a polymorphic function definition for the Maybe monad:

    MaybeFilter[filterFunc_][xs_] := Maybe@Select[xs, filterFunc[#] &];
    MaybeFilter[critFunc_][xs_Dataset] := Maybe@xs[Select[critFunc]];

    See [AA7] for more detailed examples of polymorphism in monadic programming with Mathematica / WL.

    A complete discussion can be found in [H3]. (The main message of [H3] is the poly-functional and polymorphic properties of monad implementations.)

    Polymorphic monads in R’s dplyr

    The R package dplyr, [R1], has implementations centered around monadic polymorphic behavior. The command pipelines based on dplyrcan work on R data frames, SQL tables, and Spark data frames without changes.

    Here is a diagram of a typical work-flow with dplyr:


    The diagram shows how a pipeline made with dplyr can be re-run (or reused) for data stored in different data structures.

    Monad code generation

    We can see monad code definitions like the ones for Maybe as some sort of initial templates for monads that can be extended in specific ways depending on their applications. Mathematica / WL can easily provide code generation for such templates; (see [WL1]). As it was mentioned in the introduction, we do not deal with types for monads explicitly, we generate code for monads instead.

    In this section are given examples with packages that generate monad codes. The case study sections have examples of packages that utilize generated monad codes.

    Maybe monads code generation

    The package [AA2] provides a Maybe code generator that takes as an argument a prefix for the generated functions. (Monad code generation is discussed further in the section “General work-flow of monad code generation utilization”.)

    Here is an example:

    data = {0.61, 0.48, 0.92, 0.90, 0.32, 0.11};
    AnotherMaybeUnit[data]⟹(* lift data into the monad *)
     (AnotherMaybe@Join[#, RandomInteger[8, 3]] &)⟹(* add more values *)
     AnotherMaybeEcho⟹(* display current value *)
     (AnotherMaybe @ Map[If[# < 0.4, None, #] &, #] &)(* map values that are too small to None *)
    (* {0.61,0.48,0.92,0.9,0.32,0.11,8,7,6}
       AnotherMaybeBind: Failure when applying: Function[AnotherMaybe[Map[Function[If[Less[Slot[1], 0.4], None, Slot[1]]], Slot[1]]]]
       None *)

    We see that we get the same result as above (None) and a message prompting failure.

    State monads code generation

    The State monad is also basic and its programming in Mathematica / WL is not that difficult. (See [AA3].)

    Here is the special version of the generic pipeline formula (1) for the State monad:



    Note that since the State monad pipeline caries both a value and a state, it is a good idea to have functions that manipulate them separately. For example, we can have functions for context modification and context retrieval. (These are done in [AA3].)

    This loads the package [AA3]:


    This generates the State monad for the prefix “StMon”:


    The following StMon pipeline code starts with a random matrix and then replaces numbers in the current pipeline value according to a threshold parameter kept in the context. Several times are invoked functions for context deposit and retrieval.

    StMonUnit[RandomReal[{0, 1}, {3, 2}], <|"mark" -> "TooSmall", "threshold" -> 0.5|>]⟹
      (StMon[#1 /. (x_ /; x < #2["threshold"] :> #2["mark"]), #2] &)⟹
    (* value: {{0.789884,0.831468},{0.421298,0.50537},{0.0375957,0.289442}}
       context: <|mark->TooSmall,threshold->0.5|>
       context: <|mark->TooSmall,threshold->0.5,data->{{0.789884,0.831468},{0.421298,0.50537},{0.0375957,0.289442}}|>
       value: {{0.789884,0.831468},{TooSmall,0.50537},{TooSmall,TooSmall}}
       value: {{0.789884,0.831468},{0.421298,0.50537},{0.0375957,0.289442}}
       value: TooSmall *)

    Flow control in monads

    We can implement dedicated functions for governing the pipeline flow in a monad.

    Let us look at a breakdown of these kind of functions using the State monad StMon generated above.

    Optional acceptance of a function result

    A basic and simple pipeline control function is for optional acceptance of result — if failure is obtained applying f then we ignore its result (and keep the current pipeline value.)

    Here is an example with StMonOption :

    StMonUnit[RandomReal[{0, 1}, 5]]⟹
     StMonOption[If[# < 0.3, None] & /@ # &]⟹
    (* value: {0.789884,0.831468,0.421298,0.50537,0.0375957}
       value: {0.789884,0.831468,0.421298,0.50537,0.0375957}
       StMon[{0.789884, 0.831468, 0.421298, 0.50537, 0.0375957}, <||>] *)

    Without StMonOption we get failure:

    StMonUnit[RandomReal[{0, 1}, 5]]⟹
     (If[# < 0.3, None] & /@ # &)⟹
    (* value: {0.789884,0.831468,0.421298,0.50537,0.0375957}
       StMonBind: Failure when applying: Function[Map[Function[If[Less[Slot[1], 0.3], None]], Slot[1]]]
       None *)

    Conditional execution of functions

    It is natural to want to have the ability to chose a pipeline function application based on a condition.

    This can be done with the functions StMonIfElse and StMonWhen.

    StMonUnit[RandomReal[{0, 1}, 5]]⟹
      Or @@ (# < 0.4 & /@ #) &,
      (Echo["A too small value is present.", "warning:"]; 
        StMon[Style[#1, Red], #2]) &,
      StMon[Style[#1, Blue], #2] &]⟹
     (* value: {0.789884,0.831468,0.421298,0.50537,0.0375957}
        warning: A too small value is present.
        value: {0.789884,0.831468,0.421298,0.50537,0.0375957}
        StMon[{0.789884,0.831468,0.421298,0.50537,0.0375957},<||>] *)

    Remark: Using flow control functions like StMonIfElse and StMonWhen with appropriate messages is a better way of handling computations that might fail. The silent failures handling of the basic Maybe monad is convenient only in a small number of use cases.

    Iterative functions

    The last group of pipeline flow control functions we consider comprises iterative functions that provide the functionalities of Nest, NestWhile, FoldList, etc.

    In [AA3] these functionalities are provided through the function StMonIterate.

    Here is a basic example using Nest that corresponds to Nest[#+1&,1,3]:

    StMonUnit[1]⟹StMonIterate[Nest, (StMon[#1 + 1, #2]) &, 3]
    (* StMon[4, <||>] *)

    Consider this command that uses the full signature of NestWhileList:

    NestWhileList[# + 1 &, 1, # < 10 &, 1, 4]
    (* {1, 2, 3, 4, 5} *)

    Here is the corresponding StMon iteration code:

    StMonUnit[1]⟹StMonIterate[NestWhileList, (StMon[#1 + 1, #2]) &, (#[[1]] < 10) &, 1, 4]
    (* StMon[{1, 2, 3, 4, 5}, <||>] *)

    Here is another results accumulation example with FixedPointList :

     StMonIterate[FixedPointList, (StMon[(#1 + 2/#1)/2, #2]) &]
    (* StMon[{1., 1.5, 1.41667, 1.41422, 1.41421, 1.41421, 1.41421}, <||>] *)

    When the functions NestList, NestWhileList, FixedPointList are used with StMonIterate their results can be stored in the context. Here is an example:

     StMonIterate[FixedPointList, (StMon[(#1 + 2/#1)/2, #2]) &, "fpData"]
    (* StMon[{1., 1.5, 1.41667, 1.41422, 1.41421, 1.41421, 1.41421}, <|"fpData" -> {StMon[1., <||>], 
        StMon[1.5, <||>], StMon[1.41667, <||>], StMon[1.41422, <||>], StMon[1.41421, <||>], 
        StMon[1.41421, <||>], StMon[1.41421, <||>]} |>] *)

    More elaborate tests can be found in [AA8].

    Partial pipelines

    Because of the associativity law we can design pipeline flows based on functions made of “sub-pipelines.”

    fEcho = Function[{x, ct}, StMonUnit[x, ct]⟹StMonEchoValue⟹StMonEchoContext];
    fDIter = Function[{x, ct}, 
       StMonUnit[y^x, ct]⟹StMonIterate[FixedPointList, StMonUnit@D[#, y] &, 20]];
      value: 7
      context: <||>
      value: {y^7,7 y^6,42 y^5,210 y^4,840 y^3,2520 y^2,5040 y,5040,0,0}
      context: <||> *)

    General work-flow of monad code generation utilization

    With the abilities to generate and utilize monad codes it is natural to consider the following work flow. (Also shown in the diagram below.)

    1. Come up with an idea that can be expressed with monadic programming.
    2. Look for suitable monad implementation.
    3. If there is no such implementation, make one (or two, or five.)
    4. Having a suitable monad implementation, generate the monad code.
    5. Implement additional pipeline functions addressing envisioned use cases.
    6. Start making pipelines for the problem domain of interest.
    7. Are the pipelines are satisfactory? If not go to 5. (Or 2.)


    Monad templates

    The template nature of the general monads can be exemplified with the group of functions in the package StateMonadCodeGenerator.m, [4].

    They are in five groups:

    1. base monad functions (unit testing, binding),
    2. display of the value and context,
    3. context manipulation (deposit, retrieval, modification),
    4. flow governing (optional new value, conditional function application, iteration),
    5. other convenience functions.

    We can say that all monad implementations will have their own versions of these groups of functions. The more specialized monads will have functions specific to their intended use. Such special monads are discussed in the case study sections.

    Software design with monadic programming

    The application of monadic programming to a particular problem domain is very similar to designing a software framework or designing and implementing a Domain Specific Language (DSL).

    The answers of the question “When to use monadic programming?” can form a large list. This section provides only a couple of general, personal viewpoints on monadic programming in software design and architecture. The principles of monadic programming can be used to build systems from scratch (like Haskell and Scala.) Here we discuss making specialized software with or within already existing systems.

    Framework design

    Software framework design is about architectural solutions that capture the commonality and variability in a problem domain in such a way that: 1) significant speed-up can be achieved when making new applications, and 2) a set of policies can be imposed on the new applications.

    The rigidness of the framework provides and supports its flexibility — the framework has a backbone of rigid parts and a set of “hot spots” where new functionalities are plugged-in.

    Usually Object-Oriented Programming (OOP) frameworks provide inversion of control — the general work-flow is already established, only parts of it are changed. (This is characterized with “leave the driving to us” and “don’t call us we will call you.”)

    The point of utilizing monadic programming is to be able to easily create different new work-flows that share certain features. (The end user is the driver, on certain rail paths.)

    In my opinion making a software framework of small to moderate size with monadic programming principles would produce a library of functions each with polymorphic behaviour that can be easily sequenced in monadic pipelines. This can be contrasted with OOP framework design in which we are more likely to end up with backbone structures that (i) are static and tree-like, and (ii) are extended or specialized by plugging-in relevant objects. (Those plugged-in objects themselves can be trees, but hopefully short ones.)

    DSL development

    Given a problem domain the general monad structure can be used to shape and guide the development of DSLs for that problem domain.

    Generally, in order to make a DSL we have to choose the language syntax and grammar. Using monadic programming the syntax and grammar commands are clear. (The monad pipelines are the commands.) What is left is “just” the choice of particular functions and their implementations.

    Another way to develop such a DSL is through a grammar of natural language commands. Generally speaking, just designing the grammar — without developing the corresponding interpreters — would be very helpful in figuring out the components at play. Monadic programming meshes very well with this approach and applying the two approaches together can be very fruitful.

    Contextual monad classification (case study)

    In this section we show an extension of the State monad into a monad aimed at machine learning classification work-flows.


    We want to provide a DSL for doing machine learning classification tasks that allows us:

    1. to do basic summarization and visualization of the data,
    2. to control splitting of the data into training and testing sets;
    3. to apply the built-in classifiers;
    4. to apply classifier ensembles (see [AA9] and [AA10]);
    5. to evaluate classifier performances with standard measures and
    6. ROC plots.

    Also, we want the DSL design to provide clear directions how to add (hook-up or plug-in) new functionalities.

    The package [AA4] discussed below provides such a DSL through monadic programming.

    Package and data loading

    This loads the package [AA4]:


    This gets some test data (the Titanic dataset):

    dataName = "Titanic";
    ds = Dataset[Flatten@*List @@@ ExampleData[{"MachineLearning", dataName}, "Data"]];
    varNames = Flatten[List @@ ExampleData[{"MachineLearning", dataName}, "VariableDescriptions"]];
    varNames = StringReplace[varNames, "passenger" ~~ (WhitespaceCharacter ..) -> ""];
    If[dataName == "FisherIris", varNames = Most[varNames]];
    ds = ds[All, AssociationThread[varNames -> #] &];

    Monad design

    The package [AA4] provides functions for the monad ClCon — the functions implemented in [AA4] have the prefix “ClCon”.

    The classifier contexts are Association objects. The pipeline values can have the form:

    ClCon[ val, context:(_String|_Association) ]

    The ClCon specific monad functions deposit or retrieve values from the context with the keys: “trainData”, “testData”, “classifier”. The general idea is that if the current value of the pipeline cannot provide all arguments for a ClCon function, then the needed arguments are taken from the context. If that fails, then an message is issued. This is illustrated with the following pipeline with comments example.


    The pipeline and results above demonstrate polymorphic behaviour over the classifier variable in the context: different functions are used if that variable is a ClassifierFunction object or an association of named ClassifierFunction objects.

    Note the demonstrated granularity and sequentiality of the operations coming from using a monad structure. With those kind of operations it would be easy to make interpreters for natural language DSLs.

    Another usage example

    This monadic pipeline in this example goes through several stages: data summary, classifier training, evaluation, acceptance test, and if the results are rejected a new classifier is made with a different algorithm using the same data splitting. The context keeps track of the data and its splitting. That allows the conditional classifier switch to be concisely specified.

    First let us define a function that takes a Classify method as an argument and makes a classifier and calculates performance measures.

    ClSubPipe[method_String] :=
      Function[{x, ct},
       ClConUnit[x, ct]⟹
         ClassifierInformation[#["classifier"], Method] &]⟹
        ClConEchoFunctionContext["training time:", ClassifierInformation[#["classifier"], "TrainingTime"] &]⟹
        ClConClassifierMeasurements[{"Accuracy", "Precision", "Recall"}]⟹
         ClConToNormalClassifierData[#["testData"]], "ROCCurve"] &]

    Using the sub-pipeline function ClSubPipe we make the outlined pipeline.

    res =
       ClConEchoFunctionValue["summaries:", ColumnForm[Normal[RecordsSummary /@ #]] &]⟹
        MatrixForm[CrossTensorate[Count == varNames[[1]] + varNames[[-1]], #]] & /@ # &]⟹
       (If[#1["Accuracy"] > 0.8,
          Echo["Good accuracy!", "Success:"]; ClConFail,
          Echo["Make a new classifier", "Inaccurate:"]; 
          ClConUnit[#1, #2]] &)⟹


    Tracing monad pipelines (case study)

    The monadic implementations in the package MonadicTracing.m, [AA5] allow tracking of the pipeline execution of functions within other monads.

    The primary reason for developing the package was the desire to have the ability to print a tabulated trace of code and comments using the usual monad pipeline notation. (I.e. without conversion to strings etc.)

    It turned out that by programming MonadicTracing.m I came up with a monad transformer; see [Wk2], [H2].

    Package loading

    This loads the package [AA5]:


    Usage example

    This generates a Maybe monad to be used in the example (for the prefix “Perhaps”):


    In following example we can see that pipeline functions of the Perhaps monad are interleaved with comment strings. Producing the grid of functions and comments happens “naturally” with the monad function TraceMonadEchoGrid.

    data = RandomInteger[10, 15];
    TraceMonadUnit[PerhapsUnit[data]]⟹"lift to monad"⟹
      PerhapsFilter[# > 3 &]⟹"filter current value"⟹
      PerhapsEcho⟹"display current value"⟹
      PerhapsWhen[#[[3]] > 3 &, 
       PerhapsEchoFunction[Style[#, Red] &]]⟹
      (Perhaps[#/4] &)⟹
      PerhapsEcho⟹"display current value again"⟹
      TraceMonadEchoGrid[Grid[#, Alignment -> Left] &];

    Note that :

    1. the tracing is initiated by just using TraceMonadUnit;
    2. pipeline functions (actual code) and comments are interleaved;
    3. putting a comment string after a pipeline function is optional.

    Another example is the ClCon pipeline in the sub-section “Monad design” in the previous section.


    This document presents a style of using monadic programming in Wolfram Language (Mathematica). The style has some shortcomings, but it definitely provides convenient features for day-to-day programming and in coming up with architectural designs.

    The style is based on WL’s basic language features. As a consequence it is fairly concise and produces light overhead.

    Ideally, the packages for the code generation of the basic Maybe and State monads would serve as starting points for other more general or more specialized monadic programs.


    Monadic programming

    [Wk1] Wikipedia entry: Monad (functional programming), URL: https://en.wikipedia.org/wiki/Monad_(functional_programming) .

    [Wk2] Wikipedia entry: Monad transformer, URL: https://en.wikipedia.org/wiki/Monad_transformer .

    [Wk3] Wikipedia entry: Software Design Pattern, URL: https://en.wikipedia.org/wiki/Software_design_pattern .

    [H1] Haskell.org article: Monad laws, URL: https://wiki.haskell.org/Monad_laws.

    [H2] Sheng Liang, Paul Hudak, Mark Jones, “Monad transformers and modular interpreters”, (1995), Proceedings of the 22nd ACM SIGPLAN-SIGACT symposium on Principles of programming languages. New York, NY: ACM. pp. 333[Dash]343. doi:10.1145/199448.199528.

    [H3] Philip Wadler, “The essence of functional programming”, (1992), 19’th Annual Symposium on Principles of Programming Languages, Albuquerque, New Mexico, January 1992.


    [R1] Hadley Wickham et al., dplyr: A Grammar of Data Manipulation, (2014), tidyverse at GitHub, URL: https://github.com/tidyverse/dplyr . (See also, http://dplyr.tidyverse.org .)

    Mathematica / Wolfram Language

    [WL1] Leonid Shifrin, “Metaprogramming in Wolfram Language”, (2012), Mathematica StackExchange. (Also posted at Wolfram Community in 2017.) URL of the Mathematica StackExchange answer: https://mathematica.stackexchange.com/a/2352/34008 . URL of the Wolfram Community post: http://community.wolfram.com/groups/-/m/t/1121273 .


    [AA1] Anton Antonov, “Implementation of Object-Oriented Programming Design Patterns in Mathematica”, (2016) MathematicaForPrediction at GitHub, https://github.com/antononcube/MathematicaForPrediction.

    [AA2] Anton Antonov, Maybe monad code generator Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MaybeMonadCodeGenerator.m .

    [AA3] Anton Antonov, State monad code generator Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/StateMonadCodeGenerator.m .

    [AA4] Anton Antonov, Monadic contextual classification Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicContextualClassification.m .

    [AA5] Anton Antonov, Monadic tracing Mathematica package, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicTracing.m .

    [AA6] Anton Antonov, MathematicaForPrediction utilities, (2014), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MathematicaForPredictionUtilities.m .

    [AA7] Anton Antonov, Simple monadic programming, (2017), MathematicaForPrediction at GitHub. (Preliminary version, 40% done.) URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/Documentation/Simple-monadic-programming.pdf .

    [AA8] Anton Antonov, Generated State Monad Mathematica unit tests, (2017), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/UnitTests/GeneratedStateMonadTests.m .

    [AA9] Anton Antonov, Classifier ensembles functions Mathematica package, (2016), MathematicaForPrediction at GitHub. URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/ClassifierEnsembles.m .

    [AA10] Anton Antonov, “ROC for classifier ensembles, bootstrapping, damaging, and interpolation”, (2016), MathematicaForPrediction at WordPress. URL: https://mathematicaforprediction.wordpress.com/2016/10/15/roc-for-classifier-ensembles-bootstrapping-damaging-and-interpolation/ .

    RSparseMatrix for sparse matrices with named rows and columns


    In the last few years I have used a lot R’s base library Matrix that has implementation of sparse matrix objects and efficient computations. To the sparse matrices from R’s Matrix library one can assign and retrieve row names and column names with the functions colnames and rownames. Sometimes I miss this in Mathematica so I started a Mathematica package that implements similar functionalities. The package is named RSparseMatrix.m has purely Mathematica language implementations (i.e. it does not use RLink ). It can be loaded/downloaded from MathematicaForPrediction at GitHub:

    Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/Misc/RSparseMatrix.m"]The package provides functions to create and do operations over  RSparseMatrix objects of that are basically SparseArray objects with row and column names. A major design decision is to restrict these functionalities to two dimensional sparse arrays and lists of strings as row and column names. (Note that the package is not finished and in some functions the row and column names are ignored.)

    The package attempts to cover as many as possible of the functionalities for sparse matrix objects that are provided by R’s Matrix library. (Sub-matrix extraction by row and column names, row and column names propagation for dot products, row and column binding sparse matrices, row and column sums, etc.) This document has examples and tests for RSparseMatrix.m .

    My participation in WTC 2015 with a talk about Mathematica and R comparison was one the main motivators to write this blog post. Another is this Mathematica StackExchange discussion. (And a third one is seeing tonight the impressive movie “The Martian” — such a display of the triumph of the humans over space and nature using technology and science in a creative way made me wanna discuss how to make some programming objects more convenient.)

    Update 2018-04-05

    The package RSparseMatrix.m is superseded by the package SSparseMatrix.m.
    All operations in the code below work with the latter package if “RSparseMatrix” is replaced with “SSparseMatrix”.
    Also see the unit tests file SSparseMatrix-tests.wlt.

    Basic examples


    rmat = MakeRSparseMatrix[
     {{1, 1} -> 1, {2, 2} -> 2, {4, 3} -> 3, {1, 4} -> 4, {3, 5} -> 2},
     "ColumnNames" -> {"a", "b", "c", "d", "e"},
     "RowNames" -> {"A", "B", "C", "D"},
     "DimensionNames" -> {"U", "V"}]

    The function MatrixForm shows the RSparseMatrix objects with their row and column names:

    rmat // MatrixForm

    The RSparseMatrix objects can be created from SparseArray objects:

    Query functions

    These functions can be used to retrieve the names of rows, columns, and dimensions. They correspond to R’s functions rownames, colnames, dimnames.

    In[154]:= RowNames[rmat]
    Out[154]= {"A", "B", "C", "D"}
    In[155]:= ColumnNames[rmat]
    Out[155]= {"a", "b", "c", "d", "e"}
    In[156]:= DimensionNames[rmat]
    Out[156]= {"U", "V"}

    Functions that work on SparseArray

    Of course since RSparseMatrix is based on SparseArray we would expect the functions that work on SparseArray objects to work RSpaseMatrix objects too. E.g. Dimensions, ArrayRules, Transpose, Total, and others.

    In[157]:= Dimensions[rmat]
    Out[157]= {4, 5}
    In[158]:= ArrayRules[rmat]
    Out[158]= {{1, 1} -> 1, {1, 4} -> 4, {2, 2} -> 2, {3, 5} -> 2, {4, 3} -> 3, {_, _} -> 0}

    Dot product

    Row names and column names are respected for dot products if that leads to meaningful assignments. The examples below demonstrate a general principle:

    When a matrix operation can be performed on the underlying sparse arrays but the row names or column names do not coincide the names are dropped.

    In the tables with examples below the last rows show the heads of the results.

    Matrix by vector


    Matrix by matrix




    A major useful feature is to have Part work with row and column names. The implementation of that additional functionality for Part is demonstrated below.

    In the cases when the dimension drops sparse arrays or numbers are returned. In R the operation “[” has the parameter “drop” — the expression “smat[1,,drop=F]” is going to be a sparse matrix, the expression “smat[1,,drop=T]” is going to be a dense vector. The corresponding implementation is to have the option “Drop->True|False” for Part, but that does not seem a good idea. And we can easily emulate the “drop” option in R using “{_?AtomQ}” inside Part.


    Neat example

    Consider this incidence matrix that represents a bi-partite graph of relationships of actors starring in movies:


    We can use a RSparseMatrix object of it with named rows and columns (rBiMat).

    Here is the corresponding graph:


    If we want to see which actors have participated in movies together with Orlando Bloom we can do the following:


    Movie genre associations

    In this post we are going to look at genre associations deduced by extracting association rules from a catalog of movies. For example, we might want to confirm that most romance movies are also dramas, and we want to find similar rules. For more details see this user guide https://github.com/antononcube/MathematicaForPrediction/blob/master/Documentation/MovieLens%20genre%20associations.pdf at Mathematica for Prediction at GitHub .

    The movie data was taken from the page MovieLens Data Sets (http://www.grouplens.org/taxonomy/term/14) of the site GroupLens Research (http://www.grouplens.org). More precisely, the data set named “MovieLens 10M Data Set” was taken.

    We are interested in the movie-genre relations only and if we look only at the movie-genre relations of “MovieLens 10M Data Set” the movies are poorly interconnected. Approximately 40% of the movies have only one genre. We use MovieLens since it is publicly available, easy to ingest, and the presented results can be reproduced.

    Here is a sample of the movie-genre data:
    MovieLens 10k movie-genre data sample

    Let us first look into some descriptive statistics of the data set.

    We have 10681 movies, and 18 genres. Here is a breakdown of the movies across the genres:
    MovieLens 10k movie-genre data genre breakdown

    Here are a table of descriptive statistics and a histogram of the distribution of the number of genres:
    MovieLens 10k Descriptive statistics for the number of genres

    Here are a table of descriptive statistics and a histogram with the distribution of the movies across the release years:
    MovieLens 10k Release years descriptive statistics

    I applied to the movie-genre data the algorithm Apriori which is an associative rules learning algorithm. The Mathematica implementation is available at this link: https://github.com/antononcube/MathematicaForPrediction/blob/master/AprioriAlgorithm.m

    With the Apriori algorithm we can find frequent association genre sets. In order to apply Apriori from each data row only the genres are taken. In this way we can see each movie as a “basket” of genres or as a “transaction” of genres, and the total movie catalog as a set of transactions.

    In order to extract association rules from each frequent set we apply different measures. The GitHub package provides five measures: Support, Confidence, Lift, Leverage, and Conviction. The measure Support follows the standard mathematical definition (fraction of the total number of transactions) and it is used to find the association sets. Conviction is considered to be the best for uncovering interesting rules. The definition and interpretation of the measures are given in these tables:
    Tables of definitions and properties of association rules measures

    I implemented a dynamic interface to browse the association sets that have support higher than 0.25% :
    Association sets dynamic interface

    This 2×2 table of interface snapshots shows the association sets that have the largest support:
    Association sets interface snapshots

    We can see that — as expected — “Romance” and “Drama” are highly associated. Other expected associations are {“Comedy”, “Drama”, “Romance”} , {“Crime”, “Drama”, “Thriller”}, and {“Action”, “Crime”, “Thriller”}.

    I also implemented a dynamic interface for browsing the association rules extracted from the frequent sets. Here is a list of snapshots of that interface:
    1. Association rules of 2 items for all genres ordered by Conviction:
    2 item rules for All ordered by Conviction
    2. Association rules of 3 items for all genres ordered by Conviction:
    3 item rules for All ordered by Conviction
    3. Association rules of 2 items with “Drama” ordered by Conviction:
    2 item rules for Drama ordered by Conviction
    4. Association rules of 3 items with “Drama” ordered by Conviction:
    3 item rules for Drama ordered by Conviction

    Again, the results we see are expected. For example, looking at the measure Confidence we can see that for the MovieLens 10k catalog 82% of the romance-war movies are also dramas, and 73% of the war movies are dramas. In a certain sense, “War” and {“Romance”, “War”} function like sub-genres of “Drama”.

    Digit recognition interface with an RPN calculator

    This post is a follow-up of my previous blog post “Classification of handwritten†digits” about recognition of digit drawings.

    A friend of mine pointed out that “the recognition of digits was one of the assignments in the Coursera Machine Learning course, using Matlab or Octave.” (Octave is free, open source version of Matlab.)

    So I decided to do a digit recognition and interpretation interactive interface in/with Mathematica, which is not something that can be done with Matlab or Octave. Here is a link to a video demonstrating the interface: http://youtu.be/iPF5Apa6OjY .

    Here is a screenshot:
    Digit recognition with an RPN calculator screenshot with 7