# Investigating COVID-19 with R: data analysis and simulations

Methodological presentation
R-Ladies Miami Meetup, May 28th 2020

The extended abstract of the presentation was loosely followed. Here is the presentation mind-map:

(Note that mind-map’s PDF has hyperlinks. Also, see the folder Presentation-aids. )

The organizers and I did a poll for what people want to hear. After discussing the results of the 15 votes from that poll we decided the presentation to be a methodological one instead of a know-how one.

Approximately 30% of the presentation was based on the R-project “COVID-19-modeling-in-R”, [AA1].

Approximately 30% of the presentation was based on an R-programmed software monad for epidemiology compartmental models, ECMMon-R, [AAr2].

For the rest were used frameworks, simulations, and graphics made with Mathematica, [AAr1].

The presentation was given online (because of COVID-19) using Zoom. 90 people registered. Nearly 40 showed up (and maybe 20 stayed throughout.)

Here is a link to the video recording.

## Screenshots

Here are screenshots of statistics used in the introduction:

## References

### Coronavirus

[Wk1] Wikipedia entry, Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2).

[Wk2] Wikipedia entry, Coronavirus disease 2019.

### Modeling

[Wk3] Wikipedia entry, Compartmental models in epidemiology.

[Wk4] Wikipedia entry, System dynamics.

### R code/software

[KS1] Karline Soetaert, Thomas Petzoldt, R. Woodrow Setzer, “deSolve: Solvers for Initial Value Problems of Differential Equations (‘ODE’, ‘DAE’, ‘DDE’)”, CRAN.

[AA1] Anton Antonov, “COVID-19-modeling-in-R”, 2020, SystemModeling at GitHub.

[AAr1] Anton Antonov, Coronavirus-propagation-dynamics, 2020, SystemModeling at GitHub.

[AAr2] Anton Antonov, Epidemiology Compartmental Modeling Monad in R, 2020, ECMMon-R at GitHub.

# Apple mobility trends data visualization (for COVID-19)

## Introduction

I this notebook we ingest and visualize the mobility trends data provided by Apple, [APPL1].

We take the following steps:

2. Import the data and summarise it
3. Transform the data into long form
4. Partition the data into subsets that correspond to combinations of geographical regions and transportation types
5. Make contingency matrices and corresponding heat-map plots
6. Make nearest neighbors graphs over the contingency matrices and plot communities
7. Plot the corresponding time series

### Data description

#### From Apple’s page https://www.apple.com/covid19/mobility

About This Data The CSV file and charts on this site show a relative volume of directions requests per country/region or city compared to a baseline volume on January 13th, 2020. We define our day as midnight-to-midnight, Pacific time. Cities represent usage in greater metropolitan areas and are stably defined during this period. In many countries/regions and cities, relative volume has increased since January 13th, consistent with normal, seasonal usage of Apple Maps. Day of week effects are important to normalize as you use this data. Data that is sent from users’ devices to the Maps service is associated with random, rotating identifiers so Apple doesn’t have a profile of your movements and searches. Apple Maps has no demographic information about our users, so we can’t make any statements about the representativeness of our usage against the overall population.

### Observations

The observations listed in this subsection are also placed under the relevant statistics in the following sections and indicated with “Observation”.

• The directions requests volumes reference date for normalization is 2020-01-13 : all the values in that column are $100$.
• From the community clusters of the nearest neighbor graphs (derived from the time series of the normalized driving directions requests volume) we see that countries and cities are clustered in expected ways. For example, in the community graph plot corresponding to “{city, driving}” the cities Oslo, Copenhagen, Helsinki, Stockholm, and Zurich are placed in the same cluster. In the graphs corresponding to “{city, transit}” and “{city, walking}” the Japanese cities Tokyo, Osaka, Nagoya, and Fukuoka are clustered together.
• In the time series plots the Sundays are indicated with orange dashed lines. We can see that from Monday to Thursday people are more familiar with their trips than say on Fridays and Saturdays. We can also see that on Sundays people (on average) are more familiar with their trips or simply travel less.

``````Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/DataReshape.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/Misc/HeatmapPlot.m"]``````

## Data ingestion

Apple mobile data was provided in this WWW page: https://www.apple.com/covid19/mobility , [APPL1]. (The data has to be download from that web page – there is an “agreement to terms”, etc.)

``dsAppleMobility = ResourceFunction["ImportCSVToDataset"]["~/Downloads/applemobilitytrends-2020-04-14.csv"]``

Observation: The directions requests volumes reference date for normalization is 2020-01-13 : all the values in that column are $100$.

Data dimensions:

``````Dimensions[dsAppleMobility]

(*{395, 96}*)``````

Data summary:

``Magnify[ResourceFunction["RecordsSummary"][dsAppleMobility], 0.6]``

Number of unique “country/region” values:

``````Length[Union[Normal[dsAppleMobility[Select[#["geo_type"] == "country/region" &], "region"]]]]

(*63*)``````

Number of unique “city” values:

``````Length[Union[Normal[dsAppleMobility[Select[#["geo_type"] == "city" &], "region"]]]]

(*89*)``````

All unique geo types:

``````lsGeoTypes = Union[Normal[dsAppleMobility[All, "geo_type"]]]

(*{"city", "country/region"}*)``````

All unique transportation types:

``````lsTransportationTypes = Union[Normal[dsAppleMobility[All, "transportation_type"]]]

(*{"driving", "transit", "walking"}*)``````

## Transform data

It is better to have the data in long form (narrow form). For that I am using the package “DataReshape.m”, [AAp1].

``````lsIDColumnNames = {"geo_type", "region", "transportation_type"};
dsAppleMobilityLongForm = ToLongForm[dsAppleMobility, lsIDColumnNames, Complement[Keys[dsAppleMobility[[1]]], lsIDColumnNames]];
Dimensions[dsAppleMobilityLongForm]

(*{36735, 5}*)``````

Remove the rows with “empty” values:

``````dsAppleMobilityLongForm = dsAppleMobilityLongForm[Select[#Value != "" &]];
Dimensions[dsAppleMobilityLongForm]

(*{36735, 5}*)``````

Rename the column “Variable” to “Date” and add a related “DateObject” column:

``````AbsoluteTiming[
dsAppleMobilityLongForm =
dsAppleMobilityLongForm[All, Join[KeyDrop[#, "Variable"], <|"Date" -> #Variable, "DateObject" -> DateObject[#Variable]|>] &];
]

(*{16.9671, Null}*)``````

Add “day name” (“day of the week”) field:

``dsAppleMobilityLongForm = dsAppleMobilityLongForm[All, Join[#, <|"DayName" -> DateString[#DateObject, {"DayName"}]|>] &];``

Here is sample of the transformed data:

``````SeedRandom[3232]
RandomSample[dsAppleMobilityLongForm, 12]``````

Here is summary:

``ResourceFunction["RecordsSummary"][dsAppleMobilityLongForm]``

Partition the data into geo types × transportation types:

``````aQueries =
Association@
Flatten@Outer[
Function[{gt, tt}, {gt, tt} ->
dsAppleMobilityLongForm[Select[#["geo_type"] == gt && #["transportation_type"] == tt &]]], lsGeoTypes, lsTransportationTypes];``````

## Basic data analysis

We consider relative volume o directions requests for the last date only. (The queries can easily adjusted for other dates.)

``````lastDate = Last@Sort@Normal@dsAppleMobilityLongForm[All, "Date"]

(*"2020-04-14"*)``````
``````aDayQueries =
Association@
Flatten@Outer[
Function[{gt, tt}, {gt, tt} -> dsAppleMobilityLongForm[Select[#["geo_type"] == gt && #Date == lastDate && #["transportation_type"] == tt &]]],
lsGeoTypes, lsTransportationTypes];``````
``````Dimensions /@ aDayQueries

(*<|{"city", "driving"} -> {89, 7}, {"city", "transit"} -> {64, 7}, {"city", "walking"} -> {89, 7},
{"country/region", "driving"} -> {63, 7}, {"country/region", "transit"} -> {27, 7},
{"country/region", "walking"} -> {63, 7}|>*)``````

Here we plot histograms and Pareto principle adherence:

``````opts = {PlotRange -> All, ImageSize -> Medium};
Grid[
Function[{columnName},
{Histogram[#, 12, PlotLabel -> columnName, opts],
ResourceFunction["ParetoPrinciplePlot"][#, PlotLabel -> columnName, opts]} &@Normal[#[All, "Value"]]
] /@ {"Value"},
Dividers -> All, FrameStyle -> GrayLevel[0.7]] & /@ aDayQueries``````

## Heat-map plots

We can visualize the data using heat-map plots. Here we use the package “HeatmapPlot.m”, [AAp2].

Remark: Using the contingency matrices prepared for the heat-map plots we can do further analysis, like, finding correlations or nearest neighbors. (See below.)

Cross-tabulate dates with regions:

``aMatDateRegion = ResourceFunction["CrossTabulate"][#[All, {"Date", "region", "Value"}], "Sparse" -> True] & /@ aQueries;``

Make a heat-map plot by sorting the columns of the cross-tabulation matrix (that correspond to countries):

``````aHeatMapPlots =
Association@
KeyValueMap[#1 ->
Rasterize[
HeatmapPlot[#2, PlotLabel -> #1,
DistanceFunction -> {None, CosineDistance},
AspectRatio -> 1/1.6, ImageSize -> 1600]] &, aMatDateRegion]``````

(We use Rasterize in order to reduce the size of the notebook.)

Here we take closer look to one of the plots:

``aHeatMapPlots[{"country/region", "driving"}]``

## Nearest neighbors graphs

### Graphs overview

Here we create nearest neighbor graphs of the contingency matrices computed above and plot cluster the nodes:

``````Manipulate[
Multicolumn[
Normal@Map[
CommunityGraphPlot @ NearestNeighborGraph[Normal[Transpose[#SparseMatrix]], nns, ImageSize -> Medium] &, aMatDateRegion], 2, Dividers -> All],
{{nns, 5, "Number of nearest neighbors:"}, 2, 30, 1, Appearance -> "Open"}, SaveDefinitions -> True]``````

### Closer look into the graphs

Here we endow each nearest neighbors graph with appropriate vertex labels:

``````aNNGraphs =
Map[NearestNeighborGraph[Normal[Transpose[#SparseMatrix]], 3, VertexLabels -> Thread[Rule[Normal[Transpose[#SparseMatrix]], #ColumnNames]]] &,
aMatDateRegion];``````

Here we plot the graphs with clusters:

``````ResourceFunction["GridTableForm"][
List @@@ Normal[CommunityGraphPlot[#, ImageSize -> 800] & /@ aNNGraphs],
TableHeadings -> {"region & transportation type", "communities of nearest neighbors graph"}, Background -> White,
Dividers -> All]``````

Observation: From the community clusters of the nearest neighbor graphs (derived from the time series of the normalized driving directions requests volume) we see that countries and cities are clustered in expected ways. For example in the community graph plot corresponding to “{city, driving}” the cities Oslo, Copenhagen, Helsinki, Stockholm, and Zurich are placed in the same cluster. In the graphs corresponding to “{city, transit}” and “{city, walking}” the Japanese cities Tokyo, Osaka, Nagoya, and Fukuoka are clustered together.

## Time series analysis

### Time series

In this section for each date we sum all cases over the region-transportation pairs, make a time series, and plot them.

Remark: In the plots the Sundays are indicated with orange dashed lines.

Here we make the time series:

``````aTSDirReqByCountry =
Map[
Function[{dfQuery},
TimeSeries@(List @@@
Normal[GroupBy[Normal[dfQuery], #DateObject &,
Total[#Value & /@ #] &]])
],
aQueries
]``````

Here we plot them:

``````opts = {PlotTheme -> "Detailed", PlotRange -> All, AspectRatio -> 1/4,
ImageSize -> Large};
Association@KeyValueMap[
Function[{transpType, ts},
transpType ->
DateListPlot[ts,
GridLines -> {AbsoluteTime /@
Union[Normal[ dsAppleMobilityLongForm[Select[#DayName == "Sunday" &], "DateObject"]]], Automatic},
GridLinesStyle -> {Directive[Orange, Dashed],
Directive[Gray, Dotted]}, PlotLabel -> Capitalize[transpType],
opts]
],
aTSDirReqByCountry
]``````

Observation: In the time series plots the Sundays are indicated with orange dashed lines. We can see that from Monday to Thursday people are more familiar with their trips than say on Fridays and Saturdays. We can also see that on Sundays people (on average) are more familiar with their trips or simply travel less.

### “Forecast”

He we do “forecast” for code-workflow demonstration purposes – the forecasts should not be taken seriously.

Fit a time series model to the time series:

``aTSModels = TimeSeriesModelFit /@ aTSDirReqByCountry``

Plot data and forecast:

``````Map[DateListPlot[{#["TemporalData"], TimeSeriesForecast[#, {10}]},
opts, PlotLegends -> {"Data", "Forecast"}] &, aTSModels]``````

## References

[APPL1] Apple Inc., Mobility Trends Reports, (2020), apple.com.

[AA1] Anton Antonov, “NY Times COVID-19 data visualization”, (2020), SystemModeling at GitHub.

[AAp1] Anton Antonov, Data reshaping Mathematica package, (2018), MathematicaForPrediciton at GitHub.

[AAp2] Anton Antonov, Heatmap plot Mathematica package, (2018), MathematicaForPrediciton at GitHub.

# NY Times COVID-19 data visualization

Yesterday in one of the forums I frequent it was announced that New York Times has published COVID-19 data on GitHub. I decided to make a Mathematica notebook that gives data links and related code for data ingestions. (And rudimentary data analysis.)

Here is the Markdown version of the notebook: “NY Times COVID-19 data visualization”.

Here is a screenshot of the WL notebook that also links to it:

Screenshot of an interactive interface:

# Conference abstracts similarities

## Introduction

In this MathematicaVsR project we discuss and exemplify finding and analyzing similarities between texts using Latent Semantic Analysis (LSA). Both Mathematica and R codes are provided.

The LSA workflows are constructed and executed with the software monads `LSAMon-WL`, [AA1, AAp1], and `LSAMon-R`, [AAp2].

The illustrating examples are based on conference abstracts from rstudio::conf and Wolfram Technology Conference (WTC), [AAd1, AAd2]. Since the number of rstudio::conf abstracts is small and since rstudio::conf 2020 is about to start at the time of preparing this project we focus on words and texts from RStudio’s ecosystem of packages and presentations.

## Statistical thesaurus for words from RStudio’s ecosystem

Consider the focus words:

``{"cloud","rstudio","package","tidyverse","dplyr","analyze","python","ggplot2","markdown","sql"}``

Here is a statistical thesaurus for those words:

Remark: Note that the computed thesaurus entries seem fairly “R-flavored.”

## Similarity analysis diagrams

As expected the abstracts from rstudio::conf tend to cluster closely – note the square formed top-left in the plot of a similarity matrix based on extracted topics:

Here is a similarity graph based on the matrix above:

Here is a clustering (by “graph communities”) of the sub-graph highlighted in the plot above:

## Comparison observations

### LSA pipelines specifications

The packages `LSAMon-WL`, [AAp1], and `LSAMon-R`, [AAp2], make the comparison easy – the codes of the specified workflows are nearly identical.

Here is the Mathematica code:

``````lsaObj =
LSAMonMakeDocumentTermMatrix[{}, Automatic]⟹
LSAMonEchoDocumentTermMatrixStatistics⟹
LSAMonApplyTermWeightFunctions["IDF", "TermFrequency", "Cosine"]⟹
LSAMonExtractTopics["NumberOfTopics" -> 36, "MinNumberOfDocumentsPerTerm" -> 2, Method -> "ICA", MaxSteps -> 200]⟹
LSAMonEchoTopicsTable["NumberOfTableColumns" -> 6];``````

Here is the R code:

``````lsaObj <-
LSAMonUnit(lsDescriptions) %>%
LSAMonMakeDocumentTermMatrix( stemWordsQ = FALSE, stopWords = stopwords::stopwords() ) %>%
LSAMonApplyTermWeightFunctions( "IDF", "TermFrequency", "Cosine" )
LSAMonExtractTopics( numberOfTopics = 36, minNumberOfDocumentsPerTerm = 5, method = "NNMF", maxSteps = 20, profilingQ = FALSE ) %>%
LSAMonEchoTopicsTable( numberOfTableColumns = 6, wideFormQ = TRUE ) ``````

### Graphs and graphics

Mathematica’s built-in graph functions make the exploration of the similarities much easier. (Than using R.)

Mathematica’s matrix plots provide more control and are more readily informative.

### Sparse matrix objects with named rows and columns

R’s built-in sparse matrices with named rows and columns are great. `LSAMon-WL` utilizes a similar, specially implemented sparse matrix object, see [AA1, AAp3].

## References

### Articles

[AA1] Anton Antonov, A monad for Latent Semantic Analysis workflows, (2019), MathematicaForPrediction at GitHub.

[AA2] Anton Antonov, Text similarities through bags of words, (2020), SimplifiedMachineLearningWorkflows-book at GitHub.

### Data

[AAd1] Anton Antonov, RStudio::conf-2019-abstracts.csv, (2020), SimplifiedMachineLearningWorkflows-book at GitHub.

[AAd2] Anton Antonov, Wolfram-Technology-Conference-2016-to-2019-abstracts.csv, (2020), SimplifiedMachineLearningWorkflows-book at GitHub.

### Packages

[AAp1] Anton Antonov, Monadic Latent Semantic Analysis Mathematica package, (2017), MathematicaForPrediction at GitHub.

[AAp2] Anton Antonov, Latent Semantic Analysis Monad R package, (2019), R-packages at GitHub.

[AAp3] Anton Antonov, SSparseMatrix Mathematica package, (2018), MathematicaForPrediction at GitHub.

# Pets licensing data analysis

## Introduction

This notebook / document provides ground data analysis used to make or confirm certain modeling conjectures and assumptions of a Pets Retail Dynamics Model (PRDM), [AA1]. Seattle pets licensing data is used, [SOD2].

We want to provide answers to the following questions.

• Does the Pareto principle manifests for pets breeds?

• Does the Pareto principle manifests for ZIP codes?

• Is there an upward trend for becoming a pet owner?

All three questions have positive answers, assuming the retrieved data, [SOD2], is representative. See the last section for an additional discussion.

We also discuss pet adoption simulations that are done using Quantile Regression, [AA2, AAp1].

This notebook/document is part of the SystemsModeling at GitHub project “Pets retail dynamics”, [AA1].

## Data

The ZIP code coordinates data was taken from a GitHub repository,
“US Zip Codes from 2013 Government Data”, https://gist.github.com/erichurst/7882666.

``dsPetLicenses=ResourceFunction["ImportCSVToDataset"]["~/Datasets/Seattle/Seattle_Pet_Licenses.csv"]``

Convert “Licence Issue Date” values into DateObjects.

``dsPetLicenses=dsPetLicenses[All,Prepend[#,"DateObject"->DateObject[{#[[1]],{"Month","Day","Year"}}]]&];``

#### Summary

``ResourceFunction["RecordsSummary"][dsPetLicenses]``

#### Keep dogs and cats only

Since the number of animals that are not cats or dogs is very small we remove them from the data in order to produce more concise statistics.

``````dsPetLicenses=dsPetLicenses[Select[MemberQ[{"Cat","Dog"},#Species]&]];

(*{49191,8}*)``````

### ZIP code geo-coordinates

``````dsZIPCodes=ImportCSVToDataset["~/Datasets/USAZipCodes/US-Zip-Codes-from-2013-Government-Data.csv"];
Dimensions[dsZIPCodes]

(*{33144,3}*)``````
``````aZipLatLon=Association[Normal[Query[#ZIP->{#LAT,#LON}&]/@dsZIPCodes]];
aZipLatLon=KeyMap[ToString,aZipLatLon];
Length[aZipLatLon]

(*33144*)``````

#### Summary

``ResourceFunction["RecordsSummary"][dsZIPCodes]``
``ResourceFunction["RecordsSummary"][aZipLatLon,Thread->True]``

In this section we apply the Pareto principle statistic in order to see does the Pareto principle manifests over the different columns of the pet licensing data.

### Breeds

We see a typical Pareto principle adherence for both dog breeds and cat breeds. For example, 20% of the dog breeds correspond to 80% of all registered dogs.

Note that the number of unique cat breeds is 4 times smaller than the number of unique dog breeds.

``focusColumnName="Primary Breed";``
``````dsQuery=Query[GroupBy[#Species&],GroupBy[#[focusColumnName]&],Length]@dsPetLicenses;
dsQuery=Dataset[ReverseSort/@Normal[dsQuery]]``````
``KeyValueMap[ResourceFunction["ParetoPrinciplePlot"][Values[#2],PlotLabel->Row[{#1,Spacer[3],focusColumnName}],ImageSize->Medium,opts]&,Normal[dsQuery]]``

### Animal names

We see a typical Pareto principle adherence for the frequencies of the pet names. For dogs, 10% of the unique names correspond to ~65% of the pets.

``focusColumnName="Animal's Name";``
``````dsQuery=Query[GroupBy[#Species&],GroupBy[#[focusColumnName]&],Length]@dsPetLicenses;
dsQuery=Dataset[ReverseSort/@Normal[dsQuery]]``````
``KeyValueMap[ResourceFunction["ParetoPrinciplePlot"][Values[#2],PlotLabel->Row[{#1,Spacer[3],focusColumnName}],ImageSize->Medium,opts]&,Normal[dsQuery]]``

### Zip codes

We see typical – even exaggerated – manifestation of the Pareto principle over ZIP codes of the registered pets.

``focusColumnName="ZIP Code";``
``````dsQuery=Query[GroupBy[#Species&],GroupBy[#[focusColumnName]&],Length]@dsPetLicenses;
dsQuery=Dataset[ReverseSort/@Normal[dsQuery]];``````
``KeyValueMap[ResourceFunction["ParetoPrinciplePlot"][Values[#2],PlotLabel->Row[{#1,Spacer[3],focusColumnName}],ImageSize->Medium,opts]&,Normal[dsQuery]]``

## Geo-distribution

In this section we visualize the pets licensing geo-distribution with geo-histograms.

``````city=Entity["City",{"Seattle","Washington","UnitedStates"}];
GeoPosition[city]

(*GeoPosition[{47.6205,-122.351}]*)``````

### Both cats and dogs

``lsCoords=Map[If[KeyExistsQ[aZipLatLon,#],aZipLatLon[#],Nothing]&,Select[ToString/@Normal[dsPetLicenses[All,"ZIP Code"]],StringQ[#]&&StringLength[#]>=5&]];``
``GeoHistogram[lsCoords,GeoCenter->city,GeoRange->Quantity[20,"Miles"],PlotLegends->Automatic,ColorFunction->(Hue[2/3,2/3,1-#]&),opts]``

### Dogs and cats separately

``````lsCoords=Map[If[KeyExistsQ[aZipLatLon,#],aZipLatLon[#],Nothing]&,Select[ToString/@Normal[dsPetLicenses[Select[#Species=="Dog"&],"ZIP Code"]],StringQ[#]&&StringLength[#]>=5&]];
gr1=GeoHistogram[lsCoords,GeoCenter->city,GeoRange->Quantity[20,"Miles"],PlotLegends->Automatic,ColorFunction->(Hue[2/3,2/3,1-#]&),opts];``````
``````lsCoords=Map[If[KeyExistsQ[aZipLatLon,#],aZipLatLon[#],Nothing]&,Select[ToString/@Normal[dsPetLicenses[Select[#Species=="Cat"&],"ZIP Code"]],StringQ[#]&&StringLength[#]>=5&]];
gr2=GeoHistogram[lsCoords,GeoCenter->city,GeoRange->Quantity[20,"Miles"],PlotLegends->Automatic,ColorFunction->(Hue[2/3,2/3,1-#]&),opts];``````
``ResourceFunction["GridTableForm"][{gr1,gr2},TableHeadings->{"Dogs","Cats"},Background->White]``

### Pet stores

In this subsection we show the distribution of pet stores (in Seattle).

It is better instead of image retrieval to show corresponding geo-markers in the geo-histograms above. (This is not considered that important in the first version of this notebook/document.)

``WebImage["https://www.google.com/maps/search/pet+stores+in+Seattle,+WA/@47.6326975,-122.4227211,12.05z"]``

## Time series

In this section we visualize the time series corresponding to the pet registrations.

### Time series objects

Here we make time series objects:

``````dsQuery=Query[GroupBy[#Species&],GroupBy[#DateObject&],Length]@dsPetLicenses;
aTS=TimeSeries/@(List@@@Normal[#]&/@Normal[dsQuery])``````

### Time series plots of all registrations

Here are time series plots corresponding to all registrations:

``DateListPlot[#,opts]&/@aTS``

### Time series plots of most recent registrations

It is an interesting question why the number of registrations is much higher in volume and frequency in the years 2018 and later.

``DateListPlot[TimeSeriesWindow[#,{{2017,1,1},{2020,1,1}}],opts]&/@aTS``

### Upward trend

Here we apply both Linear Regression and Quantile Regression:

``````QRMonUnit[TimeSeriesWindow[#,{{2018,1,1},{2020,1,1}}]]⟹
QRMonLeastSquaresFit[{1,x}]⟹
QRMonQuantileRegressionFit[4,0.5]⟹
QRMonDateListPlot[opts,"Echo"->False]⟹
QRMonTakeValue&/@aTS``````

We can see that there is clear upward trend for both dogs and cats.

## Quantile regression application

In this section we investigate the possibility to simulate the pet adoption rate. We plan to use simulations of the pet adoption rate in PRDM.

We do that using the software monad `QRMon`, [AAp1]. A list of steps follows.

• Split the time series into windows corresponding to the years 2018 and 2019.

• Find the difference between the two years.

• Apply Quantile Regression to the difference using a reasonable grid of probabilities.

• Simulate the difference.

• Add the simulated difference to year 2019.

### Simulation

In this sub-section we simulate the differences between the time series for 2018 and 2019, then we add the simulated difference to the time series of the year 2019.

``````ts1=TimeSeriesResample[TimeSeriesWindow[aTS[[1]],{{2018,1,1},{2019,1,1}}],"Day"];
ts1["Path"][[All,2]];``````
``````ts2=TimeSeriesResample[TimeSeriesWindow[aTS[[1]],{{2019,1,1},{2020,1,1}}],"Day"];
ts2["Path"][[All,2]];``````
``ts3=TimeSeries[Transpose[{ts1["Path"][[All,1]],ts2["Path"][[All,2]]-ts1["Path"][[All,2]]}]]``
``````qrObj=
QRMonUnit[ts3]⟹
QRMonEchoDataSummary⟹
QRMonQuantileRegression[20,Join[Range[0.1,0.9,0.1],{0.03,0.93}],InterpolationOrder->2]⟹
QRMonDateListPlot[opts];``````
``````qrObj=
qrObj⟹
QRMonEchoFunctionContext[DateListPlot[#data,PlotLabel->"Original data",opts]&]⟹
QRMonSimulate[ts2["Path"]//Length]⟹
QRMonEchoFunctionValue[DateListPlot[#,PlotLabel->"Simulated data",opts]&];``````

Take the simulated time series difference:

``tsSimDiff=TimeSeries[qrObj⟹QRMonTakeValue];``

Add the simulated time series difference to year 2019, clip the values less than zero, shift the result to 2020:

``````tsSim=MapThread[{#1[[1]],#1[[2]]+#2[[2]]}&,{ts2["Path"],tsSimDiff["Path"]}];
tsSim[[All,2]]=Clip[tsSim[[All,2]],{0,Max[tsSim[[All,2]]]}];
tsSim=TimeSeriesShift[TimeSeries[tsSim],Quantity[365,"Days"]];
DateListPlot[tsSim,opts]``````

### Plot all years together

``DateListPlot[{ts1,ts2,tsSim},opts,PlotLegends->{2018,2019,2020}]``

## Discussion

This section has subsections that correspond to additional discussion questions. Not all questions are answered, the plan is to progressively answer the questions with the subsequent versions of the this notebook / document.

#### □ Too few pets

The number of registered pets seems too few. Seattle is a large city with more than 600000 citizens; approximately 50% of the USA households have dogs; hence the registered pets are too few (~50000).

#### □ Why too few pets?

Seattle is a high tech city and its citizens are too busy to have pets?

Most people do not register their pets? (Very unlikely if they have used veterinary services.)

Incomplete data?

#### □ Registration rates

Why the number of registrations is much higher in volume and frequency in the years 2018 and later?

Can we tell apart the adoption rates of pet-less people and people who already have pets?

## Preliminary definitions

``opts=Sequence@@{PlotRange->All,ImageSize->Medium,PlotTheme->"Detailed"};``
``Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicQuantileRegression.m"]``

#### References

[AA1] Anton Antonov, Pets retail dynamics project, (2020), SystemModeling at GitHub.

[AA2] Anton Antonov, A monad for Quantile Regression workflows, (2018), MathematicaForPrediction at WordPress.

[AAp1] Anton Antonov, Monadic Quantile Regression Mathematica package, (2018), MathematicaForPrediction at GitHub.

# Wolfram Live-Coding Series on Quantile Regression workflows

A month or so ago I was invited to make Quantile Regression presentations at the Wolfram Research Twitch channel.

## The live-coding sessions

1. In the first
live-streaming / live-coding session
I demonstrated how to make
Quantile Regression
workflows using the software monad `QRMon`
and some of the underlying software design principles. (Namely
2. In the follow up live-coding session I discussed topics like outliers removal (data cleaning), anomaly detection, and structural breaks.
3. In the third live-coding session:
• First, we demonstrate and explain how to do QR-based time series simulations and their applications in Operations Research.
• Next, we discuss QR in 2D and 3D and a related application.
4. In the fourth live-coding session we discussed the following the topics.
• Brief review of previous sessions.
• Proclaiming the upcoming `ResourceFunction["QuantileRegression"]`.
• Predict tomorrow from today’s data.
• Using NLP techniques on time series.
• Generation of QR workflows with natural language commands.

## Notebooks

• The notebook of the 2nd session is also attached. (I added a “References” section to it.)

## Update (2019-11-13)

A few days ago the Wolfram Function Repository entry `QuantileRegression` was approved. The resource description page has many of the topics discussed in the live-coding sessions on Quantile regression.

# A monad for Latent Semantic Analysis workflows

## Introduction

In this document we describe the design and implementation of a (software programming) monad, [Wk1], for Latent Semantic Analysis workflows specification and execution. The design and implementation are done with Mathematica / Wolfram Language (WL).

What is Latent Semantic Analysis (LSA)? : A statistical method (or a technique) for finding relationships in natural language texts that is based on the so called Distributional hypothesis, [Wk2, Wk3]. (The Distributional hypothesis can be simply stated as “linguistic items with similar distributions have similar meanings”; for an insightful philosophical and scientific discussion see [MS1].) LSA can be seen as the application of Dimensionality reduction techniques over matrices derived with the Vector space model.

The goal of the monad design is to make the specification of LSA workflows (relatively) easy and straightforward by following a certain main scenario and specifying variations over that scenario.

The monad is named `LSAMon` and it is based on the State monad package “StateMonadCodeGenerator.m”, [AAp1, AA1], the document-term matrix making package “DocumentTermMatrixConstruction.m”, [AAp4, AA2], the Non-Negative Matrix Factorization (NNMF) package “NonNegativeMatrixFactorization.m”, [AAp5, AA2], and the package “SSparseMatrix.m”, [AAp2, AA5], that provides matrix objects with named rows and columns.

The data for this document is obtained from WL’s repository and it is manipulated into a certain ready-to-utilize form (and uploaded to GitHub.)

The monadic programming design is used as a Software Design Pattern. The `LSAMon` monad can be also seen as a Domain Specific Language (DSL) for the specification and programming of machine learning classification workflows.

Here is an example of using the `LSAMon` monad over a collection of documents that consists of 233 US state of union speeches.

The table above is produced with the package “MonadicTracing.m”, [AAp2, AA1], and some of the explanations below also utilize that package.

As it was mentioned above the monad `LSAMon` can be seen as a DSL. Because of this the monad pipelines made with `LSAMon` are sometimes called “specifications”.

Remark: In this document with “term” we mean “a word, a word stem, or other type of token.”

Remark: LSA and Latent Semantic Indexing (LSI) are considered more or less to be synonyms. I think that “latent semantic analysis” sounds more universal and that “latent semantic indexing” as a name refers to a specific Information Retrieval technique. Below we refer to “LSI functions” like “IDF” and “TF-IDF” that are applied within the generic LSA workflow.

### Contents description

The document has the following structure.

• The sections “Package load” and “Data load” obtain the needed code and data.
• The sections “Design consideration” and “Monad design” provide motivation and design decisions rationale.
• The sections “LSAMon overview”, “Monad elements”, and “The utilization of SSparseMatrix objects” provide technical descriptions needed to utilize the `LSAMon` monad .
• (Using a fair amount of examples.)
• The section “Unit tests” describes the tests used in the development of the `LSAMon` monad.
• (The random pipelines unit tests are especially interesting.)
• The section “Future plans” outlines future directions of development.
• The section “Implementation notes” just says that `LSAMon`’s development process and this document follow the ones of the classifications workflows monad ClCon, [AA6].

Remark: One can read only the sections “Introduction”, “Design consideration”, “Monad design”, and “LSAMon overview”. That set of sections provide a fairly good, programming language agnostic exposition of the substance and novel ideas of this document.

The following commands load the packages [AAp1–AAp7]:

``````Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicLatentSemanticAnalysis.m"]

In this section we load data that is used in the rest of the document. The text data was obtained through WL’s repository, transformed in a certain more convenient form, and uploaded to GitHub.

The text summarization and plots are done through `LSAMon`, which in turn uses the function RecordsSummary from the package “MathematicaForPredictionUtilities.m”, [AAp7].

### Hamlet

``````textHamlet =
ToString /@
Flatten[Import["https://raw.githubusercontent.com/antononcube/MathematicaVsR/master/Data/MathematicaVsR-Data-Hamlet.csv"]];

TakeLargestBy[
Tally[DeleteStopwords[ToLowerCase[Flatten[TextWords /@ textHamlet]]]], #[[2]] &, 20]

(* {{"ham", 358}, {"lord", 225}, {"king", 196}, {"o", 124}, {"queen", 120},
{"shall", 114}, {"good", 109}, {"hor", 109}, {"come",  107}, {"hamlet", 107},
{"thou", 105}, {"let", 96}, {"thy", 86}, {"pol", 86}, {"like", 81}, {"sir", 75},
{"'t", 75}, {"know", 74}, {"enter", 73}, {"th", 72}} *)

LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix⟹LSAMonEchoDocumentTermMatrixStatistics;``````

### USA state of union speeches

``````url = "https://github.com/antononcube/MathematicaVsR/blob/master/Data/MathematicaVsR-Data-StateOfUnionSpeeches.JSON.zip?raw=true";
str = Import[url, "String"];
filename = First@Import[StringToStream[str], "ZIP"];
aStateOfUnionSpeeches = Association@ImportString[Import[StringToStream[str], {"ZIP", filename, "String"}], "JSON"];

lsaObj =
LSAMonUnit[aStateOfUnionSpeeches]⟹
LSAMonMakeDocumentTermMatrix⟹
LSAMonEchoDocumentTermMatrixStatistics["LogBase" -> 10];``````
``````TakeLargest[ColumnSumsAssociation[lsaObj⟹LSAMonTakeDocumentTermMatrix], 12]

(* <|"government" -> 7106, "states" -> 6502, "congress" -> 5023,
"united" -> 4847, "people" -> 4103, "year" -> 4022,
"country" -> 3469, "great" -> 3276, "public" -> 3094, "new" -> 3022,
"000" -> 2960, "time" -> 2922|> *)``````

### Stop words

In some of the examples below we want to explicitly specify the stop words. Here are stop words derived using the built-in functions `DictionaryLookup` and `DeleteStopwords`.

``````stopWords = Complement[DictionaryLookup["*"], DeleteStopwords[DictionaryLookup["*"]]];

Short[stopWords]

"you'll", "your", "you're", "yours", "yourself", "yourselves", "you've" } *)
``````

## Design considerations

The steps of the main LSA workflow addressed in this document follow.

1. Get a collection of documents with associated ID’s.
2. Create a document-term matrix.
1. Here we apply the Bag-or-words model and Vector space model.
1. The sequential order of the words is ignored and each document is represented as a point in a multi-dimensional vector space.
2. That vector space axes correspond to the unique words found in the whole document collection.
2. Consider the application of stemming rules.
3. Consider the removal of stop words.
3. Apply matrix-entries weighting functions.
1. Those functions come from LSI.
2. Functions like “IDF”, “TF-IDF”, “GFIDF”.
4. Extract topics.
1. One possible statistical way of doing this is with Dimensionality reduction.
2. We consider using Singular Value Decomposition (SVD) and Non-Negative Matrix Factorization (NNMF).
5. Make and display the topics table.
6. Extract and display a statistical thesaurus of selected words.
7. Map search queries or unseen documents over the extracted topics.
8. Find the most important documents in the document collection. (Optional.)

The following flow-chart corresponds to the list of steps above.

• the introduction of new elements in LSA workflows,
• workflows elements variability, and
• workflows iterative changes and refining,

it is beneficial to have a DSL for LSA workflows. We choose to make such a DSL through a functional programming monad, [Wk1, AA1].

Here is a quote from [Wk1] that fairly well describes why we choose to make a classification workflow monad and hints on the desired properties of such a monad.

[…] The monad represents computations with a sequential structure: a monad defines what it means to chain operations together. This enables the programmer to build pipelines that process data in a series of steps (i.e. a series of actions applied to the data), in which each action is decorated with the additional processing rules provided by the monad. […]

Monads allow a programming style where programs are written by putting together highly composable parts, combining in flexible ways the possible actions that can work on a particular type of data. […]

Remark: Note that quote from [Wk1] refers to chained monadic operations as “pipelines”. We use the terms “monad pipeline” and “pipeline” below.

The monad we consider is designed to speed-up the programming of LSA workflows outlined in the previous section. The monad is named `LSAMon` for “Latent Semantic Analysis** Mon**ad”.

We want to be able to construct monad pipelines of the general form:

`LSAMon` is based on the State monad, [Wk1, AA1], so the monad pipeline form (1) has the following more specific form:

This means that some monad operations will not just change the pipeline value but they will also change the pipeline context.

In the monad pipelines of `LSAMon` we store different objects in the contexts for at least one of the following two reasons.

1. The object will be needed later on in the pipeline, or
2. The object is (relatively) hard to compute.

Such objects are document-term matrix, Dimensionality reduction factors and the related topics.

Let us list the desired properties of the monad.

• Rapid specification of non-trivial LSA workflows.
• The text data supplied to the monad can be: (i) a list of strings, or (ii) an association with string values.
• The monad uses the Linear vector space model .
• The document-term frequency matrix can be created after removing stop words and/or word stemming.
• It is easy to specify and apply different LSI weight functions. (Like “IDF” or “GFIDF”.)
• The monad can do dimension reduction with SVD and NNMF and corresponding matrix factors are retrievable with monad functions.
• Documents (or query strings) external to the monad are easily mapped into monad’s Linear vector space of terms and Linear vector space of topics.
• The monad allows of cursory examination and summarization of the data.
• The pipeline values can be of different types. (Most monad functions modify the pipeline value; some modify the context; some just echo results.)
• It is easy to obtain the pipeline value, context, and different context objects for manipulation outside of the monad.
• It is easy to tabulate extracted topics and related statistical thesauri.

The `LSAMon` components and their interactions are fairly simple.

The main `LSAMon` operations implicitly put in the context or utilize from the context the following objects:

• document-term matrix,
• the factors obtained by matrix factorization algorithms,
• LSI weight functions specifications,
• extracted topics.

Note the that the monadic set of types of `LSAMon` pipeline values is fairly heterogenous and certain awareness of “the current pipeline value” is assumed when composing `LSAMon` pipelines.

Obviously, we can put in the context any object through the generic operations of the State monad of the package “StateMonadGenerator.m”, [AAp1].

## LSAMon overview

When using a monad we lift certain data into the “monad space”, using monad’s operations we navigate computations in that space, and at some point we take results from it.

With the approach taken in this document the “lifting” into the `LSAMon` monad is done with the function LSAMonUnit. Results from the monad can be obtained with the functions `LSAMonTakeValue`, `LSAMonContext`, or with the other `LSAMon` functions with the prefix “LSAMonTake” (see below.)

Here is a corresponding diagram of a generic computation with the `LSAMon` monad:

Remark: It is a good idea to compare the diagram with formulas (1) and (2).

Let us examine a concrete `LSAMon` pipeline that corresponds to the diagram above. In the following table each pipeline operation is combined together with a short explanation and the context keys after its execution.

Here is the output of the pipeline:

The `LSAMon` functions are separated into four groups:

• operations,
• setters and droppers,
• takers,

### Monad functions interaction with the pipeline value and context

An overview of the those functions is given in the tables in next two sub-sections. The next section, “Monad elements”, gives details and examples for the usage of the `LSAMon` operations.

Here are the `LSAMon` State Monad functions (generated using the prefix “LSAMon”, [AAp1, AA1].)

Here are the usage descriptions of the main (not monad-supportive) `LSAMon` functions, which are explained in detail in the next section.

In this section we show that `LSAMon` has all of the properties listed in the previous section.

The monad head is `LSAMon`. Anything wrapped in `LSAMon` can serve as monad’s pipeline value. It is better though to use the constructor `LSAMonUnit`. (Which adheres to the definition in [Wk1].)

``LSAMon[textHamlet, <||>]⟹LSAMonMakeDocumentTermMatrix[Automatic, Automatic]⟹LSAMonEchoFunctionContext[Short];``

### Lifting data to the monad

The function lifting the data into the monad `LSAMon` is `LSAMonUnit`.

The lifting to the monad marks the beginning of the monadic pipeline. It can be done with data or without data. Examples follow.

``````LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix⟹LSAMonTakeDocumentTermMatrix

LSAMonUnit[]⟹LSAMonSetDocuments[textHamlet]⟹LSAMonMakeDocumentTermMatrix⟹LSAMonTakeDocumentTermMatrix``````

(See the sub-section “Setters, droppers, and takers” for more details of setting and taking values in `LSAMon` contexts.)

Currently the monad can deal with data in the following forms:

• vectors of strings,
• associations with string values.

Generally, WL makes it easy to extract columns datasets order to obtain vectors or matrices, so datasets are not currently supported in `LSAMon`.

### Making of the document-term matrix

As it was mentioned above with “term” we mean “a word or a stemmed word”. Here is are examples of stemmed words.

``WordData[#, "PorterStem"] & /@ {"consequential", "constitution", "forcing", ""}``

The fundamental model of `LSAMon` is the so called Vector space model (or the closely related Bag-of-words model.) The document-term matrix is a linear vector space representation of the documents collection. That representation is further used in `LSAMon` to find topics and statistical thesauri.

Here is an example of ad hoc construction of a document-term matrix using a couple of paragraphs from “Hamlet”.

``````inds = {10, 19};

MatrixForm @ CrossTabulate[Flatten[KeyValueMap[Thread[{#1, #2}] &, TextWords /@ ToLowerCase[aTempText]], 1]]``````

When we construct the document-term matrix we (often) want to stem the words and (almost always) want to remove stop words. `LSAMon`’s function LSAMonMakeDocumentTermMatrix makes the document-term matrix and takes specifications for stemming and stop words.

``````lsaObj =
LSAMonUnit[textHamlet]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> Automatic, "StopWords" -> Automatic]⟹
LSAMonEchoFunctionContext[ MatrixPlot[#documentTermMatrix] &]⟹
LSAMonEchoFunctionContext[TakeLargest[ColumnSumsAssociation[#documentTermMatrix], 12] &];``````

We can retrieve the stop words used in a monad with the function LSAMonTakeStopWords.

``Short[lsaObj⟹LSAMonTakeStopWords]``

We can retrieve the stemming rules used in a monad with the function LSAMonTakeStemmingRules.

``Short[lsaObj⟹LSAMonTakeStemmingRules]``

The specification `Automatic` for stemming rules uses `WordData[#,"PorterStem"]&`.

Instead of the options style signature we can use positional signature.

• Options style: `LSAMonMakeDocumentTermMatrix["StemmingRules" -> {}, "StopWords" -> Automatic]` .
• Positional style: `LSAMonMakeDocumentTermMatrix[{}, Automatic]` .

### LSI weight functions

After making the document-term matrix we will most likely apply LSI weight functions, [Wk2], like “GFIDF” and “TF-IDF”. (This follows the “standard” approach used in search engines for calculating weights for document-term matrices; see [MB1].)

#### Frequency matrix

We use the following definition of the frequency document-term matrix F.

Each entry fij of the matrix F is the number of occurrences of the term j in the document i.

#### Weights

Each entry of the weighted document-term matrix M derived from the frequency document-term matrix F is expressed with the formula

where gj – global term weight; lij – local term weight; di – normalization weight.

Various formulas exist for these weights and one of the challenges is to find the right combination of them when using different document collections.

Here is a table of weight functions formulas.

#### Computation specifications

`LSAMon` function LSAMonApplyTermWeightFunctions delegates the LSI weight functions application to the package “DocumentTermMatrixConstruction.m”, [AAp4].

Here is an example.

``````lsaHamlet = LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix;
wmat =
lsaHamlet⟹
LSAMonApplyTermWeightFunctions["IDF", "TermFrequency", "Cosine"]⟹
LSAMonTakeWeightedDocumentTermMatrix;

TakeLargest[ColumnSumsAssociation[wmat], 6]``````

Instead of using the positional signature of LSAMonApplyTermWeightFunctions we can specify the LSI functions using options.

``````wmat2 =
lsaHamlet⟹
LSAMonApplyTermWeightFunctions["GlobalWeightFunction" -> "IDF", "LocalWeightFunction" -> "TermFrequency", "NormalizerFunction" -> "Cosine"]⟹
LSAMonTakeWeightedDocumentTermMatrix;

TakeLargest[ColumnSumsAssociation[wmat2], 6]``````

Here we are summaries of the non-zero values of the weighted document-term matrix derived with different combinations of global, local, and normalization weight functions.

``````Magnify[#, 0.8] &@Multicolumn[Framed /@ #, 6] &@Flatten@
Table[
(wmat =
lsaHamlet⟹
LSAMonApplyTermWeightFunctions[gw, lw, nf]⟹
LSAMonTakeWeightedDocumentTermMatrix;
RecordsSummary[SparseArray[wmat]["NonzeroValues"],
List@StringRiffle[{gw, lw, nf}, ", "]]),
{gw, {"IDF", "GFIDF", "Binary", "None", "ColumnStochastic"}},
{lw, {"Binary", "Log", "None"}},
{nf, {"Cosine", "None", "RowStochastic"}}]
AutoCollapse[]``````

### Extracting topics

Streamlining topic extraction is one of the main reasons `LSAMon` was implemented. The topic extraction correspond to the so called “syntagmatic” relationships between the terms, [MS1].

#### Theoretical outline

The original weighed document-term matrix M is decomposed into the matrix factors W and H.

M ≈ W.H, W ∈ ℝm × k, H ∈ ℝk × n.

The i-th row of M is expressed with the i-th row of W multiplied by H.

The rows of H are the topics. SVD produces orthogonal topics; NNMF does not.

The i-the document of the collection corresponds to the i-th row W. Finding the Nearest Neighbors (NN’s) of the i-th document using the rows similarity of the matrix W gives document NN’s through topic similarity.

The terms correspond to the columns of H. Finding NN’s based on similarities of H’s columns produces statistical thesaurus entries.

The term groups provided by H’s rows correspond to “syntagmatic” relationships. Using similarities of H’s columns we can produce term clusters that correspond to “paradigmatic” relationships.

#### Computation specifications

Here is an example using the play “Hamlet” in which we specify additional stop words.

``````stopWords2 = {"enter", "exit", "[exit", "ham", "hor", "laer", "pol", "oph", "thy", "thee", "act", "scene"};

SeedRandom[2381]
lsaHamlet =
LSAMonUnit[textHamlet]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> Automatic, "StopWords" -> Join[stopWords, stopWords2]]⟹
LSAMonApplyTermWeightFunctions["GlobalWeightFunction" -> "IDF", "LocalWeightFunction" -> "None", "NormalizerFunction" -> "Cosine"]⟹
LSAMonExtractTopics["NumberOfTopics" -> 12, "MinNumberOfDocumentsPerTerm" -> 10, Method -> "NNMF", "MaxSteps" -> 20]⟹
LSAMonEchoTopicsTable["NumberOfTableColumns" -> 6, "NumberOfTerms" -> 10];``````

Here is an example using the USA presidents “state of union” speeches.

``````SeedRandom[7681]
lsaSpeeches =
LSAMonUnit[aStateOfUnionSpeeches]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> Automatic,  "StopWords" -> Automatic]⟹
LSAMonApplyTermWeightFunctions["GlobalWeightFunction" -> "IDF", "LocalWeightFunction" -> "None", "NormalizerFunction" -> "Cosine"]⟹
LSAMonExtractTopics["NumberOfTopics" -> 36, "MinNumberOfDocumentsPerTerm" -> 40, Method -> "NNMF", "MaxSteps" -> 12]⟹
LSAMonEchoTopicsTable["NumberOfTableColumns" -> 6, "NumberOfTerms" -> 10];``````

Note that in both examples:

1. stemming is used when creating the document-term matrix,
2. the default LSI re-weighting functions are used: “IDF”, “None”, “Cosine”,
3. the dimension reduction algorithm NNMF is used.

Things to keep in mind.

1. The interpretability provided by NNMF comes at a price.
2. NNMF is prone to get stuck into local minima, so several topic extractions (and corresponding evaluations) have to be done.
3. We would get different results with different NNMF runs using the same parameters. (NNMF uses random numbers initialization.)
4. The NNMF topic vectors are not orthogonal.
5. SVD is much faster than NNMF, but it topic vectors are hard to interpret.
6. Generally, the topics derived with SVD are stable, they do not change with different runs with the same parameters.
7. The SVD topics vectors are orthogonal, which provides for quick to find representations of documents not in the monad’s document collection.

The document-topic matrix W has column names that are automatically derived from the top three terms in each topic.

``````ColumnNames[lsaHamlet⟹LSAMonTakeW]

(* {"player-plai-welcom", "ro-lord-sir", "laert-king-attend",
"end-inde-make", "state-room-castl", "daughter-pass-love",
"hamlet-ghost-father", "father-thou-king",
"rosencrantz-guildenstern-king", "ophelia-queen-poloniu",

Of course the row names of H have the same names.

``````RowNames[lsaHamlet⟹LSAMonTakeH]

(* {"player-plai-welcom", "ro-lord-sir", "laert-king-attend",
"end-inde-make", "state-room-castl", "daughter-pass-love",
"hamlet-ghost-father", "father-thou-king",
"rosencrantz-guildenstern-king", "ophelia-queen-poloniu",

### Extracting statistical thesauri

The statistical thesaurus extraction corresponds to the “paradigmatic” relationships between the terms, [MS1].

Here is an example over the State of Union speeches.

``````entryWords = {"bank", "war", "economy", "school", "port", "health", "enemy", "nuclear"};

lsaSpeeches⟹
LSAMonExtractStatisticalThesaurus["Words" -> Map[WordData[#, "PorterStem"] &, entryWords], "NumberOfNearestNeighbors" -> 12]⟹
LSAMonEchoStatisticalThesaurus;``````

In the code above: (i) the options signature style is used, (ii) the statistical thesaurus entry words are stemmed first.

We can also call `LSAMonEchoStatisticalThesaurus` directly without calling `LSAMonExtractStatisticalThesaurus` first.

`````` lsaSpeeches⟹
LSAMonEchoStatisticalThesaurus["Words" -> Map[WordData[#, "PorterStem"] &, entryWords], "NumberOfNearestNeighbors" -> 12];``````

### Mapping queries and documents to terms

One of the most natural operations is to find the representation of an arbitrary document (or sentence or a list of words) in monad’s Linear vector space of terms. This is done with the function `LSAMonRepresentByTerms`.

Here is an example in which a sentence is represented as a one-row matrix (in that space.)

``````obj =
lsaHamlet⟹
LSAMonRepresentByTerms["Hamlet, Prince of Denmark killed the king."]⟹
LSAMonEchoValue;``````

Here we display only the non-zero columns of that matrix.

``````obj⟹
LSAMonEchoFunctionValue[MatrixForm[Part[#, All, Keys[Select[SSparseMatrix`ColumnSumsAssociation[#], # > 0& ]]]]& ];``````

#### Transformation steps

Assume that `LSAMonRepresentByTerms` is given a list of sentences. Then that function performs the following steps.

1. The sentence is split into a list of words.

2. If monad’s document-term matrix was made by removing stop words the same stop words are removed from the list of words.

3. If monad’s document-term matrix was made by stemming the same stemming rules are applied to the list of words.

4. The LSI global weights and the LSI local weight and normalizer functions are applied to sentence’s contingency matrix.

#### Equivalent representation

Let us look convince ourselves that documents used in the monad to built the weighted document-term matrix have the same representation as the corresponding rows of that matrix.

Here is an association of documents from monad’s document collection.

``````inds = {6, 10};
queries = Part[lsaHamlet⟹LSAMonTakeDocuments, inds];
queries

(* <|"id.0006" -> "Getrude, Queen of Denmark, mother to Hamlet. Ophelia, daughter to Polonius.",
"id.0010" -> "ACT I. Scene I. Elsinore. A platform before the Castle."|> *)

lsaHamlet⟹
LSAMonRepresentByTerms[queries]⟹
LSAMonEchoFunctionValue[MatrixForm[Part[#, All, Keys[Select[SSparseMatrix`ColumnSumsAssociation[#], # > 0& ]]]]& ];``````
``````lsaHamlet⟹
LSAMonEchoFunctionContext[MatrixForm[Part[Slot["weightedDocumentTermMatrix"], inds, Keys[Select[SSparseMatrix`ColumnSumsAssociation[Part[Slot["weightedDocumentTermMatrix"], inds, All]], # > 0& ]]]]& ];``````

### Mapping queries and documents to topics

Another natural operation is to find the representation of an arbitrary document (or a list of words) in monad’s Linear vector space of topics. This is done with the function `LSAMonRepresentByTopics`.

Here is an example.

``````inds = {6, 10};
queries = Part[lsaHamlet⟹LSAMonTakeDocuments, inds];
Short /@ queries

(* <|"id.0006" -> "Getrude, Queen of Denmark, mother to Hamlet. Ophelia, daughter to Polonius.",
"id.0010" -> "ACT I. Scene I. Elsinore. A platform before the Castle."|> *)

lsaHamlet⟹
LSAMonRepresentByTopics[queries]⟹
LSAMonEchoFunctionValue[MatrixForm[Part[#, All, Keys[Select[SSparseMatrix`ColumnSumsAssociation[#], # > 0& ]]]]& ];``````
``````lsaHamlet⟹
LSAMonEchoFunctionContext[MatrixForm[Part[Slot["W"], inds, Keys[Select[SSparseMatrix`ColumnSumsAssociation[Part[Slot["W"], inds, All]], # > 0& ]]]]& ];``````

#### Theory

In order to clarify what the function LSAMonRepresentByTopics is doing let us go through the formulas it is based on.

The original weighed document-term matrix M is decomposed into the matrix factors W and H.

M ≈ W.H, W ∈ ℝm × k, H ∈ ℝk × n

The i-th row of M is expressed with the i-th row of W multiplied by H.

mi ≈ wi.H.

For a query vector q0 ∈ ℝm we want to find its topics representation vector x ∈ ℝk:

q0 ≈ x.H.

Denote with H( − 1) the inverse or pseudo-inverse matrix of H. We have:

q0.H( − 1) ≈ (x.H).H( − 1) = x.(H.H( − 1)) = x.I,

x ∈ ℝk, H( − 1) ∈ ℝn × k, I ∈ ℝk × k.

In `LSAMon` for SVD H( − 1) = HT; for NNMF H( − 1) is the pseudo-inverse of H.

The vector x obtained with `LSAMonRepresentByTopics`.

### Tags representation

Sometimes we want to find the topics representation of tags associated with monad’s documents and the tag-document associations are one-to-many. See [AA3].

Let us consider a concrete example – we want to find what topics correspond to the different presidents in the collection of State of Union speeches.

Here we find the document tags (president names in this case.)

``````tags = StringReplace[
RowNames[
lsaSpeeches⟹LSAMonTakeDocumentTermMatrix],
RegularExpression[".\\d\\d\\d\\d-\\d\\d-\\d\\d"] -> ""];
Short[tags]``````

Here is the number of unique tags (president names.)

``````Length[Union[tags]]
(* 42 *)``````

Here we compute the tag-topics representation matrix using the function `LSAMonRepresentDocumentTagsByTopics`.

``````tagTopicsMat =
lsaSpeeches⟹
LSAMonRepresentDocumentTagsByTopics[tags]⟹
LSAMonTakeValue``````

Here is a heatmap plot of the tag-topics matrix made with the package “HeatmapPlot.m”, [AAp11].

``HeatmapPlot[tagTopicsMat[[All, Ordering@ColumnSums[tagTopicsMat]]], DistanceFunction -> None, ImageSize -> Large]``

### Finding the most important documents

There are several algorithms we can apply for finding the most important documents in the collection. `LSAMon` utilizes two types algorithms: (1) graph centrality measures based, and (2) matrix factorization based. With certain graph centrality measures the two algorithms are equivalent. In this sub-section we demonstrate the matrix factorization algorithm (that uses SVD.)

Definition: The most important sentences have the most important words and the most important words are in the most important sentences.

That definition can be used to derive an iterations-based model that can be expressed with SVD or eigenvector finding algorithms, [LE1].

Here we pick an important part of the play “Hamlet”.

``````focusText =
First@Pick[textHamlet, StringMatchQ[textHamlet, ___ ~~ "to be" ~~ __ ~~ "or not to be" ~~ ___, IgnoreCase -> True]];
Short[focusText]

(* "Ham. To be, or not to be- that is the question: Whether 'tis ....y.
O, woe is me T' have seen what I have seen, see what I see!" *)

LSAMonUnit[StringSplit[ToLowerCase[focusText], {",", ".", ";", "!", "?"}]]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules" -> {}, "StopWords" -> Automatic]⟹
LSAMonApplyTermWeightFunctions⟹
LSAMonFindMostImportantDocuments[3]⟹
LSAMonEchoFunctionValue[GridTableForm];``````

### Setters, droppers, and takers

The values from the monad context can be set, obtained, or dropped with the corresponding “setter”, “dropper”, and “taker” functions as summarized in a previous section.

For example:

``````p = LSAMonUnit[textHamlet]⟹LSAMonMakeDocumentTermMatrix[Automatic, Automatic];

p⟹LSAMonTakeMatrix``````

If other values are put in the context they can be obtained through the (generic) function LSAMonTakeContext, [AAp1]:

``````Short@(p⟹QRMonTakeContext)["documents"]

(* <|"id.0001" -> "1604", "id.0002" -> "THE TRAGEDY OF HAMLET, PRINCE OF DENMARK", <<220>>, "id.0223" -> "THE END"|> *) ``````

Another generic function from [AAp1] is LSAMonTakeValue (used many times above.)

Here is an example of the “data dropper” LSAMonDropDocuments:

``````Keys[p⟹LSAMonDropDocuments⟹QRMonTakeContext]

(* {"documentTermMatrix", "terms", "stopWords", "stemmingRules"} *)``````

(The “droppers” simply use the state monad function LSAMonDropFromContext, [AAp1]. For example, LSAMonDropDocuments is equivalent to LSAMonDropFromContext[“documents”].)

## The utilization of SSparseMatrix objects

The LSAMon monad heavily relies on `SSparseMatrix` objects, [AAp6, AA5], for internal representation of data and computation results.

A `SSparseMatrix` object is a matrix with named rows and columns.

Here is an example.

``````n = 6;
rmat = ToSSparseMatrix[
SparseArray[{{1, 2} -> 1, {4, 5} -> 1}, {n, n}],
"RowNames" -> RandomSample[CharacterRange["A", "Z"], n],
"ColumnNames" -> RandomSample[CharacterRange["a", "z"], n]];
MatrixForm[rmat]``````

In this section we look into some useful SSparseMatrix idioms applied within `LSAMon`.

### Visualize with sorted rows and columns

In some situations it is beneficial to sort rows and columns of the (weighted) document-term matrix.

``````docTermMat =
lsaSpeeches⟹LSAMonTakeDocumentTermMatrix;
MatrixPlot[docTermMat[[Ordering[RowSums[docTermMat]],  Ordering[ColumnSums[docTermMat]]]], MaxPlotPoints -> 300, ImageSize -> Large]``````

The most popular terms in the document collection can be found through the association of the column sums of the document-term matrix.

``````TakeLargest[ColumnSumsAssociation[lsaSpeeches⟹LSAMonTakeDocumentTermMatrix], 10]

(* <|"state" -> 8852, "govern" -> 8147, "year" -> 6362, "nation" -> 6182,
"congress" -> 5040, "unit" -> 5040, "countri" -> 4504,
"peopl" -> 4306, "american" -> 3648, "law" -> 3496|> *)
``````

Similarly for the lest popular terms.

``````TakeSmallest[
ColumnSumsAssociation[
lsaSpeeches⟹LSAMonTakeDocumentTermMatrix], 10]

(* <|"036" -> 1, "027" -> 1, "_____________________" -> 1, "0111" -> 1,
"006" -> 1, "0000" -> 1, "0001" -> 1, "______________________" -> 1,
"____" -> 1, "____________________" -> 1|> *)``````

### Showing only non-zero columns

In some cases we want to show only columns of the data or computation results matrices that have non-zero elements.

Here is an example (similar to other examples in the previous section.)

``````lsaHamlet⟹
LSAMonRepresentByTerms[{"this country is rotten",
"where is my sword my lord",
"poison in the ear should be in the play"}]⟹
LSAMonEchoFunctionValue[ MatrixForm[#1[[All, Keys[Select[ColumnSumsAssociation[#1], #1 > 0 &]]]]] &];``````

In the pipeline code above: (i) from the list of queries a representation matrix is made, (ii) that matrix is assigned to the pipeline value, (iii) in the pipeline echo value function the non-zero columns are selected with by using the keys of the non-zero elements of the association obtained with ColumnSumsAssociation.

### Similarities based on representation by terms

Here is way to compute the similarity matrix of different sets of documents that are not required to be in monad’s document collection.

``````sMat1 =
lsaSpeeches⟹
LSAMonRepresentByTerms[ aStateOfUnionSpeeches[[ Range[-5, -2] ]] ]⟹
LSAMonTakeValue

sMat2 =
lsaSpeeches⟹
LSAMonRepresentByTerms[ aStateOfUnionSpeeches[[ Range[-7, -3] ]] ]⟹
LSAMonTakeValue

MatrixForm[sMat1.Transpose[sMat2]]``````

### Similarities based on representation by topics

Similarly to weighted Boolean similarities matrix computation above we can compute a similarity matrix using the topics representations. Note that an additional normalization steps is required.

``````sMat1 =
lsaSpeeches⟹
LSAMonRepresentByTopics[ aStateOfUnionSpeeches[[ Range[-5, -2] ]] ]⟹
LSAMonTakeValue;
sMat1 = WeightTermsOfSSparseMatrix[sMat1, "None", "None", "Cosine"]

sMat2 =
lsaSpeeches⟹
LSAMonRepresentByTopics[ aStateOfUnionSpeeches[[ Range[-7, -3] ]] ]⟹
LSAMonTakeValue;
sMat2 = WeightTermsOfSSparseMatrix[sMat2, "None", "None", "Cosine"]

MatrixForm[sMat1.Transpose[sMat2]]``````

Note the differences with the weighted Boolean similarity matrix in the previous sub-section – the similarities that are less than 1 are noticeably larger.

## Unit tests

The development of `LSAMon` was done with two types of unit tests: (i) directly specified tests, [AAp7], and (ii) tests based on randomly generated pipelines, [AA8].

The unit test package should be further extended in order to provide better coverage of the functionalities and illustrate – and postulate – pipeline behavior.

### Directly specified tests

Here we run the unit tests file “MonadicLatentSemanticAnalysis-Unit-Tests.wlt”, [AAp8].

``````AbsoluteTiming[
]``````

The natural language derived test ID’s should give a fairly good idea of the functionalities covered in [AAp3].

``````Values[Map[#["TestID"] &, testObject["TestResults"]]]

"Make-document-term-matrix-1", "Make-document-term-matrix-2",
"Apply-term-weights-1", "Apply-term-weights-2", "Topic-extraction-1",
"Topic-extraction-2", "Topic-extraction-3", "Topic-extraction-4",
"Statistical-thesaurus-1", "Topics-representation-1",
"Take-document-term-matrix-1", "Take-weighted-document-term-matrix-1",
"Take-document-term-matrix-2", "Take-weighted-document-term-matrix-2",
"Take-terms-1", "Take-Factors-1", "Take-Factors-2", "Take-Factors-3",
"Take-Factors-4", "Take-StopWords-1", "Take-StemmingRules-1"} *)``````

### Random pipelines tests

Since the monad `LSAMon` is a DSL it is natural to test it with a large number of randomly generated “sentences” of that DSL. For the `LSAMon` DSL the sentences are `LSAMon` pipelines. The package “MonadicLatentSemanticAnalysisRandomPipelinesUnitTests.m”, [AAp9], has functions for generation of `LSAMon` random pipelines and running them as verification tests. A short example follows.

Generate pipelines:

``````SeedRandom[234]
pipelines = MakeLSAMonRandomPipelines[100];
Length[pipelines]

(* 100 *)``````

Here is a sample of the generated pipelines:

Here we run the pipelines as unit tests:

``````AbsoluteTiming[
res = TestRunLSAMonPipelines[pipelines, "Echo" -> False];
]``````

From the test report results we see that a dozen tests failed with messages, all of the rest passed.

``rpTRObj = TestReport[res]``

(The message failures, of course, have to be examined – some bugs were found in that way. Currently the actual test messages are expected.)

## Future plans

### Dimension reduction extensions

It would be nice to extend the Dimension reduction functionalities of `LSAMon` to include other algorithms like Independent Component Analysis (ICA), [Wk5]. Ideally with `LSAMon` we can do comparisons between SVD, NNMF, and ICA like the image de-nosing based comparison explained in [AA8].

Another direction is to utilize Neural Networks for the topic extraction and making of statistical thesauri.

### Conversational agent

Since `LSAMon` is a DSL it can be relatively easily interfaced with a natural language interface.

Here is an example of natural language commands parsed into LSA code using the package [AAp13].

## Implementation notes

The implementation methodology of the `LSAMon` monad packages [AAp3, AAp9] followed the methodology created for the `ClCon` monad package [AAp10, AA6]. Similarly, this document closely follows the structure and exposition of the `ClCon monad document “A monad for classification workflows”, [AA6].

A lot of the functionalities and signatures of `LSAMon` were designed and programed through considerations of natural language commands specifications given to a specialized conversational agent.

## References

### Packages

[AAp1] Anton Antonov, State monad code generator Mathematica package, (2017), MathematicaForPrediction at GitHub*.

[AAp2] Anton Antonov, Monadic tracing Mathematica package, (2017), MathematicaForPrediction at GitHub*.

[AAp3] Anton Antonov, Monadic Latent Semantic Analysis Mathematica package, (2017), MathematicaForPrediction at GitHub.

[AAp5] Anton Antonov, Non-Negative Matrix Factorization algorithm implementation in Mathematica, (2013), MathematicaForPrediction at GitHub.

[AAp6] Anton Antonov, SSparseMatrix Mathematica package, (2018), MathematicaForPrediction at GitHub.

[AAp7] Anton Antonov, MathematicaForPrediction utilities, (2014), MathematicaForPrediction at GitHub.

[AAp8] Anton Antonov, Monadic Latent Semantic Analysis unit tests, (2019), MathematicaVsR at GitHub.

[AAp9] Anton Antonov, Monadic Latent Semantic Analysis random pipelines Mathematica unit tests, (2019), MathematicaForPrediction at GitHub.

[AAp10] Anton Antonov, Monadic contextual classification Mathematica package, (2017), MathematicaForPrediction at GitHub.

[AAp11] Anton Antonov, Heatmap plot Mathematica package, (2017), MathematicaForPrediction at GitHub.

[AAp12] Anton Antonov,
Independent Component Analysis Mathematica package, MathematicaForPrediction at GitHub.

[AAp13] Anton Antonov, Latent semantic analysis workflows grammar in EBNF, (2018), ConverasationalAgents at GitHub.

### MathematicaForPrediction articles

[AA1] Anton Antonov, “Monad code generation and extension”, (2017), MathematicaForPrediction at GitHub.

[AA2] Anton Antonov, “Topic and thesaurus extraction from a document collection”, (2013), MathematicaForPrediction at GitHub.

[AA3] Anton Antonov, “The Great conversation in USA presidential speeches”, (2017), MathematicaForPrediction at WordPress.

[AA4] Anton Antonov, “Contingency tables creation examples”, (2016), MathematicaForPrediction at WordPress.

[AA5] Anton Antonov, “RSparseMatrix for sparse matrices with named rows and columns”, (2015), MathematicaForPrediction at WordPress.

[AA6] Anton Antonov, “A monad for classification workflows”, (2018), MathematicaForPrediction at WordPress.

[AA7] Anton Antonov, “Independent component analysis for multidimensional signals”, (2016), MathematicaForPrediction at WordPress.

[AA8] Anton Antonov, “Comparison of PCA, NNMF, and ICA over image de-noising”, (2016), MathematicaForPrediction at WordPress.

### Other

[Wk2] Wikipedia entry, Latent semantic analysis,

[Wk3] Wikipedia entry, Distributional semantics,

[Wk4] Wikipedia entry, Non-negative matrix factorization,

[LE1] Lars Elden, Matrix Methods in Data Mining and Pattern Recognition, 2007, SIAM. ISBN-13: 978-0898716269.

[MB1] Michael W. Berry & Murray Browne, Understanding Search Engines: Mathematical Modeling and Text Retrieval, 2nd. ed., 2005, SIAM. ISBN-13: 978-0898715811.

[MS1] Magnus Sahlgren, “The Distributional Hypothesis”, (2008), Rivista di Linguistica. 20 (1): 33[Dash]53.

[PS1] Patrick Scheibe, Mathematica (Wolfram Language) support for IntelliJ IDEA, (2013-2018), Mathematica-IntelliJ-Plugin at GitHub.

# Finding all structural breaks in time series

## Introduction

In this document we show how to find the so called “structural breaks”, [Wk1], in a given time series. The algorithm is based in on a systematic application of Chow Test, [Wk2], combined with an algorithm for local extrema finding in noisy time series, [AA1].

The algorithm implementation is based on the packages “MonadicQuantileRegression.m”, [AAp1], and “MonadicStructuralBreaksFinder.m”, [AAp2]. The package [AAp1] provides the software monad QRMon that allows rapid and concise specification of Quantile Regression workflows. The package [AAp2] extends QRMon with functionalities related to structural breaks finding.

### What is a structural break?

It looks like at least one type of “structural breaks” are defined through regression models, [Wk1]. Roughly speaking a structural break point of time series is a regressor point that splits the time series in such way that the obtained two parts have very different regression parameters.

One way to test such a point is to use Chow test, [Wk2]. From [Wk2] we have the definition:

The Chow test, proposed by econometrician Gregory Chow in 1960, is a test of whether the true coefficients in two linear regressions on different data sets are equal. In econometrics, it is most commonly used in time series analysis to test for the presence of a structural break at a period which can be assumed to be known a priori (for instance, a major historical event such as a war).

### Example

Here is an example of the described algorithm application to the data from [Wk2].

``QRMonUnit[data]⟹QRMonPlotStructuralBreakSplits[ImageSize -> Small];``

Here we load the packages [AAp1] and [AAp2].

``````Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicQuantileRegression.m"]

## Data used

In this section we assign the data used in this document.

### Illustration data from Wikipedia

Here is the data used in the Wikipedia article “Chow test”, [Wk2].

``````data = {{0.08, 0.34}, {0.16, 0.55}, {0.24, 0.54}, {0.32, 0.77}, {0.4,
0.77}, {0.48, 1.2}, {0.56, 0.57}, {0.64, 1.3}, {0.72, 1.}, {0.8,
1.3}, {0.88, 1.2}, {0.96, 0.88}, {1., 1.2}, {1.1, 1.3}, {1.2,
1.3}, {1.3, 1.4}, {1.4, 1.5}, {1.4, 1.5}, {1.5, 1.5}, {1.6,
1.6}, {1.7, 1.1}, {1.8, 0.98}, {1.8, 1.1}, {1.9, 1.4}, {2.,
1.3}, {2.1, 1.5}, {2.2, 1.3}, {2.2, 1.3}, {2.3, 1.2}, {2.4,
1.1}, {2.5, 1.1}, {2.6, 1.2}, {2.6, 1.4}, {2.7, 1.3}, {2.8,
1.6}, {2.9, 1.5}, {3., 1.4}, {3., 1.8}, {3.1, 1.4}, {3.2,
1.4}, {3.3, 1.4}, {3.4, 2.}, {3.4, 2.}, {3.5, 1.5}, {3.6,
1.8}, {3.7, 2.1}, {3.8, 1.6}, {3.8, 1.8}, {3.9, 1.9}, {4., 2.1}};
ListPlot[data]``````

### S&P 500 Index

Here we get the time series corresponding to S&P 500 Index.

``````tsSP500 = FinancialData[Entity["Financial", "^SPX"], {{2015, 1, 1}, Date[]}]
DateListPlot[tsSP500, ImageSize -> Medium]``````

## Application of Chow Test

The Chow Test statistic is implemented in [AAp1]. In this document we rely on the relative comparison of the Chow Test statistic values: the larger the value of the Chow test statistic, the more likely we have a structural break.

Here is how we can apply the Chow Test with a QRMon pipeline to the [Wk2] data given above.

``````chowStats =
QRMonUnit[data]⟹
QRMonChowTestStatistic[Range[1, 3, 0.05], {1, x}]⟹
QRMonTakeValue;``````

We see that the regressor points \$Failed and 1.7 have the largest Chow Test statistic values.

``````Block[{chPoint = TakeLargestBy[chowStats, Part[#, 2]& , 1]},
ListPlot[{chowStats, chPoint}, Filling -> Axis, PlotLabel -> Row[{"Point with largest Chow Test statistic:",
Spacer[8], chPoint}]]]``````

The first argument of QRMonChowTestStatistic is a list of regressor points or Automatic. The second argument is a list of functions to be used for the regressions.

Here is an example of an automatic values call.

``````chowStats2 = QRMonUnit[data]⟹QRMonChowTestStatistic⟹QRMonTakeValue;
ListPlot[chowStats2, GridLines -> {
Part[
Part[chowStats2, All, 1],
OutlierIdentifiers`OutlierPosition[
Part[chowStats2, All, 2],  OutlierIdentifiers`SPLUSQuartileIdentifierParameters]], None}, GridLinesStyle -> Directive[{Orange, Dashed}], Filling -> Axis]``````

For the set of values displayed above we can apply simple 1D outlier identification methods, [AAp3], to automatically find the structural break point.

``````chowStats2[[All, 1]][[OutlierPosition[chowStats2[[All, 2]], SPLUSQuartileIdentifierParameters]]]
(* {1.7} *)

OutlierPosition[chowStats2[[All, 2]], SPLUSQuartileIdentifierParameters]
(* {20} *)``````

We cannot use that approach for finding all structural breaks in the general time series cases though as exemplified with the following code using the time series S&P 500 Index.

``````chowStats3 = QRMonUnit[tsSP500]⟹QRMonChowTestStatistic⟹QRMonTakeValue;
DateListPlot[chowStats3, Joined -> False, Filling -> Axis]``````
``````OutlierPosition[chowStats3[[All, 2]], SPLUSQuartileIdentifierParameters]
(* {} *)

OutlierPosition[chowStats3[[All, 2]], HampelIdentifierParameters]
(* {} *)``````

In the rest of the document we provide an algorithm that works for general time series.

## Finding all structural break points

Consider the problem of finding of all structural breaks in a given time series. That can be done (reasonably well) with the following procedure.

1. Chose functions for testing for structural breaks (usually linear.)
2. Apply Chow Test over dense enough set of regressor points.
3. Make a time series of the obtained Chow Test statistics.
4. Find the local maxima of the Chow Test statistics time series.
5. Determine the most significant break point.
6. Plot the splits corresponding to the found structural breaks.

QRMon has a function, QRMonFindLocalExtrema, for finding local extrema; see [AAp1, AA1]. For the goal of finding all structural breaks, that semi-symbolic algorithm is the crucial part in the steps above.

## Computation

### Chose fitting functions

``fitFuncs = {1, x};``

### Find Chow test statistics local maxima

The computation below combines steps 2,3, and 4.

``````qrObj =
QRMonUnit[tsSP500]⟹
QRMonFindChowTestLocalMaxima["Knots" -> 20,
"NearestWithOutliers" -> True,
"NumberOfProximityPoints" -> 5, "EchoPlots" -> True,
"DateListPlot" -> True,
ImageSize -> Medium]⟹
QRMonEchoValue;``````

### Find most significant structural break point

``splitPoint = TakeLargestBy[qrObj⟹QRMonTakeValue, #[[2]] &, 1][[1, 1]]``

### Plot structural breaks splits and corresponding fittings

Here we just make the plots without showing them.

``````sbPlots =
QRMonUnit[tsSP500]⟹
QRMonPlotStructuralBreakSplits[(qrObj⟹ QRMonTakeValue)[[All, 1]],
"LeftPartColor" -> Gray, "DateListPlot" -> True,
"Echo" -> False,
ImageSize -> Medium]⟹
QRMonTakeValue;
``````

The function QRMonPlotStructuralBreakSplits returns an association that has as keys paired split points and Chow Test statistics; the plots are association’s values.

Here we tabulate the plots with plots with most significant breaks shown first.

``````Multicolumn[
KeyValueMap[
Show[#2, PlotLabel ->
Grid[{{"Point:", #1[[1]]}, {"Chow Test statistic:", #1[[2]]}}, Alignment -> Left]] &, KeySortBy[sbPlots, -#[[2]] &]], 2]``````

## Future plans

We can further apply the algorithm explained above to identifying time series states or components. The structural break points are used as knots in appropriate Quantile Regression fitting. Here is an example.

The plan is to develop such an identifier of time series states in the near future. (And present it at WTC-2019.)

## References

### Articles

[Wk1] Wikipedia entry, Structural breaks.

[Wk2] Wikipedia entry, Chow test.

[AA1] Anton Antonov, “Finding local extrema in noisy data using Quantile Regression”, (2019), MathematicaForPrediction at WordPress.

[AA2] Anton Antonov, “A monad for Quantile Regression workflows”, (2018), MathematicaForPrediction at GitHub.

### Packages

[AAp1] Anton Antonov, Monadic Quantile Regression Mathematica package, (2018), MathematicaForPrediction at GitHub.

[AAp2] Anton Antonov, Monadic Structural Breaks Finder Mathematica package, (2019), MathematicaForPrediction at GitHub.

[AAp3] Anton Antonov, Implementation of one dimensional outlier identifying algorithms in Mathematica, (2013), MathematicaForPrediction at GitHub.

### Videos

[AAv1] Anton Antonov, Structural Breaks with QRMon, (2019), YouTube.