Work with EventWindow

Many failure prediction applications use risk scores produced by Machine Learning (ML) models. These are almost always represented as a time series. However, from a scoring perspective, it is more desirable to look at windows of time, rather than points.

This topic discusses the concept of an EventWindow and describes the helper utilities that have been developed around creating EventWindows, and also scoring them.

Topics:

• create EventWindows from time series (by for example, using a risk score as a threshold)
• create EventWindows from list of observed events (by for example, specifying 4-day window ending 24 hrs prior to event)
• generate classification of True Positive, False Positive values from list of predicted EventWindows`` and observed EventWindows``

Copy link to this section

Do processing on event scores

Assuming there is some defined metric that represents score, we would like to get EventWindows from it. To do that TimeseriesToEventWindowsSpec should be filled and passed into the EventWindow#fromTimeseries method. If you would like to have more than one start trigger (for example, 0.35 < metric < 0.45 and 0.8 < metric), then you should do separate calls for each of the triggers and merge them into one list using EventWindow#mergeEventWindows, which has implemented different EventWindowMergeSpec#mergeStrategyies.

Example:

TimeseriesToEventWindowsSpec spec = TimeseriesToEventWindowsSpec
  .builder()
  .objId(powerTs.id())
  .source(powerTs.type().myTypeRef())
  .startTimestamp(dataStart.withTimeAtStartOfDay())
  .endTimestamp(dataEnd.withTimeAtStartOfDay())
  .startAndEndExpression("rollingDiff(PowerConsumptionEWTest > 0.35 && PowerConsumptionEWTest < 0.45)")
  .build();

TimeseriesToEventWindowsSpec specWithSecondaryTriggers = spec
   .withStartAndEndExpression("rollingDiff(PowerConsumptionEWTest > 0.8)");

Arry<Arry<EventWindows>> arry = Arry.of(EventWindows.arry().collectionType());
arry.add(EventWindows.fromTimeseries(spec));
arry.add(EventWindows.fromTimeseries(specWithSecondaryTriggers));

Arry<EventWindows> windows = EventWindow.mergeEventWindows(arry);

Copy link to this section

Persisting EventWindow instances to database

EventWindows ids get hydrated according to the EventWindow start date and TimeseriesToEventWindowsSpec properties, so no collisions should be expected when directly persisting EventWindows.

Example:

Arry<EventWindows> windows = EventWindow.mergeEventWindows(arry);
EventWindow.mergeBatch(windows);

Copy link to this section

Continuous processing

To evaluate EventWindow#fromTimeseries you can set the startTimestamp field on TimeseriesToEventWindowsSpec before the last opened EventWindow start timestamp if you have any, or before the previous endTimestamp on TimeseriesToEventWindowsSpec.

Copy link to this section

Preparing tests data

ObservedEvent type represent user-labeled event. EventWindow#fromObservedEvents should be used to create "labeled" EventWindows. Such EventWindows can be modified and persisted, if needed.

Example:

Arry<ObservedEvent> events = ...;
Arry<EventWindows> observedEventWindows = EventWindow.fromObservedEvents(events);

Copy link to this section

Binary classification

After creating a list of EventWindows that were generated by EventWindow#fromTimeseries and a list of "labeled" EventWindows, you can pass them into EventWindowBinaryClassification#classify to produce a list of EventWindowBinaryScoreResults.

Each EventWindowBinaryScoreResult could have one of the following outcomes:

• TP: A True Positive is a EventWindow occurs right before a ObservedEvent.
• FP: A False Positive is a EventWindow far from any ObservedEvents.
• FN: A False Negative is a EventWindow before some ObservedEvent.

Aggregation of this scores can be directly used for evaluating quality of event score.

Example:

Arry<EventWindows> monitoredEventWindows = ...;
Arry<EventWindowBinaryScoreResult> scoreResults = 
   EventWindowBinaryClassification.classify(monitoredEventWindows, observedEventWindows);

double precision = EventWindowBinaryScoreResult.precision(scoreResults);
double recall = EventWindowBinaryScoreResult.recall(scoreResults);
double f1Score = EventWindowBinaryScoreResult.f1Score(scoreResults);

Copy link to this section

Persisting monitoring event windows with observed events

Sometimes it is required to persist EventWindows with classified events. This can be done with the following:

EventWindow.mergeBatch(classify.map(EventWindow.myReferenceType(), EventWindowBinaryScoreResult::monitoredEventWindow));