Using compound metric tester

The compound metric tester allows you to pick an instance of a source type of a compound metric, and save at the current point in time the following:

1. All the data relevant to that compound metric from the instance of the source needed to evaluate the metric, stored in a map of metric names to arrays of doubles. No other traceable information about the instance is stored.
2. The current result of evaluating this metric on the instance.

This saved set of data can later be used to verify that the compound metric still evaluates to the saved result, testing to see if any basic metric functionality has changed since the saved result evaluation. For example, it would evaluate differently if the "window" function started to behave differently than previously expected and appears in the compound metric being tested.

Copy link to this section

How to use compound metric tester

1. Find an instance of the desired source type of the compound metric with well-defined data.
2. Call MetricTestData.extractTestData, passing in the required metric name, source, and evalMetricSpec. This can return a fully defined MetricTestData object (which should be saved as seed data in the package which the metric belongs to).
3. This saved MetricTestData can then be used to test the metric in the future using TestApi.validateMetricTestData, which allows the user to filter on the metrics to fetch and test as a batch. Theoretically this would be used in a short jasmine test for a package with saved MetricTestData.

Copy link to this section

Relevant types

• MetricTestData: Stores the data necessary for testing a compound metric. One completed instance of this Type needs to be created per compound metric. Field truth stores the expected result of evaluation (taken at the time that the metric was evaluated during creation of the MetricTestData instance. Think of it as a snapshot at that point in time).
• MetricTestData.extractTestData: Extracts data needed to evaluate compound metric and the current evaluation of the metric, as described above.
• MetricTestData.validate: Compares the saved result (the truth) with a fresh evaluation of the metric on the saved bindings from MetricTestData.extractTestData. Returns 1 if these match, and 0 otherwise.
• MetricTestBindings: A means of storing an array of doubles in a type accepted by the any type so that bindings in the EvalMetricSpec can be set (currently cannot be passed in directly as an array of doubles).
• TestApi.validateMetricTestData: Allows user to filter on metric names by include (pass in the entire metric's name), starts with, and name includes (string appearing in the names of returned metrics). Fetches all MetricTestData meeting this criteria, and calls MetricTestData.validate on them.

Copy link to this section

Dependencies

TestApi.validateMetricTestData exists within type TestApi.