Exposing the hidden links between AI training metrics

AI experiment metrics

The result of ML training runs are the ML models. The models are giant matrices that are hard to read and make sense of (mostly). However there is a need to debug and verify the model performance systematically during and after the training.

AI researchers collect dozens of metrics during training runs in order to

• sense-check the runs - does the loss decrease with more epochs and runs?
• analyze them - does the model generalize or overfit (regularization, etc)?
• compare when fine-tuning the model - using diff parallel coordinate plots, grouping and other ways to make sense of many runs

Metrics are a numeric time-series data collected during an iterative process.

ML training is an iterative process and so lots of metrics (we have seen 100s per training run among Aim users) are collected with different meaning and purpose behind them.

The metrics context

A considerable amount of tracked metrics are the same metric but collected in different ways with different meanings. For instance a loss could be collected in so many different ways. A few examples:

• during train, validation and test phases
• with different frequencies (once every epoch, iteration etc)
• for different tasks in multi-task training environment

Even though these are all losses, however each one has a different context associated with it. Therefore they all have different role when analyzing and comparing the training runs.

There is also a relationship between these metrics. Most of the times they are collected so that they could be compared together. For instance, one of the most basic requirements for the experiment tracking tools is to be able to compare train loss and validation loss as it represents whether the model generalizes well or not.

Metric context adds totally new dimension to metrics and their relationship.

How do we use or represent this relationship in current experiment tracking tools?

Metric context in existing tools

AI researchers and engineers have been historically encoding all these additional pieces of information in the metric name.

So in most of them (including TensorBoard, MLFlow, Weights and Biases) metrics are treated as numeric sequences with a name.

What are the issues with such representation?

• No systematic built-in way of encoding the relationship between metrics
• This results in super-long metric names (and UI that needs to handle them)
• Sometimes you’d want to group, aggregate several dozens of l2_losses and recons_losses separately to compare them.
  • What if you have deeper context such as l2_epoch_loss?
• Or when you’d like to search both losses by just one context you will have to write a regular expression for strings.

Metrics as objects with properties

What if the experiment tracking tools could represent metrics as full-fledged objects with context and additional metadata attached to them.

Instead of multispectral_loss_l2 we would have just a loss with the following properties and the values:

• loss.context.func where the value of this property would be l2
• loss.context.type where the value would be multispectral or spectral

You could access these properties both on the main UI of the tool while searching through metrics and on the API when automating over some metrics during evaluation time.

For instance you could search by loss type and submit a query like this:

SELECT loss IF loss.context.type == multispectral

This would select all the multispectral losses available. In contrast to that if you ran

SELECT loss

The system would return all the losses ever recorded by your experiments.

This is a relatively simple example of the conveniences it would add when treating metrics as an object with a context.

Conclusion

Having been a super-passionate TensorBoard user in the past, I would always wonder why metrics would not have more “life” to them?

Metrics being represented as full-fledged objects aren’t just beneficial for the experiment tracking visualization/debugging but also for further automations at the evaluation step - just much better API to work with than just parsing files from TensorBoard. In this way it’s just a lot more systematic and efficient.

Shameless Plug

We have channeled these “frunstrations” into the way we have designed the Aim metrics. You can use the track function to pass as many context variables as you’d like (described in here) and use them to query, group and do other manipulations with your metrics just like in this article.

I am Gev, an AI engineer, co-author of Aim. We have been super-lucky to have folks use Aim and share feedback, use-cases about experiment tracking. I am using these Substack series to share more about our experience in building Aim, generally AI dev tools, design decisions and dilemmas. You can find me on Twitter here.

Originally published on Substack on August 6, 2021.