Author(s): Will Whitney, Min Jae Song, David Brandfonbrener, Jaan Altosaar, Kyunghyun Cho
Publication date: September 24 2020
Reviewer: Cheolhyoung Lee 
Editor: Kyunghyun Cho

In the last few years, there’s been an explosion of work on learning good representations of data. From NLP^{1 2 3} to computer vision^{4 5 6} to reinforcement learning^{7 8 9}, the field has never been hotter. However, defining precisely what we mean by a good representation can be tricky. This has led to a somewhat ad-hoc approach to evaluation in the literature, with each paper choosing its own measure or set of measures and a general sense that our evaluation methods aren’t very robust.

In a recent paper, Evaluating representations by the complexity of learning low-loss predictors¹⁰, we show that many notions of the quality of a representation for a task can be expressed as a function of the loss-data curve. This perspective allows us to see the limitations of existing measures and propose new ones that are more robust.

We think that evaluation is crucially important in the field right now and we don’t want the measures that we and others have proposed to languish as purely theoretical exercises. Since these measures (ours and others) aren’t trivial to implement or to compute, we are releasing a library called Reprieve for representation evaluation that aims to standardize the evaluation of representation quality. Whether you’re using the measures that we proposed or several others, and no matter what ML library you use, you can evaluate representations with Reprieve.

Loss-data curves and existing measures

The loss-data curve, with the size of the training set on the X axis and validation loss on the Y axis, describes how an algorithm’s performance varies based on the amount of training data it’s given. Intuitively, the curve for a representation that allows the algorithm to learn efficiently (with little data) will lie to the left of the curve for a representation that makes learning less efficient. Meanwhile a representation that contains more predictive information will lead to a curve that goes lower as the training set size goes to infinity.

On the loss-data curve we can graphically show the meaning of several existing evaluation measures for representation quality (left panel).

Validation accuracy with limited data (VA) is the simplest measure. VA corresponds to picking some \(n\) for the dataset size and looking only at a vertical slice of the loss-data curve at that \(n\).

Mutual information (MI) attempts to measure the quality of a representation by its mutual information with the labels¹¹. MI is equivalent to considering only the validation loss with infinite training data.

Minimum description length (MDL) is an interesting measure recently proposed by Voita et al. (2020)¹². Given a fixed dataset, MDL measures the description length of the dataset’s labels (the vector of all the Ys) given its observations (the vector of all the Xs) according to a particular encoding scheme. In the prequential or online coding scheme, a model is trained to predict \(p(Y^k \mid X^k)\) on a dataset of size \(k\), and then used to encode the \((k+1)^{\mathrm{th}}\) point. MDL corresponds to the area under the loss-data curve up to \(n\), the full size of the dataset.

An interesting feature of all these methods is that they depend on (or specify, for MI) a particular dataset size. This can be a bit tricky: how much data should an algorithm need to solve a new task? Provide too little data and no representation will allow any learning, but provide too much and only asymptotic loss will matter, not efficiency.

Instead, we will construct an evaluation procedure that measures a property of the data distribution and the learning algorithm, not a particular dataset or dataset size.

Surplus Description Length

We’re going to build on the MDL idea to make a measure of representation quality. To do this, we measure the complexity of learning for a given data distribution and learning algorithm. We have two main goals for this representation evaluation measure:

1. It should measure a fundamental property of the data distribution and learning algorithm.
2. The measure shouldn’t depend on a particular sample of a dataset from the data distribution, the size of the dataset, or the order of the points.

Defining surplus description length

To start with, imagine trying to efficiently encode a large number of samples of some random variable \(\mathbf{e}\) which takes discrete values in \(\{1 \ldots K\}\) with probability \(p(\mathbf{e})\). The best possible code for each sample leverages knowledge of the probability of observing that sample, and assigns a code length of \(- \log p(e_i)\) to each sampled value \(e_i\). This results in an expected length per sample of \[ \mathbb{E}_\mathbf{e} [\ell_p(\mathbf{e})] = \mathbb{E}_\mathbf{e} [- \log p(\mathbf{e})] = H(\mathbf{e}) \] where we use \(\ell_p\) to denote the negative log-likelihood loss for the distribution \(p\).

If instead \(\mathbf{e}\) was encoded using some other distribution \(\hat p\), the expected length becomes \(H(\mathbf{e}) + D_{\mathrm{KL}}(p~||~\hat p)\). We call \(D_{\mathrm{KL}}(p~||~\hat p)\) the surplus description length (SDL) from encoding according to \(\hat p\) instead of \(p\). We can also write it as \[ \mathrm{SDL}(\hat p) = D_{\mathrm{KL}}(p~||~\hat p) = \mathbb{E}_{\mathbf{e} \sim p} \left[ \log p(\mathbf{e}) – \log \hat p(\mathbf{e}) \right] \] to highlight how SDL measures only the extra entropy that comes from not having the correct model.

SDL as a measure of representation quality

As our model learns we get a new \(\hat p\) at every training step. Similarly to MDL with online codes¹², we measure the SDL of the learned model at each step and then sum them up. Writing the expected loss of running algorithm \(\mathcal{A}\) on a dataset with \(i\) points as \(L(\mathcal{A}_\phi, i)\), the SDL measure of representation quality is \[ m_{\mathrm{SDL}}(\phi, \mathcal{D}, \mathcal{A}) = \sum_{i=1}^\infty \Big[ L(\mathcal{A}_\phi, i) – H(\mathbf{Y} \mid \mathbf{X}) \Big]. \]

We show in the paper that MDL is a special case of SDL which assumes that the true distribution of \(\mathbf{e}\) is a delta mass, which is to say that \(\mathbf{e}\) has no randomness at all. This leads to some odd properties with real data, which typically has noise. MDL goes to infinity with the size of the dataset even for algorithms which learn the true data distribution, which makes numbers hard to compare. More worryingly, if we rank the quality of two representations using MDL, that ranking can (and in practice does) switch as we change the dataset size. That means our conclusions about which representation is better are totally dependent on how much data we have to evaluate them!

Since in practice we don’t know the true entropy of the data distribution, we also propose a version of the SDL measure where we set some threshold \(\varepsilon\) as a criterion for success instead of using the true entropy of the data. As long as \(\varepsilon > H(\mathbf{Y} \mid \mathbf{X})\), this still has most of the same nice properties. A good way to set \(\varepsilon\) would be to run the learning algorithm on a large amount of data using the raw representation of the data, then set \(\varepsilon\) to the loss of that model plus a small slack term for estimation error.

We also propose a simpler measure called \(\varepsilon\) sample complexity, or \(\varepsilon\)SC, which is the number of training points required for the expected loss to drop below \(\varepsilon\). For full details on that check out the paper!

Representation evaluation in practice

With our tools in hand, we can examine some practical representations. Looking first at MNIST, we compare using the raw pixels to using neural encoders pretrained on supervised CIFAR classification or trained without supervision as a low-dimensional VAE on MNIST.

As you can see from the loss-data curve (right), these representations perform very differently! While the VAE representation allows the quickest learning at first, it makes achieving very low loss hard. Meanwhile the CIFAR pretrained representation supports learning that’s more efficient than raw pixels for any loss.

Looking at the evaluation measures, we see that the existing measures like validation loss and MDL tend to switch their rankings when larger datasets are used for evaluation. Meanwhile SDL and \(\varepsilon\)SC know when there isn’t enough data available to evaluate a representation, and once they make a judgement, it sticks.

To show that this phenomenon isn’t just limited to vision tasks or small datasets, we also provide experiments on a part of speech classification task using pretrained representations from ELMo². Just like on MNIST, validation loss and MDL make very different predictions with small evaluation datasets than with large ones.

Better representation evaluation for everyone

Existing measures of representation quality, which are functions of a particular dataset rather than the data distribution, can have some tricky behavior. Whether you use our measures or not, we urge our fellow members of the representation learning community to think carefully about the measures and procedures that you use to evaluate representations.

Reprieve, our library for representation evaluation, is one tool that we think can help. By using the powerful program transformations provided by JAX, Reprieve is able to train the ~100 or so small networks required to construct a loss-data curve in parallel on one GPU in about two minutes. From there it can compute all of the measures that we mentioned today.

We hope that by standardizing on one codebase for evaluation, we in the representation learning community can move faster while producing results that are more comparable and more reproducible. If Reprieve is missing a measure that you think is important, submit a pull request!

Author(s): Vid Kocijan and Samuel R. Bowman
Publication date: August 27 2020
Reviewer: Alex Wang 
Editor: Kyunghyun Cho

One of the ways to improve the performance of a neural model on a task with scarce data is to pre-train it on a related task first. Large-scale language models, such as RoBERTa (Liu et al. ’19) are pre-trained on large corpora of text, using unsupervised training objectives, such as masked token prediction. Fine-tuning such a pre-trained language model on the target tasks usually outperforms a model trained on the target task data only. In certain cases, e.g. when the target task training-set is small, it can be beneficial to train the pre-trained language model on an “intermediate task” first and only then fine-tune it on the target task (Pruksachatkun et al. ’20). In this project, we took a closer look at this scenario. Since the intermediate task may come from a domain unrelated to that of the target task, we suspect that not all examples in the training set of the intermediate task benefit the target task.

To identify and filter out the examples that do not positively impact the performance of the model, we used influence functions (Cook et al. ’80). Influence functions are a method from robust statistics that measure the dependence of the estimator on the value of a single point in the sample. Informally, they estimate how the removal (or an addition) of a training example impacts the predictions of the model making them the obvious choice to approach the problem. In deep learning, they have already been successfully applied as explanations (Koh et al. ’17), and as an estimation of the quality of the training samples (Wang et al. ’18, Yang et al. ’20). The use in the context of transfer learning thus seems like a natural progression.

There are two potential problems of using influence functions in the context of transfer learning of deep neural networks. Firstly, they have only been proven to work for models with a convex optimization criterion. Secondly, they were not designed for a two-stage training with the model architecture changing during the stages (the last layer is a task-specific classification layer, so it changes between tasks). A few previous works have successfully used influence functions even with neural networks, despite the non-convex optimization criterion (Koh et al. ’17, Wang et al. ’18, Yang et al. ’20). We designed our experiments in a way that avoids the second issue, as the model was only trained on one training set only (SNLI training set (Bowman et al. ’15)) and evaluated on a validation set with the same output format (MNLI matched validation set (Williams et al. ’18)), avoiding the need to change the last layer.

We found that training RoBERTa (Liu et al. ’19) on SNLI training set, filtered using influence functions w.r.t. MNLI validation set resulted in a large performance drop on that same validation set. Moreover, it seems like a dataset consisting of examples with either highest or lowest influence scores resulted in a similar drop in performance. Using middle-ranked examples, on the other hand, resulted in performance similar to, or slightly better, than random downsampling. However, none of the results outperformed training on the full training set.

Experiments

We trained an instance of RoBERTa on the SNLI training set while validating it on the MNLI (matched) validation set. We used Influence functions to estimate the influence of each training example on the validation set loss. Simplified, each example is assigned a real number estimating how much the example contributes to the validation loss. By retaining only the examples with positive influence, we retain approximately 40% of the training set. The distribution of all influences can be found in Figure 1.

Figure 1: Distribution of influences of examples in the training set. We can see that the large majority of the examples are centred around 0.

To gain more insight into the impact of the influence functions, we additionally experiment with only keeping the 25%, 50%, and 75% best-ranked examples. Additionally, we conducted experiments with 25%, 50%, 75% of the worst-ranked examples, as well as an experiment with exactly all examples that were estimated to be detrimental (60% of the dataset). For additional comparison, two more series of experiments were conducted, one with randomly downsampled examples and one with exactly 50% and 25% of the dataset by taking the middle-most ranked examples. The results can be found on Figure 1.

Using subsets with only detrimental or only beneficial examples significantly reduces the performance of the model. All experiments were conducted with the same sets of hyperparameters that worked best for the full dataset. We did an additional hyperparameter search to investigate whether training on filtered dataset requires an additional hyperparameter search, however, the accuracy of the re-trained models improved marginally or did not improve at all.

Figure 2: Performance of RoBERTa on the MNLI matched validation set, fine-tuned on subsets of the SNLI according to the influence of examples. We can see that the computed influence of examples does not correlate to the performance of the trained model as subsets of either only beneficial or detrimental examples give results significantly worse than random downsampling.

Discussion

The results of the experiments showed that the signal from the influence functions is not noise as the difference from random downsampling is too large to be a coincidence. It is, however, unclear what causes this and how it could be useful. There are several potential explanations for why these experiments could fail to yield positive results, e.g. Basu et al. (2020) note that fragility of influence functions rises with the size of a network. RoBERTa most definitely constitutes a very large neural network by the standards of that paper. Moreover, it is well known that an increase/decrease in validation loss does not always correspond to an increase/decrease in classification accuracy on the validation set.

However, all potential explanations in the previous paragraph can only explain potential noise in the results, but not such an enormous drop of performance. We have manually observed and analysed the data, but we were not able to spot any obvious patterns in the filtered datasets that could explain this phenomenon. Since we could not find any potential use of such behaviour, research on this project was finished and left for future research.

Revision history

• August 27 2020: Initial publication