Résumé
The representational differences between generalizing
networks and intentionally flawed models can be insightful
on the dynamics of network training. Do memorizing
networks, e.g. networks that learn random label correspondences,
focus on specific patterns in the data to memorize
the labels? Are the features learned by a generalizing network
affected by randomization of the model parameters?
In high-risk applications such as medical, legal or financial
domains, highlighting the representational differences
that help generalization may be even more important than
the model performance itself. In this paper, we probe the
activations of intermediate layers with linear classification
and regression. Results show that the bias towards simple
solutions of generalizing networks is maintained even when
statistical irregularities are intentionally introduced.