ICLR-2019
means being highly related to my personal research interest.
Core/Fundamental Deep Learning (Informative/Forgettable Example, Uncertain Examples)
-
Informative/Forgettable Examples: An Empirical Study of Example Forgetting during Deep Neural Network Learning-Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes*, Adam Trischler, Yoshua Bengio, Geoffrey J. Gordon
- We define a ``forgetting event’’ to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. We show that catastrophic forgetting occurs within what is considered to be a single task and find that examples that are not prone to forgetting can be removed from the training set without loss of generalization.
- Across several benchmark data sets, we find that:
- (i) certain examples are forgotten with high frequency, and some not at all;
- (ii) a data set’s (un)forgettable examples generalize across neural architectures;
- (iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.
Their finding: Harder/Support/informative samples are learned later and may include noisy examples.
Then a question is arouse: How to differentiate informative and noisy examples? Our work provides a solution for this question. Derivative Manipulation for General Example Weighting- Detailed findings of theirs:
- Support examples = forgettable examples = Informative examples that cannot be removed. Removing unforgottable examples do not hurt the generalisation performance when training a model from scratch on the remained subset.
- The properties of support/informative examples: a) learnt later; b) larget misclassification margin when forgotten; c) perceptually ambiguous; d) tend to be noisy.
-
Uncertain Examples-NeurIPS2017: Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples-Haw-Shiuan Chang, Erik Learned-Miller, Andrew McCallum
- How to define uncertain examples: predicitons in the history.
