ICCV-2019

29 Dec 2019 in Reading Notes

means being highly related to my personal research interest.

Noisy Labels, …

• Deep Self-Learning From Noisy Labels: Self means `without extra supervision’
  • The proposed SMP trains in an iterative manner which contains two phases: the first phase is to train a network with the original noisy label and corrected label generated in the second phase.
  • By extracting multiple prototypes for a category, we demonstrate that more prototypes would get a better representation of a class and obtain better label-correction results.
• Co-Mining: Deep Face Recognition With Noisy Labels: We propose a novel co-mining framework, which employs two peer networks to detect the noisy faces, exchanges the high-confidence clean faces and reweights the clean faces in a mini-batch fashion.
• NLNL: Negative Learning for Noisy Labels: Input image belongs to this label–Positive Learning; Negative Learning (NL)–CNNs are trained using a complementary label as in “input image does not belong to this complementary label.”
• Symmetric Cross Entropy for Robust Learning With Noisy Labels: Already compared in our method.
• O2U-Net: A Simple Noisy Label Detection Approach for Deep Neural Networks–Overall, this method is complex:
  • We propose a novel noisy label detection approach, named O2U-net, without human annotation and verification. It only requires adjusting the hyper-parameters of the deep network to make its status transfer from overfitting to underfitting (O2U) cyclically. By calculating and ranking the normalized average loss of every sample, the mislabeled samples can be identified.
  • The losses of each sample are recorded during iterations. The higher the normalized average loss of a sample, the higher the probability of being noisy labels. => Is it scalable to large datasets?
  • O2U-net is naturally compatible with active learning and other human annotation approaches. This introduces extra flexibility for learning with noisy labels.
  • The whole training process:
    • Pre-training: Train the network directly on the original dataset including noisy labels. At this step, a common constant learning rate is applied. A large batch size is applied to reduce the impact of label noise Deep Learning is Robust to Massive Label Noise. We use a validation set to monitor the performance of training. The network is trained until the accuracy in the validation set stays stable. (Validation Data is Needed!)
    • Cyclical Training: A small batch size–A smaller batch size is chosen to make the network more easily transfer from overfitting to underfitting. After the whole cyclical training, the average of the normalized losses of every sample is computed. All the average losses are then ranked in descending order. The top k% of samples are removed from the original dataset as noisy labels, where k depends on the prior knowledge on the dataset. Such prior knowledge can be obtained by manually verifying a small group of randomly selected samples.
    • Training on Clean Data: Lastly, we re-initialize the parameters of the network, and re-train it on the cleansing dataset ordinarily until achieving stable accuracy and loss in the validation set.

Robustness

• Human uncertainty makes classification more robust–From Labels to Label Distributions
  • We suggest an alternative objective: not just trying to capture the most likely label, but trying to capture the full distribution over labels. Although there has been much work scaling the number of images in datasets [18], and investigating label noise [40, 12, 48], little effort has been put into identifying the benefits from increasing the richness of (informative) label distributions for image classification tasks.
  • Soft Labels: One of the core contributions of our work is around using the soft labels provided through human confusion as a replacement for one-hot label encodings.
  • Our approach proposes utilizing these human disagreements to improve the accuracy and robustness of a model, complementing existing work aimed at leveraging “errors” in human labeling
  • As accuracy gains have begun to asymptote at near-perfect levels [11], there has been increasing focus on out-of-training-set performance—in particular, the ability to generalize to related stimuli [39], and robustness to adversarial examples [29]. On these tasks, by contrast, CNNs tend to perform rather poorly, whereas humans continue to perform well.

• Subspace Structure-aware Spectral Clustering for Robust Subspace Clustering

Adversarial Robustness

• Scalable Verified Training for Provably Robust Image Classification
  • Train deep neural networks that are provably robust to norm-bounded adversarial perturbations.

• Improving Adversarial Robustness via Guided Complement Entropy

• Bilateral Adversarial Training: Towards Fast Training of More Robust Models Against Adversarial Attacks

Deep Metric Learning, …