arXiv-2020
means being highly related to my personal research interest.
Foundation of Deep Learning
Neural Bayes: A Generic Parameterization Method for Unsupervised Representation Learning
-
A Simple Framework for Contrastive Learning of Visual Representations
- Data augmentation: composition of data augmentations plays a critical role in defining effective predictive tasks;
- Auxiliary transformation: introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations;
- Larger batch size and more training steps: contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning.
- Results: By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy.
-
Instance Cross Entropy for Deep Metric Learning and its application in SimCLR-A Simple Framework for Contrastive Learning of Visual Representations
I am very glad to highlight that: our proposed ICE is simple and effective, which has also been demonstrated in recent work SimCLR, in the context of self-supervised learning: A Simple Framework for Contrastive Learning of Visual Representations
- Its loss expression NT-Xent (the normalized temperature-scaled cross entropy loss) is a fantastic application of our recently proposed Instance Cross Entropy for Deep Metric Learning, in the context of self-supervised learnining. I am very excited about this.
- #InstanceCrossEntropy #TemperatureScaling #RepresentationLearning
- Research Gate
- Open Review
-
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature
- Unsupervised Embedding Learning via Invariant and SpreadingInstance Feature is even closer, also in the context of self-supervised learning, maximising the agreement over augmentations of one instance.
- Instance Cross Entropy for Deep Metric Learning is in the context of supervised discriminative representation learning, maximising the agreement over augmentaions of multiple images i.e. a class.
- ICLR 2019: Learning deep representations by mutual information estimation and maximization
Noisy labels
- Learning Adaptive Loss for Robust Learning with Noisy Labels
- DivideMix: Learning with Noisy Labels as Semi-supervised Learning
- Group-Teaching: Learning Robust CNNs From Extremely Noisy Labels
- []
