Self-Supervised
YouTube search... ...Google search
- Supervised Learning ... Semi-Supervised ... Self-Supervised ... Unsupervised
- Learning Techniques
- Process Supervision
- Artificial Intelligence (AI) ... Generative AI ... Machine Learning (ML) ... Deep Learning ... Neural Network ... Reinforcement ... Learning Techniques
- Conversational AI ... ChatGPT | OpenAI ... Bing | Microsoft ... Bard | Google ... Claude | Anthropic ... Perplexity ... You ... Ernie | Baidu
- Self-supervised learning is the future of AI | Ben Dickson
Self-supervised learning refers to an unsupervised learning problem that is framed as a supervised learning problem in order to apply supervised learning algorithms to solve it. Supervised learning algorithms are used to solve an alternate or pretext task, the result of which is a model or representation that can be used in the solution of the original (actual) modeling problem. A common example of self-supervised learning is computer vision where a corpus of unlabeled images is available and can be used to train a supervised model, such as making images grayscale and having a model predict a color representation (colorization) or removing blocks of the image and have a model predict the missing parts (inpainting). 14 Different Types of Learning in Machine Learning | Jason Brownlee - Machine Learning Mastery
Given a task and enough labels, supervised learning can solve it really well. Good performance usually requires a decent amount of labels, but collecting manual labels is expensive (i.e. ImageNet) and hard to be scaled up. Considering the amount of unlabelled data (e.g. free text, all the images on the Internet) is substantially more than a limited number of human curated labelled Datasets, it is kinda wasteful not to use them. However, unsupervised learning is not easy and usually works much less efficiently than supervised learning. What if we can get labels for free for unlabelled data and train unsupervised dataset in a supervised manner? We can achieve this by framing a supervised learning task in a special form to predict only a subset of information using the rest. In this way, all the information needed, both inputs and labels, has been provided. This is known as self-supervised learning. This idea has been widely used in language modeling. The default task for a language model is to predict the next word given the past sequence. Bidirectional Encoder Representations from Transformers (BERT) adds two other auxiliary tasks and both rely on self-generated labels.
Why Self-Supervised Learning? Self-supervised learning empowers us to exploit a variety of labels that come with the data for free. The motivation is quite straightforward. Producing a dataset with clean labels is expensive but unlabeled data is being generated all the time. To make use of this much larger amount of unlabeled data, one way is to set the learning objectives properly so as to get supervision from the data itself. The self-supervised task, also known as pretext task, guides us to a supervised loss function. However, we usually don’t care about the final performance of this invented task. Rather we are interested in the learned intermediate representation with the expectation that this representation can carry good semantic or structural meanings and can be beneficial to a variety of practical downstream tasks. Self-Supervised Representation Learning | Lilian Weng - Lil'Log
In the context of self-supervised learning, a self-supervised loss is a type of loss function that is used to train a predictive model based on a self-supervised task. A self-supervised loss function is designed to take advantage of the supervisory signal provided by the self-supervised task, which allows the algorithm to learn representative features of the data that can be used for downstream tasks. The precise form of the self-supervised loss function depends on the specifics of the self-supervised task being used, but typically involves minimizing the difference between the predicted output and the actual output.
