Difference between revisions of "BLIP-2"
m |
m |
||
| Line 15: | Line 15: | ||
* [[Large Language Model (LLM)#Multimodal|Multimodal Language Model]]s | * [[Large Language Model (LLM)#Multimodal|Multimodal Language Model]]s | ||
* [[Large Language Model (LLM)]] ... [[Natural Language Processing (NLP)]] ...[[Natural Language Generation (NLG)|Generation]] ... [[Natural Language Classification (NLC)|Classification]] ... [[Natural Language Processing (NLP)#Natural Language Understanding (NLU)|Understanding]] ... [[Language Translation|Translation]] ... [[Natural Language Tools & Services|Tools & Services]] | * [[Large Language Model (LLM)]] ... [[Natural Language Processing (NLP)]] ...[[Natural Language Generation (NLG)|Generation]] ... [[Natural Language Classification (NLC)|Classification]] ... [[Natural Language Processing (NLP)#Natural Language Understanding (NLU)|Understanding]] ... [[Language Translation|Translation]] ... [[Natural Language Tools & Services|Tools & Services]] | ||
| − | * [[Assistants]] ... [[Agents]] ... [[Negotiation | + | * [[Assistants]] ... [[Agents]] ... [[Negotiation]] ... [[LangChain]] |
* [[Attention]] Mechanism ...[[Transformer]] ...[[Generative Pre-trained Transformer (GPT)]] ... [[Generative Adversarial Network (GAN)|GAN]] ... [[Bidirectional Encoder Representations from Transformers (BERT)|BERT]] | * [[Attention]] Mechanism ...[[Transformer]] ...[[Generative Pre-trained Transformer (GPT)]] ... [[Generative Adversarial Network (GAN)|GAN]] ... [[Bidirectional Encoder Representations from Transformers (BERT)|BERT]] | ||
* [[Generative AI]] ... [[Conversational AI]] ... [[OpenAI]]'s [[ChatGPT]] ... [[Perplexity]] ... [[Microsoft]]'s [[Bing]] ... [[You]] ...[[Google]]'s [[Bard]] ... [[Baidu]]'s [[Ernie]] | * [[Generative AI]] ... [[Conversational AI]] ... [[OpenAI]]'s [[ChatGPT]] ... [[Perplexity]] ... [[Microsoft]]'s [[Bing]] ... [[You]] ...[[Google]]'s [[Bard]] ... [[Baidu]]'s [[Ernie]] | ||
Revision as of 22:01, 19 May 2023
YouTube ... Quora ...Google search ...Google News ...Bing News
- BLIP-2 | Salesforce Research
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models | J. Li, D. Li, S. Savarese, & S. Hoi
- Multimodal Language Models
- Large Language Model (LLM) ... Natural Language Processing (NLP) ...Generation ... Classification ... Understanding ... Translation ... Tools & Services
- Assistants ... Agents ... Negotiation ... LangChain
- Attention Mechanism ...Transformer ...Generative Pre-trained Transformer (GPT) ... GAN ... BERT
- Generative AI ... Conversational AI ... OpenAI's ChatGPT ... Perplexity ... Microsoft's Bing ... You ...Google's Bard ... Baidu's Ernie
- Capabilities
- Development ...AI Pair Programming Tools ... Analytics ... Visualization ... Diagrams for Business Analysis
- Prompt Engineering (PE)
- Foundation Models (FM)
- Singularity ... Sentience ... AGI ... Curious Reasoning ... Emergence ... Moonshots ... Explainable AI ... Automated Learning
- Zero-shot image-to-text generation with BLIP-2 | Maria Khalusova - Hugging Face ... how to use BLIP-2 for image captioning, prompted image captioning, visual question-answering, and chat-based prompting
- Interactive ChatCaptioner for image and video | Vision-CAIR - GitHub
BLIP-2 is a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models1. It achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. BLIP-2 bridges the modality gap between vision and language models by adding a lightweight Querying Transformer (Q-Former) between an off-the-shelf frozen pre-trained image encoder and a frozen large language model. Q-Former is the only trainable part of BLIP-2; both the image encoder and language model remain frozen.
- Q-Former is a transformer model that consists of two submodules that share the same self-attention layers: an image transformer that interacts with the frozen image encoder for visual feature extraction and a text transformer that can function as both a text encoder and a text decoder.
One key difference between BLIP-2 and other vision-language models is that BLIP-2 introduces a new visual-language pre-training paradigm that can potentially leverage any combination of pre-trained vision encoder and LLM without having to pre-train the whole architecture end to end. This enables achieving state-of-the-art results on multiple visual-language tasks while significantly reducing the number of trainable parameters and pre-training costs.
The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 is pre-trained in two stages:
- 1st stage: bootstraps vision-language representation learning from a frozen image encoder.
- 2nd stage bootstraps vision-to-language generative learning from a frozen language model.
BLIP-2 is a zero-shot visual-language model that can be used for multiple image-to-text tasks with image and image and text prompts. Some of the image-to-text tasks that visual language models can tackle include image captioning, image-text retrieval, and visual question answering. Image captioning can aid the visually impaired, create useful product descriptions, identify inappropriate content beyond text, and more. Image-text retrieval can be applied in multimodal search, as well as in applications such as autonomous driving. Visual question-answering can aid in education, enable multimodal chatbots, and assist in various domain-specific information retrieval applications.
- Visual question answering (VQA) is a task in which a model is presented with an image and a natural language question about the image. The model must then provide a natural language answer to the question based on the content of the image. VQA can aid in education, enable multimodal chatbots, and assist in various domain-specific information retrieval applications. It can aid in education by providing students with a tool to ask questions about visual content and receive answers. It can also enable multimodal chatbots that can answer questions about images and other visual content. VQA can also assist in various domain-specific information retrieval applications, such as helping doctors retrieve information from medical images.
Code a VISION - LLM w/ ViT, FLAN-T5 LLM & BLIP-2: Multimodal LLMs (MLLM)
