Week 2 · Day 8 · Wednesday 21 October

Vision-Language Models

Connecting images and text: captioning, joint embeddings and text-to-image generation, then BLIP, Flamingo, LLaVA, and how multimodal LLMs are trained and evaluated.

Topic Description & Objectives Resource
Vision and Text Integration Image and video captioning, multimodal models and joint embeddings with CLIP, text-to-image generation with GANs and diffusion, and classifier-free guidance Open
Vision-Language Models Image-text alignment, BLIP and BLIP-2, Flamingo, LLaVA, connecting vision to an LLM, training multimodal LLMs, and their evaluation and failure modes Open
Lab 1: A Vision LLM from Scratch Connect a vision encoder to a language model and train the result to caption Flickr8k images Notebook Colab
Optional: Image Captioning with Attention A CNN encoder and an LSTM decoder with additive attention, trained on Flickr8k with teacher forcing Notebook Colab