In this issue:

• we start the new series about text-to-image models; 

• we discuss CLIP, a neural network that can learn image representations while being trained using natural language datasets; 

• we explore Hugging Face’s CLIP implementation. 

Enjoy the learning!  

💡 ML Concept of the Day: A New Series About Text-to-Image Models  

Multidomain learning is one of the aspirational crown jewels of deep learning. Today, most neural networks remain highly specialized in a single domain such as language, speech, or computer vision. In recent years, we have seen a generation of successful models that can operate with datasets from different domains. Among those, text-image models have proven to be particularly successful in combining recent breakthroughs in both language and computer vision. Today, we are starting a brief series about a new generation of text-image models and their underlying techniques.  

Combining language and images seems like a natural evolution for deep learning methods. After all, the association between images and text is natural to human cognition. Recent advancements in transformer architectures, few-shot learning and language pretrained models have pushed the boundaries of natural language understanding (NLU) and are making inroads in computer vision. Additionally, methods such as generative adversarial networks (GANs) have achieved major milestones in photorealistic image generation. Combining the two, we have a new wave of models that are able to produce high-fidelity images from textual inputs.  

The key to text-image models is...