The Sequence Chat: Rohan Taori on Stanford's Alpaca, Alpaca Farm and the Future of LLMs
Was this email forwarded to you? Sign up here The Sequence Chat: Rohan Taori on Stanford's Alpaca, Alpaca Farm and the Future of LLMsAlpaca was one of the first open LLMs to incorporate instruction following capabilities. Now one of the project's main researchers shares his insights about modern LLMs.👤 Quick bio
I’m currently a PhD student at Stanford, where my research has focused around understanding and improving the use of real-world machine learning systems. My interest in ML started during my undergrad at Berkeley where I met a fantastic group of people (who later became good friends) who introduced me to DeepDream, a method for style transfer with image classification models. Though these models are nothing compared to the generative text-to-image models we have today, I was enthralled by the art pieces created by DeepDream, and that set me on a path delving into deep learning. 🛠 ML Work
Alpaca was really sort of a happy surprise along the way of our bigger project, AlpacaFarm. Our goal for AlpacaFarm was to study methods for learning from human feedback (e.g. RLHF) in depth, as this process has been used to train models like ChatGPT. The first step for any RLHF method is actually to perform supervised fine-tuning on a small set of initial questions and answers, which is the baseline that any method builds off of. Alpaca is precisely this baseline! We were stunned at how just a little bit of fine-tuning made the base LLaMA 7B model so much easier to interact with, and so we released the model as a way to share this information with everyone.
Alpaca itself did not have any RLHF component and was only supervised fine-tuned on a small question-answer set. To build this dataset, we queried text-davinci-003 to generate questions and answers to open-ended instructions in a format similar to a set of seed questions written by the authors of SelfInstruct (https://arxiv.org/abs/2212.10560). Our subsequent project, AlpacaFarm, studied the RLHF component in more detail. In this stage, we asked the model to generate two outputs for a set of instructions, and then asked humans to select which output they preferred. We then trained a binary classifier (or reward model) with this data, and optimized against it with a variety of methods, including PPO.
AlpacaFarm is a simulator that enables research on learning from feedback quickly and cheaply. The primary component is a set of LLM prompts designed to simulate humans - we ask GPT-4 and ChatGPT for the output preferences to train the reward model, and we designed this automated feedback to align with learning from real human feedback. We also include a standardized evaluation protocol for judging the performance of any instruction-following model, and a set of reference implementations for PPO and other RLHF methods. AlpacaFarm is really cool because the rankings of different methods developed in simulation roughly line up with the rankings of the methods when trained on actual human feedback, which means that the simulator can serve as an effective way to study and develop new RLHF methods.
I think that tool learning is a really exciting direction, as it equips LLMs with the ability to execute very complex tasks. At the same time, there is a greater security risk in giving an automated agent control over various external resources. So developing new methods for tool use and mitigating the security risks is a really important research discussion for the near future. While Alpaca doesn’t come with tool use directly, AlpacaFarm shows that simulation is a viable way to easily study real-world LM behaviors, and I’m excited to see what this direction can bring.
Alpaca is not actually open-source - it came with a non-commercial license due to the assets involved in its creation. However, since then, other truly open-source instruction-following models have been released to the community, which is great. Just like any new technology, there are new risks and consequences with LLMs. My view is that having better models more openly available will enable research on these important questions. Many of the issues surrounding their use involve different communities, different norms, and different cultures; open releases are a great way of inviting the broader participation needed in the development of these models. 💥 Miscellaneous – a set of rapid-fire questions
Having worked in image classification for a while, I have to say that I am quite fond of CLIP (https://arxiv.org/abs/2103.00020) - a contrastive learning paradigm to align text and images found on the internet. Constructing a new classifier for a downstream task can now be much easier, and potentially even accomplished with just a bit of prompting.
I really don’t know too much about the intricacies of different ML architectures. As a broader vision, I am pretty excited to see video understanding capabilities be added into language models, and there are lots of questions around how to best encode video data efficiently that have yet to be answered.
This question is hard to answer, in part because we don’t know how big GPT-4 is :). More generally, as a community, we lack a good understanding of how the quality of pre-training data impacts model performance. For example, what sources should be included in the dataset? What types of automated filters should we train, and how aggressively should we use them? How feasible is it to learn from multi-lingual data? All these and many more data engineering questions remain, and we will probably see model capabilities steadily improve as we solve more pieces of this puzzle.
AlpacaFarm is really the completion arc for Alpaca! I’m really excited to see the methods innovation that the simulator can bring in the following months. Apart from that, I have a few loose ideas about what to work on next, but nothing concrete decided for now. You’re on the free list for TheSequence Scope and TheSequence Chat. For the full experience, become a paying subscriber to TheSequence Edge. Trusted by thousands of subscribers from the leading AI labs and universities. |
Key phrases
Older messages
Edge 295: Self-Instruct Models
Tuesday, May 30, 2023
What if LLMs could auto improve their own instruction following capabilities?
📝 Guest Post: How to build a responsible code LLM with crowdsourcing*
Monday, May 29, 2023
In this post Toloka showcases Human-in-the-Loop using StarCoder, a code LLM, as an example. They address PII risks by training a PII reduction model through crowdsourcing, employing strategies like
GPT-Microsoft
Sunday, May 28, 2023
Sundays, The Sequence Scope brings a summary of the most important research papers, technology releases and VC funding deals in the artificial intelligence space.
Announcing Turing Post
Saturday, May 27, 2023
When we launched TheSequence back in 2020, AI and machine learning were not as widely discussed or covered. Our goal from the start was to make AI knowledge accessible in bite-sized pieces, helping
📢 Event: ML practitioners from Affirm, Block, Remitly, Tide & more share their learnings from building risk & fra…
Friday, May 26, 2023
Want to connect with the ML engineering community and learn best practices from ML practitioners on how to build risk and fraud detection systems? Then join us on May 30 for apply(risk), a free half-
You Might Also Like
Software Testing Weekly - Issue 217
Monday, April 29, 2024
How do you deal with conflicts in QA? ⚔️ View on the Web Archives ISSUE 217 April 29th 2024 COMMENT Welcome to the 217th issue! How do you deal with conflicts in QA? Ideally, you'd like to know how
📧 Did you watch the free MMA chapters? (1+ hours of content)
Monday, April 29, 2024
Did you watch the free MMA chapters? Hey there! 👋 I wish you a fantastic start to the week. Last week, I launched Modular Monolith Architecture. More than 300+ students are already deep into the MMA
WP Weekly 191 - Essentials - Duplicate in Core, White Label Kadence, Studio for Mac
Monday, April 29, 2024
Read on Website WP Weekly 191 / Essentials It seems many essential features are being covered in-house, be it the upcoming duplicate posts/pages feature in the WordPress core or the launch of Studio
SRE Weekly Issue #422
Monday, April 29, 2024
View on sreweekly.com A message from our sponsor, FireHydrant: FireHydrant is now AI-powered for faster, smarter incidents! Power up your incidents with auto-generated real-time summaries,
Quick question
Sunday, April 28, 2024
I want to learn how I can better serve you
Kotlin Weekly #404 (NOT FOUND)
Sunday, April 28, 2024
ISSUE #404 28st of April 2024 Announcements Kotlin Multiplatform State of the Art Survey 2024 Help to shape and understand the Kotlin Multiplatform Ecosystem! It takes 4 minutes to fill this survey.
📲 Why Is It Called Bluetooth? — Check Out This AI Text to Song Generator
Sunday, April 28, 2024
Also: What to Know About Emulating Games on iPhone, and More! How-To Geek Logo April 28, 2024 📩 Get expert reviews, the hottest deals, how-to's, breaking news, and more delivered directly to your
Daily Coding Problem: Problem #1425 [Easy]
Sunday, April 28, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Microsoft. Suppose an arithmetic expression is given as a binary tree. Each leaf is an
PD#571 Software Design Principles I Learned the Hard Way
Sunday, April 28, 2024
If there's two sources of truth, one is probably wrong. And yes, please repeat yourself.
When Procrastination is Productive & Ghost integrating with ActivityPub
Sunday, April 28, 2024
Automattic, Texts, and Beeper join forces to build world's best inbox, Reflect launches its iOS app, how to start small rituals, and a lot more in this week's issue of Creativerly. Creativerly