🔹◽️Edge#162: EleutherAI’s GPT-NeoX-20B, one of the largest open-source language models
This is an example of TheSequence Edge, a Premium newsletter that our subscribers receive every Tuesday and Thursday. On Thursdays, we dive deep into one of the freshest research papers or technology frameworks that is worth your attention. 💥 What’s New in AI: EleutherAI’s GPT-NeoX-20B, one of the largest open-source language modelsWe’ve seen the accelerating race to build massively large transformer models. From GPT-2 with 1.5 billion parameters, to Microsoft’s Turing NLG with 17 billion parameters, to GPT-3 passing 175 billion parameters and Google’s Switch Transformer reaching 1.6 trillion parameters, and now, Wu Dao 2.0 containing a mind-blowing 1.75 trillion parameters. Most of these transformer models are not open-source and are difficult to fully access. Just yesterday, in an effort to democratize access to large language models, EleutherAI, in partnership with CoreWeave, released a 20 billion parameter open-source language model named GPT-NeoX-20B. For those who are looking to serve GPT-NeoX-20B without managing infrastructure, CoreWeave has partnered with Anlatan, the creators of NovelAI, to create GooseAI, a fully managed inference service delivered by API. Let’s dive deeper into the practical implementation of GPT-NeoX-20B and GooseAI. The HistoryThe research collective EleutherAI was founded in 2020 by Connor Leahy, Sid Black and Leo Gao with the goal of ensuring that large language models remain widely accessible to aid in research towards the safe use of AI systems. In comparison, the well-known GPT3 is still closed to the public at large and has prohibitively expensive training costs – both are significant hurdles to researchers interested in studying and using it and businesses building products on top of it. From the beginning, EleutherAI has been deeply committed to helping tackle AI safety while focusing on making large language models more accessible.
In 2021, CoreWeave built a state-of-the-art NVIDIA A100 cluster for distributed training. They have partnered with EleutherAI to train GPT-NeoX-20B using this cluster.
GPT-NeoX-20BWith its beta release on February 2nd on GooseAI, EleutherAI claims GPT-NeoX-20B to be the largest publicly accessible language model available. At 20 billion parameters, GPT-NeoX-20B is a powerhouse trained on EleutherAI’s curated collection of datasets, The Pile. When EleutherAI developed The Pile’s 825GB dataset, no public datasets suitable for training language models of this size existed. The Pile is now widely used as a training dataset for many current cutting-edge models, including the Beijing Academy of Artificial Intelligence’s Wu Dao (1.75T parameters, multimodal), AI21’s Jurassic-1 (178B parameters), Anthropic’s language assistant (52B parameters), and Microsoft and NVIDIA’s Megatron-Turing NLG (340B parameters). In short, GPT-NeoX-20B is more accessible to developers, researchers, and tech founders because it is fully open-source and less expensive to serve compared to similar models of its size and quality.
Performance Comparison by ModelThe following table shows a comparison of GPT-NeoX-20 B’s performance, relative to other publicly available NLP models, to answer factual questions in a variety of domains. GPT-NeoX-20B outperforms its peers by a statistically significant margin: The other interesting comparison is of factual knowledge by subject group: GPT-NeoX-20B on GooseAIBefore being fully released to the public on February 9th, GooseAI is currently offering a week-long beta of EleutherAI’s GPT-NeoX-20B. GooseAI is a fully managed inference service delivered by API. Anyone who is looking to serve GPT-NeoX-20B without managing infrastructure can start serving the model via a fully managed API endpoint on GooseAI today, before the official release. With feature parity to other well-known APIs, GooseAI delivers a plug-and-play solution for serving open-source language models at over 70% cost savings by simply changing 1 line of code, helping to unlock everything that GPT-NeoX-20B has to offer. GooseAI delivers all the advantages of CoreWeave Cloud, with zero infrastructure overhead, including impressive spin-up times and responsive auto-scaling.
ConclusionGPT-NeoX-20B is a glimpse into the next generation of what powerful AI systems could look like, and EleutherAI works on removing the current barriers to research on the understanding and safety of such models. You can join the community of NeoX-20B developers at GooseAI for a preview period to serve GPT-Neox-20B from February 2nd through February 8th, before its full, open-source release on February 9th. You’re on the free list for TheSequence Scope and TheSequence Chat. For the full experience, become a paying subscriber to TheSequence Edge. Trusted by thousands of subscribers from the leading AI labs and universities. |
Older messages
🎙SuperAnnotate's CTO Vahan Petrosyan on the present and future of ML data labeling
Wednesday, February 2, 2022
No subscription is needed
🌆🏙 Edge#161: A New Series About Deep Generative Models
Tuesday, February 1, 2022
+Optimus, a large generative model for language tasks; +ART that uses generative models to protect neural networks
💻 Meta’s AI SuperComputer
Sunday, January 30, 2022
Weekly digest curated by industry insiders
📝 Guest post: The Original Open Source Feature Store - Hopsworks*
Friday, January 28, 2022
In TheSequence Guest Post our partners explain in detail what machine learning (ML) challenges they help deal with. This article reintroduces the core concepts of a Feature Store; the dual storage
🟢 ⚪️Edge#158: A Deep Dive Into Aporia, the ML Observability Platform
Thursday, January 27, 2022
Read it without subscription
You Might Also Like
Learn How to Use AI to Reach Your Full Potential, newsletterest1!
Sunday, May 19, 2024
3 Ways AI Can Help Your Writing ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Software Testing Weekly - Issue 220
Saturday, May 18, 2024
Software Testing Conferences 📚 View on the Web Archives ISSUE 220 May 18th 2024 COMMENT Welcome to the 220th issue! Have you ever been to a testing conference? They're a great way to learn about
📶 Is a Cellular iPad Worth It? — How to Prevent YouTube From Taking Over Your Screensaver
Saturday, May 18, 2024
Also: This Robot Vacuum Can Clean Stairs, and More! How-To Geek Logo May 18, 2024 📩 Get expert reviews, the hottest deals, how-to's, breaking news, and more delivered directly to your inbox by
Weekend Reading — Objection-oriented programming
Saturday, May 18, 2024
This week we find a power-up box, replace GitHub Actions with Maven XMLs, avoid the worst website in the world, revisit RTO policies, “listen” to OpenAI employees, watch our Slack private messages, do
Daily Coding Problem: Problem #1445 [Easy]
Saturday, May 18, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Jane Street. The United States uses the imperial system of weights and measures, which
You don’t have to take our word for it…
Saturday, May 18, 2024
You can probably tell how excited we are to re-launch our Gigantic courses – which bring on-demand product management training for today's modern Product Managers and Product Leaders. In fact, we
🐍 New Python tutorials on Real Python
Saturday, May 18, 2024
Hey there, There's always something going on over at realpython.com as far as Python tutorials go. Here's what you may have missed this past week: What Is the __pycache__ Folder in Python? In
Visualized | Life Expectancy by Region (1950-2050F) 📊
Saturday, May 18, 2024
This map shows life expectancy at birth for key global regions, from 1950 to 2050F. View Online | Subscribe Presented by Voronoi: The App Where Data Tells the Story FEATURED STORY Life Expectancy by
New Wi-Fi Vulnerability Enables Network Eavesdropping via Downgrade Attacks
Saturday, May 18, 2024
THN Daily Updates Newsletter cover The DevSecOps Playbook: Deliver Continuous Security at Speed ($19.00 Value) FREE for a Limited Time A must-read guide to a new and rapidly growing field in
🐍 New Python tutorials on Real Python
Saturday, May 18, 2024
Hey there, There's always something going on over at realpython.com as far as Python tutorials go. Here's what you may have missed this past week: What Is the __pycache__ Folder in Python? In