📝 Guest post: What is a Vector Database?*
Was this email forwarded to you? Sign up here In this guest post, Frank Liu, Director of Operations & ML Architect @ Zilliz, took a quick tour of vector databases. Specifically, he looked at 1) what features go into a mature vector database, 2) how a vector database differs from vector search libraries, 3) how a vector database differs from vector search plugins in traditional databases or search systems, and 4) the key challenges associated with building a vector database. Vector databases from 1000 feetGuess how many curators it took to label the now-famous ImageNet dataset. Ready for the answer? 25000 people (that’s a lot). Being able to search across images, video, text, audio, and other forms of unstructured data via their content rather than human-generated labels or tags is exactly what vector databases were meant to solve. When combined with powerful machine learning models, vector databases such as Milvus have the ability to revolutionize e-commerce solutions, recommendation systems, computer security, pharmaceuticals, and many other industries. A vector database is a fully managed, no-frills solution for storing, indexing, and searching across a massive dataset of unstructured data that leverages the power of embeddings from machine learning models. But let’s think about it from a user perspective. What good is a piece of technology without strong usability and a good user API? In concert with the underlying technology, multi-tenancy and usability are also incredibly important attributes for vector databases. Let’s list out all of the features a mature vector database should have (many of these features overlap with those of databases for structured/semi-structured data):
Phew. That was quite a bit of info, so we’ll summarize it right here: a vector database should have the following features:
In the next two sections, we’ll follow up on this concept by comparing vector databases versus vector search libraries and vector search plugins, respectively. Vector databases versus vector search librariesA common misconception that I hear around the industry is that vector databases are merely wrappers around ANN search algorithms. This could not be further from the truth! A vector database is, at its core, a full-fledged solution for unstructured data. As we’ve already seen in the previous section, this means that user-friendly features present in today’s database management systems for structured/semi-structured data - cloud-nativity, multi-tenancy, scalability, etc - should also be attributes for a mature vector database as well. All of these features will become clear as we dive deeper into this guest post. On the other hand, projects such as FAISS, ScaNN, and HNSW are lightweight ANN libraries rather than managed solutions. The intention of these libraries is to aid in the construction of vector indices – data structures designed to significantly speed up nearest neighbor search for multi-dimensional vectors. If your dataset is small and limited, these libraries can prove to be sufficient for unstructured data processing, even for systems running in production. However, as dataset sizes increase and more users are onboarded, the problem of scale becomes increasingly difficult to solve. Vector databases also operate in a totally different layer of abstraction from vector search libraries - vector databases are full-fledged services, while ANN libraries are meant to be integrated into the application that you’re developing. In this sense, ANN libraries are one of the many components that vector databases are built on top of, similar to how Elasticsearch is built on top of Apache Lucene. To give an example of why this abstraction is so important, let’s take a look at inserting a new unstructured data element into a vector database. This is super easy in Milvus:
It’s really as easy as that - 3 lines of code. With a library such as FAISS or ScaNN, there is unfortunately no easy way of doing this without manually re-creating the entire index at certain checkpoints. Even if you could, vector search libraries still lack scalability and multi-tenancy, two of the most important vector database features. Vector search plugins for traditional databasesGreat, now that we’ve established the difference between vector search libraries and vector databases, let’s take a look at how vector databases differ from vector search plugins. An increasing number of traditional databases and search systems, such as Clickhouse and Elasticsearch, include built-in vector search plugins. Elasticsearch 8.0, for example, includes vector insertion and ANN search functionality that can be called via restful API endpoints. The problem with vector search plugins should be clear as night and day - these solutions do not take a full-stack approach to embedding management and vector search. Instead, these plugins are meant to be enhancements on top of existing architectures, thereby making them limited and unoptimized. Developing an unstructured data application atop a traditional database would be like trying to fit lithium batteries and electric motors inside a frame of a gas-powered car - not a great idea! To illustrate why this is, let’s go back to the list of features that a vector database should implement (from the first section). Vector search plugins are missing two of these features - tunability and user-friendly APIs/SDKs. I’ll continue to use Elasticsearch’s ANN engine as an example; other vector search plugins operate very similarly so I won’t go too much further into detail. Elasticsearch supports vector storage via the
Elasticsearch’s ANN plugin supports only one indexing algorithm: Hierarchical Navigable Small Worlds, also known as HNSW (I like to think that the creator was ahead of Marvel when it came to popularizing the multiverse). On top of that, only L2/Euclidean distance is supported as a distance metric. This is an okay start, but let’s compare it to Milvus, a full-fledged vector database. Using
While both Elasticsearch and Milvus have methods for creating indexes, inserting embedding vectors, and performing nearest neighbor search, it’s clear from these examples that Milvus has a more intuitive vector search API (better user-facing API) and broader vector index + distance metric support (better tunability). Milvus also plans to support more vector indices and allow for querying via SQL-like statements in the future, further improving both tunability and usability. We just blew through quite a bit of content. This section was admittedly fairly long, so for those of you who skimmed it, here’s a quick tl;dr: Milvus is better than vector search plugins because Milvus was built from the ground-up as a vector database, allowing for a richer set of features and an architecture more suited towards unstructured data. Technical challengesEarlier in this guest post, I listed the desired features a vector database should implement, before comparing vector databases to vector search libraries and vector search plugins. Now, let’s briefly go over some high-level technical challenges associated with modern vector databases. In future guest posts, we’ll provide an overview of how Milvus tackles each of these, in addition to how these technical decisions improve Milvus’ performance over other open-source vector databases. Picture an airplane. The airplane itself contains a number of interconnected mechanical, electrical, and embedded systems, all working in harmony to provide us with a smooth and pleasurable in-flight experience. Likewise, a vector database is composed of a number of evolving software components. Roughly speaking, these can be broken down into the storage, the index, and the service. Although these three components are tightly integrated, companies such as Snowflake have shown the broader storage industry that “shared nothing” database architectures are arguably superior to the traditional “shared storage” cloud database models. Thus, the first technical challenge associated with vector databases is designing a flexible and scalable data model. Great, so we have a data model. What’s next? With data already stored in a vector database, being able to search across that data, i.e. querying and indexing, is the next important component. The compute-heavy nature of machine learning and multi-layer neural networks has allowed GPUs, NPUs/TPUs, FPGAs, and other general purpose compute hardware to flourish. Vector indexing and querying is also compute-heavy, operating at maximum speed and efficiency when run on accelerators. This diverse set of compute resources gives way to the second main technical challenge, developing a heterogeneous computing architecture. With a data model and architecture in place, the last step is making sure your application can, well, read from the database - this ties closely into the API and user interface bullet points mentioned in the first section. While a new category of database necessitates a new architecture in order to extract maximal performance at minimal cost, the majority of vector database users are still acclimated to traditional CRUD operations (e.g. Note how each of the three components corresponds to a primary technical challenge. With that being said, there is no one-size-fits-all architecture for vector databases. The best vector databases will fulfill all of these technical challenges by focusing on delivering the features mentioned in the first section. Wrapping upIn this guest post, we took a quick tour of vector databases. Specifically, we looked at 1) what features go into a mature vector database, 2) how a vector database differs from vector search libraries, 3) how a vector database differs from vector search plugins in traditional databases or search systems, and 4) the key challenges associated with building a vector database. This guest post is not meant to be a deep dive into vector databases, nor is it meant to show how vector databases can be used in applications. Rather, the goal is to provide an overview of vector databases. This is where your journey truly begins! *This post was written by Frank Liu, Director of Operations & ML Architect at Zilliz, and originally posted here. We thank Zilliz for their support of TheSequence.You’re on the free list for TheSequence Scope and TheSequence Chat. For the full experience, become a paying subscriber to TheSequence Edge. Trusted by thousands of subscribers from the leading AI labs and universities. |
Key phrases
Older messages
🌅 The Era of Foundation Models is Here
Sunday, November 20, 2022
Weekly news digest curated by the industry insiders
📝 Guest post: Using One Methodology to Solve The Three Failure Modes
Friday, November 18, 2022
In this guest post, Eric Landau, CEO of Encord, discusses the three major model failure modes that prevent models from reaching the production state and solve all three problems with a single
🗣👥 Edge#244: This Google Model Combines Reasoning and Acting in a Single Language Model
Thursday, November 17, 2022
ReAct provides an architecture that triggers actions based on language reasoning paths
📌 Event: apply(recsys)—ML experts from Slack, ByteDance & more share their recommender system learnings
Wednesday, November 16, 2022
Are you building an ML recommender system or planning to? Then you won't want to miss apply(recsys)
🔂 Edge#243: Text-to-Image Synthesis Models – Recap
Tuesday, November 15, 2022
Our longest and the most popular series
You Might Also Like
Learning about Android Runtime
Thursday, April 25, 2024
View in browser 🔖 Articles Learning about Android Runtime I always enjoy reading articles that explore how something works under the hood. Here's an article that does exactly that, providing
Stripe changes its … stripes
Wednesday, April 24, 2024
TikTok on the president's docket and Nvidia acquires Run:ai View this email online in your browser By Christine Hall Wednesday, April 24, 2024 Good afternoon, and welcome to TechCrunch PM! Today
💪 You Can Use Copilot AI as a Personal Trainer — Why Your Laptop Needs a Docking Station
Wednesday, April 24, 2024
Also: Here's How to Make Your Apple ID Recoverable, and More! How-To Geek Logo April 24, 2024 📩 Get expert reviews, the hottest deals, how-to's, breaking news, and more delivered directly to
JSK Daily for Apr 24, 2024
Wednesday, April 24, 2024
JSK Daily for Apr 24, 2024 View this email in your browser A community curated daily e-mail of JavaScript news JSK Weekly - 24th April, 2024 React 19 has introduced many great functionalities and
Daily Coding Problem: Problem #1422 [Hard]
Wednesday, April 24, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Airbnb. Given a list of integers, write a function that returns the largest sum of non-
Charted | Artificial Intelligence Patents, by Country 🤖
Wednesday, April 24, 2024
This visualization shows which countries have been granted the most AI patents each year, from 2012 to 2022. View Online | Subscribe Presented by: New on VC+: Our Visual Briefing on the IMF's World
Save your seat: 1Password’s 2024 Security report insights webinar
Wednesday, April 24, 2024
Join us April 25th. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Top Tech Deals 📱 LG Flex TV, Google Pixel 7, DJI Mini 3, and More
Wednesday, April 24, 2024
Get yourself a discounted DJI drone, save on the Pixel 7, or score some PC and phone accessories. How-To Geek Logo April 24, 2024 Top Tech Deals: LG Flex TV, Google Pixel 7, DJI Mini 3, and More Find
The Protest Song Wakes Up 🎙️
Wednesday, April 24, 2024
Is this song the future of musical protest? Here's a version for your browser. Hunting for the end of the long tail • April 24, 2024 The Protest Song Wakes Up A buzzy protest song about the
JSK Weekly - 24th April, 2024
Wednesday, April 24, 2024
React 19 has introduced many great functionalities and features, among which the useOptimistic hook stands out. The useOptimistic hook offers a seamless way to manage UI states during asynchronous