What Is A Vector Database?

The past couple of years have seen significant advances in the field of artificial intelligence. A revolution that started with the launch of ChatGPT in November 2022 has been followed by a rapid number of launches of various Large Language Models(LLMs) by both corporates and Open Source communities.

The growing popularity of LLMs have also seen interest peak in technologies that support development of LLMs. One such technology is vector databases. Pinecone, one such vector database, recently raised a $100 million investment at a valuation of $750 million which put Pinecone and vector databases in the limelight.

In this blog we will explore 

• What is a vector database ?
• How does  a vector database work ?
• What is the need for a vector database ?
• How does a vector database compare to traditional databases ?
• Some popular vector databases

What Is A Vector Database ?

Going by the definition, a vector database is a database that stores information as vector embeddings and facilitates faster retrieval and search of similar data.

A vector database organizes data through high-dimensional vectors. High-dimensional vectors contain hundreds of dimensions, and each dimension corresponds to a specific feature or property of the data object it represents.

In Layman’s terms, a vector database facilitates storing of numeric representation of unstructured and semi-structured data. Machine learning models often output numeric data and hence vector databases offer an excellent way to store such data.

How Does A Vector Database Work ?

Any database not only vector database should support CRUD i.e. Create, Read, Update & Delete operation.  With that in mind let’s now understand how a vector database allows these operations.

A vector is a mathematical structure with a size and a direction. Vectors are an ideal data structure for machine learning algorithms — modern CPUs and GPUs are optimized to perform the mathematical operations needed to process them. 

Vector embeddings are numerical representations of unstructured or semi-structured data which preserve not only structure but the meaning of data. 

So, in terms of storage the operation is simple, the data is converted to vector by an embedding model or an output of a machine learning model and stored. 

For querying i.e. read operation a vector database uses a combination of different algorithms which all come together to participate in an Approximate Nearest Neighbor(ANN) search. The search is optimized by algorithms through techniques such as hashing, quantization or graph-based search. A high level overview of the process can be seen in the image below. This process forms the vector database pipeline.

The vector pipeline consists of the Indexing, Querying and Post Processing. Let us look in brief at each of these terms.

Indexing

Using hashing, quantization or  graph-based technique a vector database indexes vectors by mapping them to a given data structure. This helps in faster search. The vector database indexes not only the object but also the object’s metadata.

Querying 

When a vector database receives a query vector it looks up the indexed vectors and tries to find the nearest vector match i.e. nearest neighbor. To find these nearest neighbors a purely mathematical approach is relied upon known as similarity measures. 

Some of the measures that can be applied are Cosine similarity, Euclidean distance or Dot product

Post Processing

An optional step sometimes is to perform post processing to re-rank the results if required using a different similarity measure

What Is The Need For A Vector Database ?

The advances in LLMs have been rapid and the data required to train these LLMs is humungous and most of the times unstructured i.e. images, texts, voice, video, pdfs, excels etc.

As seen the LLMs cannot directly be trained on this unstructured data but it first needs to be converted into a mathematical representation of a vector form.

Traditional databases are designed to handle structured data well, even at scale. They are not suitable for storing vector representations. This is the primary reason for the emergence of vector databases and they are so important to LLMs

• Improved Query Performance:
  • Challenge: As datasets grow, the performance of traditional databases may degrade, leading to slower query response times.
  • Solution: Vector databases are designed with a focus on efficient vector operations, resulting in improved query performance for high-dimensional data. This is essential for applications like geospatial analysis or large-scale data analytics.

How Does A Vector Database Compare To A Traditional Database ?

For years, relational databases have been the backbone of software. Then with the rapid change of technology NoSQL databases also started being part of the software architecture and now the AI revolution has given prominence to vector databases as well. 

Let us quickly understand how a vector database stacks up against traditional databases.

High-Dimensional Data Handling

Challenge: Traditional databases, especially relational databases, struggle with efficiently storing and querying high-dimensional data.

Solution: Vector databases are specifically designed to handle high-dimensional vectors. This makes them well-suited for applications like image recognition, where each image can be represented as a high-dimensional vector.

Efficient Similarity Searches

Challenge: Traditional databases may not efficiently support similarity searches, which are essential for tasks like finding similar images, documents, or patterns.

Solution: Vector databases excel at similarity searches. By representing data as vectors, these databases can quickly identify and retrieve similar items based on mathematical operations in the vector space. This is crucial for recommendation systems and content similarity analysis.

Optimized for Machine Learning Applications

Challenge: Machine learning algorithms often involve extensive vector operations, such as calculating distances between data points or performing matrix manipulations.

Solution: Vector databases are optimized for such vector-centric operations, providing a performance boost for machine learning tasks. This makes them an ideal choice for applications ranging from natural language processing to recommendation engines.

Flexible Schema for Dynamic Data

Challenge: Many traditional databases have rigid schemes that may not accommodate the dynamic nature of certain types of data, especially in real-time applications.

Solution: Vector databases often offer more flexibility in terms of data schema. This is beneficial for applications where data structures may evolve over time, such as in social media analytics or IoT (Internet of Things) environments.

Having said that, traditional databases have also started taking strides to support vector search. 

Some Popular Vector Databases

Some popular names that currently dominate the vector database space are

• Qdrant which powers X (formerly twitter) GrokAI
• Pinecone
• Faiss from Facebook
• Weaviate
• Chroma 

Conclusion

The rise in prominence of vector databases is directly related to the AI advancements since 2022. They provide the right means to power complex LLMs which offer text generation or image generation or LLMs which are multi-modal.

As the data landscape continues to expand, the need for specialized databases becomes increasingly evident. Vector databases, with their focus on high-dimensional data and efficient vector operations, are positioned to play a crucial role in shaping the future of data management.

By understanding their origin, capabilities, and comparing them to traditional database models, organizations can make informed decisions about adopting vector databases to meet the demands of their data-intensive applications.