Best practices for querying vector data for gen AI apps in PostgreSQL (DAT423)

Here is a detailed summary of the video transcription in markdown format:

Introduction

• This is an L400 talk to dive deep into best practices for recording vector data for generative AI applications and using PostgreSQL.
• The speaker, Jonathan Katz, emphasizes that this talk will cover a gentle ramp-up, a lot of data, charts, and explanations.
• The talk will be recorded, and the speaker is open to answering questions after the talk.

Why Does This Matter?

• Generative AI models can provide rich, personalized responses, but we need a way to tie the data in our databases to the context required for these responses.
• Retrieve Augmented Generation (RAG) is an effective technique to bring in the necessary context from our databases.
• Vector embeddings are used to represent unstructured data (text, images, video) in a way that can be searched and compared.

Vector Embeddings and RAG Workflow

1. Ingestion: Unstructured data is preprocessed and passed through an embedding model to generate vector representations, which are then stored in a database (e.g., Amazon Aurora).
2. Agentic Workflow: When a user query comes in, it is also converted to a vector representation, which is used to query the database and retrieve the most similar results. These results are then used to augment the response from the generative AI model.

Challenges with Vector Data

• Generating vector embeddings can be time-consuming, especially for large datasets.
• Modern embeddings can be fairly large (e.g., 768-dimensional vectors are 6 KB in size), which can lead to "data inflation" compared to traditional relational data.
• Querying vector data involves expensive distance computations, which can be slow, especially for large datasets.

Approximate Nearest Neighbor Search

• To address the performance challenges of vector data, approximate nearest neighbor search techniques are used.
• Approximate nearest neighbor search allows comparing the query vector to a subset of the overall vectors, which is faster but may not return all the expected results (reduced recall).
• Recall is a critical measure of search quality that needs to be balanced against cost, development ease, and performance.

Choosing Vector Storage and Search Strategies

• Cost, ease of development, and performance (including query latency and data ingestion) are key factors in deciding how to store and search vector data.
• These factors are in tension with each other and need to be balanced based on the specific requirements of the application.

Using PostgreSQL for Vector Search

• PostgreSQL is an extensible database system that can be extended to support vector search through the PG Vector extension.
• PG Vector provides a vector data type and the necessary operations to perform vector searches, indexing, and more.
• PG Vector is open-source, actively developed, and the second-largest contributor is AWS.

PG Vector Features and Best Practices

1. Indexing Methods: PG Vector supports two approximate nearest neighbor search methods: IVF Flat (cluster-based) and HNSW (graph-based). HNSW has become a popular choice.
2. Storage Considerations: Vector data can be stored in-line with the table data (plane storage), in a separate "toast" table (external storage), or in a compressed form (extended storage). The storage method impacts performance.
3. Index Build Parameters: The key parameters for HNSW indexing are M (the number of connections per node) and EF_construction (the search radius during index building). These parameters impact index build time, search quality (recall), and query performance.
4. Data Ingestion: Bulk inserts, the COPY command (with binary format), and parallel ingestion can all improve data ingestion performance.
5. Quantization: Reducing the precision of vector data (quantization) can decrease storage requirements but may impact recall.

Filtering with Vector Data

• Combining vector search with filters in the query can be challenging due to the "over-filtering" problem, where the filters return too few or no results.
• PG Vector 0.8 introduced new features to address this, including the HNSW iterative scan parameter and better query planning estimation.
• Strategies for filtering depend on the selectivity of the filters:
  • High selectivity: Use a B-tree index rather than the vector index.
  • Low selectivity: Use the vector index with the iterative scan parameter.
  • Mixed selectivity: Use a combination of indexes (vector index and B-tree).

Amazon Aurora Features for Vector Search

• Amazon Aurora's Optimized Reads feature, which uses a local NVMe cache, can significantly improve performance for highly concurrent vector workloads.
• Amazon Aurora Serverless v2 (Limitless) can automatically scale the database to handle large vector datasets that exceed the capacity of a single instance.
• Amazon Bedrock Knowledge Bases can be used to automate the pipeline of ingesting vector data (e.g., from S3) into an Aurora database.

Future Developments in PG Vector

• Filtering and pre-filtering techniques are an active area of research and development.
• Support for additional vector data types (e.g., 16-bit floating-point, 8-bit integer) is planned.
• Streaming I/O and parallel query support are potential future improvements.

Conclusion

• Recall is the critical measure that should anchor all decisions around storing and searching vector data.
• Balancing recall, cost, ease of development, and performance is key when designing vector search-based applications.
• PG Vector provides a mature and extensible solution for vector search in PostgreSQL, and Amazon Aurora offers features to help scale and manage these workloads.
• Vector search technology is rapidly evolving, and staying up-to-date with the latest developments is important for building effective generative AI applications.