Inside Alibaba's engineering push to make pgvector work at billion-vector scale

Vector databases went through three phases since 2023: first, supporting vector types and basic distance operators. Then came approximate nearest neighbor indexes like IVF and HNSW, pulling response times down to single-digit milliseconds. The third phase, production readiness at scale, turned out to be the hardest. Alibaba Cloud's PolarDB for PostgreSQL team documented its approach in a technical post that unpacks exactly what it took to turn the open-source pgvector extension into an engine that can store billions of vectors and return results in milliseconds. This effort represents one of many open-source projects powering the AI stack, as cataloged in a recent roundup of 30 essential GitHub repositories.

Quantization: fitting larger vectors into smaller footprints

The first bottleneck is storage and compute cost. A single float32 vector of 768 dimensions weighs about 3 KB. Multiply that by a billion and you are looking at 3 TB of raw data before indexes. PolarDB for PostgreSQL now supports three quantization schemes alongside IVF and HNSW indexes: Product Quantization (PQ), Scalar Quantization (SQ), and Random Bit Quantization (RaBitQ). Each compresses a vector to between one-quarter and one-sixty-fourth of its original size, while keeping precision controllable. The team emphasized that no single scheme fits all scenarios; the right choice depends on dataset size, feature concentration, and dimensionality. PQ splits vectors into subsegments and trains a codebook per segment, offering high compression but heavy training cost. SQ maps each float32 dimension linearly to int8 or int4, making it the most cost-effective option. RaBitQ uses random orthogonal transformation plus 1-bit quantization, with a provable precision lower bound in theory.

Performance enhancements: I/O and free space management at billion scale

Index builds on standard pgvector follow the PostgreSQL buffer manager: every page written passes through the shared buffer, acquires a lock, generates write-ahead logs, and is flushed asynchronously. When indexes hit hundreds of gigabytes, that path becomes the primary bottleneck. PolarDB introduced batch read-write for index builds and a prefetch window for queries. Internal testing showed a 1.5 to 2 times improvement in build and query speed at billion-vector scale. A second pain point emerged under sustained inserts: when free pages appear in an index, a new insert had to scan data pages sequentially to find available space. The team introduced an efficient Free Space Map (FSM) management mechanism. In tests with sustained inserts at billion-row scale, I/O contention dropped by 90 percent and p99 write latency became noticeably more stable. Combined with a reworked index cleanup framework that moves bloat from linear growth to a steady state, the system maintains long-term stability even under high-write and high-delete workloads. This complements Alibaba's separate work on secondary prefix compression, which reduced index bloat in PolarDB-X by up to 70%.

Ecosystem enhancements: polar_vectorboost automates hybrid search

Teams adopting vector retrieval often had to write large amounts of fusion query logic by hand, embedding columns, tokenization, backfilling data, recommending indexes, and then dealing with inconsistent scales, missing fusion ranking, and recall stability. PolarDB's in-house extension polar_vectorboost compresses those high-frequency SQL operations into a single function call. A single call can add an embedding column, automatically tokenize text, backfill data, recommend an index, and register metadata. For hybrid search, it supports two fusion algorithms: Reciprocal Rank Fusion (RRF), which is scale-insensitive, and a weighted, min-max-normalized approach. It also automatically joins tsquery terms with OR to avoid the recall collapse that plainto_tsquery's default AND joining can cause. A set of diagnostic functions, checking column types, dimension