Alibaba's Panda Index gives InnoDB unique keys a version history they never had

Unique keys are the quiet bottleneck of online databases. Order IDs, phone numbers, and transaction serial numbers lean on unique key (UK) indexes to stay non-duplicated, and under high concurrency the enforcement starts to cost. In InnoDB, the engine behind MySQL and Alibaba's MySQL-compatible PolarDB-X, the problem is structural: secondary indexes carry no version information, so the engine compensates with gap locks and table lookups.

Alibaba's PolarDB-X team has published a fix. Panda Index, a new-generation UK index for the engine's data nodes, gives unique keys native multi-version capability with transaction columns and an independent undo chain on each record. The ApsaraDB team's technical post describes a simpler insert path and true index-only scans.

The InnoDB unique-key bottleneck

PolarDB-X is a distributed database on a unified centralized-and-distributed architecture, fully MySQL-compatible, with a row store handling OLTP per data node. InnoDB's clustered index manages history the standard way: each record carries a TRX_ID and a ROLL_PTR into an undo chain, updates happen in place, and the B+ tree keeps only the latest version. Secondary indexes get none of that. UK records carry no TRX_ID, no ROLL_PTR, and no version information.

So a unique key update uses delete-mark-then-insert: the engine marks the old record deleted, then inserts a new one. Until the purge thread runs, several physical records with the same key coexist on the index. That breaks uniqueness enforcement in two ways.

The first problem is locking. Insert conflict detection cannot tell which of several records sharing a key is live, so it scans them all and places gap locks across the interval. Logically non-conflicting transactions end up waiting on each other, sometimes deadlocking. The problem is especially acute in trading systems, balance deductions, and flash sales, the ApsaraDB engineers note.

The second is visibility. With no version information on UK records, an MVCC read cannot judge visibility alone, even when the index covers every column in the query. It must return to the clustered index and walk the undo chain, as must implicit lock detection and purge. Write-heavy workloads add a third penalty: purge lags behind deletions, and residual delete-marked records widen the next conflict check's scan range.

Inside Panda Index: four system columns and an independent undo chain

Panda Index pushes the clustered index's versioning down into the UK index. Each record gains four system columns: TRX_ID, the transaction ID of the most recent modification; ROLL_PTR, a pointer to an independent undo record; SCN, a commit sequence number from the Lizard transaction system; and UBA, an undo block address. Their semantics match clustered index records, so the engine reuses Lizard's existing visibility framework, Vision.

ROLL_PTR points to an undo chain owned by the UK index alone. Updates happen in place: no delete-mark-then-insert, one physical record per unique key, and historical versions parked in a separate undo tablespace, reachable through ROLL_PTR without a table lookup. Forward DML, rollback, and purge each follow dedicated logic, which decouples the UK index's version lifecycle from the clustered index's.

One record lock instead of a row of gap locks

In-place updates produce a useful side effect: at most one physical record exists per unique key. Insert conflict detection collapses to a check. If no record exists, the insert succeeds immediately. If one exists, the engine places a single record lock on it and checks whether it is delete-marked. No gap locks, and a much smaller lock footprint.

The post includes a worked example. A transaction deletes a row where c1 = 1 and inserts a new row with the same value; a second transaction then inserts c1 = 2, which does not conflict. Under standard InnoDB behavior, the second insert times out, with four locks on the index, three of them gap locks. Under Panda Index, it succeeds at once with a single X,REC_NOT_GAP record lock.

Visibility gets the same simplification: consistent reads complete inside the index B+ tree, covering-index queries get true index-only scans, and implicit lock detection no longer reaches the clustered index.

The high-concurrency payoff, and what's still missing

The change aligns UK version management with the clustered index, which InnoDB already handles well. Alibaba has published results for neighboring database index work: secondary prefix compression cut PolarDB-X index space by 30 to 70 percent, with up to 47 percent sysbench throughput gains under I/O pressure, and separate PolarDB vector-index engineering reported 1.5 to 2 times faster builds and queries, with I/O contention down 90 percent.

Panda Index itself has no published numbers yet. The post includes no benchmarks, and the lock-table evidence covers a single hand-built scenario. Four extra system columns and a second undo chain add storage and purge overhead of their own, and whether those stay flat under UK-heavy writes is something only measurements can answer. The honest summary: the mechanism looks right, and the evidence base is Alibaba's own.