How 1 Millisecond Broke Our Rewards System

A real-world lesson in database concurrency, optimistic locking, and lost updates.

---

The Bug That Shouldn’t Exist

In a recent system update, something strange happened.

A user completed their profile flow and ended up with -50 reward points instead of 0 or +50.

At first, this didn’t make sense. The business logic was simple:

• Delete profile step → -50 points
• Re-upload ID → +50 points

So the final state should always cancel out.

But under load and fast user actions, the system broke.

The culprit: a 1 millisecond race condition.

---

The Heart of the Problem: The Storage Engine

To understand this, we need to look at the database internals.

Inside every DBMS lies the storage engine, with two key components:

Transaction Manager

The coordinator.
Ensures all operations respect ACID properties and consistency rules.

Lock Manager

The guard.
Controls access to shared data so concurrent transactions don’t corrupt state.

---

Latches vs Locks

Databases use two different synchronization mechanisms:

Latches

• Very fast and short-lived
• Protect physical memory structures (pages, buffers)
• Used internally by the DB engine

Locks

• Slower but logical
• Protect data correctness at transaction level
• Ensure consistency across queries and transactions

Think of it like:

• Latches → internal engine safety
• Locks → business data safety

---

Why We Use Optimistic Concurrency Control

Our system uses Optimistic Concurrency Control (OCC).

The idea:

Assume conflicts are rare, and validate only before committing.

It works in three phases:

1. Read Phase

The transaction reads data and performs operations locally.

2. Validation Phase

Before committing, the DB checks:

“Has this data changed since I read it?”

3. Write Phase

If validation succeeds, changes are saved.

If not, the transaction is rejected.

---

The 1ms Bug: A Lost Update

Here’s what actually happened.

Two operations were triggered almost simultaneously:

• Transaction A: Delete → -50 points
• Transaction B: Re-upload → +50 points

Both started within ~1 millisecond.

Timeline

1. Transaction A commits first → user becomes -50
2. Transaction B reaches validation phase
3. DB detects data was modified
4. Transaction B is rejected (optimistic lock failure)

Final state

• -50 applied
• +50 rejected

➡️ User ends up at -50 points

This is a classic lost update under optimistic concurrency control.

---

Why This Happens in Real Systems

This isn’t a database bug.

It’s a trade-off.

Optimistic concurrency assumes:

• Conflicts are rare
• Retrying is cheaper than locking everything

But under extreme timing (like 1ms bursts), assumptions break.

---

What We Learned

• Concurrency bugs are timing bugs
• Optimistic locking requires retry logic
• Rejected transactions must be handled properly
• Race conditions only show up under real load

---

Reference

Based on concepts from:

• Database Internals — Alex Petrov

Especially:

• Concurrency Control
• Transaction Management
• Locking vs Optimistic strategies

---

Closing Thought

A 1ms difference didn’t break the logic.

It broke the assumption that nothing else would happen at the same time.

And in distributed systems, that assumption is always the first thing to fail.