Problem
The multi-tenant workspace platform bills on usage. Each plan grants a limit per feature: record groups, records per group, automations and active users. Every metered action checks the workspace's counter for the current period against that limit before it's allowed through.
The first prototype did the obvious thing: load the counter row, compare it with the limit in Java, add one and save the row back. That's correct when requests arrive one at a time, which is exactly how they arrive in a demo.
Under concurrency it breaks twice over. Two requests can both read the same count and both save count + 1, so one increment silently disappears and the counter undercounts. And both can pass the limit check on that same stale count, so a workspace with one unit left gets two.
Approach
01Two bugs in one read-modify-write
Check-then-act across separate statements leaves a window between the read and the write. Any request that lands inside it works from stale state. Here that stale state causes two different failures: a lost update (the counter is wrong) and a broken invariant (the limit is exceeded).
Prototype: read, compare, save
Both pass the check and both save 100: two allowed with one unit left, one increment lost, no error anywhere.
Proposed: guarded UPDATE
The database serializes the writes. B's condition fails against A's committed value, so it gets a typed quota error.
WorkspaceUsageCounter counter = repository.find(workspaceId, feature, periodStart);
if (counter.getCount() + 1 > limit) {
throw new QuotaExceededException(...);
}
// Two requests can both reach this line with the same count:
// one increment is lost, and both were allowed.
counter.setCount(counter.getCount() + 1);
repository.save(counter);02Shipped: let SQL do the increment
The increment moved into a single UPDATE that adds to whatever value is committed when it runs. PostgreSQL's row lock serializes concurrent increments, so every allowed request is now counted. The counter can no longer drift.
What this didn't change is where the limit is checked. PolicyService still reads the count and compares it with the limit in Java before the record is saved, then the increment happens later in the same transaction. Two requests at the boundary can both pass that check before either increments. The counter will correctly show the workspace is over its limit, but it's over.
// 1. Check, in Java, against a plain read of the counter
policy.assertUnderQuota(workspaceId, MAX_RECORDS_PER_RECORD_GROUP, 1);
// ... validate and save the record ...
// 2. Increment, atomically, with no condition
@Modifying
@Query("""
UPDATE WorkspaceUsageCounter w
SET w.count = w.count + :delta
WHERE w.workspaceId = :workspaceId
AND w.featureKey = :featureKey
AND w.periodStart = :periodStart
""")
int incrementCountByDelta(...);The first prototype: load the counter, compare in Java, save count + 1.
workspace_usage_counters
- count
- 99
- limit
- 100
- requests allowed
- 0
- // B will read before A commits. Press send.
03Next: put the limit in the UPDATE
The fix I'd make next removes the window instead of narrowing it: pass the period's limit into the UPDATE and only increment when there's room. The row lock still serializes the competing writes, and the second one re-evaluates its WHERE clause against the first one's committed value, so it updates zero rows.
The service then stops deciding whether the quota allows the action. It asks the database to consume a unit and reads the affected-row count: 1 means allowed, 0 means throw QuotaExceededException, which rolls back the record saved in the same transaction. Unlimited plans (a limit of -1) skip the guard.
@Modifying
@Query("""
UPDATE WorkspaceUsageCounter w
SET w.count = w.count + :delta
WHERE w.workspaceId = :workspaceId
AND w.featureKey = :featureKey
AND w.periodStart = :periodStart
AND w.count + :delta <= :limit
""")
int tryConsume(UUID workspaceId, FeatureKey featureKey,
LocalDate periodStart, long delta, long limit);
if (counters.tryConsume(workspaceId, feature, periodStart, 1, limit) == 0) {
throw new QuotaExceededException(...);
}04Version plans as snapshots
Plan changes are versioned. Each subscription period carries its own snapshot of the plan's entitlements, and usage for that period is measured against the snapshot. Switching plans mid-period can't rewrite what the workspace already consumed under the old terms.
05Make failures specific
A generic 'request failed' gives a client nothing to branch on. Quota, policy and resource-lock violations each return their own RFC 9457 ProblemDetail with its own status, so the frontend can show an upgrade prompt, a plan message or an unlock hint as appropriate.
Quota exceeded · 429
The plan's limit for this feature is used up for the period.
Policy violation · 422
The plan doesn't include this feature at all.
Resource locked · 423
Over the limit of a downgraded plan; locked until an upgrade.
Outcome
Counting is fixed: moving the increment into SQL means concurrent requests can no longer lose updates, so usage figures are right.
Enforcement isn't finished. The limit is still checked in Java before the increment, so a burst of requests at the boundary can push a workspace slightly over. The guarded UPDATE above closes that by construction rather than by timing, and it's the change I'd ship next.
Lessons
- An atomic increment fixes lost updates, not invariants. If a limit matters, the check and the write have to be the same statement.
- Code that is 'correct' one request at a time is a hypothesis. Concurrency is the test.