Why My MongoDB Stalled for 2 Minutes: WiredTiger Tickets

One of my side projects stores a few hundred gigabytes of data in MongoDB. For weeks, users of that app would hit a page and watch it hang for about two minutes. Then everything would recover on its own. No crash, no restart. Just a recurring two-minute hole in the afternoon.

This is the story of how I chased that stall, what I got wrong along the way, and the one configuration mistake that turned out to be hiding in plain sight.

The symptom

The pattern was consistent. Requests that normally came back in a few hundred milliseconds would pile up, queue, and then all complete at once about two minutes later. CPU on the box was unremarkable. Memory looked fine. The database was not down, and the application logs showed nothing except slow queries.

The numbers for the database:

  • 362 GB of data
  • 42 GB of indexes
  • 154.6 million documents
  • a WiredTiger cache of 12 GiB

Write those numbers next to each other and the problem is already visible, but I did not see it yet. The working set I eventually measured was about 404 GB. The cache was 12 GiB. That is roughly 3 percent of the hot data fitting in memory.

What I wrongly assumed

My first theory was a missing index. An earlier audit of the cluster had flagged a collection with what looked like a terrible scan ratio: 86 documents examined for every one returned. That is a classic “you are doing a collection scan” smell, and it fit the story of a database that sometimes does enormous amounts of work for a small answer.

I spent a while building a case around it. I looked at the slow queries, found ones that did a fetch-then-filter on low-cardinality fields, and started drafting index changes.

Then I tried to reproduce the 86:1 figure and could not. The real number, measured over a short window, was 6.66 to 1. That is not great, but it is nowhere near a smoking gun. The 86:1 number came from a cumulative counter that had been accumulating since the process started, and it was dominated by old, unrepresentative history.

This is a mistake I have made more than once: reading a cumulative counter as if it described the present. Counters that span weeks of uptime are averages over everything that ever happened. If you want to know what the system is doing now, diff two snapshots a minute apart.

How I found the real cause

With the index theory deflated, I went looking at what the storage engine itself was doing during a stall. WiredTiger limits how many operations can be inside the storage engine at once using a ticket system. The classic default is 128 read tickets and 128 write tickets. If you run out of tickets, new operations queue up behind the ones in flight.

In newer MongoDB versions the number of tickets is no longer a fixed setting. The server adjusts it dynamically with an algorithm called throughput probing. It tries a concurrency level, measures throughput, and moves up or down looking for the best result. That is a sensible idea when the machine is CPU-bound. It behaves very differently when the machine is disk-bound.

On modern versions the place to look moved. The old wiredTiger.concurrentTransactions path in serverStatus is not where this lives now. You want the execution queues:

// mongosh
const s = db.serverStatus().queues.execution;
printjson(s);
// look at read.available / read.out and write.available / write.out
// and compare against the 128 default you may be assuming

What I found was that during a stall the ticket pool had been clamped to 6 or 7 read tickets and 6 or 7 write tickets. Not 128. Six.

This was throughput probing doing exactly what it was designed to do. The disk was the bottleneck. Adding more concurrent operations only made each of them wait longer for the same saturated disk, so the algorithm backed concurrency off to a handful. The result was a database that could only have about six operations in flight at a time, while the web app kept sending more. The queue behind those six tickets grew until it flushed.

The numbers behind the disk-bound claim:

  • roughly 3.5 TB per day was being read from disk
  • query threads were spending an enormous amount of time blocked on disk reads, on the order of 46 thread-days over the process lifetime
  • dirty cache pressure was trivial, about 96 MB, so this was not a write-flush problem

That last point mattered. It ruled out the other usual suspect for periodic stalls, which is the checkpoint flushing too much dirty data at once. The cache was not being forced to write. It was being forced to read, over and over, because the data it needed was almost never already in memory.

The mistake that was hiding in plain sight

Here is the part that stung. WiredTiger’s own default cache size is roughly half of RAM minus 1 GB, which on this machine works out to a little over 15 GiB. My configuration had pinned cacheSizeGB to 12.

So I had set the cache lower than what the database would have chosen for itself. At some point I had capped it to be conservative and then forgot about it. The box was not short of memory. Dirty pressure was tiny. There was headroom sitting unused.

# mongod.conf
storage:
  wiredTiger:
    engineConfig:
      cacheSizeGB: 20   # was 12; the engine's own default would have been ~15.4

I raised it to 20 GiB. It needs a restart to take effect, so I did it in a quiet window.

The second finding: 45 indexes on one collection

While I was in there, the index audit turned up something more useful than the phantom scan ratio. The busiest collection, about 4.76 million documents that receive a very high volume of updates (hundreds of millions over its lifetime), carried 45 indexes.

Every write to a document has to update every index that covers the changed fields. Forty-five indexes means a lot of extra random writes on a disk that is already the bottleneck. When I went through them:

  • 10 were single-field indexes that were just a prefix of an existing compound index, so they were fully redundant
  • 4 had zero recorded use
  • 1 more needed verification before I would trust dropping it

That is 15 candidates, worth about 936 MB, around 24 percent of that collection’s index footprint, and about a third less index maintenance on every write. A prefix index being redundant is one of the few index-cleanup rules that is safe to apply mechanically: if you have an index on (a, b, c), a separate index on (a) is almost always dead weight, because queries on a alone can use the compound one.

// find indexes that have never been used since the last restart
db.mycollection.aggregate([{ $indexStats: {} }])
  .toArray()
  .filter(i => i.accesses.ops === 0)
  .map(i => i.name);

One caveat that I now repeat to myself: $indexStats counters reset on restart. Zero uses since the last restart is a reason to look closer, not proof that an index is useless. A monthly report query could be the only thing that touches it.

What the numbers say, and what I do not know

I want to be honest about the result. After raising the cache to 20 GiB, the ticket queues were quiet when I sampled them, with zero queued and zero tickets in use on later checks. But I do not have a rigorous before-and-after on how often the two-minute stalls occurred. Stalls were intermittent to begin with, so a quiet afternoon after a change proves little.

What I can say with confidence is that the configuration was wrong in a way that made a disk-bound system worse, and that fixing it is cheap and reversible. When the working set is many times larger than the cache, a larger cache buys headroom. It does not change the shape of the problem.

Lessons

  • Diff counters, do not read them cold. A cumulative number that spans weeks of uptime will confidently tell you about a past that no longer exists. My 86:1 scan ratio was really 6.66:1 when I measured a short window.
  • When throughput probing clamps tickets to single digits, the database is telling you it is waiting on something. More concurrency would not help. Look at disk reads and the ratio of working set to cache.
  • Check your cache setting against the engine default. If you pinned it below what the database would pick for itself, you need a reason, and the reason needs to still be true.
  • Every index is a tax on every write. Forty-five indexes on a hot collection is a write amplifier. Prefix-redundant indexes are free to remove.
  • Intermittent problems resist proof of a fix. Say so in the write-up instead of declaring victory.

If you run MongoDB on a box where the data is much bigger than memory, spend ten minutes looking at queues.execution during your next slow spell. If the ticket count is a single digit, you do not have a concurrency problem. You have a disk problem wearing a concurrency costume.

By Alex Rivers

I write hands-on guides to web development and programming: JavaScript and React patterns, Python scripting, CSS layout and the developer tools that tie it all together. I like explanations that start with working code and end with knowing why it works. If a post saves you an afternoon of debugging, it did its job.

Leave a Reply