Exit Code 143: My Liveness Probe Was Killing a Next.js App

A Next.js frontend in my cluster kept restarting, and the exit code told me a comfortable lie. It said 143, which is SIGTERM, which sounds like a polite request to shut down. It was not OOMKilled. So I assumed nothing was wrong with memory. I had it backwards, and it took me a while to see why the container was really dying.

The symptom

One of my side projects has a Next.js frontend, and it was restarting a few times a day. The restart count went from 3 to 6 in a single 24-hour stretch. Of all the Next.js frontends in my cluster, this was the only one that restarted at all. Its peers were calm:

  • one frontend used 992 MiB of a 4 GiB limit
  • another used 178 MiB of a 768 MiB limit

The terminated container reported exit code 143 with reason “Error”. Not “OOMKilled”, which would be exit 137. So the memory limit apparently was not being hit.

What I wrongly assumed

I read 143 as “something outside the process asked it to stop.” That is literally true, but my brain filed it under “graceful, external, probably a node drain or a rollout.” When I checked, there was no drain and no rollout. I then started wondering about the node.

The other thing I assumed was that the earlier fix had worked. I had already raised this container’s memory limit once, from 512 MiB to 2 GiB, because it had been OOMKilled before. That bump had fixed the clean signal. I thought I was done.

What the bump had really done was take away the good diagnostic. An OOMKill is unambiguous: it shows up in the pod status and the kernel log. Replacing it with something quieter made the symptom harder to read, not rarer.

How I found the real cause

The clue was a number. The container’s uptime before it died was about 175 seconds from the point it started failing. That matched, almost exactly, the liveness probe configuration:

livenessProbe:
  httpGet:
    path: /health
    port: http
  periodSeconds: 30
  timeoutSeconds: 10
  failureThreshold: 6    # 6 x 30s = 180s

Six consecutive failures at thirty-second intervals is 180 seconds. My observed 175 seconds is as close as the timing of a restart could be expected to get. The kubelet was not being gracefully polite. The kubelet was killing the container because the health endpoint had stopped answering for three minutes, and it sends SIGTERM when it does that. Exit code 143.

Why would a health endpoint stop answering? Because the Node.js event loop was stalled. And memory explained that. I sampled the container over several days:

  • 575 MiB after 11 hours of uptime
  • 1,613 MiB (79 percent of the 2 GiB limit) after 15 hours
  • 1,936 MiB (94.5 percent) after 27 hours
  • 1,441 MiB after 3 days and 10 hours, with no restart

As the process approached its limit, garbage collection began working harder and harder, and long GC pauses froze the event loop. A frozen event loop cannot answer /health inside ten seconds. Six failures later, the kubelet pulled the plug.

That last data point matters. The memory came back down, from 94.5 percent to about 70, without a restart. So this was not a monotonic leak. It was a sawtooth: a cache that grows, gets reclaimed, and grows again. A restart happens when a peak in the sawtooth coincides with a burst of traffic, which is why restarts clustered instead of arriving on a clock. One consequence is that a single high reading, like 94.5 percent, tells you almost nothing on its own.

The twist: it was not only memory

A few days later the exit code changed. A restart finished with 139, which is SIGSEGV, a native crash. That is neither a liveness kill nor a memory kill, so I had to throw out the previous explanation for that one event. The memory limit had also been raised again, to 3 GiB. Lesson from that: capture the previous container’s output before theorising.

Then I found the more important thing. Restarts were not the main damage. The main damage was readiness flapping. One warning event had fired 1,943 times in a week, about 12 per hour:

Readiness probe failed: Get "http://.../health": context deadline exceeded

The pod showed Ready between flaps, so a quick look at the pod list looked healthy. But whenever it flapped with nothing else ready behind the Service, the endpoint list emptied and users got 502s and 503s. It was continuous low-grade degradation, invisible unless I checked event counts rather than pod status.

Memory was not the binding constraint any more: the container was at 1,742 MiB of 3 GiB, 57 percent. What had changed was the node. That pod had been moved to a worker that was running at about 91 percent CPU during a batch speech-to-text workload, with CPU limits across the pods on that node summing to 183 percent of its capacity. The event loop was being CPU-throttled, and the health endpoint was timing out for the same reason.

I covered the cause of that oversubscription in another post: one transcription deployment was running 8 threads where 4 would have been faster, burning 58 percent of the node’s cores for 11 percent of the output. Fixing that returned about three cores to the node, and as far as I can tell that is what let the frontend settle down. As of my last check it has been quiet.

What did not work, and what I avoided

  • Loosening the liveness probe. That would only have let the process run to a real OOMKill instead of getting restarted early. It would have hidden the problem while making it worse.
  • Raising the limit forever. It bought time. It did not fix growth that is not bounded.
  • Treating this as an app bug first. The app was one of the victims. The loud neighbour on the node was the cause of the flapping.

The durable fix

The proper long-term fix for the memory side is to bound the growth. A cache with no size limit is the usual suspect for a sawtooth like this, and capping it gives the sawtooth a ceiling. Running more than one replica means one pod’s flap cannot empty the Service.

How I would read this next time

Three questions, in order. What was the exit code, and which component produces it? 137 is the kernel killing for memory, 143 is a polite stop request and the kubelet sends it after failed liveness checks, 139 is the process itself crashing. Then: does the time to death match a probe schedule? Multiply the failure threshold by the period and compare. Finally: did anything else on the node change? Probe timeouts are sensitive to scheduling delay as much as to application slowness, and a health endpoint is often the first thing to notice when the node gets busy.

Lessons

  • Exit 143 does not mean “polite.” It means something sent SIGTERM. If nothing in the platform drained or rolled out, it was probably the kubelet acting on a probe.
  • Do the arithmetic on your probe. failureThreshold times periodSeconds gave me the exact lifetime to match against. A timing coincidence is worth chasing.
  • A bigger limit can make debugging harder. It replaces a loud, obvious failure with a quiet, indirect one.
  • Sawtooth is not leak. Look at memory over several days before choosing a story.
  • Count events instead of eyeballing pod status. A single replica that flaps twelve times an hour looks Ready every time you look.
  • Noisy neighbours cause application symptoms. A health check timing out can be a node-level CPU problem entirely unrelated to your code.

By Alex Rivers

I write hands-on guides to web development and programming: JavaScript and React patterns, Python scripting, CSS layout and the developer tools that tie it all together. I like explanations that start with working code and end with knowing why it works. If a post saves you an afternoon of debugging, it did its job.

Leave a Reply