We ran a large ETL system that turned emails into structured data. Producers pushed email jobs into a queue, a fleet of Python workers on Kubernetes processed them, and an autoscaler grew and shrank the fleet with the load.

When we reconciled the numbers, they didn’t add up. For every 100,000 emails that went in, 10-odd fewer came out the other end. No errors in the logs, no failed jobs, no alerts. Just a tiny, steady gap.

A gap of 0.01% is easy to shrug off. It shouldn’t be. At a billion records a day, 0.01% is 100,000 records a day, and a gap with no error message means something in the pipeline is dropping data silently. This post is about how that happened, how we fixed it, and a local experiment that reproduces it with real numbers.

The setup

The queue was a home-grown message queue built on Redis lists:

• Producers LPUSH a message (an email ID plus metadata) onto a list.
• Workers block on BRPOP, take one message, parse and enrich the email, and write the result downstream.
• A Kubernetes HorizontalPodAutoscaler scales the worker Deployment up and down with the backlog.

Simple, fast, and it handled the volume well. It also has one property that turned out to matter: BRPOP removes the message from Redis. There’s no acknowledgement step. From the moment a worker pops a message until it writes the result, the only copy of that message lives in the worker’s memory.

Finding where the messages went

Counting at the start and end of a pipeline tells you that data is missing. To find out why, you need to know which records are missing and when they disappeared:

1. Reconcile by ID, not just by count. If every message carries a unique ID, the set difference between IDs produced and IDs written gives you the exact missing records.
2. Find the stage where they vanish. Count at every hop: produced, popped, processed, written.
3. Line up the timing of the losses with what else was happening in the cluster.

In our case the trail led to the workers, and to the moments when the autoscaler scaled them down. That explains everything. When the HPA scales a Deployment down, Kubernetes terminates pods: it sends SIGTERM, waits for terminationGracePeriodSeconds, then sends SIGKILL. Our workers didn’t handle SIGTERM in a way that protected the message they were working on. Each pod that was terminated mid-message was killed halfway through, and the message it had popped died with it.

It also explains why the gap was so small and so steady. A busy worker holds exactly one message at a time. Every terminated pod loses at most one message. So the number of lost messages tracks the number of scale-down terminations, not the volume of data.

Reproducing it locally

To check this explanation, and to give this post real numbers, I rebuilt the mechanism on my laptop:

• Redis 7 in Docker, a list with 100,000 message IDs, and 50 Python worker processes doing BRPOP.
• Most messages take 5–14 ms to process. 1% take 1–2.5 seconds, standing in for the occasional huge email with attachments.
• A small orchestrator plays the autoscaler. It runs three scale-down events that terminate 5 workers each (15 in total) the way Kubernetes does: SIGTERM, wait the grace period, then SIGKILL. Each scale-down is followed by a scale-up.
• Time is compressed so a run takes about a minute: the grace period is 1 second instead of 30.

Then it reconciles: how many of the 100,000 IDs reached the sink?

Worker	Pods terminated	Messages lost	Handed back to fallback	Processed twice
No SIGTERM handling	15	15	0	0
Graceful shutdown	15	2	0	0
Graceful shutdown + fallback queue	15	0	3	0

With no handling, 15 terminations lost exactly 15 messages, the same 10-odd per 100,000 we saw in production. A graceful shutdown fixed most of it, but not all of it. Only adding the fallback queue brought the loss to zero.

Fix, part 1: a graceful exit

The first part is the standard one. When SIGTERM arrives, the worker must stop taking new messages and finish the one it has. (The mechanics, including why your app needs to be PID 1 and how to size terminationGracePeriodSeconds, are in Graceful Shutdown for ETL Workers on Kubernetes.)

This took us from 15 lost messages to 2. Why not 0? Look at the two that were still lost: both were slow messages. The worker received SIGTERM, kept working as it should, and still needed more time than the grace period allowed. Kubernetes sent SIGKILL, and the message was gone.

You can raise the grace period, but you can’t make it unlimited. Node drains and spot interruptions have their own limits, and a very long grace period makes every deploy and scale-down slow. Some messages will always take longer than the time you have left.

Fix, part 2: hand unfinished messages back

So the second part: if a worker can’t finish a message in time, it puts the message back before it exits. We used a separate Redis list as a fallback queue:

import signal
import time

import redis

QUEUE, FALLBACK = "emails", "emails:fallback"
GRACE_SECONDS = 30      # match terminationGracePeriodSeconds
SAFETY_SECONDS = 5      # time reserved for handing the message back

r = redis.Redis(host="redis", port=6379)
state = {"stopping": False, "deadline": None}


def on_sigterm(signum, frame):
    # Only set flags here; the main loop does the work.
    state["stopping"] = True
    state["deadline"] = time.monotonic() + GRACE_SECONDS - SAFETY_SECONDS


def hand_back(msg):
    r.lpush(FALLBACK, msg)


def process(msg):
    """Run the processing steps; return False if we had to give the message back."""
    for step in steps_for(msg):              # parse, extract, enrich, ...
        if state["stopping"] and time.monotonic() >= state["deadline"]:
            hand_back(msg)                   # out of time: return it untouched
            return False
        step()
    write_result(msg)                        # idempotent: upsert keyed by message ID
    return True


signal.signal(signal.SIGTERM, on_sigterm)

while not state["stopping"]:
    # BRPOP checks keys in order, so the fallback queue is drained first
    item = r.brpop([FALLBACK, QUEUE], timeout=1)
    if item is None:
        continue
    _, msg = item
    if state["stopping"]:
        hand_back(msg)                       # popped just as SIGTERM arrived
        break
    if not process(msg):
        break

A few details matter here:

• Keep working until the deadline, not just until SIGTERM. Most messages finish well inside the grace period, so the worker tries to finish first. It hands back only what it can’t finish.
• Reserve time for the hand-back. The deadline is the grace period minus a safety margin, so LPUSH has time to complete before SIGKILL. Remember that the grace period starts counting when the pod is marked for deletion, so a preStop hook uses up part of it.
• Check between steps, not in the middle of one. The message is handed back in its original form, and nothing partial is written downstream.
• Make the write idempotent. A message that was handed back will be processed again from the start. An upsert keyed by message ID means that if a message is ever processed twice, the result is still correct.
• Drain the fallback first. BRPOP with several keys checks them in order, so handed-back messages are picked up before new ones.

What this doesn’t cover

A SIGTERM handler only runs if the process gets a SIGTERM. Some ways a pod dies skip it entirely:

• OOMKilled: the kernel kills the process with no warning.
• Node failure: the machine disappears.
• A bug that crashes the worker in the middle of a message.

In each of these, a popped message is still lost, because Redis no longer has a copy. If you need stronger guarantees, the queue itself has to keep a copy until the work is done:

• BLMOVE (formerly BRPOPLPUSH): atomically move the message to a per-worker “processing” list instead of popping it, remove it with LREM when the work is done, and have a reaper return messages from processing lists whose workers have died.
• Redis Streams with consumer groups: XREADGROUP to read, XACK when done, and XAUTOCLAIM to take over messages that a dead consumer never acknowledged.

Both give you at-least-once delivery, so idempotent writes become essential. The graceful exit is still worth having: it handles the most common case, routine scale-downs, cleanly and quickly.

Keep the reconciliation running

The most useful thing in this story is the check that found the problem. Comparing counts in and out for every batch, and raising an alert when they don’t match, caught a bug that no log line or error rate would have shown. It’s worth keeping as a permanent data quality check; I wrote about that kind of check in Automated Data Quality Checks.

Checklist

• Reconcile input and output by ID, per batch, and alert on any mismatch
• Know whether your queue removes a message on read (Redis BRPOP, many home-grown queues) or keeps it until acknowledged
• Workers handle SIGTERM: stop taking messages, finish in-flight work up to a deadline
• Unfinished messages go back to a fallback queue before the deadline
• Writes are idempotent, keyed by message ID
• terminationGracePeriodSeconds covers typical messages, with time left for the hand-back
• For crash safety, use BLMOVE plus a reaper, or Redis Streams with XACK

Takeaways

• A small, steady gap with no errors usually means a silent loss somewhere. Find which records are missing and when.
• With a queue that deletes on read, a pod killed mid-message loses that message. Each scale-down termination can cost you one.
• A graceful exit fixes most of it. A fallback queue handles messages that can’t finish within the grace period.
• Only a queue with acknowledgements protects you from crashes and OOM kills.

How the numbers were made

The experiment in this post ran on a laptop with Redis 7 in Docker and Python 3. It used 100,000 synthetic message IDs, 50 worker processes, and three scale-down events of 5 pods each. The grace period was 1 second, and slow messages took 1–2.5 seconds, so time was compressed about 30×. The production numbers (10-odd missing per 100,000) are from the original system.

More from PipePrompt

• Graceful Shutdown for ETL Workers on Kubernetes: Stop Losing Data on Scale-Down

  Oct 14, 2025 · kubernetes, etl, aws, python, go

  Handle SIGTERM in Go and Python ETL workers, size terminationGracePeriodSeconds, and avoid the pitfalls that still lose data when EKS scales down.

• Building an Automated Data Quality Framework: Rules, Trend Detection, and LLM False-Alarm Filtering

  Oct 28, 2025 · data quality, data engineering, llm, python

  How to build automated data quality checks: rule-based checkers, regression trend detection, ID pattern analysis, and LLM false-alarm filtering.

• Migrating Production Spark Pipelines off Databricks: What Broke, and How We Proved the Data Matched

  Oct 13, 2026 · databricks, spark, migration, iceberg, data engineering, ai

  Moving dozens of Spark pipelines off Databricks: silent SQL translation bugs, hidden state, stale inputs, hash-based parity checks, and how AI helped.