Deploying queue workers without losing a single message

Every team that runs background workers on a message queue eventually has to ship new code while messages are in flight. Most teams accept a few dropped messages or go straight to blue-green infrastructure. There's a middle path that's embarrassingly simple.

At some point you're going to have a background worker that pulls messages off a queue, does something expensive with each one, and writes the result somewhere. And at some point you're going to need to ship a new version of that worker while there are messages sitting in the queue and possibly one being processed right now.

This is the moment most teams pick a lane. You either accept that a message or two might get lost during deployment and build retry logic to compensate, or you set up blue-green deployment slots with traffic shifting and health probes and suddenly your deployment infrastructure is more complex than the worker itself.

I want to walk through a third option that I've been running in production for a while now. It's not clever. It doesn't require any additional infrastructure. It works by taking advantage of something the queue broker already gives you for free: the ability to pause delivery.

The setup

The system is an Azure Function triggered by Azure Service Bus. A request arrives as a JSON message on a queue. The function picks it up, runs it through a processing pipeline that can take anywhere from a few seconds to several minutes — database lookups, external API calls, ML inference, the usual chain of async operations — and writes the result to a database. The function is containerized: Docker image gets built in CI, pushed to a container registry, and the Function App points at a specific image tag.

The host configuration enforces sequential processing:

{
  "extensions": {
    "serviceBus": {
      "maxConcurrentCalls": 1,
      "maxConcurrentSessions": 1,
      "transportType": "amqpWebSockets"
    }
  },
  "functionTimeout": "01:00:00"
}

One message at a time. One session at a time. Up to one hour per message. That last number matters — this requires a Premium or Dedicated plan, since Consumption maxes out at ten minutes — and the processing pipeline involves multiple external service calls that can't just be killed mid-flight without leaving orphaned state.

So the deployment problem is: how do you swap the container image without interrupting whatever's happening right now, and without losing the messages that are waiting in line?

The trick: pause the queue, not the worker

Service Bus queues have a status field. You can set it to Active (normal operation), Disabled (nothing in, nothing out), or ReceiveDisabled — which is the interesting one. ReceiveDisabled means the queue keeps accepting new messages from producers, but stops delivering them to consumers.

That distinction is the entire deployment strategy. Here's what the pipeline does:

Step 1: Set the queue to ReceiveDisabled.

current_status=$(az servicebus queue show \
  --resource-group "$RESOURCE_GROUP" \
  --namespace-name "$SB_NAMESPACE" \
  --name "$QUEUE_NAME" \
  --query status -o tsv)

if [ "$current_status" = "ReceiveDisabled" ]; then
  echo "Queue is already paused. No action needed."
else
  az servicebus queue update \
    --resource-group "$RESOURCE_GROUP" \
    --namespace-name "$SB_NAMESPACE" \
    --name "$QUEUE_NAME" \
    --status ReceiveDisabled

  sleep 2

  # Verify it actually took effect
  new_status=$(az servicebus queue show \
    --resource-group "$RESOURCE_GROUP" \
    --namespace-name "$SB_NAMESPACE" \
    --name "$QUEUE_NAME" \
    --query status -o tsv)

  if [ "$new_status" != "ReceiveDisabled" ]; then
    echo "ERROR: Failed to pause queue" >&2
    exit 1
  fi
fi

At this point, three things are true simultaneously:

  • Upstream systems keep sending messages. They don't know anything changed. Their publishes succeed normally.
  • Messages accumulate in the queue. No delivery attempts, no lock expiry, no dead-lettering, no TTL surprises.
  • If the function was already processing a message, it finishes undisturbed. Nothing interrupts it.

That last point is subtle and important. You're not stopping the Function App. You're not killing the container. You're just telling Service Bus to stop handing out new messages. The worker keeps running, finishes whatever it's doing, and then sits idle because nothing new is coming in.

Step 2: Swap the container image.

CONTAINER_IMAGE="${ACR_NAME}.azurecr.io/my-app/my-worker:${IMAGE_TAG}"

az functionapp config set \
  --name "$FUNCTION_APP" \
  --resource-group "$RESOURCE_GROUP" \
  --linux-fx-version "DOCKER|${CONTAINER_IMAGE}"

Azure pulls the new image, restarts the function app, and cold-starts the new container. This typically takes 10–30 seconds. During this window, it doesn't matter that the app is restarting — the queue is paused, so there are no messages to miss.

Step 3: Verify the deployment landed.

az functionapp show \
  --name "$FUNCTION_APP" \
  --resource-group "$RESOURCE_GROUP" \
  --query "{name:name,state:state,lastModifiedTimeUtc:lastModifiedTimeUtc,linuxFxVersion:siteConfig.linuxFxVersion}" \
  -o table

Quick sanity check: is the app running? Does the linuxFxVersion show the tag you just deployed?

Step 4: Resume the queue.

This is the part that surprised people when they first saw it — the deploy pipeline doesn't do this step. The queue stays paused until either a human explicitly resumes it or an automated pipeline flips it back. The reason is practical: we wanted an operator to look at the deployment, confirm the new version started cleanly, and then open the floodgates. It's a deliberate pause, not an accident.

When you're ready:

az servicebus queue update \
  --resource-group "$RESOURCE_GROUP" \
  --namespace-name "$SB_NAMESPACE" \
  --name "$QUEUE_NAME" \
  --status Active

Every message that accumulated during the pause immediately starts flowing. The new container picks them up one by one. No messages were lost. No messages were double-processed. The upstream systems never noticed.

Why ReceiveDisabled and not just stopping the Function App

I tried the other approach first. az functionapp stop and az functionapp start around the deployment. Three problems showed up immediately.

First, stopping the function app kills in-flight executions. If a message was 40 minutes into processing, that work is gone. The message lock gets abandoned, the delivery count increments, and the message reappears on the queue — but the database might already have partial writes from the interrupted run. Now you need idempotency logic to handle the half-finished state, which is the kind of thing that's simple in theory and miserable in practice.

Second, when the function app is stopped, Service Bus still tries to deliver. The function host isn't listening, so peek-locks expire, delivery counts tick up, and messages can land in the dead-letter queue if the restarts push them past MaxDeliveryCount. Now your deployment procedure includes "check the DLQ and replay anything that ended up there," which is not a sentence you want in a runbook.

Third, stopping and starting the app means the cold start happens while messages are being delivered. The function runtime isn't ready yet, delivery attempts fail, you get transient errors in the logs, and it's hard to tell whether those errors are from the deployment or from an actual bug.

ReceiveDisabled sidesteps all three. The app never stops, in-flight work finishes, and the queue is a perfectly fine buffer while the container restarts.

The deliberate manual resume

Not having an automatic resume in the deploy pipeline is a design choice I want to defend for a second, because the first instinct is always to automate the full cycle.

A failed deployment with a paused queue is a safe state. Nothing is being lost — messages accumulate. Nothing is being processed incorrectly — the broken version isn't eating from the queue. You have time to investigate, roll back if needed, and resume when you're confident.

A failed deployment with an auto-resumed queue means the new (possibly broken) code immediately starts consuming messages. If the bug is subtle — wrong output format, corrupted data, silent logic error — you might not catch it until a lot of damage is done.

The manual resume is a circuit breaker. It forces a human checkpoint between "code is deployed" and "code is processing real work."

If that's too manual for your taste, you can absolutely automate it with a health check:

for i in {1..30}; do
  if curl -sf "https://${FUNCTION_APP}.azurewebsites.net/api/health" > /dev/null; then
    echo "Function is healthy. Resuming queue."
    az servicebus queue update \
      --resource-group "$RESOURCE_GROUP" \
      --namespace-name "$SB_NAMESPACE" \
      --name "$QUEUE_NAME" \
      --status Active
    break
  fi
  sleep 10
done

Just know what you're trading. We've had exactly one incident where someone deployed on a Friday afternoon, forgot to resume, and Monday morning started with a weekend's worth of queued messages. The monitoring caught it — we have an alert on queue depth — but it's the kind of thing that happens once and then you add a calendar reminder. Still better than the alternative.

Reusing the pattern for batch processing windows

The same ReceiveDisabled trick turned out to be useful beyond deployments. We have nightly batch jobs that pre-compute data in a warehouse. Running those while the worker is also processing live messages causes consistency issues — the batch job updates tables that running pipelines are reading from.

So we built an automated maintenance pipeline on a cron schedule:

  • 23:55 UTC: Queue → ReceiveDisabled
  • 00:00–02:00 UTC: Batch processing runs against clean, stable data
  • 02:10 UTC: Queue → Active, accumulated messages start flowing

No coordination between the batch job and the worker beyond the queue status. No distributed locks. No "check if the batch is running" flags. The queue is just a buffer that you can pause when you need to.

The pipeline also supports manual triggers with parameters — choose pause, resume, or auto-detect based on time, and select which environments to target. Useful for emergency maintenance.

Wrapping it into a CI/CD template

In Azure Pipelines, this becomes a reusable template. The deploy pipeline calls it with environment-specific parameters:

# templates/deploy-function.yml
parameters:
  - name: environment
    type: string
  - name: functionAppName
    type: string
  - name: acrName
    type: string
  - name: imageTag
    type: string
  - name: serviceBusNamespace
    type: string
  - name: queueName
    type: string
    default: 'my-queue'

jobs:
  - deployment: DeployWorker
    environment: '${{ parameters.environment }}'
    strategy:
      runOnce:
        deploy:
          steps:
            - task: AzureCLI@2
              displayName: 'Pause Service Bus queue'
              inputs:
                azureSubscription: $(azureServiceConnection)
                scriptType: 'bash'
                scriptLocation: 'inlineScript'
                inlineScript: |
                  # ... pause logic from above

            - task: AzureCLI@2
              displayName: 'Update container and restart'
              inputs:
                azureSubscription: $(azureServiceConnection)
                scriptType: 'bash'
                scriptLocation: 'inlineScript'
                inlineScript: |
                  # ... swap + verify logic from above

The calling pipeline validates inputs before it reaches this template — it blocks latest tags (you must deploy a specific version), verifies the image actually exists in the container registry, and ensures at least one component is selected for deployment. By the time the pause-swap-verify template runs, you know the image is real and tagged.

What this costs you

The downside is throughput during the pause window. If your queue is processing 100 messages per second and you pause for 60 seconds, you've got 6,000 messages waiting when you resume. Your worker needs to be able to drain that backlog in a reasonable time.

In our case it's not an issue. Traffic is steady but not bursty, a typical deployment pause accumulates a handful of messages, and sequential processing drains them in minutes.

The other cost is conceptual. The first time someone on the team sees a queue in ReceiveDisabled state and doesn't know why, there's a moment of panic. Good documentation and monitoring on queue depth solve this, but it's something you need to actively communicate.

When this doesn't work

If your worker processes messages in parallel at high concurrency, pausing the queue still works, but the drain time after pause is harder to predict. You'd need to wait until all concurrent handlers finish before it's safe to swap.

If you have multiple queues feeding the same function app, you need to pause all of them. Miss one, and you're back to the "messages arriving while the container restarts" problem.

If your system can't tolerate any processing delay — even the minute or two of pause time — you need a real blue-green deployment with traffic shifting. The queue-pause approach is zero message loss, but it's not zero latency impact.

And if you're using a broker that doesn't have a broker-level equivalent of ReceiveDisabled — Kafka, for instance, only has a client-side pause() that stops fetching but requires the consumer to keep heartbeating — you'd need a different approach entirely, probably consumer group management or partition reassignment.

The point

The thing I keep coming back to is how simple this is. You don't need a deployment framework. You don't need Kubernetes rolling updates or sidecar proxies or traffic management policies. The queue is already a buffer. ReceiveDisabled just makes the buffer explicit during the window where you need it.

I spent weeks early on researching proper blue-green deployments for Azure Functions. Deployment slots, warm-up triggers, connection draining configurations. All of it valid, all of it significantly more infrastructure to maintain. Then someone on the team pointed out that we could just pause the queue, and the entire problem collapsed into two CLI commands.

Sometimes the right engineering decision is the boring one that uses what's already there.