Email, webhooks, and file processing fail for dull reasons. A job is ready for production when running it twice does not do the work twice.
Store the intent
Write a row for the thing you mean to do — send this invoice, fetch this callback — and let the worker claim it. If the process dies after the send and before the acknowledgement, the row is still there and the next run can see that the provider already accepted the first attempt.
Backoff is part of the design
A provider that is down does not need a hundred attempts in a minute. Space the retries, cap them, and surface the last error on the request a person can open. Silent infinite retries fill the queue and hide the outage.
Do not retry what is not safe
Creating a charge, sending an SMS, and posting to a ledger are not safe to repeat unless you carry an idempotency key. Reading a status is safe. The worker should know which is which without reading the function name.
A job that can run twice
Assume the worker will die after the side effect and before it records success. The next attempt must notice the work is already done. Store an idempotency key on the job, and make the handler check it before it sends the email, charges the invoice, or writes the row.
Python · jobs/handlers.py
def send_invoice(job):
invoice = Invoice.objects.get(pk=job.payload["invoice_id"])
if invoice.emailed_at:
return
deliver_invoice_email(invoice)
invoice.emailed_at = timezone.now()
invoice.save(update_fields=["emailed_at"])What to do this week
- Give each job a row and a status.
- Cap retries and show the last error.
- Mark which jobs are safe to run twice.