FastAPI’s BackgroundTasks runs in the same event loop as the request handler. That is fine for sending a 200ms confirmation email after a successful signup. It is the wrong tool for anything CPU-bound, anything that takes longer than the request, anything you need to retry, anything you need to monitor, or anything you need to survive a process restart. The “background” in BackgroundTasks is misleading. The task runs after the response is sent, in the same process, in the same event loop, and dies the moment the worker dies. It is a fire-and-forget helper, not a job queue.
This post is the ladder from BackgroundTasks to the right tool. The right tool depends on three things: how long the task takes, whether it needs to be retried, and whether it needs to survive a restart. Most teams that use BackgroundTasks for anything past a quick post-response hook end up rewriting it within a year. The rewrite is the time to know which tool you should have picked.
The interesting thing about FastAPI’s background task story is that the framework gives you the easy option in the box and expects you to reach for a queue when you need it. Most tutorials do not make that boundary clear, and most teams learn it the hard way. The hard way involves a memory leak, a 504 from a load balancer that killed a worker mid-task, and a postmortem.
Table of contents
- The direct answer
- What BackgroundTasks actually does
- The ladder: when to use what
- Level 1: BackgroundTasks for short post-response work
- Level 2: asyncio.create_task for in-process async work
- Level 3: ARQ for lightweight durable jobs
- Level 4: Celery for the heavy work
- Level 5: external services for the things that should not be in your process
- The opinion this post is built on
- FAQ
The direct answer
Pick the lowest level that does the job:
- Send a confirmation email after a signup? BackgroundTasks.
- Call a webhook after a payment? BackgroundTasks.
- Generate a PDF report that takes 30s? ARQ.
- Resize 1000 uploaded images? Celery + Redis.
- Run a multi-step ML pipeline? external service.The boundary that matters: if the task needs to outlive the request, or if it can fail in a way you need to retry, you need a queue. If it is a sub-second post-response hook, BackgroundTasks is the right answer.
The rest of the post is the reasoning behind the ladder and the failure mode at each level that pushes you up to the next.
What BackgroundTasks actually does
The mental model that the docs do not give you:
BackgroundTasks is a list of coroutines that the FastAPI request handler adds to. After the response is sent to the client, FastAPI iterates the list and awaits each coroutine, in the same event loop, in the same process. The coroutines have access to the request’s database session, the request’s context variables, and the request’s exception handler. They do not have access to the request’s response (which has already been sent).
What this means in practice:
- The task runs in the request worker’s event loop. If the task blocks the event loop for 5 seconds, no other request can be served by that worker for 5 seconds.
- The task shares the worker’s memory. A 500MB image-processing task consumes 500MB of worker RAM, reducing the number of workers the host can run.
- The task dies when the worker dies. A deploy that cycles the worker pool mid-task kills the task. There is no retry, no requeue, no log entry.
- The task shares the database session with the request. The session is closed by the time the task runs, which is the source of “the object is detached from the session” errors that show up in production.
- The task’s exceptions are swallowed. If the task raises, the client already has a 200 OK. The exception is logged if you set up the logging, and otherwise it disappears.
BackgroundTasks is a useful primitive. It is not a job queue. The naming and the docs are the source of most of the confusion.
The ladder: when to use what
There are five levels, and most production code lives on level 1 or 3. The trick is matching the level to the failure modes you can tolerate.
| Level | Tool | Lifetime | Survives restart? | Retries? | Cost |
|---|---|---|---|---|---|
| 1 | BackgroundTasks | Until response sent | No | No | Free |
| 2 | asyncio.create_task | Until worker dies | No | No | Free |
| 3 | ARQ / Dramatiq | Until processed | Yes | Yes | One Redis |
| 4 | Celery | Until processed | Yes | Yes | Redis + workers |
| 5 | External service | Forever | Yes | Yes | Paid |
The decision tree:
- Will the task take less than a second, run after the response, and not need a retry? Use
BackgroundTasks. - Will the task take seconds, run while the response is in flight, and not need a retry? Use
asyncio.create_task. - Will the task need to survive a worker restart, or need a retry? Use ARQ.
- Will the task be CPU-heavy, take minutes, or have complex scheduling? Use Celery.
- Will the task be best run by a service that already does this well? Use that service.
The mistake is to start at the top of the ladder for a task that could be at the bottom. Celery for a confirmation email is a four-hour setup for a 200ms task. The mistake the other way — BackgroundTasks for a 10-minute report — is a memory leak and a process restart every deploy.
Level 1: BackgroundTasks for short post-response work
The right use case:
from fastapi import BackgroundTasks, FastAPI
app = FastAPI()
def send_confirmation_email(email: str):
# 200ms, talks to SMTP, never blocks the event loop for long
...
@app.post("/signup")
async def signup(email: str, background_tasks: BackgroundTasks):
user = create_user(email)
background_tasks.add_task(send_confirmation_email, user.email)
return {"id": user.id}The user gets a 200, the email is sent a moment later in the same process, the worker is free for the next request. If the email fails, the client never knows. If the worker dies mid-email, the email is lost. Both are acceptable for a confirmation email that the user can request again.
The wrong use case looks the same but is not:
def generate_annual_report_pdf(user_id: int):
# 8 minutes of CPU work, 1.2GB of memory
...
@app.post("/reports")
async def generate_report(user_id: int, background_tasks: BackgroundTasks):
background_tasks.add_task(generate_annual_report_pdf, user_id)
return {"status": "started"}The user gets a 200 in 20ms. The worker spends 8 minutes blocked on a single task. The next 100 requests queue up behind the PDF generation. The worker OOMs because the report’s memory plus the request memory plus the FastAPI machinery exceeds the worker’s limit. The deploy platform restarts the worker. The report is lost.
That is the use case for level 3 or 4.
Level 2: asyncio.create_task for in-process async work
The right use case: a long-running async task that you want to fire and forget without going through the FastAPI dependency injection. Common examples: opening a WebSocket connection to an external service, running a long-polling loop, prefetching data into a cache.
import asyncio
async def keep_websocket_alive():
while True:
await ws.ping()
await asyncio.sleep(30)
@app.on_event("startup")
async def startup():
asyncio.create_task(keep_websocket_alive())The wrong use case: a CPU-bound task. asyncio.create_task runs in the same event loop, and a CPU-bound task blocks the event loop the same way BackgroundTasks does. For CPU-bound work, use a process pool (loop.run_in_executor) or a real worker.
The boundary between level 1 and level 2 is whether the task should be tied to a request lifetime. BackgroundTasks is tied to a request. asyncio.create_task is tied to the worker’s lifetime. Use BackgroundTasks for per-request work. Use asyncio.create_task for work that should outlive the request but should not outlive the worker.
Level 3: ARQ for lightweight durable jobs
ARQ is a lightweight async job queue built on Redis. It is the right answer for “I need a job that survives a worker restart and I do not want to run Celery.” The setup is small, the worker is async, and the Redis dependency is one most production stacks already have.
# worker.py
from arq.worker import create_worker
from arq import cron
async def process_upload(ctx, upload_id: int):
# do the work
...
class WorkerSettings:
functions = [process_upload]
cron_jobs = []# main.py
from arq import create_pool
from arq.connections import RedisSettings
async def enqueue_upload(upload_id: int):
redis = await create_pool(RedisSettings())
await redis.enqueue_job('process_upload', upload_id)The win is durability: the job is in Redis, the worker can restart, the job runs. The retry is configurable: max_tries, retry_delay, exponential backoff. The cost is operational: a Redis to run, a worker process to manage, a queue to monitor.
For most teams that outgrow BackgroundTasks but do not need Celery’s full feature set, ARQ is the right next step. It is also the right answer for periodic jobs that need to survive a deploy — see the RunxBuild platform’s cron job support for the same pattern with a different operational shape.
Level 4: Celery for the heavy work
Celery is the heavy option. It has been around for a decade, it has every feature a job queue could plausibly have, and it is the right answer for “I need a job that takes 30 minutes, needs to be retried 5 times with exponential backoff, and needs to be scheduled with a cron expression.”
The setup is heavier than ARQ. A Celery deployment is a Redis or RabbitMQ broker, one or more worker processes, a results backend (Redis, Postgres, or a custom store), and a beat process for scheduled jobs. The reward is the feature set: routing, chains, chords, groups, rate limits, ETA scheduling, and the operational tools to monitor all of it.
For most teams the choice between ARQ and Celery is operational philosophy, not feature. ARQ is small and async-native. Celery is the default and the most well-trodden. If the team has Celery experience, use Celery. If the team is starting fresh and the job is straightforward, use ARQ. If the job is genuinely complex, use Celery.
Level 5: external services for the things that should not be in your process
The right answer for “send a transactional email” is not a job queue, it is an email service. The right answer for “transcribe an audio file” is not a job queue, it is a transcription service. The right answer for “render a 10-minute video” is not a job queue, it is a video service.
The pattern: when the work is a commodity that someone else does well, use them. Your job queue is for the work that is unique to your product. The confirmation email is the same email everyone sends; the annual report is a thing only you can generate. The first is a level-5 problem. The second is a level-4 problem.
For a sanity check on the deploy cost of running your own workers, the hosting cost calculator gives a real number to compare against. Workers are not free, and the difference between “the work runs in my process” and “the work runs in a managed worker” is the difference between “I am buying a CPU hour” and “I am buying a CPU hour plus the operational knowledge to use it well.”
The opinion this post is built on
BackgroundTasks is a 90% solution. For the 90% of tasks that take less than a second and run after the response, it is the right answer, the simplest answer, and the one that does not require a Redis. For the 10% of tasks that take longer, need retries, or need to survive a restart, it is the wrong answer, and the wrong answer is the one that costs the team a week of debugging the first time the worker restarts mid-task.
The right pattern is to start at level 1, know where the boundary is, and have level 3 (ARQ) ready when the boundary is crossed. Celery is the next step for the teams that need its feature set, but most teams that think they need Celery actually need ARQ. The teams that need Celery know who they are: they have a beat process, they have rate limits per queue, they have a results dashboard, and they have an SRE who owns the broker.
The level you choose also tells you something about the platform. A platform that runs FastAPI on a worker but does not have a managed queue is a platform that pushes the level-3-and-above work back onto the developer. A platform with a managed queue, a managed worker, and a managed scheduler is a platform where the level choice is a code decision, not an operational decision. The RunxBuild platform is built around the second model.
Pick the lowest level that does the job. Promote the task to the next level when the failure mode you can no longer tolerate demands it. Do not start at Celery because someone on the team heard it was “the right way to do background jobs.” Most of the time, it is not.
FAQ
When should I use FastAPI’s BackgroundTasks?
For tasks that take less than a second, run after the response is sent, do not need a retry, and do not need to survive a worker restart. Confirmation emails, single webhook calls, and minor post-write side effects are the canonical cases. Anything past that is a job queue problem.
What is the difference between BackgroundTasks and asyncio.create_task?
BackgroundTasks is tied to a request lifetime and runs in the FastAPI worker after the response is sent. asyncio.create_task is tied to the worker’s lifetime and runs as a background coroutine in the same event loop. Use BackgroundTasks for per-request post-response work. Use asyncio.create_task for work that should outlive the request but not the worker.
When should I use Celery or ARQ instead?
When the task needs to survive a worker restart, needs a retry, needs to be scheduled, or needs to be monitored. The boundary is durability: if the task can be lost without consequence, BackgroundTasks is fine. If the task matters, use a queue. ARQ is the lighter option; Celery is the heavier option with more features.
Can BackgroundTasks run on a different process?
No. BackgroundTasks runs in the same process and the same event loop as the request. If the worker dies, the task is lost. The only way to make a background task survive a worker restart is to use a queue — Redis-backed (ARQ, Dramatiq, RQ), broker-backed (Celery), or service-backed (SQS, Cloud Tasks, etc.).
Why does my FastAPI background task raise “object is detached from the session”?
Because the database session is closed when the request handler returns, and the background task runs after. The fix is to open a new session inside the task. For SQLAlchemy:
def send_email(user_id: int):
with Session() as session:
user = session.get(User, user_id)
...For most teams the cleaner fix is to use a queue (ARQ or Celery) and let the worker own its own session lifecycle.
Can I use BackgroundTasks with async def functions?
Yes. The task can be a regular function or an async coroutine. FastAPI awaits async tasks and runs sync tasks in a thread pool (via run_in_threadpool). The thread pool is the right answer for blocking I/O, but it is still bounded, and a sync task that blocks for 30 seconds is still a problem.