Ephemeral Storage: What It Is and What It Is Not

Ephemeral storage is the writable filesystem that comes with a container instance, a Kubernetes pod, or a serverless function. It exists for the lifetime of that instance. When the instance is replaced, scaled down, or restarted, the data is gone. That is the short version. The longer version is that ephemeral storage is not a bug. It is a design choice. Treating it as durable storage is a bug, and it is one of the most common production mistakes in modern cloud architecture.

The container model assumes the application is stateless. The filesystem is a cache, not a record. The data that has to survive a restart — uploaded files, database tables, message queues, build artifacts — lives somewhere else: a managed database, an object store, a network volume. The ephemeral layer is fast, cheap, and tied to the lifecycle of the instance. The persistent layer is slower, more expensive, and tied to the lifecycle of the data.

This post is a working engineer’s take on what ephemeral storage is, what it is for, where the line is between ephemeral and persistent, and how to design an app that does not lose data when the container restarts. It assumes you are deploying containers (Docker, Kubernetes, or a PaaS that uses them under the hood), that you have hit at least one “where did my data go” moment, and that you want the mental model that prevents the next one.

Table of contents

• The short version
• What “ephemeral” actually means
• Where ephemeral storage comes from
• The right uses for ephemeral storage
• The wrong uses (and the data they lose)
• How to tell which layer your data belongs on
• The persistent volume options
• The cost trap nobody warns you about
• The pattern that prevents the 2 a.m. page
• FAQ

The short version

Ephemeral storage is the writable layer inside a container or a pod. It is fast, it is local to the instance, and it disappears the moment the instance is replaced. Use it for things that can be rebuilt: caches, intermediate files, in-flight computations, build artifacts, logs that have already been shipped. Do not use it for the only copy of anything.

If your app needs durable storage, the data has to live outside the container. A managed database, an object store, a network volume, or a backup target — the right choice depends on the shape of the data, but the principle is the same: durable data lives on a service that survives the container.

The mistake this post is trying to prevent is the one that erases an upload directory, a SQLite database, a queued message, or a session store on a routine deploy. Every experienced engineer has been bitten by this at least once. The fix is not a smarter container. The fix is knowing which layer the data belongs on before the first line of code is written.

What “ephemeral” actually means

The word ephemeral in cloud architecture means “tied to the lifetime of the compute instance, not the lifetime of the data.” When the instance is replaced, the data is gone. When the instance is restarted, the data is gone. When the instance is scaled down, the data is gone. When the instance crashes, the data is gone.

Concretely:

• In a Docker container, the writable layer lives in the container’s overlay filesystem. When the container exits, the writable layer is removed.
• In Kubernetes, the pod’s writable filesystem is an emptyDir volume by default. When the pod is rescheduled (because the node failed, the deployment rolled, or the autoscaler scaled it down), the data is gone.
• In a serverless function, the /tmp directory is the writable filesystem. It is wiped between invocations and may be wiped mid-invocation on a cold start.
• In a PaaS like RunxBuild, each service runs in a container. The container’s writable layer is ephemeral. Anything that has to survive a deploy, a restart, or a scale event has to live in a managed database, an object store, or a persistent volume.

The principle is the same across all of these. The instance is a temporary execution environment. The data is not.

Where ephemeral storage comes from

Every compute instance has it. The layer is part of the operating system’s process model. The interesting question is how much of it is available, how fast it is, and what happens to it on failure.

Docker containers get a writable layer on top of the image. The layer is backed by the host’s storage driver (overlay2, btrfs, zfs, depending on the host). The size is configurable. The performance is host-disk-bound. On container exit, the layer is discarded.

Kubernetes pods get an emptyDir volume by default, mounted at the pod’s working directory. The volume is backed by the node’s local disk (or RAM, if configured). On pod deletion, the volume is removed. If the pod is rescheduled to a different node, the data is gone even if the pod “looks” the same.

AWS Lambda and similar get 512 MB of /tmp per function. The directory is wiped on a cold start and is not guaranteed to survive an execution. The 512 MB is also billed only for the duration of the function, which is a useful pricing model for transient work.

PaaS services like RunxBuild behave like Kubernetes. The service runs in a container, the writable layer is local to the instance, and the data is gone when the instance is replaced. The platform exposes a managed database and an object store as the durable layers.

The shape of the ephemeral layer is consistent across these. What varies is the durability story: how do you get data out of the ephemeral layer and into the persistent layer so it survives the instance?

The right uses for ephemeral storage

Ephemeral storage is not a bug. It is the right place for a specific set of things.

Build artifacts and caches. When you build a Node app, the node_modules directory is a build artifact. It is reconstructed on the next build. When you build a Python app, the __pycache__ and .venv directories are the same. These live in the ephemeral layer, because if the layer is wiped, the next build recreates them.

Intermediate computation results. A data pipeline that processes a file, generates intermediate results, and writes the final result to a database can use ephemeral storage for the intermediate files. The pipeline is restarted on failure, the intermediates are regenerated, and the final result is durable.

Hot caches. A cache that is faster than the source but optional (the source can regenerate it) lives in the ephemeral layer. Examples: a parsed configuration file, a compiled regex set, an in-memory data structure backed by a file. On restart, the cache is empty; the next request rebuilds it.

Log buffers. Logs that have already been shipped to a centralized log service do not need to live in the container. The container can write to stdout (which the platform captures and ships), and the local log buffer is wiped on restart. The logs are not lost because they were already shipped.

Temporary files during a request. Image resizing, file conversion, CSV parsing, PDF generation — these are operations that produce a temporary file. The file lives in /tmp for the duration of the request, gets sent to the client or to durable storage, and is deleted. The next request creates a new temporary file.

The pattern is the same in all of these: the data can be reconstructed from somewhere else, or the data is not needed after the operation completes. The ephemeral layer is the right place for it.

The wrong uses (and the data they lose)

Every experienced engineer has been bitten by at least one of these. The patterns repeat because the data is “small” and “not worth a database” — until the restart happens.

The SQLite database file. A small app stores everything in a SQLite file at /data/app.db. The database is “just a file,” and the file is on the container’s writable layer. The container is restarted. The database is gone. The user data is gone. The fix is to use a managed database, or to mount a persistent volume, or to back up the SQLite file to object storage on every commit. None of those are optional.

The upload directory. An app accepts file uploads and writes them to /uploads. The uploads are served back to users. The container is scaled down. The uploads are gone. The fix is an object store (S3, Cloudflare R2, Backblaze B2) with the uploads going directly to the bucket.

The session store. A Node app keeps user sessions in memory or in a file. The container is replaced. Every user is logged out. The fix is a real session store (Redis, Postgres, a database), with sessions on the persistent layer.

The build cache that is not a cache. A Docker image has a COPY step that copies a generated file into the image. The file is not regenerated; it is the output of a previous step that ran on the developer’s laptop. The image is rebuilt on the server. The file is missing. The build fails. The fix is to regenerate the file in the build step or to ship it as a build artifact that the build can pull.

The queued message. A background worker reads jobs from a file or a local queue. The container is restarted. The in-flight jobs are lost. The fix is a real queue (Redis, RabbitMQ, SQS, Postgres-backed) on the persistent layer.

The user-generated content. A comments system, a profile picture, a blog post draft, a generated PDF, a screenshot, a model checkpoint. Anything a user produces is durable data. The container is not the right place for it. The right place is the database or the object store.

The pattern in every case is the same: a small amount of data that “feels like a file” and is small enough to be tempting to leave on the container’s filesystem. The container does not care how small the data is. The data is gone on the next restart.

How to tell which layer your data belongs on

A useful decision tree, sharpened by the production mistakes the industry has already made.

Can the data be reconstructed from somewhere else? If yes, ephemeral is fine. A cache that can be rebuilt from the database is ephemeral. A build artifact that is regenerated by the build is ephemeral. A log buffer that is shipped to a log service is ephemeral.

Does the data have to survive a restart, a redeploy, a scale event, or a node failure? If yes, persistent. Uploads, user data, configuration that took time to set up, queued messages, model checkpoints, generated PDFs — all persistent.

Is the data small and low-value enough that “we’ll just regenerate it” is a real answer? If yes, ephemeral. Session tokens that are short-lived, idempotency keys that expire, one-shot computation results — ephemeral.

Is the data small and high-value enough that losing it would matter? If yes, persistent. Even a 10 KB SQLite file with three users’ worth of data is persistent. The size does not matter. The value does.

Does the data have to be shared with other instances? If yes, persistent. A file on the container’s filesystem is visible only to that container. A second container cannot read it. The only way to share state is to put it on a shared layer: a database, an object store, a network volume.

The last point is the one teams miss most often. A team runs a single container, writes a file to the container’s filesystem, and the file is there. The team scales to two containers behind a load balancer. The user uploads a file, the file lands on container A, the next request lands on container B, and the file is missing. The fix is not “make the file shareable.” The fix is to put the file on a shared layer from the start.

The persistent volume options

For data that has to survive the container, the options are roughly the same across providers, with different names and price points.

Managed databases. Postgres, MySQL, MongoDB, Redis. The data lives in a service the platform operates. The application connects over the network. The service handles replication, backups, and failover. This is the right answer for any structured data that fits a database.

Object stores. S3, Cloudflare R2, Backblaze B2, Google Cloud Storage, Azure Blob. The data is a key and a blob. The service handles durability (typically 11 nines), replication, and access control. This is the right answer for files, images, videos, model checkpoints, and any unstructured blob.

Network file systems. NFS, EFS, Azure Files, Cloudflare R2 with the right adapter. The data is a file on a network mount. The container mounts the volume, reads and writes the file, and the data persists across container restarts. The right answer for workloads that genuinely need a POSIX filesystem.

Block storage. EBS, persistent disks, Cinder volumes. The data is a block device. The container mounts the volume, formats it, and uses it as a filesystem. The right answer for stateful workloads that need block-level access (databases, in particular).

Backup targets. S3 Glacier, Backblaze B2 with versioning, tape. The right answer for data that has to survive a regional disaster.

The choice depends on the data shape and the access pattern. The principle is the same: durable data lives on a service that survives the container.

On a PaaS like RunxBuild, the managed database and the object store are exposed as first-class resources. The application code does not know or care that there is a container under the hood. The data lives on the platform’s persistent layer. The container is just the runtime that reads and writes the data.

The cost trap nobody warns you about

Ephemeral storage is cheap because it is local to the instance. Persistent storage is more expensive because it is replicated, backed up, and available across instances. The cost difference is real and worth understanding.

A 100 GB ephemeral disk is essentially free. It is part of the instance. The instance is already paid for. The disk is “free” in the sense that it does not add a line item to the bill.

A 100 GB managed database is a line item. Postgres on a managed service starts at a few dollars a month for a small instance and grows with the storage, the IOPS, and the backup retention. The same is true of a managed Redis, a managed MongoDB, an object store with egress fees, and a network file system.

A 1 TB persistent volume is significant. Network volumes and block storage scale linearly with capacity. 1 TB of EBS in us-east-1 is around $100/month for the storage, plus the IOPS charges, plus the snapshot charges. The same 1 TB of ephemeral storage is part of an instance that costs around the same.

The trap is the opposite of what most teams expect. Most teams assume persistent storage is cheap, because “it’s just files.” Persistent storage is the line item. Ephemeral storage is the part that does not show up on the bill.

A useful exercise: run the RunxBuild hosting calculator for your workload with the database size you actually need. The number is usually larger than the team expects, because the database size includes the working set, the indexes, the backups, and the replication. A 50 GB database is not a 50 GB line item. It is a 50 GB database, 50 GB of backups, 50 GB of replicas, and the IOPS to read and write it. The bill is the sum.

The pattern that prevents the 2 a.m. page

The pattern that prevents the data-loss outage is the one that decides the storage layer at the design stage, not the production stage.

Step 1: inventory the data the app will create or accept. User accounts, posts, comments, uploaded files, generated PDFs, queued messages, session state, configuration that took time to set up. List every category.

Step 2: classify each category as ephemeral or persistent. A comment is persistent. A queued message is persistent. An upload is persistent. A session token is persistent (or short-lived ephemeral, depending on the design). A cache that can be rebuilt is ephemeral. A log buffer that is shipped elsewhere is ephemeral.

Step 3: pick the layer for each persistent category. A comment goes in the database. An upload goes in the object store. A generated PDF goes in the object store. A session token goes in Redis or the database.

Step 4: build the deploy around the layers. The application code knows about the database and the object store. The application code does not write to the container’s filesystem except for caches, build artifacts, and intermediate computation. The container is stateless from the platform’s perspective.

Step 5: add the health check and the observability to the persistent layer, not the container. A health check that confirms the database connection at boot catches the “I deployed and the database is not reachable” failure mode before the first request lands.

Step 6: test the restart. The single most useful test in any containerized app is “what happens to the data when the container restarts?” Run it. If the answer is “we lose state,” the data is on the wrong layer.

The pattern is not new. It is the same pattern that has been in the cloud architecture playbook since the beginning. The reason to write it down again is that the same mistake keeps happening, on every platform, in every team, because the ephemeral layer is so easy to use that the line between ephemeral and persistent is invisible until it bites.

FAQ

What is ephemeral storage?

Ephemeral storage is the writable filesystem that comes with a container instance, a Kubernetes pod, or a serverless function. It exists for the lifetime of the instance. When the instance is replaced, the data is gone.

Is ephemeral storage the same as a container’s writable layer?

Yes. In Docker, the writable layer on top of the image. In Kubernetes, the emptyDir volume. In a serverless function, the /tmp directory. In all cases, the data is local to the instance and disappears when the instance is replaced.

What is the difference between ephemeral and persistent storage?

Ephemeral storage is tied to the lifetime of the compute instance. Persistent storage is tied to the lifetime of the data. A managed database, an object store, and a network volume are persistent. A container’s filesystem is ephemeral.

Can I make ephemeral storage persistent by using a Docker volume?

A Docker volume that is mounted from the host is durable across container restarts, but the durability is tied to the host. If the host is replaced (because the node failed, the VM was re-created, or the autoscaler moved the workload), the volume is gone. For real durability, use a managed database, an object store, or a network volume that is not tied to a specific host.

What is the right way to handle file uploads in a container?

Write the file directly to an object store (S3, Cloudflare R2, Backblaze B2). The application does not need to touch the container’s filesystem. The file is durable from the moment it is uploaded. The container’s writable layer is not involved.

How do I back up a database in a Kubernetes pod?

Use the database’s native backup tool (pg_dump for Postgres, mongodump for MongoDB, redis-cli for Redis) and ship the backup to an object store. The backup does not need to live in the pod. The backup is a separate, durable artifact.

Is a Docker volume ephemeral?

It depends. A Docker volume that is mounted from the host is durable across container restarts, but not across host failures. A Docker volume that is backed by a network file system (NFS, EFS, Cloudflare R2 with the right driver) is durable across host failures. The right answer for any data that matters is the network-backed volume or the managed database.

What should I store in ephemeral storage?

Build artifacts, intermediate computation results, hot caches that can be rebuilt, log buffers that have already been shipped, and temporary files during a request. Anything that can be reconstructed from a source of truth, or anything that is not needed after the operation completes.

How does RunxBuild handle ephemeral vs persistent storage?

Each RunxBuild service runs in a container with ephemeral local storage. Durable data lives in the managed database, the managed object store, or a persistent volume. The application code does not need to manage the boundary. The data layer is configured in the platform.