On-premises S3 storage platforms: MinIO (single-binary, S3-compatible, fastest to deploy), Ceph (distributed, scales to petabytes), SeaweedFS (lightweight, optimized for small files), and Rook (Kubernetes operator for Ceph). The right pick: MinIO for a single server or small cluster, Ceph for large-scale distributed storage, SeaweedFS for small-file workloads.
Table of contents
- The 3 platforms at a glance
- MinIO: the quick start
- Ceph: the distributed option
- SeaweedFS: the small-file option
- Picking the right one
- The trade-offs
- How this fits the rest of the stack
- FAQ
The 3 platforms at a glance
MinIO - single-binary S3-compatible object storage. Drop the binary, run it, point your S3 client at it. The team that wants S3 on-prem in 5 minutes uses MinIO.
Ceph - distributed object, block, and file storage. Scales from a few TB to many PB. More complex to deploy, but the right answer for large-scale storage. The team that has petabytes of data and a team to operate it uses Ceph.
SeaweedFS - lightweight, optimized for small files. The team that has millions of small files (images, logs, small objects) uses SeaweedFS.
Rook - a Kubernetes operator for Ceph. The team that runs Ceph on Kubernetes uses Rook to manage the cluster.
MinIO: the quick start
Install:
curl -O https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /var/minio --console-address ":9001"
Single-node MinIO is up. The dashboard is on port 9001, the S3 API is on port 9000. Default credentials are minioadmin:minioadmin - change them immediately.
For a cluster: 4+ nodes with erasure coding. MinIO distributes data across nodes, tolerates node failures.
Client usage: the same S3 SDKs work. Just point at the MinIO endpoint:
import boto3
s3 = boto3.client('s3',
endpoint_url='http://minio.example.com:9000',
aws_access_key_id='minioadmin',
aws_secret_access_key='minioadmin')
The team that uses MinIO has full S3 compatibility - same APIs, same SDKs, same tools.
Ceph: the distributed option
Ceph is harder. Components:
MONs (monitors) - maintain the cluster map. 3+ for quorum.
OSDs (object storage daemons) - one per disk. Each disk is an OSD.
MDSs (metadata servers) - for the CephFS file system. Optional.
MGRs (managers) - for cluster statistics and dashboards. 2+ for HA.
Deploy with cephadm (the modern way):
cephadm bootstrap --mon <ip> --cluster-network <cidr>
Then add OSDs: ceph orch daemon add osd <host>:<device>. The team that uses cephadm has a managed cluster, not a hand-rolled one.
S3 access: install RGW (RADOS Gateway). It exposes an S3-compatible API on top of the Ceph storage. The team that uses RGW has S3 access to the Ceph cluster.
For Kubernetes: Rook. kubectl apply -f rook-ceph-operator.yaml, then a CephCluster CRD. The team that has Kubernetes uses Rook to manage Ceph.
SeaweedFS: the small-file option
Install:
curl -L -o seaweedfs.tar.gz https://github.com/seaweedfs/seaweedfs/releases/latest/download/linux_amd64.tar.gz
tar xzf seaweedfs.tar.gz
./weed server -dir=/var/seaweed -s3
The default config gives you a master + volume + S3 gateway. SeaweedFS is optimized for small files - it can store billions of small objects efficiently.
The team that has a workload with millions of small files (e.g., a chat app with avatars, a CMS with thumbnails, an ML training set with millions of small images) uses SeaweedFS.
Picking the right one
Single server, S3 on-prem, fastest deploy: MinIO. The team that has a single server and wants S3 has it in 5 minutes.
Large-scale distributed storage, multi-PB: Ceph. The team that has the ops capacity and the data volume uses Ceph.
Kubernetes-native, large-scale: Rook (Ceph on Kubernetes). The team that has Kubernetes and wants Ceph on it uses Rook.
Small files (millions of objects under 1 MB): SeaweedFS. The team that has a small-file-heavy workload uses SeaweedFS.
Just want S3 on a single host for development: MinIO single-node, or even a small S3 mock (localstack, minio in Docker).
The trade-offs
MinIO:
-
Pros: S3-compatible out of the box, single binary, fast to deploy.
-
Cons: single point of failure in single-node mode, cluster mode needs 4+ nodes.
-
The team that uses MinIO has the fastest S3-on-prem path; the team that needs petabytes moves to Ceph.
Ceph:
-
Pros: distributed, scales to PB, multiple interfaces (object, block, file).
-
Cons: complex, lots of moving parts, ops burden.
-
The team that has the ops capacity uses Ceph; the team that does not finds MinIO.
SeaweedFS:
-
Pros: lightweight, fast for small files, simple to operate.
-
Cons: less mature than Ceph or MinIO, smaller community.
-
The team that has a small-file workload and wants simplicity uses SeaweedFS.
FAQ
Is MinIO really S3-compatible?
Yes - MinIO implements the full S3 API. The team that uses the AWS SDK, boto3, aws-cli, or any S3 client can point it at MinIO with no code changes.
Can I use Ceph as a drop-in for AWS S3?
Yes - install the RGW (RADOS Gateway) component. RGW exposes an S3-compatible API. The team that uses boto3 or the AWS SDK can point it at Ceph with the same code.
Which is faster, MinIO or Ceph?
MinIO is faster for small deployments (single binary, no consensus overhead). Ceph is faster at scale (parallel data path, optimized for large clusters). The team that benchmarks on their specific workload gets the only correct answer.
Can I run MinIO on Kubernetes?
Yes - the official MinIO Operator deploys MinIO on Kubernetes with one CRD. The team that runs Kubernetes uses the Operator for production; the team that does not uses the single-binary install.
What is the smallest Ceph cluster?
3 MONs + 3 OSDs (typically on the same 3 hosts) is the minimum. The team that wants to try Ceph on a single host uses the cephadm bootstrap --single-host-defaults mode.
How this fits the rest of the stack
For a sense of what the full project costs before it commits, the RunxBuild hosting calculator shows the line items together. The API, the database, the storage, the worker, the bandwidth - each one is a separate number, and the team’s mental model for the platform is the sum of those numbers.
Useful related references: