Ceph Storage in Production: Architecture, Management, and Enterprise Support

Jul 6
8 min read

Ceph Storage in Production: Architecture, Management, and Enterprise Support

Ceph is now the most widely used open-source distributed storage solution in European enterprise cloud infrastructures. It's used by cloud providers, telecom operators, universities, hospitals, and banks—often without end users knowing, because Ceph runs under the hood of OpenStack , Kubernetes , and dozens of other platforms.

Yet, despite its widespread adoption, Ceph remains one of the most complex technologies to manage in production. The difference between a well-configured and a poorly managed Ceph cluster isn't measured in functionality—it's measured in performance, availability, and, in the worst cases, data loss.

What is Ceph and why has it become the de facto standard?

Ceph is a software-defined distributed storage platform that unifies three types of storage into a single infrastructure: block storage (RBD), object storage (S3/Swift compatible), and shared storage (CephFS). This unification is one of its key strengths—rather than managing three separate systems for three different needs, Ceph solves them all with a single platform, a single management API, and a single pool of hardware.

The choice of Ceph as the reference storage for OpenStack is no coincidence. Cinder, OpenStack's block storage service, integrates natively with Ceph RBD. Glance, the image management service, uses Ceph as its backend. Nova, the compute service, can run VMs directly on Ceph volumes with copy-on-write—reducing provisioning times from minutes to seconds.

The same goes for Kubernetes: Rook-Ceph is one of the most mature storage operators for Kubernetes today, and allows you to manage persistent storage for containers with the same reliability as a proprietary enterprise system.

The result is that those who choose Ceph don't just choose storage—they choose the foundation on which to build an entire open-source cloud infrastructure, scalable horizontally, with no vendor lock-in and full data sovereignty.

Ceph Architecture: The Essential Components

To understand how Ceph works in production, you need to know its main components and the role each plays in the cluster.

OSD — Object Storage Daemon. This is the component that physically manages data. Each OSD typically corresponds to a physical disk—HDD or NVMe—and is responsible for reading, writing, and replicating data. A production Ceph cluster can have anywhere from a few OSDs to several thousand. The health of the OSDs is the first indicator of the cluster's health: an OSD down means reduced replication capacity and, if there are sufficient numbers, loss of data access.

Monitors — MON. Monitors maintain the cluster map—the CRUSH map structure that defines how data is distributed across OSDs. In a production cluster, at least three Monitors are always used to ensure quorum. Losing quorum means the cluster enters a read-only state until recovery—a critical condition for any production workload.

Manager — MGR. The Manager collects cluster metrics, exposes REST APIs, and manages monitoring and orchestration modules. In a modern cluster, the Manager is also the access point to the Ceph web dashboard and the Cephadm orchestrator.

MDS — Metadata Server. The Metadata Server is only needed when using CephFS, Ceph's distributed filesystem. It manages filesystem metadata—directories, permissions, attributes—while the actual data remains in the OSDs.

RADOS Gateway — RGW. The RADOS Gateway exposes an S3- and Swift-compatible API for object storage. For those with applications that already use the AWS S3 API, RGW allows you to maintain the same code while pointing to an on-premises Ceph cluster—without any application modifications.

In summary:

Component	Role	Minimum in production
OSD	Physical data management	3+ per rack fault tolerance
Monitor	Cluster Map and Quorum	3 (always odd)
Manager	Metrics, APIs, dashboards	2 (active + standby)
MDS	CephFS Metadata	2+ if using CephFS
RGW	S3/Swift API compatible	2+ for high availability

CRUSH Map: The Heart of Data Distribution

The CRUSH Map is the algorithm that decides where data is written in the cluster. It's not a simple round-robin—it's a deterministic algorithm that takes into account the cluster's physical topology: which OSD is located on which host, which rack, which server room, and which data center.

Configuring the CRUSH Map correctly is one of the most critical—and most overlooked—aspects of a production Ceph cluster. A poorly configured CRUSH Map can cause replicas of a piece of data to end up on the same physical disks, negating data loss protection. An optimized CRUSH Map ensures that each replica is on a different host, a different rack, and, for multi-site deployments, a different data center.

Ceph's default replication rule is three replicas—every data written to the cluster is replicated three times. This means that a cluster with 100 TB of raw capacity has a usable capacity of approximately 33 TB. For workloads where capacity is more important than redundancy, Ceph also supports erasure coding—a RAID-like mechanism that achieves storage efficiencies of 80% or higher, at the expense of increased computational cost for write operations.

BlueStore: The modern Ceph storage backend

Until Ceph Luminous, the OSD storage backend was FileStore—based on a standard Linux filesystem (typically XFS) with a separate journaling layer. FileStore was reliable but had significant performance limitations, especially with NVMe drives.

From Ceph Luminous onward, the default backend is BlueStore—a completely custom backend that writes directly to block devices, bypassing the operating system filesystem. The advantages are significant: lower latency, higher throughput, native checksums for each data written, and no double-write journaling overhead.

BlueStore uses RocksDB to manage OSD metadata. In a high-performance NVMe cluster, separating the RocksDB device from the data device is one of the most effective tuning interventions—it reduces read latency and increases overall cluster throughput.

Ceph use cases in production

Ceph isn't a one-size-fits-all solution for every use case. Understanding when Ceph excels and when other approaches are more appropriate is crucial to designing an efficient storage infrastructure.

Storage for OpenStack infrastructure. This is the use case for which Ceph was originally optimized. Native integration with Cinder, Glance, and Nova, the ability to manage thousands of concurrent volumes, and horizontal scalability make Ceph the natural choice for any medium to large OpenStack cloud.

Persistent storage for Kubernetes. With Rook-Ceph, the Ceph cluster is managed as a native Kubernetes operator. PersistentVolumeClaims are automatically satisfied by Ceph RBD, with support for ReadWriteOnce and, with CephFS, ReadWriteMany. It is the most mature choice for persistent storage on Kubernetes in on-premises environments.

S3-compatible object storage. With RGW, Ceph offers on-premises S3-compatible object storage ideal for backup, data archiving, image and video storage, log data, and any workload that would use S3 in the public cloud. S3 compatibility allows you to use the same clients, SDKs, and tools you already use with AWS—without modifications.

Backup and disaster recovery. Ceph supports asynchronous replication between geographically dispersed clusters—RBD mirroring for block volumes and multi-site RGW for object storage. This allows you to build disaster recovery architectures where data is continuously replicated to a second site, with RPOs measured in seconds.

HPC and AI/ML workloads. CephFS with an optimized NVMe cluster can deliver sequential throughput in the GB/s range—sufficient for HPC workloads and AI/ML model training that require fast access to large datasets distributed across multiple compute nodes.

Common issues in production Ceph clusters

Years of experience with production Ceph clusters have taught us that the most common problems follow recurring patterns.

Placement Group in a degraded or undersized state. This is the most common sign of a cluster losing redundancy. It can be caused by an OSD down, a CRUSH Map failing to satisfy replication rules, or a cluster rebalancing after adding or removing nodes. A cluster with a degraded PG has not lost data—but has reduced data loss protection until recovery.

Abnormal performance on random I/O workloads. Ceph is optimized for sequential workloads. Workloads with many small random operations—typical of OLTP databases—can saturate the cluster's IOPS capacity before saturating throughput. Tuning the number of PGs, separating NVMe journals from HDD data, and configuring cache tiering are the most effective interventions in these scenarios.

OSD flapping. A situation where one or more OSDs cycle in and out of the up/down state. The most common causes are network problems between cluster nodes, disks with bad sectors causing I/O timeouts, or insufficient memory causing an OOM kill of the OSD process. OSD flapping generates a high volume of recovery operations that can saturate the cluster's network bandwidth.

Excessive rebalancing after cluster changes. Every time you add or remove an OSD, Ceph recalculates the data distribution and moves the PGs to balance the cluster. In a large cluster, this can generate hundreds of GB of network traffic. Correctly configuring the target_max_bytes parameter and using the osd reweight tool instead of adding large OSDs all at once is crucial to maintaining performance during expansion phases.

How Epic Edge Supports Ceph Clusters in Production

The Epic Edge team has been working on production Ceph clusters for cloud providers, telcos, and enterprise infrastructure for years. Our support covers the entire cluster lifecycle—from initial design to day-to-day management, upgrades, and expansions.

Assessment and health check. The starting point for any intervention is a comprehensive analysis of the cluster's status: Ceph version, OSD configuration, Monitor status, CRUSH Map, PG distribution, performance metrics, and recent incident logs. Within 24-48 hours, the customer receives a detailed report detailing the identified critical issues and recommended corrective actions, prioritized based on potential impact.

Troubleshooting and incident response. For clusters experiencing a crisis—OSD down, PGs stuck in an inconsistent state, or loss of Monitor quorum—the Epic Edge team responds with a guaranteed SLA. The response time for P1 critical incidents is 30 minutes. The goal of each intervention is to restore the cluster with minimal impact on production workloads.

Performance optimization. Tuning a Ceph cluster is an iterative process that requires a deep understanding of the specific workload. We optimize BlueStore configuration, PG sizing, NVMe device separation for RocksDB metadata, replication policies, and erasure coding based on the cluster's real-world usage profile.

Upgrades between major versions. Ceph upgrades between major versions—from Pacific to Quincy, from Quincy to Reef—require careful planning and a tested rolling process. Epic Edge plans and executes upgrades within agreed-upon maintenance windows, with systematic testing at every step and rollback procedures in place if necessary.

Expansion and scaling. Adding new OSDs, hosts, and racks to an existing cluster requires editing the CRUSH map, rebalancing data, and verifying post-expansion performance. Epic Edge manages the entire process, ensuring the cluster remains operational during the expansion phase.

24/7 proactive monitoring. The Epic Edge NOC monitors the health of managed customers' Ceph clusters with proactive alerts on anomalies, capacity thresholds, OSD warnings, and performance degradation. The goal is to intervene before problems become critical incidents.

Ceph and data sovereignty in Europe

An increasingly important issue for European organizations is data localization. Ceph, as an open-source, software-defined platform, can be deployed on any hardware, in any data center, under any jurisdiction.

This makes it the natural choice for those who need to ensure their data remains in Europe—without being dependent on a US cloud provider subject to the US Cloud Act. With Ceph deployed on customer infrastructure or in European data centers, data sovereignty is structural, not contractually declared by a vendor who could be forced to violate it.

GDPR and NIS2 compliance becomes easier to demonstrate when the entire data custody chain—from hardware to storage software—is under the direct control of the organization or a certified European provider.

Conclusion: Ceph is mature, but requires expertise

Ceph is an extraordinarily capable platform. It's also a platform that requires specific expertise to manage properly in production—not because it's flawed, but because the complexity of a large-scale distributed system can't be eliminated, it's managed.

The difference between a functioning Ceph cluster and one that performs poorly lies in the quality of the initial setup, understanding the specific workload, and the ability to quickly intervene when something doesn't go as expected.

For organizations that don't have this expertise in-house—or want to leverage it with a specialized team—enterprise support from a partner with real-world experience with Ceph clusters in production isn't an additional cost. It's an investment in business continuity.