OpenStack without Ceph isn't a true cloud: why distributed storage is the foundation of enterprise IaaS

OpenStack without Ceph isn't a true cloud: why distributed storage is the foundation of enterprise IaaS

Those new to OpenStack tend to focus on compute—Nova, VMs, and networking with Neutron. Storage comes second, treated as an implementation detail to be resolved once everything else is working.

It's a mistake that costs dearly in production.

Storage isn't an afterthought in an enterprise cloud infrastructure. It's the foundation upon which everything else rests—VMs, images, persistent volumes, object storage. And in a serious OpenStack infrastructure, that foundation is called Ceph .

Why storage is the real difference between a lab and a production cloud

An OpenStack cluster works even without Ceph. It can be configured with local storage on hypervisors, with shared NFS, or with a proprietary SAN array. In a test or development environment, these solutions are perfectly adequate.

In production, with real workloads, the situation changes radically.

Local storage on hypervisors means that VMs are physically tied to the node they run on — live migration becomes impossible or very limited, automatic failover doesn't work as it should, and any hardware failure brings with it the risk of data loss.

Shared NFS introduces a single point of failure and a performance bottleneck that manifests under load. It works as long as the number of VMs and IOPS remain low—then it collapses.

Proprietary SAN arrays solve some of these problems but introduce new ones: high costs, vendor lock-in, rigid scalability, and a hardware dependency that doesn’t mesh well with OpenStack’s open-source philosophy.

Ceph was born to solve these problems structurally, not with patches or workarounds.

Native integration between OpenStack and Ceph

The reason Ceph has become the reference storage for OpenStack isn't just technical—it's architectural. The two systems were designed to work together, and the integration between OpenStack and Ceph components is native, deep, and optimized.

Nova and Ceph RBD — VM Provisioning

When Nova, OpenStack's compute service, is configured with Ceph as its backend, VMs are no longer created by copying an image to the hypervisor's local disk. Instead, they are booted directly onto RBD volumes in the Ceph cluster, with copy-on-write functionality.

The practical result is immediate: the time to provision a new VM goes from minutes—the time it takes to copy a tens of GB image—to seconds. The base image disk remains in the Ceph cluster as a read-only snapshot, and each new VM simply receives a copy-on-write layer on top of it. No copying, no data transfer, no waiting.

This fundamentally changes the user experience of the cloud — and the operational efficiency for those who manage it.

Cinder and Ceph RBD — the persistent volumes

Cinder, OpenStack's block storage service, natively integrates with Ceph RBD for managing persistent volumes. Each Cinder volume is an RBD image in the Ceph cluster—replicated across three nodes by default, independent of the hypervisor running the VM using it.

This decoupling is crucial. A VM with a Cinder volume can be migrated live from one hypervisor to another without any service disruption—the volume remains in the Ceph cluster, and the VM's new location reaches it over the network. Automatic failover in the event of a hypervisor failure becomes possible and transparent to the end user.

Cinder volume snapshots on a Ceph backend are nearly instantaneous operations—they don't require a physical copy of the data, but only the creation of an RBD snapshot, which is a metadata-level operation. This enables frequent snapshot policies without impacting performance.

Glance and Ceph — Image Management

Glance, OpenStack's image management service, uses Ceph as a backend to store VM images. This has two concrete advantages.

The first is efficient image sharing—an Ubuntu or CentOS image uploaded once to the Ceph cluster is available across all hypervisors without having to copy it locally to each. In a cluster with tens or hundreds of compute nodes, this greatly reduces network traffic during provisioning.

The second is the Nova integration for copy-on-write described above — when Nova boots a VM from a Glance image on a Ceph backend, the image is simply cloned as an RBD snapshot, without any physical copy of the data.

RGW — S3-compatible object storage

Ceph's RADOS Gateway exposes an S3 and Swift-compatible API directly from your existing Ceph cluster. For an enterprise OpenStack cloud, this means native S3-compatible object storage without having to manage a separate system.

For workloads that already use the S3 API—backup, archive, log storage, application data—RGW allows you to continue using the same clients and SDKs unchanged, pointing to an on-premises cluster instead of AWS S3. Data sovereignty is built-in, not contractually mandated.

Performance: Where Ceph Changes the Game

A common misconception about Ceph is that it is “good enough” storage for cloud workloads, but falls short of the performance of enterprise SAN arrays for the most demanding workloads.

In a properly sized and optimized cluster, this is not true.

BlueStore and NVMe

Since Luminous, Ceph has used BlueStore as the default backend for OSDs—a custom backend that writes directly to block devices without the overhead of an intermediate filesystem. With NVMe drives, a Ceph cluster with BlueStore optimized achieves latencies in the hundreds of microseconds range and throughput in the GB/s range for a single OSD.

Separating the RocksDB device—which manages OSD metadata—on a dedicated NVMe drive from the data is one of the most effective tuning interventions. In a cluster with HDDs for data and NVMe for metadata, read latencies are dramatically reduced because metadata lookup operations don't compete with data I/O operations.

Horizontal scalability without bottlenecks

The fundamental difference between Ceph and a traditional SAN array is the scalability model. A SAN array scales vertically—you add capacity and performance to the same physical system, up to the array's hardware limits. When you reach that limit, you buy a larger array.

Ceph scales horizontally—add OSD nodes, and capacity and performance grow linearly with the number of nodes. There's no architectural limit—the world's largest Ceph clusters manage exabytes of data across thousands of OSDs. And adding new nodes happens while the cluster is live, with no service interruption.

When Ceph makes a difference in production

There are specific scenarios where the choice between Ceph and simpler alternatives determines the success or failure of the infrastructure.

High availability and automatic failover

In an OpenStack cluster with Ceph, hypervisor failover can be managed completely automatically. When a compute node goes offline, the VMs running on it can be automatically restarted on other hypervisors—their Ceph volumes are already available in the cluster, requiring no storage recovery.

In a cluster with local storage, the same scenario requires first recovering data from the failed node—assuming it's possible—before restarting VMs elsewhere. Recovery times are measured in hours rather than minutes.

Transparent live migration

Live migration—moving a VM from one hypervisor to another without service interruption—requires that the VM's storage be accessible to all compute nodes. With Ceph, this is structurally guaranteed: all compute nodes see the same Ceph cluster over the network, and live migration is a transparent operation for the VM and the end user.

Multi-tenancy and isolation

In a multitenant cloud—where multiple customers or business units share the same physical infrastructure—storage must ensure complete isolation between tenants. With Ceph, each tenant has its own pools, with its own capacity quotas, replication policies, and snapshots. Isolation is guaranteed at the cluster level, not just at the application level.

Why Not All Ceph Deployments Work Well

Ceph is powerful, but the difference between a functioning Ceph cluster and a performing one lies in the quality of the configuration and understanding of the workload.

The most common problems we see on production clusters are almost always the same: CRUSH Map configured without considering the physical topology of the hardware, number of Placement Groups sized poorly compared to the cluster capacity, lack of separation between public and cluster networks, OSD HDD and NVMe mixed in the same pool without cache tiering, BlueStore not optimized for the type of workload.

Each of these errors has a measurable impact on performance and stability — a poorly configured Ceph cluster can degrade performance by 50-70% of its actual potential, and can exhibit anomalous behavior under load that isn't seen in initial testing.

Properly configuring an enterprise Ceph cluster isn't a plug-and-play operation. It requires a deep understanding of the architecture, the specific workload, and the interactions between components.

Conclusion

OpenStack is an extraordinarily capable IaaS platform. But its production capabilities—high availability, live migration, fast provisioning, horizontal scalability, multi-tenancy—are only fully realized when the storage is up to par.

Ceph isn't just another storage option for an enterprise OpenStack cluster. It's the foundation upon which that capacity is built.

Those who choose OpenStack without properly planning for Ceph storage are building on sand—and will find out when production workloads start to push against the architecture’s weak points.

Do you have an OpenStack cluster or are you planning an enterprise IaaS infrastructure?

Epic Edge designs and manages OpenStack infrastructure with Ceph on customer hardware, with 99.99% SLA and 24/7 NOC/SOC support.

→ Request an assessment