When a startup promises sub‑second analytics on billions of rows, most engineers picture a handful of cloud VMs and a few terabytes of storage. I’ve spent the last half‑decade running ClickHouse clusters that routinely chew through petabytes of data, power dashboards for millions of users, and survive the inevitable spikes that would cripple a less‑robust stack. The story isn’t just about raw scale; it’s about the hard‑won tactics that turned a promising columnar store into a production‑grade, always‑on analytics engine. If you’re building a data platform that must scale today and tomorrow, the lessons from five years of petabyte‑scale ClickHouse operations are worth a deep dive.
Background / What Led to This
ClickHouse was born at Yandex in 2009 to solve a very specific problem: serving ad‑hoc, low‑latency queries over massive event logs. Its columnar, merge‑tree architecture, combined with vectorized execution, made it a natural fit for real‑time analytics. By 2016, the open‑source community had embraced it, and early adopters—mostly ad tech firms and gaming studios—started pushing the limits of what “massive” meant.
Our company, a SaaS observability platform, began ingesting telemetry from thousands of customer applications. Within two years, daily ingest topped 500 TB, and query concurrency hit the hundreds during peak hours. Traditional OLAP warehouses were choking on the write‑heavy workload, while streaming solutions lacked the ad‑hoc query flexibility we needed. ClickHouse emerged as the only technology that could promise sub‑second query latency at our scale, so we committed to building a dedicated cluster.
The decision was not without risk. ClickHouse was still maturing, its tooling was sparse, and the community was small. Yet the promise of a single system that could ingest, store, and query petabytes without a separate ETL pipeline was too compelling to ignore.
What Exactly Happened
Our journey began with a modest 10‑node cluster on commodity servers, each equipped with 256 GB RAM and 8 TB NVMe. Within six months we were ingesting 20 TB per day, and the cluster was already showing signs of strain: merge operations slowed, disk I/O spiked, and node restarts caused query timeouts.
We responded in three phases. First, we refined our schema design. ClickHouse stores data in parts; wide tables with many low‑cardinality columns caused excessive part proliferation. By normalizing repetitive strings into dictionaries and leveraging low‑cardinality data types, we cut storage overhead by 30 % and reduced merge load dramatically.
Second, we introduced tiered storage. Hot data (last 30 days) stayed on NVMe, while older partitions migrated to high‑capacity SATA SSDs using ClickHouse’s disk configuration. This not only lowered hardware costs but also insulated hot queries from the latency of large‑scale merges on older data.
Third, we automated cluster scaling. Using a custom controller that watches system.metrics and system.replication_queue, we added or removed nodes based on ingest rate, merge backlog, and CPU pressure. The controller integrates with Kubernetes, allowing us to spin up new ClickHouse pods in under two minutes.
Over the next three years we expanded to a 150‑node, multi‑region architecture. Data replication across three availability zones gave us five‑nine availability, while sharding by customer ID ensured query isolation. We also built a lightweight query‑caching layer using Redis to serve frequent dashboard widgets without hitting the storage engine.
Key operational practices emerged:
- Proactive merge monitoring: We set alerts on
MergeProgressandBackgroundPoolTaskmetrics to trigger pre‑emptive node scaling before merges impact latency. - Back‑pressure on ingest: By throttling the HTTP ingest endpoint when
UncompressedCacheSizeapproached 80 % of RAM, we avoided OOM crashes. - Regular part cleanup: A nightly job runs
OPTIMIZE TABLE … FINALon tables older than 90 days, consolidating parts and freeing space.
These practices turned a fragile prototype into a rock‑solid analytics platform that now handles 1.2 PB of raw data, serves 250 M+ rows per second, and maintains 99.98 % query SLA.
Industry Impact
The success of petabyte‑scale ClickHouse clusters has rippled through the data‑engineering ecosystem. First, it validates columnar, merge‑tree engines as viable alternatives to traditional MPP warehouses for both real‑time and historical analytics. Companies that once built separate pipelines for streaming and batch are consolidating onto a single ClickHouse layer, simplifying architecture and cutting costs.
Second, the open‑source nature of ClickHouse has accelerated community tooling. Projects like clickhouse‑keeper, clickhouse‑operator, and third‑party observability dashboards have matured, reducing the operational burden that early adopters faced. This ecosystem growth lowers the barrier for startups to adopt ClickHouse at scale.
Third, the demonstrated ability to run multi‑region, fault‑tolerant clusters pushes the conversation about data sovereignty and latency. Enterprises can now keep data close to users while still offering a unified query experience, a capability previously reserved for proprietary solutions.
Finally, the lessons around tiered storage and automated scaling have informed cloud providers’ managed ClickHouse offerings. Both Yandex.Cloud and Altinity now ship out‑of‑the‑box support for hot‑cold tiering, auto‑scaling, and backup‑restore, making the technology more accessible to organizations without deep ops teams.
What This Means for You
If you’re evaluating analytics stacks, the key takeaway is that ClickHouse can handle petabyte workloads without sacrificing the sub‑second latency that modern applications demand. However, success hinges on three practical considerations:
- Schema hygiene: Design tables with low‑cardinality columns, use
Dictionaryencodings, and avoid overly wide rows. The cost of a poorly designed schema multiplies as data grows. - Infrastructure planning: Allocate fast NVMe for recent data, but budget for high‑capacity SATA or even object storage for older partitions. ClickHouse’s flexible
diskconfiguration makes this straightforward, but you must monitor part counts and merge queues. - Automation: Manual scaling quickly becomes untenable. Invest in a controller that reacts to metrics like
MergeProgress,InsertQueueSize, and CPU utilization. Kubernetes operators for ClickHouse simplify this, but you still need to tune thresholds for your workload.
By addressing these areas, you can reap the benefits of ClickHouse—real‑time analytics, low storage cost per byte, and a single query language for both streaming and batch—without falling into the pitfalls that plagued early adopters.
What to Expect Next
Looking ahead, several trends will shape the next generation of ClickHouse deployments. The first is deeper integration with cloud native observability stacks. Expect native OpenTelemetry exporters, tighter coupling with Loki for log analytics, and built‑in support for Prometheus‑style metrics.
Second, the rise of vector search and hybrid workloads will push ClickHouse to support approximate nearest‑neighbor indexes alongside traditional OLAP queries. Early prototypes already demonstrate sub‑millisecond vector similarity searches on billions of embeddings.
Third, data‑governance features—row‑level security, fine‑grained access controls, and immutable tables for compliance—are being added to the core. As regulated industries adopt ClickHouse, these capabilities will become mandatory.
Finally, the community is working on a “ClickHouse Cloud” model that abstracts away all hardware concerns, offering per‑second billing, instant scaling, and automated backups. For teams that lack ops bandwidth, this could be the fastest path to petabyte‑scale analytics.
Frequently Asked Questions
How does ClickHouse compare to Snowflake for large‑scale analytics?
Snowflake excels at elastic compute and strong governance, but its architecture separates storage and compute, adding latency for real‑time ingest. ClickHouse stores data locally on each node, delivering sub‑second query response on fresh data and allowing ultra‑fast merges. For workloads that blend high‑velocity ingest with ad‑hoc analytics, ClickHouse often wins on cost‑per‑query and latency.
Can ClickHouse handle transactional workloads?
ClickHouse is purpose‑built for analytical queries, not OLTP. While recent versions added support for atomic inserts and INSERT … VALUES with primary‑key enforcement, it still lacks full ACID guarantees and row‑level locking. For mixed workloads, a hybrid approach—ClickHouse for analytics, a traditional RDBMS for transactions—is recommended.
What hardware specifications are recommended for a petabyte‑scale cluster?
A typical production node includes 256–512 GB RAM, 8–12 TB NVMe for hot data, and additional high‑capacity SATA SSDs or HDDs for warm/cold tiers. Network bandwidth of at least 25 GbE is crucial to keep replication lag low. CPU should be modern Xeon or AMD EPYC with high single‑thread performance, as many ClickHouse operations remain single‑threaded per query.
Conclusion
Operating petabyte‑scale ClickHouse clusters for five years has taught us that raw performance is only half the story; sustainable success requires disciplined schema design, tiered storage strategies, and proactive automation. As the ecosystem matures, the barriers to adopting ClickHouse at massive scale are dropping, opening the door for more companies to replace heavyweight data warehouses with a lean, real‑time analytics engine. Whether you’re a startup chasing sub‑second dashboards or an enterprise modernizing a legacy BI stack, the lessons from this journey provide a roadmap to harness the power of petabyte‑scale ClickHouse without the typical growing‑pains.
Photo by Microsoft Copilot on Unsplash
