Get in Touch
 Duration 35 hours

Course Outline

Day 1 – Real-Time Analytics Architecture and ClickHouse Foundations

Modern real-time analytics architecture

  • Batch analytics versus streaming analytics
  • Operational databases versus analytical databases
  • OLTP and OLAP workload characteristics
  • Responsibilities of Kafka, Spark and ClickHouse
  • Reference architecture for on-premises deployment
  • Data flow, latency, throughput and durability considerations
  • Planning for datasets exceeding 100 million records

ClickHouse architecture

  • Column-oriented database concepts
  • ClickHouse server, client and interfaces
  • Data parts, granules, background merges and sparse indexes
  • Single-node and distributed deployment models
  • Storage, memory and CPU considerations
  • Native protocol, HTTP interface and JDBC connectivity

Self-managed installation and configuration

  • Linux installation options
  • Package and repository management
  • Directory structure and configuration hierarchy
  • Server and user configuration files
  • Starting, stopping and validating services
  • Network ports and remote connections
  • Users, roles and access permissions
  • Configuration overrides and environment separation

Hands-on laboratory

  • Install the specified ClickHouse LTS release
  • Configure server and client connectivity
  • Create users, roles and databases
  • Load an initial analytical dataset
  • Execute baseline analytical queries

Day 2 – ClickHouse Data Modelling, Engines and Performance

Data types and schema design

  • Numeric, decimal, string, date and time types
  • Arrays, tuples, maps, nested structures and JSON
  • LowCardinality, Enum and Nullable considerations
  • Selecting efficient types for storage and performance
  • Schema design for event and transactional data

ClickHouse table engines

  • MergeTree family architecture
  • MergeTree and ReplicatedMergeTree
  • ReplacingMergeTree
  • SummingMergeTree and AggregatingMergeTree
  • CollapsingMergeTree and VersionedCollapsingMergeTree
  • Distributed tables
  • Selecting the appropriate engine for a workload

Ordering, partitioning and indexing

  • Primary keys and ordering keys in ClickHouse
  • Sparse primary indexes
  • Partitioning strategies
  • Data-skipping indexes
  • Projections
  • Materialized views
  • TTL-based retention and movement
  • Avoiding excessive partitions and small parts

Large-volume ingestion and optimisation

  • Bulk inserts and asynchronous inserts
  • Insert batch sizing
  • Compression codecs
  • Query execution and EXPLAIN
  • Identifying excessive reads
  • Optimising filters, aggregations and joins
  • Monitoring parts and merges
  • Benchmarking methodology

Hands-on laboratory

  • Generate and load over 100 million synthetic records
  • Compare alternative ordering and partitioning strategies
  • Measure compression and query performance
  • Implement a materialized aggregation
  • Diagnose and optimise inefficient queries

Day 3 – ClickHouse Administration, Backup, Recovery and Migration

Production administration

  • Configuration management
  • Storage policies and volumes
  • User and role administration
  • Resource limits and quotas
  • Workload isolation
  • Logging and system tables
  • Query monitoring and cancellation
  • Capacity and disk-space monitoring
  • Managing background merges and mutations

Security

  • Authentication and role-based access control
  • Network restrictions
  • TLS considerations
  • Protecting credentials and configuration
  • Least-privilege service accounts
  • Audit and query logging

Backup and recovery

  • Backup objectives, RPO and RTO
  • Full and incremental backups
  • Synchronous and asynchronous backups
  • Local-disk and object-storage destinations
  • Database, table and access-management backups
  • Restore procedures and validation
  • Automating backup execution
  • Recovery testing and operational runbooks

Migration

  • Schema and data migration approaches
  • Native and file-based migration methods
  • Parallel loading strategies
  • Validating row counts, checksums and query results
  • Minimising downtime
  • Migration rehearsal and rollback planning
  • Upgrade and patch-management considerations

Hands-on laboratory

  • Configure a backup destination
  • Execute full and incremental backups
  • Simulate data loss
  • Restore into a separate environment
  • Perform a controlled database migration
  • Validate data completeness and recovery time

Day 4 – ClickHouse High Availability and Distributed Operations

High-availability architecture

  • Replication versus sharding
  • Designing shards and replicas
  • ClickHouse Keeper architecture
  • Keeper quorum and failure tolerance
  • ReplicatedMergeTree engines
  • Distributed table architecture
  • Distributed DDL
  • Read and write routing
  • Load balancing and failover

Cluster deployment and operation

  • Cluster topology configuration
  • Creating replicated databases and tables
  • Adding and removing replicas
  • Replica synchronisation
  • Monitoring replication queues
  • Detecting delayed or inactive replicas
  • Handling broken and detached parts
  • Network partitions and quorum behaviour
  • Rolling maintenance and upgrades
  • Disaster-recovery architecture

Hands-on laboratory

  • Deploy a multi-node ClickHouse cluster
  • Configure ClickHouse Keeper
  • Create replicated and distributed tables
  • Load a large dataset across shards
  • Simulate a failed replica
  • Restore availability and validate consistency
  • Test routing, failover and replica recovery

Day 5 – Apache Kafka Setup, Producers, Consumers and Operations

Kafka fundamentals

  • Event-streaming concepts
  • Brokers, controllers, topics and partitions
  • Records, keys, offsets and ordering
  • Replication and in-sync replicas
  • Producers, consumers and consumer groups
  • Kafka durability and delivery semantics
  • Kafka as an event log versus a traditional message queue

Kafka 3.9 installation and configuration

  • Java and operating-system requirements
  • KRaft architecture
  • Controller and broker roles
  • Storage formatting and cluster IDs
  • Broker and listener configuration
  • Creating and managing topics
  • Partition counts and replication factors
  • Retention by time and size
  • Log compaction
  • Producer and consumer configuration
  • Acknowledgements, retries, batching and compression
  • Consumer offsets, rebalancing and lag

Availability, security and monitoring

  • Broker and partition failure scenarios
  • Minimum in-sync replicas
  • Unclean leader-election considerations
  • TLS and SASL concepts
  • Access-control lists
  • Monitoring throughput, lag and broker health
  • Capacity and retention planning
  • Troubleshooting producers and consumers

Hands-on laboratory

  • Deploy a multi-broker Kafka 3.9 cluster
  • Create replicated topics
  • Configure command-line and application producers
  • Configure consumer groups
  • Test partitioning and message ordering
  • Simulate a broker failure
  • Measure and troubleshoot consumer lag

Day 6 – Kafka-to-ClickHouse Ingestion and Spark Foundations

Kafka-to-ClickHouse ingestion

  • Direct ingestion options
  • ClickHouse Kafka table engine
  • Kafka engine configuration
  • Materialized views for persistent ingestion
  • Parsing JSON, Avro and other event formats
  • Consumer-group configuration
  • Controlling ingestion parallelism
  • Delivery semantics and duplicate handling
  • Replay and offset management
  • Schema evolution
  • Malformed messages and dead-letter patterns
  • Monitoring Kafka ingestion from ClickHouse
  • Kafka Connect ClickHouse sink overview

Hands-on ingestion laboratory

  • Publish high-volume events to Kafka
  • Create a Kafka engine table
  • Create destination MergeTree tables
  • Implement materialized-view ingestion
  • Stop and restart consumers
  • Replay events safely
  • Diagnose malformed messages and ingestion lag

Apache Spark foundations

  • Spark driver, executors, workers and applications
  • Spark standalone, YARN and Kubernetes deployment options
  • Installing Spark 4.0.4
  • Java 17 and Scala 2.13 requirements
  • SparkSession and Spark configuration
  • Spark SQL and DataFrames
  • Transformations, actions and lazy evaluation
  • Partitions, stages, tasks and shuffles
  • Reading and writing large datasets

Day 7 – Spark Streaming, Workload Management and Capstone

Spark Structured Streaming

  • Structured Streaming processing model
  • Kafka source configuration
  • Parsing and transforming Kafka events
  • Event time and processing time
  • Windows and aggregations
  • Watermarks and late-arriving data
  • Output modes
  • Checkpointing and recovery
  • Trigger intervals and micro-batching
  • Failure and restart behaviour

Spark and ClickHouse integration

  • Official ClickHouse Spark connector
  • Connector and JDBC approaches
  • Spark DataSource V2
  • Reading ClickHouse tables
  • Writing transformed data to ClickHouse
  • Batch sizing and parallel writes
  • Avoiding excessive small inserts
  • Schema and type mapping
  • Performance considerations

Workload and resource management

  • spark-submit
  • Driver and executor memory
  • Executor cores and parallelism
  • Static and dynamic resource allocation
  • FIFO and FAIR scheduling
  • Fair-scheduler pools
  • Shuffle management
  • Handling data skew
  • Spark UI, event logs and metrics
  • Diagnosing failed and slow jobs

End-to-end capstone

Participants will:

  • Generate or receive a high-volume event stream
  • Publish the events to Kafka
  • Consume and transform selected events using Spark
  • Ingest events into ClickHouse
  • Create an optimised analytical model
  • Process a dataset exceeding 100 million rows
  • Execute real-time analytical queries
  • Monitor Kafka lag, Spark workloads and ClickHouse performance
  • Simulate a component failure
  • Recover the pipeline without losing analytical availability
  • Present the final architecture and operational runbook

Requirements

Participants should have:

  • Working knowledge of SQL
  • Basic Linux command-line and system administration skills
  • Basic understanding of databases and data modelling
  • Familiarity with application development concepts
  • Basic knowledge of Python, Java or Scala

For the practical Spark exercises, PySpark is recommended.

Previous ClickHouse, Kafka or Spark experience is not required.

Testimonials (3)

Related Categories