ClickHouse on Kubernetes – Part 3 | Running ClickHouse cluster on Google Kubernetes Engine

sakthi.sri
Posted on December 1, 2022

Kubernetes orchestration simplifies many common operational concerns like scheduling, auto-scaling, and failover. Usually, databases that support replication, sharding, and auto-scaling are well-suited for Kubernetes. ClickHouse and Kubernetes can perform better together.

At ChistaData, we are interested in writing the following series of blogs to explain the ClickHouse on Kubernetes topic.

  • ClickHouse on Minikube
  • ClickHouse on Google Kubernetes Engine ( GKE )
  • ClickHouse on Amazon Elastic Kubernetes Service ( Amazon EKS )

This is the third part of the series. You can find the previous parts below.

In this blog post, we will explain the complete details of the Installation and configuration process of the ClickHouse cluster on Google Kubernetes Engine.

Overview of the Setup:

To complete the setup, we need to work on the following steps.

  • Enable the Kubernetes Engine API
  • Gcloud cli Installation
  • Kubectl Installation
  • Install Auth plugin
  • Create the GKE Cluster
  • Gcloud Configurations
  • Cloning ClickHouse cluster configs and configurations
  • Testing the connections and cluster status

Enable the Kubernetes Engine API

The first step we would say is to login your Google Cloud account and enable the Kubernetes Engine API as shown below.

Once the API enabled, we are able to create the GKE cluster.

Gcloud CLI Installation

The Google Cloud CLI is a set of tools to create and manage Google Cloud resources. You can use these tools to perform many common platform tasks from the command line or through scripts and other automation. The following steps can be used to install the gcloud cli on Ubuntu servers.

sudo apt-get install apt-transport-https ca-certificates gnupg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt-get update
sudo apt-get install google-cloud-cli

Ref: https://cloud.google.com/sdk/docs/install#deb

Kubectl Installation:

The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. The following steps can be used to install the “kubectl” client tool.

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

Install Auth plugin:

“google-cloud-sdk-gke-gcloud-auth-plugin” is the authentication plugin used to authenticate the GKE cluster and generate the GKE related tokens. It can be installed by using the following command.

apt-get install google-cloud-sdk-gke-gcloud-auth-plugin

Create the GKE Cluster

Now, we have installed the requirements. The next step is, need to create the GKE cluster on the console. Follow the below steps to create the GKE cluster.

Search “Kubernetes” on the search tab then choose the “Kubernetes Engine” topic.

Then click on the “CREATE” option. it will displays to options ( Autopilot & Standard ). You can choose any of these. We used to choose the “Standard” option. Choosing the “, Autopilot” option will enable automatic management from GKE.

Once you select the mode, It will redirect to the configuration page. There you can choose the cluster name, node configurations, security, networking, etc. Once you have configured everything, click the “CREATE” button below.

Note: Make sure you have the enough Quotas to create the cluster. otherwise, you will get the errors.

Once you clicked on the “CREATE” button, you can see the progress of the cluster creation. It might take some 5 – 10 minutes for the entire configuration.

Once the configuration completed, you can see the GKE cluster is available with green tick mark.

Gcloud Configurations:

Now, we have created the GKE cluster. The next step is we need to configure the gcloud Clie with the cluster.

configure the project id ( project id can be get from console by clicking your project profile )

root@chista-gke:~# gcloud config set project orbital-heaven-298517
Updated property [core/project].

Update the cube config with the GKE cluster credentials so that we can access the GKE cluster using “kubectl” command.

root@chista-gke:~/ClickHouseCluster# gcloud container clusters get-credentials chista-gke-cluster --zone asia-east1-a 
Fetching cluster endpoint and auth data.
kubeconfig entry generated for chista-gke-cluster.

This will automatically updates the cube config as shown below.

root@chista-gke:~# cat .kube/config | tail -n9
- name: gke_orbital-heaven-298517_us-east1_chista-gke-cluster
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

Now, we can access the GKE cluster by using the “kubectl” command.

root@chista-gke:~/ClickHouseCluster# kubectl get namespace
NAME              STATUS   AGE
default           Active   4m9s
kube-node-lease   Active   4m12s
kube-public       Active   4m12s
kube-system       Active   4m12s

Now we are good to configure the ClickHouse cluster.

Cloning cluster configs and configurations

The next step is, we need to configure the ClickHouse cluster. The configs are publicly available in our repository. You can clone them directly as follows.

root@chista-gke:~# git clone https://github.com/ChistaDATA/clickhouse_lab.git
Cloning into 'clickhouse_lab'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 9 (delta 1), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (9/9), 29.61 KiB | 286.00 KiB/s, done.
Resolving deltas: 100% (1/1), done.

Once you have cloned the repository, you might see the following files under the folder “clickhouse_lab/ClickHouseCluster”.

root@chista-gke:~# cd ClickHouseCluster/
root@chista-gke:~/ClickHouseCluster# pwd
/root/ClickHouseCluster
root@chista-gke:~/ClickHouseCluster# ls -lrth
total 252K
-rwxr-xr-x 1 root root 241 Nov 29 07:33 create-operator
-rw-r--r-- 1 root root 6.1K Nov 29 07:33 zookeeper.yaml
-rwxr-xr-x 1 root root 199 Nov 29 07:33 create-cluster
-rw-r--r-- 1 root root 227K Nov 29 07:33 operator.yaml
-rwxr-xr-x 1 root root 254 Nov 29 07:33 create-zookeeper
-rw-r--r-- 1 root root 1.6K Nov 29 09:07 cluster.yaml

Here you can see three .yaml files and the respective bash files. We can execute the bash files one by one, it will call the respective config and build the cluster.

First we need to call the operator script as shown below,

root@chista-gke:~/ClickHouseCluster# ./create-operator 
namespace/chista-operator created
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallations.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallationtemplates.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseoperatorconfigurations.clickhouse.altinity.com created
serviceaccount/clickhouse-operator created
clusterrole.rbac.authorization.k8s.io/clickhouse-operator-chista-operator created
clusterrolebinding.rbac.authorization.k8s.io/clickhouse-operator-chista-operator created
configmap/etc-clickhouse-operator-files created
configmap/etc-clickhouse-operator-confd-files created
configmap/etc-clickhouse-operator-configd-files created
configmap/etc-clickhouse-operator-templatesd-files created
configmap/etc-clickhouse-operator-usersd-files created
deployment.apps/clickhouse-operator created
service/clickhouse-operator-metrics created

Secondly, we need to call the zookeeper script,

root@chista-gke:~/ClickHouseCluster# ./create-zookeeper 
namespace/chista-zookeeper created
service/zookeeper created
service/zookeepers created
poddisruptionbudget.policy/zookeeper-pod-disruption-budget created
statefulset.apps/zookeeper created

Finally, we need to call the cluster script as shown below,

root@chista-gke:~/ClickHouseCluster# ./create-cluster 
clickhouseinstallation.clickhouse.altinity.com/herc created

ClickHouse Cluster is created. We can verify this using the following command.

root@chista-gke:~/ClickHouseCluster# kubectl get pods -n chista-operator
NAME READY STATUS RESTARTS AGE
chi-herc-herc-cluster-0-0-0 2/2 Running 0 77m
chi-herc-herc-cluster-0-1-0 2/2 Running 0 76m
chi-herc-herc-cluster-1-0-0 2/2 Running 0 76m
chi-herc-herc-cluster-1-1-0 2/2 Running 0 75m
clickhouse-operator-58768cf654-h7c8s 2/2 Running 0 79m

As per the config ( cluster.yaml ), We have mentioned 2 replicas and 2 shards.

root@chista-gke:~# less cluster.yaml| grep -i 'repl\|shard'
          shardsCount: 2
          replicasCount: 2

Here,

  • “chi-herc-herc-cluster-0-0-0” and “chi-herc-herc-cluster-1-0-0” are the shards.
  • “chi-herc-herc-cluster-0-1-0” and “chi-herc-herc-cluster-1-1-0” are the respective replicas.

Shard 1:

chi-herc-herc-cluster-0-0-0
  \_chi-herc-herc-cluster-0-1-0

Shard 2:

chi-herc-herc-cluster-1-0-0
  \_chi-herc-herc-cluster-1-1-0

The cluster setup is completed!

Testing the connections and cluster status

You can use the following command to go directly login the ClickHouse shell.

root@chista-gke:~# kubectl exec -it chi-herc-herc-cluster-0-0-0 -n chista-operator -- clickhouse-client
Defaulted container "clickhouse" out of: clickhouse, clickhouse-log
ClickHouse client version 21.10.5.3 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 21.10.5 revision 54449.

chi-herc-herc-cluster-0-0-0.chi-herc-herc-cluster-0-0.chista-operator.svc.cluster.local :) show databases;

SHOW DATABASES

Query id: 62d8526c-9785-4f9c-99ce-81e1fb1a2ee5

┌─name────┐
│ default │
│ system  │
└─────────┘

2 rows in set. Elapsed: 0.004 sec.

You can also use the following method to login the OS shell then ClickHouse shell.

root@chista-gke:~# kubectl exec -it chi-herc-herc-cluster-0-0-0 /bin/bash -n chista-operator
clickhouse@chi-herc-herc-cluster-0-0-0:/$ clickhouse-client 
ClickHouse client version 21.10.5.3 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 21.10.5 revision 54449.

chi-herc-herc-cluster-0-0-0.chi-herc-herc-cluster-0-0.chista-operator.svc.cluster.local :) show databases;

SHOW DATABASES

Query id: da2bd4c4-e9fa-4c8c-8862-e5252fe6d476

┌─name────┐
│ default │
│ system  │
└─────────┘

2 rows in set. Elapsed: 0.003 sec.

ClickHouse cluster status:

chi-herc-herc-cluster-0-0-0.chi-herc-herc-cluster-0-0.chista-operator.svc.cluster.local :) SELECT cluster, shard_num, replica_num, host_name FROM system.clusters WHERE cluster = 'herc-cluster'

SELECT
    cluster,
    shard_num,
    replica_num,
    host_name
FROM system.clusters
WHERE cluster = 'herc-cluster'

Query id: 2c38e844-c2f3-46f5-b34f-7eca7f2754be

┌─cluster──────┬─shard_num─┬─replica_num─┬─host_name─────────────────┐
│ herc-cluster │         1 │           1 │ chi-herc-herc-cluster-0-0 │
│ herc-cluster │         1 │           2 │ chi-herc-herc-cluster-0-1 │
│ herc-cluster │         2 │           1 │ chi-herc-herc-cluster-1-0 │
│ herc-cluster │         2 │           2 │ chi-herc-herc-cluster-1-1 │
└──────────────┴───────────┴─────────────┴───────────────────────────┘

4 rows in set. Elapsed: 0.027 sec. 

From the above output, We have overall 4 nodes. For shard_num “1”, we have two nodes, and for shard_num “2” we have two nodes, and we can see the respective replicas as well. So, the configuration is perfect!

Conclusion:

Hopefully, this blog will help you understand the configurations involved in the ClickHouse cluster on Google Kubernetes Engine. Thank you!

 

Photo by Alexander Zvir: https://www.pexels.com/photo/silver-and-orange-fish-on-ice-3731945/

Leave a Reply

Your email address will not be published. Required fields are marked *