Usage

If you are not installing the operator using Helm then after installation the CRD for this operator must be created:

kubectl apply -f /etc/stackable/zookeeper-operator/crd/zookeepercluster.crd.yaml

To create a three-node Apache ZooKeeper cluster you can use the example shown below.

Please note that the version you need to specify is not only the version of ZooKeeper which you want to roll out, but has to be amended with a Stackable version as shown. This Stackable version is the version of the underlying container image which is used to execute the processes. For a list of available versions please check our image registry. It should generally be safe to simply use the latest image version that is available.

---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperCluster
metadata:
  name: simple-zk
spec:
  version: 3.8.0-stackable0.7.1
  servers:
    roleGroups:
      default:
        replicas: 3
        config: {}

Afterwards, a ZNode can be created:

---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-znode
spec:
  clusterRef:
    name: simple-zk
    namespace: default

Finally, a ConfigMap is created, containing a path that a ZooKeeper client can connect to:

$ kubectl get configmap simple-znode-nodeport -o yaml
$ $ZOOKEEPER_HOME/bin/zkCli.sh -server $(kubectl get configmap simple-znode-nodeport -o jsonpath='{.data.ZOOKEEPER}')

Encryption

The quorum and client communication are encrypted by default via TLS. This requires the Secret Operator to be present in order to provide certificates. The utilized certificates can be changed in a top-level config.

---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperCluster
metadata:
  name: simple-zk
spec:
  version: 3.8.0-stackable0.7.0
  config:
    tls:
      secretClass: tls (1)
    quorumTlsSecretClass: tls (2)
  servers:
    roleGroups:
      default:
        replicas: 3

1	The `tls.secretClass` refers to the client-to-server encryption. Defaults to the `tls` secret.
2	The `quorumTlsSecretClass` refers to the server-to-server quorum encryption. Defaults to the `tls` secret.

The tls secret is deployed from the Secret Operator and looks like this:

---
apiVersion: secrets.stackable.tech/v1alpha1
kind: SecretClass
metadata:
  name: tls
spec:
  backend:
    autoTls:
      ca:
        secret:
          name: secret-provisioner-tls-ca
          namespace: default
        autoGenerate: true

You can create your own secrets and reference them e.g. in the tls.secretClass to use different certificates.

Authentication

The quorum or server-to-server communication is authenticated via TLS per default. In order to enforce TLS authentication for client-to-server communication, you can set an AuthenticationClass reference in the custom resource provided by the Commons Operator.

---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperCluster
metadata:
  name: simple-zk
spec:
  version: 3.8.0-stackable0.7.0
  config:
    clientAuthentication:
      authenticationClass: zk-client-tls (1)
    quorumTlsSecretClass: tls
  servers:
    roleGroups:
      default:
        replicas: 3
---
apiVersion: authentication.stackable.tech/v1alpha1
kind: AuthenticationClass
metadata:
  name: zk-client-tls (2)
spec:
  provider:
    tls:
      clientCertSecretClass: zk-client-auth-secret (3)
---
apiVersion: secrets.stackable.tech/v1alpha1
kind: SecretClass
metadata:
  name: zk-client-auth-secret (4)
spec:
  backend:
    autoTls:
      ca:
        secret:
          name: secret-provisioner-tls-zk-client-ca
          namespace: default
        autoGenerate: true

1	The `config.clientAuthentication.authenticationClass` can be set to use TLS for authentication. This is optional.
2	The referenced `AuthenticationClass` that references a `SecretClass` to provide certificates.
3	The reference to a `SecretClass`.
4	The `SecretClass` that is referenced by the `AuthenticationClass` in order to provide certificates.

If both spec.config.tls.secretClass and spec.config.clientAuthentication.authenticationClass are set, the authentication class will take precedence over the secret class. The cluster will be encrypted and authenticate only against the authentication class.

Due to a bug in ZooKeeper, the clientPort property in combination with client.portUnification=true is used instead of the secureClientPort. This means that unencrypted and unauthenticated access to the ZooKeeper cluster is still possible.

Monitoring

The managed ZooKeeper instances are automatically configured to export Prometheus metrics. See Monitoring for more details.

Configuration & Environment Overrides

The cluster definition also supports overriding configuration properties and environment variables, either per role or per role group, where the more specific override (role group) has precedence over the less specific one (role).

Overriding certain properties which are set by operator (such as the ports) can interfere with the operator and can lead to problems.

Configuration Properties

For a role or role group, at the same level of config, you can specify: configOverrides for the zoo.cfg. For example, if you want to set the 4lw.commands.whitelist to allow the ruok administrative command, it can be configured in the ZookeeperCluster resource like so:

servers:
  roleGroups:
    default:
      configOverrides:
        zoo.cfg:
          4lw.commands.whitelist: "srvr, ruok"
      replicas: 1

Just as for the config, it is possible to specify this at role level as well:

routers:
  configOverrides:
    zoo.cfg:
      4lw.commands.whitelist: "srvr, ruok"
  roleGroups:
    default:
      replicas: 1

All override property values must be strings.

For a full list of configuration options we refer to the Apache ZooKeeper Configuration Reference.

Environment Variables

In a similar fashion, environment variables can be (over)written. For example per role group:

servers:
  roleGroups:
    default:
      envOverrides:
        MY_ENV_VAR: "MY_VALUE"
      replicas: 1

or per role:

servers:
  envOverrides:
    MY_ENV_VAR: "MY_VALUE"
  roleGroups:
    default:
      replicas: 1

Storage for data volumes

You can mount volumes where data is stored by specifying PersistentVolumeClaims for each individual role group:

servers:
  roleGroups:
    default:
      config:
        resources:
          storage:
            data:
              capacity: 2Gi

In the above example, all ZooKeeper nodes in the default group will store data (the location of the property dataDir) on a 2Gi volume.

By default, in case nothing is configured in the custom resource for a certain role group, each Pod will have a 1Gi large local volume mount for the data location.

Memory requests

You can request a certain amount of memory for each individual role group as shown below:

servers:
  roleGroups:
    default:
      config:
        resources:
          memory:
            limit: '2Gi'

In this example, each ZooKeeper container in the "default" group will have a maximum of 2 gigabytes of memory. To be more precise, these memory limits apply to the containers running the ZooKeeper daemons but not to any sidecar containers that are part of the pod.

Setting this property will also automatically set the maximum Java heap size for the corresponding process to 80% of the available memory. Be aware that if the memory constraint is too low, the cluster might fail to start. If pods terminate with an 'OOMKilled' status and the cluster doesn’t start, try increasing the memory limit.

For more details regarding Kubernetes memory requests and limits see: Assign Memory Resources to Containers and Pods.

CPU requests

Similarly to memory resources, you can also configure CPU limits, as shown below:

servers:
  roleGroups:
    default:
      config:
        resources:
          cpu:
            max: '500m'
            min: '250m'

For more details regarding Kubernetes CPU limits see: Assign CPU Resources to Containers and Pods.