Complete Practical Example of Kubernetes CRD, Operator & Kubebuilder

In this blog, we will delve into Kubernetes CRD (Custom Resource Definition) and Operator with hands-on examples.

When we build projects, we often hope to have a useful framework that can provide a series of tools to help developers create, test, and deploy more easily. There is such a framework for the scenarios of CRD and Operator, and it’s called Kubebuilder.

Kubernetes CRD

There are several built-in resources in Kubernetes, such as pods, deployments, configmaps, services, etc., which are managed by the internal components of K8s.

In addition to these built-in resources, K8s also provides another way for users to customize resources at will, known as CustomResourceDefinitions (CRD). For instance, with CRD, I can define resources like “mypod,” “myjob,” “myanything,” and so on.

There are several built-in resources in Kubernetes, such as pods, deployments, configmaps, services, etc., which are managed by the internal components of K8s.

In addition to these built-in resources, K8s also provides another way for users to customize resources at will, known as CustomResourceDefinitions (CRD). For instance, with CRD, I can define resources like “mypod,” “myjob,” “myanything,” and so on.

It should be noted that the term “CRD” has different meanings in different contexts.

Sometimes, it may only refer to “CustomResourceDefinitions,” which is a specific resource in K8s, and at other times, it may refer to the custom resources created by users using CRD.

In a narrow sense, “CRD” (full name CustomResourceDefinitions) is a special built-in resource in K8s, through which we can create other customized resources.

For example, we can use CRD to create a CronTab resource called:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # Name must match <plural>.<group>
  name: crontabs.stable.example.com
spec:
  # group name, used for REST API: /apis/<group>/<version>
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            #Define attributes
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
  # Scope can be Namespaced or Cluster
  scope: Namespaced
  names:
    # # Plural name, used in URL: /apis/<group>/<version>/<plural>
    plural: crontabs
    # Singular name, can be used in CLI
    singular: crontab
    # # Camel case singular, used for resource lists
    kind: CronTab
    # Name abbreviation, can be used in CLI
    shortNames:
    - ct

Once we apply this YAML file, our custom resource “CronTab” will be registered in K8s.

At this point, we can operate this custom resource at will, for example, using the file “my-crontab.yaml”:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image

Execute

kubectl apply -f my-crontab.yaml 

to create our customized CronTab, and execute

kubectl get crontab

to query our customized CronTab list.

The advantage of using CRD custom resources is that we do not need to worry about the data storage of custom resources, nor do we need to implement an additional HTTP server to expose the API interfaces for operating these custom resources. Kubernetes has already handled these aspects for us.

We only need to interact with custom resources in a manner similar to other built-in resources.

Only CRD is often not enough. For example, above we created kubectl apply -f my-crontab.yaml a crontab custom resource, but this crontab will not have any execution content (will not run any programs), and in many scenarios we hope that the custom resource can be executed.

This is when we need Operator it.

Operator

An Operator is essentially a custom resource controller. Its purpose is to monitor changes in custom resources and then execute specific operations in response.

For instance, upon detecting the creation of a custom resource, an Operator can read the properties of the custom resource, create a pod to execute a specific program, and associate the pod with the custom resource object

In what form does an Operator exist? In reality, it is similar to an ordinary service; it can be a deployment or a statefulSet.

In essence, what is commonly referred to as the Operator pattern is this model: CRD + custom controller.

Kubebuilder

When we embark on projects, we often seek a helpful framework that offers a set of tools facilitating developers in creating, testing, and deploying more efficiently. For scenarios involving CRD and Operator, such a framework exists – Kubebuilder.

Next, I will use Kubebuilder to create a small project. This project will generate a custom resource, “Foo,” and the controller will monitor changes to this resource, printing out the updates.

1. Installation

# download kubebuilder and install locally.
curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"

chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

2. Create a test directory

mkdir kubebuilder-test
cd kubebuilder-test

3. Initialise project

kubebuilder init --domain mytest.domain --repo mytest.domain/foo

4. Define CRD

Let’s suppose we want to define a CRD in the following format:

apiVersion: "mygroup.mytest.domain/v1"
kind: Foo
metadata:
  name: xxx
spec:
  image: image
  msg: message

Then we need to create a CRD (essentially creating an API):

kubebuilder create api --group mygroup --version v1 --kind Foo

After execution, enter y to confirm the generation, and then kubebuilder will automatically create some directories and files for us, including:

• api/v1/foo_types.go This CRD is defined in the file (also API).
• internal/controllers/foo_controller.go The file is the business logic that controls the CRD.

Since the automatically generated file is only a basic framework, we need to modify it accordingly to our own needs.

a. Modify the structure of CRD in code

First, modify api/v1/foo_types.go and adjust the structure of the CRD

// FooSpec defines the desired state of Foo
type FooSpec struct {
 Image string `json:"image"`
 Msg   string `json:"msg"`
}

// FooStatus defines the observed state of Foo
type FooStatus struct {
 PodName string `json:"podName"`
}

b. Automatically generate CRD yaml through commands

After executing the command kubebuilder make manifests , a file mygroup.mytest.domain_foos.yaml will be generated in the directory. config/crd/bases

This file is the yaml file in which we define CRD:

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.13.0
  name: foos.mygroup.mytest.domain
spec:
  group: mygroup.mytest.domain
  names:
    kind: Foo
    listKind: FooList
    plural: foos
    singular: foo
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: Foo is the Schema for the foos API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: FooSpec defines the desired state of Foo
            properties:
              image:
                type: string
              msg:
                type: string
            required:
            - image
            - msg
            type: object
          status:
            description: FooStatus defines the observed state of Foo
            properties:
              podName:
                type: string
            required:
            - podName
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

The specific content of the instruction execution is defined in the file Makefile :

.PHONY: manifests
manifests: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
 $(CONTROLLER_GEN) rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

From this example, it becomes evident that the tool Kubebuilder is employed to utilize controller-gen for scanning the code.

It searches for comments in a specific format (such as //+kubebuilder:…) and subsequently generates a CRD YAML file.

5. Supplement controller logic

Suppose we intend to monitor the custom resource “Foo” created by the user and print out its properties.

a. Modify the controller to supplement business logic

Modify the file internal/controllers/foo_controller.go to supplement our own business logic, as follows:

func (r *FooReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 l := log.FromContext(ctx)

 // Supplementary business logic
 foo := &mygroupv1.Foo{}
 if err := r.Get(ctx, req.NamespacedName, foo); err != nil {
  l.Error(err, "unable to fetch Foo")
  return ctrl.Result{}, client.IgnoreNotFound(err)
 }

 //Print Foo property
 l.Info("Received Foo", "Image", foo.Spec.Image, "Msg", foo.Spec.Msg)

 return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *FooReconciler) SetupWithManager(mgr ctrl.Manager) error {
 return ctrl.NewControllerManagedBy(mgr).
  For(&mygroupv1.Foo{}).
  Complete(r)

b. Carry out testing

Note: The test requires a local or remote Kubernetes cluster environment, and it will use the same environment as the current kubectl by default.

Execute make install to register the CRD. In the Makefile, you can see that it actually executes the following instructions:

.PHONY: install
install: manifests kustomize ## Install CRDs into the K8s cluster specified in ~/.kube/config.
 $(KUSTOMIZE) build config/crd | $(KUBECTL) apply -f -

Execute make run to run the controller. In the Makefile, you can see that it actually executes the following instructions:

.PHONY: run
run: manifests generate fmt vet ## Run a controller from your host.
 go run ./cmd/main.go

Then you can see the following output:

...
go fmt ./...
go vet ./...
go run ./cmd/main.go
2024-01-19T15:14:18+08:00       INFO    setup   starting manager
2024-01-19T15:14:18+08:00       INFO    controller-runtime.metrics      Starting metrics server
2024-01-19T15:14:18+08:00       INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-01-19T15:14:18+08:00       INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-01-19T15:14:18+08:00       INFO    Starting EventSource    {"controller": "foo", "controllerGroup": "mygroup.mytest.domain", "controllerKind": "Foo", "source": "kind source: *v1.Foo"}
2024-01-19T15:14:18+08:00       INFO    Starting Controller     {"controller": "foo", "controllerGroup": "mygroup.mytest.domain", "controllerKind": "Foo"}
2024-01-19T15:14:19+08:00       INFO    Starting workers        {"controller": "foo", "controllerGroup": "mygroup.mytest.domain", "controllerKind": "Foo", "worker count": 1}

Let’s submit a foo.yaml and try:

apiVersion: "mygroup.mytest.domain/v1"
kind: Foo
metadata:
  name: test-foo
spec:
  image: test-image
  msg: test-message

After executing kubectl apply -f foo.yaml, you will observe “foo” printed out in the controller’s output.

2024-01-19T15:16:00+08:00       INFO    Received Foo    {"controller": "foo", "controllerGroup": "mygroup.mytest.domain", "controllerKind": "Foo", "Foo": {"name":"test-foo","namespace":"aries"}, "namespace": "aries", "name": "test-foo", "reconcileID": "8dfd629e-3081-4d40-8fc6-bcc3e81bbb39", "Image": "test-image", "Msg": "test-message"}

This is a simple example of using Kubebuilder.

🔥 [20% Off] Linux Foundation Coupon Code for 2024 DevOps & Kubernetes Exam Vouchers (CKAD , CKA and CKS) [RUNNING NOW ]

Save 20% on all the Linux Foundation training and certification programs. This is a limited-time offer for this month. This offer is applicable for CKA, CKAD, CKS, KCNA, LFCS, PCA FINOPS, NodeJS, CHFA, and all the other certification, training, and BootCamp programs.

Check our last updated Kubernetes Exam Guides (CKAD , CKA , CKS) :

Conclusion

Kubernetes’ CRD and Operator mechanism provide users with powerful scalability. CRD allows users to customize resources, and Operators can manage these resources.

It is this extension mechanism that provides the Kubernetes ecosystem with great flexibility and adaptability, enabling it to be more widely used in various scenarios.

References:

https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/

https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

https://book.kubebuilder.io/introduction