Kubernetes operators guide

: 10 version: type: string enum: ["1.0", "1.1", "2.0"] status: type: object properties: phase: type: string nodes: type: array items: type: string scope: Namespaced names: plural: mydatabases singular: mydatabase kind: MyDatabase

#### Step 2: Implement the Controller Logic

Using `kubebuilder` (the industry standard framework for Go-based operators), the controller watches the CR and owned resources.

**Architecture Decision:** Use the Controller Runtime library. It provides a high-level abstraction for caching, client interactions, and event handling, reducing boilerplate and preventing common race conditions.

```go
package controllers

import (
	"context"
	"reflect"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	examplev1 "github.com/yourorg/myoperator/api/v1"
)

type MyDatabaseReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

// Reconcile is the core loop. It must be idempotent.
func (r *MyDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	// 1. Fetch the Custom Resource
	var mydb examplev1.MyDatabase
	if err := r.Get(ctx, req.NamespacedName, &mydb); err != nil {
		if errors.IsNotFound(err) {
			// Resource deleted. Handle finalizers if necessary.
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, err
	}

	// 2. Define the desired State (e.g., a StatefulSet)
	sts := &appsv1.StatefulSet{}
	err := r.Get(ctx, client.ObjectKey{
		Name:      mydb.Name,
		Namespace: mydb.Namespace,
	}, sts)

	if err != nil && errors.IsNotFound(err) {
		// Create the StatefulSet if it doesn't exist
		sts = r.statefulSetForCR(&mydb)
		if err := ctrl.SetControllerReference(&mydb, sts, r.Scheme); err != nil {
			return ctrl.Result{}, err
		}
		logger.Info("Creating StatefulSet", "name", sts.Name)
		return ctrl.Result{}, r.Create(ctx, sts)
	} else if err != nil {
		return ctrl.Result{}, err
	}

	// 3. Update StatefulSet if CR changed
	// Compare spec fields. If different, update.
	// This logic ensures convergence.
	if sts.Spec.Replicas == nil || *sts.Spec.Replicas != int32(mydb.Spec.Size) {
		sts.Spec.Replicas = &[]int32{int32(mydb.Spec.Size)}[0]
		logger.Info("Updating StatefulSet replicas", "replicas", mydb.Spec.Size)
		return ctrl.Result{}, r.Update(ctx, sts)
	}

	// 4. Update Status
	// Reflect actual state back to the CR
	if mydb.Status.Phase != "Running" {
		mydb.Status.Phase = "Running"
		mydb.Status.Nodes = []string{"node-0", "node-1"} // Example
		if err := r.Status().Update(ctx, &mydb); err != nil {
			return ctrl.Result{}, err
		}
	}

	return ctrl.Result{}, nil
}

func (r *MyDatabaseReconciler) statefulSetForCR(cr *examplev1.MyDatabase) *appsv1.StatefulSet {
	replicas := int32(cr.Spec.Size)
	return &appsv1.StatefulSet{
		ObjectMeta: metav1.ObjectMeta{
			Name:      cr.Name,
			Namespace: cr.Namespace,
		},
		Spec: appsv1.StatefulSetSpec{
			Replicas: &replicas,
			// ... container spec, volume claims, etc.
		},
	}
}

func (r *MyDatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&examplev1.MyDatabase{}).
		Owns(&appsv1.StatefulSet{}).
		Owns(&corev1.Service{}).
		Complete(r)
}

Key Implementation Patterns

1. Idempotency: The Reconcile function must be safe to run multiple times. It should not assume the current state; it must fetch and compare.
2. Owner References: Use ctrl.SetControllerReference to link child resources to the CR. This enables automatic garbage collection when the CR is deleted.
3. Finalizers: Implement finalizers to handle cleanup logic (e.g., deleting persistent volumes or external cloud resources) before the CR is removed.
4. Status Updates: Always update the status subresource. This provides observability into the operator's view of the system.

Pitfall Guide

Production operators fail due to subtle implementation errors. The following pitfalls are derived from real-world operator maintenance experience.

1. Non-Idempotent Reconcile Loops:

   • Mistake: Modifying resources based on assumptions or performing actions that change state without checking current state first.
   • Impact: Resource thrashing, excessive API server load, and inconsistent cluster state.
   • Fix: Always Get the resource before Update. Compare the desired state with the retrieved state.

2. Blocking the Reconcile Loop:

   • Mistake: Performing long-running operations (e.g., waiting for a backup to complete, sleeping) inside Reconcile.
   • Impact: The controller becomes unresponsive to other events. Other CRs are starved.
   • Fix: Use ctrl.Result{RequeueAfter: 5 * time.Minute} for async tasks. Return immediately and let the loop re-trigger.

3. Ignoring RBAC Scopes:

   • Mistake: Granting the operator cluster-admin or wildcard permissions.
   • Impact: Security vulnerabilities. If the operator is compromised, the attacker gains full cluster access.
   • Fix: Use minimal RBAC. Grant permissions only for the specific resources the operator manages. Use kubebuilder:rbac markers to generate precise roles.

4. Missing Finalizers for Cleanup:

   • Mistake: Deleting the CR leaves orphaned resources (PVCs, external load balancers, cloud instances).
   • Impact: Resource leaks, billing costs, and "zombie" infrastructure.
   • Fix: Add a finalizer to the CR. When a delete timestamp is detected, execute cleanup logic, then remove the finalizer to allow garbage collection.

5. Coupling Operator Logic to Specific Versions:

   • Mistake: Hardcoding logic that only works for version 1.0 of the managed application.
   • Impact: Operator breaks during upgrades or requires frequent operator releases.
   • Fix: Design the CRD schema to be version-agnostic where possible. Implement upgrade logic that inspects the spec.version and applies migration steps dynamically.

6. Lack of Integration Testing:

   • Mistake: Testing only with kubectl apply in a live cluster.
   • Impact: Flaky behavior in production. Race conditions are hard to reproduce manually.
   • Fix: Use envtest from controller-runtime. This spins up a local etcd and API server for fast, deterministic unit and integration tests.

7. Status Blindness:

   • Mistake: The operator updates resources but never updates the CR status.
   • Impact: Users cannot see the state of their application. kubectl get mydb shows no useful information.
   • Fix: Implement a status writer. Update conditions and phases based on the health of child resources.

Production Bundle

Action Checklist

• Schema Validation: Define strict openAPIV3Schema in the CRD to prevent invalid configurations from reaching the controller.
• Finalizers: Implement finalizers for all external resources or persistent data that requires cleanup.
• RBAC Minimization: Review RBAC markers. Ensure the operator only has permissions for resources it creates or manages.
• Leader Election: Enable leader election for HA deployments. This prevents multiple operator pods from reconciling simultaneously.
• Metrics Integration: Expose Prometheus metrics for reconciliation duration, error counts, and custom application metrics.
• Integration Tests: Write tests using envtest covering create, update, delete, and error scenarios.
• Documentation: Document the CRD spec, including all fields, defaults, and status conditions for end-users.

Decision Matrix

Use this matrix to determine if an Operator is the right tool for your workload.

Scenario	Recommended Approach	Why	Cost Impact
Stateless Microservice	Deployment + Helm	Operators add unnecessary complexity for stateless apps. Helm handles templating and upgrades sufficiently.	Low
Complex Stateful App (DB/Queue)	Kubernetes Operator	Requires automated backups, scaling, and self-healing. Operators encode this logic reliably.	Medium (Dev time) / Low (Ops time)
Multi-Cluster Management	Cluster API / Fleet Manager	Operators manage single-cluster scope. Multi-cluster requires federation or GitOps tools.	High
Legacy Migration	Operator + Sidecar	Wrap legacy binaries in containers and use an Operator to manage lifecycle if stateful logic is complex.	High
Configuration Management	GitOps (ArgoCD/Flux)	Operators are for runtime logic. GitOps is for declarative state synchronization. Use GitOps to deploy Operators.	Low

Configuration Template

A production-ready CRD snippet with validation and subresources.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mydatabases.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {} # Enables status subresource
        scale:
          specReplicasPath: .spec.size
          statusReplicasPath: .status.replicas
      additionalPrinterColumns:
        - name: Phase
          type: string
          description: Current phase
          jsonPath: .status.phase
        - name: Size
          type: integer
          description: Cluster size
          jsonPath: .spec.size
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["size", "version"]
              properties:
                size:
                  type: integer
                  minimum: 1
                  maximum: 50
                  description: Number of nodes in the cluster.
                version:
                  type: string
                  description: Application version.
                storage:
                  type: object
                  properties:
                    size:
                      type: string
                      pattern: '^\d+(Gi|Ti)$'
                    class:
                      type: string
            status:
              type: object
              properties:
                phase:
                  type: string
                  enum: ["Creating", "Running", "Scaling", "Failed"]
                replicas:
                  type: integer
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      type:
                        type: string
                      status:
                        type: string
                      reason:
                        type: string
  scope: Namespaced
  names:
    plural: mydatabases
    singular: mydatabase
    kind: MyDatabase
    shortNames:
      - mdb

Quick Start Guide

Get a basic operator running in under 5 minutes using the Operator SDK.

1. Initialize Project:

   operator-sdk init --domain example.com --repo github.com/myorg/myoperator
   

2. Create API:

   operator-sdk create api --group example --version v1 --kind MyDatabase --resource --controller
   

3. Edit Controller: Open internal/controller/mydatabase_controller.go. Implement the Reconcile logic to create a Deployment based on the CR spec. Add RBAC markers at the top of the file.

4. Run Locally:

   make install
   make run
   

   The operator runs locally, connecting to your active kubeconfig. This allows rapid iteration.

5. Deploy Sample CR:

   kubectl apply -f config/samples/example_v1_mydatabase.yaml
   

   Verify the operator creates the managed resources and updates the status.

Kubernetes Operators represent the maturation of cloud-native operations. By encoding domain knowledge into the control plane, teams achieve autonomy, reliability, and scalability that static manifests cannot provide. The initial investment in operator development yields compounding returns through reduced operational toil and increased system resilience.