How the Kubernetes cluster works
How the cluster works
Understanding how a Kubernetes cluster works is about understanding the complete journey of a request and how all the components orchestrate to fulfill it. Let's walk through a typical workflow from the moment you submit a request to the cluster until it's running and being monitored.
Request Lifecycle: From User to Running Pod
1. Submission Phase: Your Request Enters the Cluster
When you submit a request to create a deployment, service, or any Kubernetes object, it goes through the following path:
kubectl/UI → API Server
You typically use kubectl (command-line tool) or a UI to submit YAML configurations. For example:
kubectl apply -f deployment.yaml
This request is sent to the API Server, which is the single entry point for ALL cluster operations. The API server acts as a gatekeeper, validating your request to ensure it meets Kubernetes requirements.
What the API Server does:
- Validates the request (syntax, schema, permissions)
- Authenticates the user
- Authorizes whether the user has permission to perform this action
- Stores the object definition in etcd (the cluster database)
2. Scheduling Phase: Deciding Where Pods Should Run
Once your deployment is stored in etcd, the Controller Manager springs into action:
Controller Manager → Scheduler → Worker Nodes
The Deployment Controller (part of the Controller Manager) notices the new deployment and creates Pod objects with the desired specifications. These Pods are now in a "pending" state because they haven't been assigned to any worker node yet.
The Scheduler continuously watches for unscheduled Pods. When it finds one, it performs the following analysis:
- Scans available worker nodes - which nodes are healthy and available?
- Evaluates resource requirements - does the Pod need specific CPU, memory, or other resources?
- Checks node constraints - are there affinity rules, taints, or tolerations that restrict where the Pod can run?
- Applies scheduling policies - distributes workload evenly across nodes
Once the Scheduler decides on a suitable node, it updates the Pod object with the node assignment, and the API Server stores this information in etcd.
3. Deployment Phase: Container Starts Running
Once a Pod is assigned to a worker node, the Kubelet on that node takes over:
Kubelet → Container Runtime → Running Container
The Kubelet (the agent on each worker node) performs these steps:
- Monitors etcd changes - it continuously watches for Pods assigned to its node
- Detects the Pod assignment - sees that a new Pod is now scheduled on its node
- Communicates with the Container Runtime - sends instructions to Docker/containerd to pull the image and start the container
- Starts the container - the actual application starts running inside the container
- Reports back to API Server - updates the Pod status to "Running" in etcd
4. Networking Phase: Making the Pod Accessible
Once containers are running, the Kube Proxy handles networking:
Kube Proxy → Network Rules → Traffic Routing
The Kube Proxy establishes network rules on each worker node to:
- Create virtual IP addresses (Service IPs) for Kubernetes Services
- Map traffic from Services to the actual Pods
- Perform load balancing - distributes incoming traffic across multiple Pods
- Enable Pod-to-Pod communication - allows Pods to discover and communicate with each other
When external traffic arrives or a Pod needs to communicate with another Pod, Kube Proxy's network rules ensure the traffic reaches the correct destination.
Monitoring & State Management: The Continuous Guardian
The journey doesn't end once a Pod is running. Kubernetes has built-in mechanisms to continuously monitor the cluster and maintain the desired state.
Continuous Monitoring Loop
Kubelet monitors Pods → Reports status → Controller Manager takes action → Desired state maintained
The Kubelet's Watchdog Role:
On each worker node, the Kubelet constantly monitors all running Pods:
- Checks if containers are healthy - are they still running? Are they responding to health checks?
- Reports Pod status to the API Server at regular intervals (heartbeat)
- Detects failures - if a container crashes, dies, or becomes unresponsive
- Restarts failed containers - automatically restarts a container if it fails (respecting restart policies)
Controller Manager's Reconciliation
The Controller Manager runs a reconciliation loop for each controller:
Check desired state → Compare with actual state → Take corrective action
For example, the Deployment Controller:
- Reads desired state from etcd - "I want 3 replicas of nginx Pod"
- Reads actual state from etcd - "Currently 2 replicas are running"
- Detects mismatch - "2 ≠ 3"
- Takes action - Creates a new Pod to reach the desired state of 3 replicas
- Repeats continuously - This reconciliation loop runs every few seconds
Common Scenarios Handled by Monitoring:
Scenario 1: A Pod Crashes
- Kubelet detects the container has exited
- Kubelet restarts the container (if restart policy allows)
- Reports the failure to API Server
- If the Pod is part of a deployment, the Deployment Controller may create a new replacement Pod
Scenario 2: A Worker Node Dies
- The Kubelet on the dead node stops reporting status
- After a timeout (typically 40 seconds), the API Server marks the node as "NotReady"
- The Node Controller detects the failed node
- Automatically reschedules all Pods that were running on that node to healthy nodes
- Maintains the desired number of replicas across the cluster
Scenario 3: Manual Scale Up
kubectl scale deployment nginx --replicas=5
- Your command updates the desired replicas in etcd
- Deployment Controller sees 3 current, 5 desired
- Creates 2 new Pods
- Scheduler assigns these Pods to suitable worker nodes
- Kubelets start the containers
- All Pods are now running
Scenario 4: Rolling Updates
kubectl set image deployment/nginx nginx=nginx:1.19
- Deployment Controller creates new Pods with the new image
- Old Pods are gradually terminated
- Health checks verify new Pods are working
- Traffic is only sent to healthy new Pods
- The update completes without downtime
The Complete Data Flow
Here's a visual representation of how all components interact:
Here's how all components interact in a data flow:
- User Request → Sent to API Server
- API Server → Validates and stores in etcd
- Controller Manager → Detects changes, takes action
- Scheduler → Assigns Pods to nodes
- Kubelet → Starts containers on assigned node
- Kube Proxy → Sets up networking rules
- Container Runtime → Runs the actual application
- Kubelet → Monitors and reports status
- API Server → Updates etcd with status
- Controller Manager → Continuously reconciles desired vs actual state
- Loop continues → Maintaining the desired state indefinitely
Why This Design is Powerful
This architecture provides:
- Self-healing: Failed Pods and nodes are automatically recovered
- Scalability: Easy to add or remove Pods without manual intervention
- High Availability: Multiple replicas ensure services stay available
- Declarative State: You declare what you want; Kubernetes maintains it
- Decoupling: Components work independently, making the system fault-tolerant
- Continuous Reconciliation: The desired state is constantly enforced, even after infrastructure changes
Every component has a specific role, and together they create a system that not only starts your applications but continuously maintains them in their desired state, handling failures, scaling, and updates automatically.