Kubernetes scheduling is a key component in ensuring that Pods in the cluster run on the appropriate nodes. By flexibly configuring scheduling policies, resource utilization, load balancing, and high availability can be improved.
In this article, we’ll delve into some practical Kubernetes scheduling scenarios and provide corresponding configuration examples and best practices.
1. Basic scenario – NodeSelector
Scenario description: We have some nodes marked with SSD hard drives, and we want to schedule Pods that require high-performance storage to these nodes.
Pod configuration:
apiVersion: v1
kind: Pod
metadata:
name: high-performance-pod
spec:
containers:
- name: my-container
image: my-image
nodeSelector:
disktype: ssd2. Advanced Scenario-Node Affinity
Scenario description: We want to schedule tasks that require GPUs to nodes with GPU labels.
Pod configuration:
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
spec:
containers:
- name: my-container
image: my-image
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: gpu
operator: In
values:
- "true"3. Resource allocation – Pod priority and pre-selection scheduling
Scenario description: To ensure that critical tasks have higher priority, we can define a PriorityClass and apply it to the Pod.
PriorityClass configuration:
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000
globalDefault: false
description: "High priority class"Pod configuration:
apiVersion: v1
kind: Pod
metadata:
name: high-priority-pod
spec:
containers:
- name: my-container
image: my-image
priorityClassName: high-priority4. Prevent Pods from running on the same node – Pod Anti-Affinity
Scenario description: Through Pod Anti-Affinity, we can ensure that Pods in the same group will not be scheduled to the same node to improve high availability.
Pod configuration:
apiVersion: v1
kind: Pod
metadata:
name: anti-affinity-pod
spec:
containers:
- name: my-container
image: my-image
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- web
topologyKey: kubernetes.io/hostname5. Multi-copy topology domain distribution – Pod Topology Spread
Scenario description: Ensure that multiple Pods of the same application are distributed in different topological domains to improve availability.
Deployment configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-deployment
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- "web"
topologyKey: "kubernetes.io/hostname"6. Node Taints and Pod Tolerations
Scenario description: Through the Taints of nodes, we can mark nodes, and only Pods with corresponding Tolerations can be scheduled to these nodes.
Node configuration:
apiVersion: v1
kind: Node
metadata:
name: node1
spec:
taints:
- key: special
value: unique
effect: NoSchedulePod configuration:
apiVersion: v1
kind: Pod
metadata:
name: toleration-pod
spec:
containers:
- name: my-container
image: my-image
tolerations:
- key: "special"
operator: "Equal"
value: "unique"
effect: "NoSchedule"7. Custom Scheduler – Custom scheduler
Scenario description: Customize the scheduler to achieve specific scheduling requirements, such as based on business rules or special hardware conditions.
Custom scheduler example:
- Create a custom scheduler plugin
// my_scheduler.go
package main
import (
"k8s.io/kubernetes/pkg/scheduler"
"k8s.io/kubernetes/pkg/scheduler/framework"
"k8s.io/kubernetes/pkg/scheduler/framework/plugins/defaultbinder"
"k8s.io/kubernetes/pkg/scheduler/framework/plugins/defaultpreemption"
"k8s.io/kubernetes/pkg/scheduler/framework/plugins/names"
)
const (
// YourSchedulerName is the name of your custom scheduler
YourSchedulerName = "my-scheduler"
)
// New initializes a new scheduler with your custom plugins
func New() *scheduler.Config {
return &scheduler.Config{
Client: scheduler.NewHTTPClient(),
SchedulerName: YourSchedulerName,
PercentageOfNodesToScore: 0.25,
Profiles: []scheduler.Profile{
{
Name: YourSchedulerName,
Plugins: []scheduler.Plugin{
defaultpreemption.Name: defaultpreemption.New,
defaultbinder.Name: defaultbinder.New,
names.NewNodeResourcesFit(),
names.NewNodePorts(),
names.NewNodeAffinity(YourSchedulerName),
names.NewNodeAffinityPriority(YourSchedulerName),
},
},
},
}
}
func main() {
// Use the New() function to create a new scheduler with your custom plugins
config := New()
command := app.NewSchedulerCommand(
// Use the WithConfig function to set your custom scheduler configuration
app.WithConfig(config),
)
f := command.Flags()
f.AddGoFlagSet(flag.CommandLine)
if err := command.Execute(); err != nil {
os.Exit(1)
}
}2. Compile and run a custom scheduler
go build my_scheduler.go
./my_scheduler8. Pod Priority and Preemption – Pod priority and preemption
Scenario description: By setting the priority and preemption policy of the Pod, ensure that key tasks are prioritized.
Pod configuration:
apiVersion: v1
kind: Pod
metadata:
name: priority-pod
spec:
containers:
- name: my-container
image: my-image
priorityClassName: high-priority9. Resource Limits and Requests- Resource Limits and Requests
Scenario description: By setting resource limits and requests for Pods, the scheduler can better optimize resource utilization.
Pod configuration:
apiVersion: v1
kind: Pod
metadata:
name: resource-pod
spec:
containers:
- name: my-container
image: my-image
resources:
limits:
cpu: "2"
memory: "1Gi"
requests:
cpu: "1"
memory: "500Mi"10. Affinity and Anti-Affinity Rules – Affinity and Anti-Affinity Rules
Scenario description: Use affinity and anti-affinity rules to ensure that Pods are on specific nodes or avoid being scheduled to the same node as other Pods.
Pod configuration:
apiVersion: v1
kind: Pod
metadata:
name: affinity-pod
spec:
containers:
- name: my-container
image: my-image
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: security
operator: In
values:
- "high"
topologyKey: kubernetes.io/hostname11. Pod Disruption Budget – Pod disruption budget
Scenario description: Use the Pod interruption budget to limit the number of Pods allowed to be interrupted during maintenance to ensure system stability.
PodDisruptionBudget configuration:
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: web-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app: web12. Horizontal Pod Autoscaler – Horizontal expander
Scenario description: Use horizontal expanders to automatically adjust the number of Pods based on CPU usage or other metrics to meet application needs.
HorizontalPodAutoscaler configuration:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: web-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web-deployment
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 7013. Pod Overhead – Pod overhead
Scenario description: By setting the Pod overhead, tell the scheduler to consider the additional resources required by the Pod to avoid scheduling too many Pods on the node.
Pod configuration:
apiVersion: v1
kind: Pod
metadata:
name: overhead-pod
spec:
containers:
- name: my-container
image: my-image
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
overhead:
podFixed: 100Mi
ephemeral-storage: 1Gi14. Node Local DNS Cache – Node local DNS cache
Scenario description: Enable local DNS caching on the node to improve DNS query performance.
kubelet configuration:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDomain: cluster.local
featureGates:
CoreDNSLocalCache: true15. Pod Priority Class – Pod priority class
Scenario description: Use Pod priority classes to divide Pods into different priorities to ensure that key tasks are scheduled first.
PriorityClass configuration:
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000
globalDefault: false
description: "High priority class"
preemptionPolicy: PreemptLowerPriority🔥 [20% Off] Linux Foundation Coupon Code for 2024 DevOps & Kubernetes Exam Vouchers (CKAD , CKA and CKS) [RUNNING NOW ]
Save 20% on all the Linux Foundation training and certification programs. This is a limited-time offer for this month. This offer is applicable for CKA, CKAD, CKS, KCNA, LFCS, PCA FINOPS, NodeJS, CHFA, and all the other certification, training, and BootCamp programs.
Check our last updated Kubernetes Exam Guides (CKAD , CKA , CKS) :
Conclusion
These scenarios cover Kubernetes scheduling practical cases from basic to advanced. According to your needs, you can choose the appropriate scenario for configuration to optimize the resource utilization and performance of the cluster.
In actual applications, adjust the configuration according to specific needs to ensure that the scheduler’s strategy meets business and performance requirements.
Trivy is a powerful and easy-to-use tool for scanning Docker images, helping you maintain a secure Kubernetes environment.
Check last Kubernetes Exams (CKAD , CKA , CKS) Coupon Page to get discounts on certification registration.
