[!TIP] Ongoing and occasional updates and improvements.
apply settings belongs with worker profile
Openshift4 supports worker profile, which is a set of configurations that can be applied to the worker nodes. The worker profile contains some key parameters that can be adjusted to meet the specific requirements of the worker nodes. But the parameters is predefined, if you want to adjust the parameters, you need to find a way to do it.
- https://docs.openshift.com/container-platform/4.15/nodes/clusters/nodes-cluster-worker-latency-profiles.html
Openshift4 developers listed the cluster operator around the worker profile. This gives us a clue on how to do that.
- https://github.com/openshift/enhancements/blob/master/enhancements/worker-latency-profile/worker-latency-profile.md
[!WARNING] The method list in this doc, will break the supportability of your openshift cluster, like your cluster can not upgrade without rollback the configurations. Please use it with caution.
Contact with your redhat GPS, TAM, CEE or other redhat support team before apply the changes.
There is redhat cases about the same topic.
- https://access.redhat.com/support/cases/#/case/03903659
for KubeletConfig
Our target is to set the node-status-update-frequency to 5s
Here is the official document to change the kubeletconfig.
- https://docs.openshift.com/container-platform/4.15/post_installation_configuration/machine-configuration-tasks.html#create-a-kubeletconfig-crd-to-edit-kubelet-parameters_post-install-machine-configuration-tasks
# before apply change, check on the worker node
cat /etc/kubernetes/kubelet.conf | grep node
# "nodeStatusUpdateFrequency": "10s",
# "nodeStatusReportFrequency": "5m0s",
oc get mcp
# NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
# master rendered-master-3b06c8a6cdbb7a48ab7c3f43f08990bd True False False 1 1 1 0 16d
# worker rendered-worker-eeb0c5ee23e3b38d342372cffde47bfb True False False 0 0 0 0 16d
oc get node
# NAME STATUS ROLES AGE VERSION
# master-01-demo Ready control-plane,master,worker 16d v1.28.11+add48d0
# label the machineconfigpool
oc label machineconfigpool master custom-kubelet=set-duration
# oc label machineconfigpool master custom-kubelet-
cat << EOF > ${BASE_DIR}/data/install/kubeletconfig-profile.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
name: set-duration
spec:
machineConfigPoolSelector:
matchLabels:
custom-kubelet: set-duration
kubeletConfig:
nodeStatusUpdateFrequency: 5s
EOF
oc apply -f ${BASE_DIR}/data/install/kubeletconfig-profile.yaml
# oc delete -f ${BASE_DIR}/data/install/kubeletconfig-profile.yaml
# this will trigger ocp node reboot, after reboot,
# check node's kubelet.conf
cat kubelet.conf | grep node
# "nodeStatusUpdateFrequency": "5s",
# "nodeStatusReportFrequency": "5m0s",for KubeControllerManager
Our target is to set the node-monitor-grace-period to 20s
Here is the kcs from redhat, tell us how to do it.
- https://access.redhat.com/solutions/5686011
# before apply patch, check the kube-controller-manager pod
POD_NAME=`oc get pod -n openshift-kube-controller-manager | grep kube-controller-manager | awk '{print $1}'`
oc exec -n openshift-kube-controller-manager $POD_NAME -- ps -ef
# UID PID PPID C STIME TTY TIME CMD
# root 1 0 6 00:56 ? 00:00:38 kube-controller-manager --openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml --kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --authentication-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --authorization-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt --requestheader-client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt -v=2 --tls-cert-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt --tls-private-key-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key --allocate-node-cidrs=false --cert-dir=/var/run/kubernetes --cluster-cidr=10.132.0.0/14 --cluster-name=demo-01-rhsys-wkmd8 --cluster-signing-cert-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.crt --cluster-signing-duration=720h --cluster-signing-key-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.key --configure-cloud-routes=false --controllers=* --controllers=-bootstrapsigner --controllers=-tokencleaner --controllers=-ttl --enable-dynamic-provisioning=true --feature-gates=AdminNetworkPolicy=false --feature-gates=AlibabaPlatform=true --feature-gates=AutomatedEtcdBackup=false --feature-gates=AzureWorkloadIdentity=true --feature-gates=BuildCSIVolumes=true --feature-gates=CSIDriverSharedResource=false --feature-gates=CloudDualStackNodeIPs=true --feature-gates=ClusterAPIInstall=false --feature-gates=DNSNameResolver=false --feature-gates=DisableKubeletCloudCredentialProviders=false --feature-gates=DynamicResourceAllocation=false --feature-gates=EventedPLEG=false --feature-gates=ExternalCloudProviderAzure=true --feature-gates=ExternalCloudProviderExternal=true --feature-gates=ExternalCloudProviderGCP=true --feature-gates=GCPClusterHostedDNS=false --feature-gates=GCPLabelsTags=false --feature-gates=GatewayAPI=false --feature-gates=InsightsConfigAPI=false --feature-gates=InstallAlternateInfrastructureAWS=false --feature-gates=MachineAPIOperatorDisableMachineHealthCheckController=false --feature-gates=MachineAPIProviderOpenStack=false --feature-gates=MachineConfigNodes=false --feature-gates=ManagedBootImages=false --feature-gates=MaxUnavailableStatefulSet=false --feature-gates=MetricsServer=false --feature-gates=MixedCPUsAllocation=false --feature-gates=NetworkLiveMigration=true --feature-gates=NodeSwap=false --feature-gates=OnClusterBuild=false --feature-gates=OpenShiftPodSecurityAdmission=false --feature-gates=PrivateHostedZoneAWS=true --feature-gates=RouteExternalCertificate=false --feature-gates=SignatureStores=false --feature-gates=SigstoreImageVerification=false --feature-gates=VSphereControlPlaneMachineSet=false --feature-gates=VSphereStaticIPs=false --feature-gates=ValidatingAdmissionPolicy=false --flex-volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --kube-api-burst=300 --kube-api-qps=150 --leader-elect-renew-deadline=12s --leader-elect-resource-lock=leases --leader-elect-retry-period=3s --leader-elect=true --pv-recycler-pod-template-filepath-hostpath=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --pv-recycler-pod-template-filepath-nfs=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --root-ca-file=/etc/kubernetes/static-pod-resources/configmaps/serviceaccount-ca/ca-bundle.crt --secure-port=10257 --service-account-private-key-file=/etc/kubernetes/static-pod-resources/secrets/service-account-private-key/service-account.key --service-cluster-ip-range=172.22.0.0/16 --use-service-account-credentials=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 --tls-min-version=VersionTLS12
# root 47 0 0 01:04 pts/0 00:00:00 sh
# root 59 0 0 01:06 ? 00:00:00 ps -eflet’s see the parameters of kube-controller-manager clearly
kube-controller-manager –openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml –kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig –authentication-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig –authorization-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig –client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt –requestheader-client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt -v=2 –tls-cert-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt –tls-private-key-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key –allocate-node-cidrs=false –cert-dir=/var/run/kubernetes –cluster-cidr=10.132.0.0/14 –cluster-name=demo-01-rhsys-wkmd8 –cluster-signing-cert-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.crt –cluster-signing-duration=720h –cluster-signing-key-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.key –configure-cloud-routes=false –controllers=* –controllers=-bootstrapsigner –controllers=-tokencleaner –controllers=-ttl –enable-dynamic-provisioning=true –feature-gates=AdminNetworkPolicy=false –feature-gates=AlibabaPlatform=true –feature-gates=AutomatedEtcdBackup=false –feature-gates=AzureWorkloadIdentity=true –feature-gates=BuildCSIVolumes=true –feature-gates=CSIDriverSharedResource=false –feature-gates=CloudDualStackNodeIPs=true –feature-gates=ClusterAPIInstall=false –feature-gates=DNSNameResolver=false –feature-gates=DisableKubeletCloudCredentialProviders=false –feature-gates=DynamicResourceAllocation=false –feature-gates=EventedPLEG=false –feature-gates=ExternalCloudProviderAzure=true –feature-gates=ExternalCloudProviderExternal=true –feature-gates=ExternalCloudProviderGCP=true –feature-gates=GCPClusterHostedDNS=false –feature-gates=GCPLabelsTags=false –feature-gates=GatewayAPI=false –feature-gates=InsightsConfigAPI=false –feature-gates=InstallAlternateInfrastructureAWS=false –feature-gates=MachineAPIOperatorDisableMachineHealthCheckController=false –feature-gates=MachineAPIProviderOpenStack=false –feature-gates=MachineConfigNodes=false –feature-gates=ManagedBootImages=false –feature-gates=MaxUnavailableStatefulSet=false –feature-gates=MetricsServer=false –feature-gates=MixedCPUsAllocation=false –feature-gates=NetworkLiveMigration=true –feature-gates=NodeSwap=false –feature-gates=OnClusterBuild=false –feature-gates=OpenShiftPodSecurityAdmission=false –feature-gates=PrivateHostedZoneAWS=true –feature-gates=RouteExternalCertificate=false –feature-gates=SignatureStores=false –feature-gates=SigstoreImageVerification=false –feature-gates=VSphereControlPlaneMachineSet=false –feature-gates=VSphereStaticIPs=false –feature-gates=ValidatingAdmissionPolicy=false –flex-volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec –kube-api-burst=300 –kube-api-qps=150 –leader-elect-renew-deadline=12s –leader-elect-resource-lock=leases –leader-elect-retry-period=3s –leader-elect=true –pv-recycler-pod-template-filepath-hostpath=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml –pv-recycler-pod-template-filepath-nfs=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml –root-ca-file=/etc/kubernetes/static-pod-resources/configmaps/serviceaccount-ca/ca-bundle.crt –secure-port=10257 –service-account-private-key-file=/etc/kubernetes/static-pod-resources/secrets/service-account-private-key/service-account.key –service-cluster-ip-range=172.22.0.0/16 –use-service-account-credentials=true –tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 –tls-min-version=VersionTLS12
There is no node-monitor-grace-period in the output, so we need to check the help, whether it is supported.
oc exec -n openshift-kube-controller-manager $POD_NAME -- kube-controller-manager --help | grep node-monitor-grace-period
# --node-monitor-grace-period duration Amount of time which we allow running Node to be unresponsive before marking it unhealthy. Must be N times more than kubelet's nodeStatusUpdateFrequency, where N means number of retries allowed for kubelet to post node status. (default 40s)Yes, it is supported. Let’s manually patch the configuration
Apply the patch below.
apiVersion: operator.openshift.io/v1
kind: KubeControllerManager
metadata:
name: cluster
....
spec:
....
unsupportedConfigOverrides:
extendedArguments:
node-monitor-grace-period:
- "20s"After manually patch the config, check the kube-controller-manager pod again
oc exec -n openshift-kube-controller-manager $POD_NAME -- ps -ef
# UID PID PPID C STIME TTY TIME CMD
# root 1 0 3 01:14 ? 00:00:07 kube-controller-manager --openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml --kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --authentication-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --authorization-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt --requestheader-client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt -v=2 --tls-cert-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt --tls-private-key-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key --allocate-node-cidrs=false --cert-dir=/var/run/kubernetes --cluster-cidr=10.132.0.0/14 --cluster-name=demo-01-rhsys-wkmd8 --cluster-signing-cert-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.crt --cluster-signing-duration=720h --cluster-signing-key-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.key --configure-cloud-routes=false --controllers=* --controllers=-bootstrapsigner --controllers=-tokencleaner --controllers=-ttl --enable-dynamic-provisioning=true --feature-gates=AdminNetworkPolicy=false --feature-gates=AlibabaPlatform=true --feature-gates=AutomatedEtcdBackup=false --feature-gates=AzureWorkloadIdentity=true --feature-gates=BuildCSIVolumes=true --feature-gates=CSIDriverSharedResource=false --feature-gates=CloudDualStackNodeIPs=true --feature-gates=ClusterAPIInstall=false --feature-gates=DNSNameResolver=false --feature-gates=DisableKubeletCloudCredentialProviders=false --feature-gates=DynamicResourceAllocation=false --feature-gates=EventedPLEG=false --feature-gates=ExternalCloudProviderAzure=true --feature-gates=ExternalCloudProviderExternal=true --feature-gates=ExternalCloudProviderGCP=true --feature-gates=GCPClusterHostedDNS=false --feature-gates=GCPLabelsTags=false --feature-gates=GatewayAPI=false --feature-gates=InsightsConfigAPI=false --feature-gates=InstallAlternateInfrastructureAWS=false --feature-gates=MachineAPIOperatorDisableMachineHealthCheckController=false --feature-gates=MachineAPIProviderOpenStack=false --feature-gates=MachineConfigNodes=false --feature-gates=ManagedBootImages=false --feature-gates=MaxUnavailableStatefulSet=false --feature-gates=MetricsServer=false --feature-gates=MixedCPUsAllocation=false --feature-gates=NetworkLiveMigration=true --feature-gates=NodeSwap=false --feature-gates=OnClusterBuild=false --feature-gates=OpenShiftPodSecurityAdmission=false --feature-gates=PrivateHostedZoneAWS=true --feature-gates=RouteExternalCertificate=false --feature-gates=SignatureStores=false --feature-gates=SigstoreImageVerification=false --feature-gates=VSphereControlPlaneMachineSet=false --feature-gates=VSphereStaticIPs=false --feature-gates=ValidatingAdmissionPolicy=false --flex-volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --kube-api-burst=300 --kube-api-qps=150 --leader-elect-renew-deadline=12s --leader-elect-resource-lock=leases --leader-elect-retry-period=3s --leader-elect=true --node-monitor-grace-period=20s --pv-recycler-pod-template-filepath-hostpath=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --pv-recycler-pod-template-filepath-nfs=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --root-ca-file=/etc/kubernetes/static-pod-resources/configmaps/serviceaccount-ca/ca-bundle.crt --secure-port=10257 --service-account-private-key-file=/etc/kubernetes/static-pod-resources/secrets/service-account-private-key/service-account.key --service-cluster-ip-range=172.22.0.0/16 --use-service-account-credentials=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 --tls-min-version=VersionTLS12
# root 102 0 0 01:18 ? 00:00:00 ps -efkube-controller-manager –openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml –kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig –authentication-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig –authorization-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig –client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt –requestheader-client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt -v=2 –tls-cert-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt –tls-private-key-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key –allocate-node-cidrs=false –cert-dir=/var/run/kubernetes –cluster-cidr=10.132.0.0/14 –cluster-name=demo-01-rhsys-wkmd8 –cluster-signing-cert-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.crt –cluster-signing-duration=720h –cluster-signing-key-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.key –configure-cloud-routes=false –controllers=* –controllers=-bootstrapsigner –controllers=-tokencleaner –controllers=-ttl –enable-dynamic-provisioning=true –feature-gates=AdminNetworkPolicy=false –feature-gates=AlibabaPlatform=true –feature-gates=AutomatedEtcdBackup=false –feature-gates=AzureWorkloadIdentity=true –feature-gates=BuildCSIVolumes=true –feature-gates=CSIDriverSharedResource=false –feature-gates=CloudDualStackNodeIPs=true –feature-gates=ClusterAPIInstall=false –feature-gates=DNSNameResolver=false –feature-gates=DisableKubeletCloudCredentialProviders=false –feature-gates=DynamicResourceAllocation=false –feature-gates=EventedPLEG=false –feature-gates=ExternalCloudProviderAzure=true –feature-gates=ExternalCloudProviderExternal=true –feature-gates=ExternalCloudProviderGCP=true –feature-gates=GCPClusterHostedDNS=false –feature-gates=GCPLabelsTags=false –feature-gates=GatewayAPI=false –feature-gates=InsightsConfigAPI=false –feature-gates=InstallAlternateInfrastructureAWS=false –feature-gates=MachineAPIOperatorDisableMachineHealthCheckController=false –feature-gates=MachineAPIProviderOpenStack=false –feature-gates=MachineConfigNodes=false –feature-gates=ManagedBootImages=false –feature-gates=MaxUnavailableStatefulSet=false –feature-gates=MetricsServer=false –feature-gates=MixedCPUsAllocation=false –feature-gates=NetworkLiveMigration=true –feature-gates=NodeSwap=false –feature-gates=OnClusterBuild=false –feature-gates=OpenShiftPodSecurityAdmission=false –feature-gates=PrivateHostedZoneAWS=true –feature-gates=RouteExternalCertificate=false –feature-gates=SignatureStores=false –feature-gates=SigstoreImageVerification=false –feature-gates=VSphereControlPlaneMachineSet=false –feature-gates=VSphereStaticIPs=false –feature-gates=ValidatingAdmissionPolicy=false –flex-volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec –kube-api-burst=300 –kube-api-qps=150 –leader-elect-renew-deadline=12s –leader-elect-resource-lock=leases –leader-elect-retry-period=3s –leader-elect=true –node-monitor-grace-period=20s –pv-recycler-pod-template-filepath-hostpath=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml –pv-recycler-pod-template-filepath-nfs=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml –root-ca-file=/etc/kubernetes/static-pod-resources/configmaps/serviceaccount-ca/ca-bundle.crt –secure-port=10257 –service-account-private-key-file=/etc/kubernetes/static-pod-resources/secrets/service-account-private-key/service-account.key –service-cluster-ip-range=172.22.0.0/16 –use-service-account-credentials=true –tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 –tls-min-version=VersionTLS12
You can see the parameters now.
for KubeAPIServer
Our target is to set the default-not-ready-toleration-seconds to 20s
# before apply patch, check the kube-controller-manager pod
POD_NAME=`oc get pod -n openshift-kube-apiserver | grep kube-apiserver | awk '{print $1}'`
oc exec -n openshift-kube-apiserver $POD_NAME -- ps -ef
# UID PID PPID C STIME TTY TIME CMD
# root 1 0 0 00:56 ? 00:00:00 watch-termination --termination-touch-file=/var/log/kube-apiserver/.terminating --termination-log-file=/var/log/kube-apiserver/termination.log --graceful-termination-duration=15s --kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-cert-syncer-kubeconfig/kubeconfig -- hyperkube kube-apiserver --openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml --advertise-address=192.168.50.23 -v=2 --permit-address-sharing
# root 19 1 93 00:56 ? 00:35:11 kube-apiserver --openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml --advertise-address=192.168.50.23 -v=2 --permit-address-sharing
# root 63 0 0 01:33 ? 00:00:00 ps -ef
oc exec -n openshift-kube-apiserver $POD_NAME -- cat /etc/kubernetes/static-pod-resources/configmaps/config/config.yaml
# the output is a jsonLet’s see the configuration in detail
{
"admission": {
"pluginConfig": {
"PodSecurity": {
"configuration": {
"apiVersion": "pod-security.admission.config.k8s.io/v1",
"defaults": {
"audit": "restricted",
"audit-version": "latest",
"enforce": "privileged",
"enforce-version": "latest",
"warn": "restricted",
"warn-version": "latest"
},
"exemptions": {
"usernames": [
"system:serviceaccount:openshift-infra:build-controller"
]
},
"kind": "PodSecurityConfiguration"
}
},
"network.openshift.io/ExternalIPRanger": {
"configuration": {
"allowIngressIP": false,
"apiVersion": "network.openshift.io/v1",
"externalIPNetworkCIDRs": null,
"kind": "ExternalIPRangerAdmissionConfig"
},
"location": ""
},
"network.openshift.io/RestrictedEndpointsAdmission": {
"configuration": {
"apiVersion": "network.openshift.io/v1",
"kind": "RestrictedEndpointsAdmissionConfig",
"restrictedCIDRs": [
"10.132.0.0/14",
"172.22.0.0/16"
]
}
}
}
},
"apiServerArguments": {
"allow-privileged": [
"true"
],
"anonymous-auth": [
"true"
],
"api-audiences": [
"https://kubernetes.default.svc"
],
"audit-log-format": [
"json"
],
"audit-log-maxbackup": [
"10"
],
"audit-log-maxsize": [
"200"
],
"audit-log-path": [
"/var/log/kube-apiserver/audit.log"
],
"audit-policy-file": [
"/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-audit-policies/policy.yaml"
],
"authentication-token-webhook-config-file": [
"/etc/kubernetes/static-pod-resources/secrets/webhook-authenticator/kubeConfig"
],
"authentication-token-webhook-version": [
"v1"
],
"authorization-mode": [
"Scope",
"SystemMasters",
"RBAC",
"Node"
],
"client-ca-file": [
"/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt"
],
"enable-admission-plugins": [
"CertificateApproval",
"CertificateSigning",
"CertificateSubjectRestriction",
"DefaultIngressClass",
"DefaultStorageClass",
"DefaultTolerationSeconds",
"LimitRanger",
"MutatingAdmissionWebhook",
"NamespaceLifecycle",
"NodeRestriction",
"OwnerReferencesPermissionEnforcement",
"PersistentVolumeClaimResize",
"PersistentVolumeLabel",
"PodNodeSelector",
"PodTolerationRestriction",
"Priority",
"ResourceQuota",
"RuntimeClass",
"ServiceAccount",
"StorageObjectInUseProtection",
"TaintNodesByCondition",
"ValidatingAdmissionWebhook",
"ValidatingAdmissionPolicy",
"authorization.openshift.io/RestrictSubjectBindings",
"authorization.openshift.io/ValidateRoleBindingRestriction",
"config.openshift.io/DenyDeleteClusterConfiguration",
"config.openshift.io/ValidateAPIServer",
"config.openshift.io/ValidateAuthentication",
"config.openshift.io/ValidateConsole",
"config.openshift.io/ValidateFeatureGate",
"config.openshift.io/ValidateImage",
"config.openshift.io/ValidateOAuth",
"config.openshift.io/ValidateProject",
"config.openshift.io/ValidateScheduler",
"image.openshift.io/ImagePolicy",
"network.openshift.io/ExternalIPRanger",
"network.openshift.io/RestrictedEndpointsAdmission",
"quota.openshift.io/ClusterResourceQuota",
"quota.openshift.io/ValidateClusterResourceQuota",
"route.openshift.io/IngressAdmission",
"scheduling.openshift.io/OriginPodNodeEnvironment",
"security.openshift.io/DefaultSecurityContextConstraints",
"security.openshift.io/SCCExecRestrictions",
"security.openshift.io/SecurityContextConstraint",
"security.openshift.io/ValidateSecurityContextConstraints",
"storage.openshift.io/CSIInlineVolumeSecurity"
],
"enable-aggregator-routing": [
"true"
],
"enable-logs-handler": [
"false"
],
"endpoint-reconciler-type": [
"lease"
],
"etcd-cafile": [
"/etc/kubernetes/static-pod-resources/configmaps/etcd-serving-ca/ca-bundle.crt"
],
"etcd-certfile": [
"/etc/kubernetes/static-pod-resources/secrets/etcd-client/tls.crt"
],
"etcd-healthcheck-timeout": [
"9s"
],
"etcd-keyfile": [
"/etc/kubernetes/static-pod-resources/secrets/etcd-client/tls.key"
],
"etcd-prefix": [
"kubernetes.io"
],
"etcd-readycheck-timeout": [
"9s"
],
"etcd-servers": [
"https://192.168.50.23:2379",
"https://localhost:2379"
],
"event-ttl": [
"3h"
],
"feature-gates": [
"AdminNetworkPolicy=false",
"AlibabaPlatform=true",
"AutomatedEtcdBackup=false",
"AzureWorkloadIdentity=true",
"BuildCSIVolumes=true",
"CSIDriverSharedResource=false",
"CloudDualStackNodeIPs=true",
"ClusterAPIInstall=false",
"DNSNameResolver=false",
"DisableKubeletCloudCredentialProviders=false",
"DynamicResourceAllocation=false",
"EventedPLEG=false",
"ExternalCloudProvider=true",
"ExternalCloudProviderAzure=true",
"ExternalCloudProviderExternal=true",
"ExternalCloudProviderGCP=true",
"GCPClusterHostedDNS=false",
"GCPLabelsTags=false",
"GatewayAPI=false",
"InsightsConfigAPI=false",
"InstallAlternateInfrastructureAWS=false",
"MachineAPIOperatorDisableMachineHealthCheckController=false",
"MachineAPIProviderOpenStack=false",
"MachineConfigNodes=false",
"ManagedBootImages=false",
"MaxUnavailableStatefulSet=false",
"MetricsServer=false",
"MixedCPUsAllocation=false",
"NetworkLiveMigration=true",
"NodeSwap=false",
"OnClusterBuild=false",
"OpenShiftPodSecurityAdmission=false",
"PrivateHostedZoneAWS=true",
"RouteExternalCertificate=false",
"SignatureStores=false",
"SigstoreImageVerification=false",
"VSphereControlPlaneMachineSet=false",
"VSphereStaticIPs=false",
"ValidatingAdmissionPolicy=false"
],
"goaway-chance": [
"0"
],
"http2-max-streams-per-connection": [
"2000"
],
"kubelet-certificate-authority": [
"/etc/kubernetes/static-pod-resources/configmaps/kubelet-serving-ca/ca-bundle.crt"
],
"kubelet-client-certificate": [
"/etc/kubernetes/static-pod-certs/secrets/kubelet-client/tls.crt"
],
"kubelet-client-key": [
"/etc/kubernetes/static-pod-certs/secrets/kubelet-client/tls.key"
],
"kubelet-preferred-address-types": [
"InternalIP"
],
"kubelet-read-only-port": [
"0"
],
"kubernetes-service-node-port": [
"0"
],
"max-mutating-requests-inflight": [
"1000"
],
"max-requests-inflight": [
"3000"
],
"min-request-timeout": [
"3600"
],
"proxy-client-cert-file": [
"/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.crt"
],
"proxy-client-key-file": [
"/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.key"
],
"requestheader-allowed-names": [
"kube-apiserver-proxy",
"system:kube-apiserver-proxy",
"system:openshift-aggregator"
],
"requestheader-client-ca-file": [
"/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"
],
"requestheader-extra-headers-prefix": [
"X-Remote-Extra-"
],
"requestheader-group-headers": [
"X-Remote-Group"
],
"requestheader-username-headers": [
"X-Remote-User"
],
"send-retry-after-while-not-ready-once": [
"true"
],
"service-account-issuer": [
"https://kubernetes.default.svc"
],
"service-account-jwks-uri": [
"https://api.demo-01-rhsys.wzhlab.top:6443/openid/v1/jwks"
],
"service-account-lookup": [
"true"
],
"service-account-signing-key-file": [
"/etc/kubernetes/static-pod-certs/secrets/bound-service-account-signing-key/service-account.key"
],
"service-node-port-range": [
"30000-32767"
],
"shutdown-delay-duration": [
"0s"
],
"shutdown-send-retry-after": [
"true"
],
"storage-backend": [
"etcd3"
],
"storage-media-type": [
"application/vnd.kubernetes.protobuf"
],
"strict-transport-security-directives": [
"max-age=31536000,includeSubDomains,preload"
],
"tls-cert-file": [
"/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt"
],
"tls-private-key-file": [
"/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key"
]
},
"apiVersion": "kubecontrolplane.config.openshift.io/v1",
"authConfig": {
"oauthMetadataFile": "/etc/kubernetes/static-pod-resources/configmaps/oauth-metadata/oauthMetadata"
},
"consolePublicURL": "",
"corsAllowedOrigins": [
"//127\\.0\\.0\\.1(:|$)",
"//localhost(:|$)"
],
"imagePolicyConfig": {
"internalRegistryHostname": "image-registry.openshift-image-registry.svc:5000"
},
"kind": "KubeAPIServerConfig",
"projectConfig": {
"defaultNodeSelector": ""
},
"serviceAccountPublicKeyFiles": [
"/etc/kubernetes/static-pod-resources/configmaps/sa-token-signing-certs",
"/etc/kubernetes/static-pod-resources/configmaps/bound-sa-token-signing-certs"
],
"servicesSubnet": "172.22.0.0/16",
"servingInfo": {
"bindAddress": "0.0.0.0:6443",
"bindNetwork": "tcp4",
"cipherSuites": [
"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
],
"minTLSVersion": "VersionTLS12",
"namedCertificates": [
{
"certFile": "/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt",
"keyFile": "/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key"
},
{
"certFile": "/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt",
"keyFile": "/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key"
},
{
"certFile": "/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt",
"keyFile": "/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key"
},
{
"certFile": "/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt",
"keyFile": "/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key"
},
{
"certFile": "/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt",
"keyFile": "/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key"
}
]
}
}there is no default-not-ready-toleration-seconds in the configuration
# confirm the parameter is not there.
oc exec -n openshift-kube-apiserver $POD_NAME -- cat /etc/kubernetes/static-pod-resources/configmaps/config/config.yaml | grep default-not-ready-toleration-seconds
# no output
# and kube-apiserver support the parameter
oc exec -n openshift-kube-apiserver $POD_NAME -- kube-apiserver --help | grep default-not-ready-toleration-seconds
# --default-not-ready-toleration-seconds int Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300)Apply the patch below.
apiVersion: operator.openshift.io/v1
kind: KubeAPIServer
name: cluster
....
spec:
....
unsupportedConfigOverrides:
apiServerArguments:
default-not-ready-toleration-seconds:
- "20"And check the configuration again
# go back to check the configuration again, we can see the parameter now
oc exec -n openshift-kube-apiserver $POD_NAME -- cat /etc/kubernetes/static-pod-resources/configmaps/config/config.yaml | jq . | grep -A 2 default-not-ready-toleration-seconds
# "default-not-ready-toleration-seconds": [
# "20"
# ],