openshift/k8s, 远程shell / oc exec, 原理

我们在日常运维openshift/k8s的时候,经常会运行oc exec命令,比如

oc exec -it $pod_name -- bash

但是,有的时候,这个session会连不上,有的时候,这个session会突然中断,接下来我们就尝试看看这个命令背后的原理

oc exec -v 6 -it pod/du-deployment1-58944f9f85-8m49m -- bash
# I1230 14:38:39.347429  188014 loader.go:372] Config loaded from file:  /data/install/auth/kubeconfig
# I1230 14:38:39.370718  188014 round_trippers.go:454] GET https://api.ocp4s.redhat.ren:6443/api/v1/namespaces/default/pods/du-deployment1-58944f9f85-8m49m 200 OK in 10 milliseconds
# I1230 14:38:39.376109  188014 podcmd.go:88] Defaulting container name to du-container1
# I1230 14:38:39.396350  188014 round_trippers.go:454] POST https://api.ocp4s.redhat.ren:6443/api/v1/namespaces/default/pods/du-deployment1-58944f9f85-8m49m/exec?command=bash&container=du-container1&stdin=true&stdout=true&tty=true 101 Switching Protocols in 19 milliseconds
#                                                                                                                             [root@du-deployment1-58944f9f85-8m49m /]#


oc exec -v 7 -it pod/du-deployment1-58944f9f85-8m49m -- bash
# I1230 14:39:13.441167  188023 loader.go:372] Config loaded from file:  /data/install/auth/kubeconfig
# I1230 14:39:13.450807  188023 round_trippers.go:432] GET https://api.ocp4s.redhat.ren:6443/api/v1/namespaces/default/pods/du-deployment1-58944f9f85-8m49m
# I1230 14:39:13.450830  188023 round_trippers.go:438] Request Headers:
# I1230 14:39:13.450837  188023 round_trippers.go:442]     Accept: application/json, */*
# I1230 14:39:13.450842  188023 round_trippers.go:442]     User-Agent: oc/4.9.0 (linux/amd64) kubernetes/96e95ce
# I1230 14:39:13.465425  188023 round_trippers.go:457] Response Status: 200 OK in 14 milliseconds
# I1230 14:39:13.473072  188023 podcmd.go:88] Defaulting container name to du-container1
# I1230 14:39:13.475155  188023 round_trippers.go:432] POST https://api.ocp4s.redhat.ren:6443/api/v1/namespaces/default/pods/du-deployment1-58944f9f85-8m49m/exec?command=bash&container=du-container1&stdin=true&stdout=true&tty=true
#                                                                                  I1230 14:39:13.475182  188023 round_trippers.go:438] Request Headers:
#    I1230 14:39:13.475187  188023 round_trippers.go:442]     X-Stream-Protocol-Version: v4.channel.k8s.io
#                                                                                                         I1230 14:39:13.475191  188023 round_trippers.go:442]     X-Stream-Protocol-Version: v3.channel.k8s.io
#                                                           I1230 14:39:13.475195  188023 round_trippers.go:442]     X-Stream-Protocol-Version: v2.channel.k8s.io
#             I1230 14:39:13.475199  188023 round_trippers.go:442]     X-Stream-Protocol-Version: channel.k8s.io
#                                                                                                               I1230 14:39:13.475203  188023 round_trippers.go:442]     User-Agent: oc/4.9.0 (linux/amd64) kubernetes/96e95ce
#                                                                          I1230 14:39:13.496289  188023 round_trippers.go:457] Response Status: 101 Switching Protocols in 21 milliseconds
#                                       [root@du-deployment1-58944f9f85-8m49m /]#

上面2个命令,我们打开了log,等级设置不同,可以看到oc exec命令,其实是调用了api server上的pod接口,然后通道协议切换到了x-stream

那么我们在项目上,发现oc exec不稳定,那就要先去看api server是不是正常,在通往api server的通路上,是不是有haproxy之类的代理,代理是否正常。这样逐步的排查。

reference

  • https://www.cnblogs.com/a00ium/p/10905279.html
  • https://cloud.redhat.com/blog/executing-commands-in-pods-using-k8s-api
  • https://docs.openshift.com/container-platform/4.9/rest_api/workloads_apis/pod-core-v1.html#apiv1namespacesnamespacepodsnameexec