王楠博客

遇事无难易，而勇于敢为。

0%

Docker--k8s证书过期解决方案

发表于 分类于 ， ， ， Valine：

查看集群证书过期情况

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 29, 2021 06:53 UTC   358d                                    no
apiserver                  Dec 29, 2021 06:53 UTC   358d            ca                      no
apiserver-etcd-client      Dec 29, 2021 06:53 UTC   358d            etcd-ca                 no
apiserver-kubelet-client   Dec 29, 2021 06:53 UTC   358d            ca                      no
controller-manager.conf    Dec 29, 2021 06:53 UTC   358d                                    no
etcd-healthcheck-client    Dec 29, 2021 06:53 UTC   358d            etcd-ca                 no
etcd-peer                  Dec 29, 2021 06:53 UTC   358d            etcd-ca                 no
etcd-server                Dec 29, 2021 06:53 UTC   358d            etcd-ca                 no
front-proxy-client         Dec 29, 2021 06:53 UTC   358d            front-proxy-ca          no
scheduler.conf             Dec 29, 2021 06:53 UTC   358d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 23, 2029 11:49 UTC   8y              no
etcd-ca                 Dec 23, 2029 11:49 UTC   8y              no
front-proxy-ca          Dec 23, 2029 11:49 UTC   8y              no

查看根CA证书的有效期

1
2
3
4
5
cd /etc/kubernetes/pki
# 当前证书是10年的证书,可以直接生成, 如果和上面`EXPIRES` 日期是一样的是不适用
ls | grep ca.crt | xargs -I {} openssl x509 -text -in {} | grep "Not After"
            Not After : Dec 23 11:49:43 2029 GMT
            Not After : Dec 23 11:49:44 2029 GMT

查看证书目录结构

kubelet 上一般不会明确指定服务端证书, 而是只指定 ca 根证书, 让 kubelet 根据本地主机信息自动生成服务端证书并保存到配置的cert-dir文件夹中。

Kubernetes 集群根证书

== /etc/kubernetes/pki/ca.crt 根证书 ==
== /etc/kubernetes/pki/ca.key 根证书 ==

其他证书均为根证书签发

  • kube-apiserver 组件持有的服务端证书
    　　/etc/kubernetes/pki/apiserver.crt
    　　/etc/kubernetes/pki/apiserver.key
  • kubelet 组件持有的客户端证书
    　　/etc/kubernetes/pki/apiserver-kubelet-client.crt
    　　/etc/kubernetes/pki/apiserver-kubelet-client.key

    汇聚层(aggregator)证书

    == /etc/kubernetes/pki/front-proxy-ca.crt ==
    == /etc/kubernetes/pki/front-proxy-ca.key ==
  • 代理端使用的客户端证书, 用作代用户与 kube-apiserver 认证
    /etc/kubernetes/pki/front-proxy-client.crt
    /etc/kubernetes/pki/front-proxy-client.key

    etcd 集群根证书

    == /etc/kubernetes/pki/etcd/ca.crt ==
    == /etc/kubernetes/pki/etcd/ca.key ==
  • etcd server 持有的服务端证书
    /etc/kubernetes/pki/etcd/server.crt
    /etc/kubernetes/pki/etcd/server.key
  • peer 集群中节点互相通信使用的客户端证书
    /etc/kubernetes/pki/etcd/peer.crt
    /etc/kubernetes/pki/etcd/peer.key
  • pod 中定义 Liveness 探针使用的客户端证书
    /etc/kubernetes/pki/etcd/healthcheck-client.crt
    /etc/kubernetes/pki/etcd/healthcheck-client.key
  • 配置在 kube-apiserver 中用来与 etcd server 做双向认证的客户端证书
    /etc/kubernetes/pki/apiserver-etcd-client.crt
    /etc/kubernetes/pki/apiserver-etcd-client.key

    Serveice Account秘钥

    == /etc/kubernetes/pki/sa.key ==
    == /etc/kubernetes/pki/sa.pub ==
    这组的密钥对儿仅提供给 kube-controller-manager 使用. kube-controller-manager 通过 sa.key 对 token 进行签名, master 节点通过公钥 sa.pub 进行签名的验证.
    API Server的authenticating环节支持多种身份校验方式：client cert、bearer token、static password auth等，这些方式中有一种方式通过authenticating（Kubernetes API Server会逐个方式尝试），那么身份校验就会通过。一旦API Server发现client发起的request使用的是service account token的方式，API Server就会自动采用signed bearer token方式进行身份校验。而request就会使用携带的service account token参与验证。该token是API Server在创建service account时用API server启动参数：–service-account-key-file的值签署(sign)生成的。如果–service-account-key-file未传入任何值，那么将默认使用–tls-private-key-file的值，即API Server的私钥（server.key）。
    通过authenticating后，API Server将根据Pod username所在的group：system:serviceaccounts和system:serviceaccounts:(NAMESPACE)的权限对其进行authority 和admission control两个环节的处理。在这两个环节中，cluster管理员可以对service account的权限进行细化设置。
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    # `pki`目录下属于根证书目录
    # /etc/kubernetes/pki/ca.crt 根证书
    # /etc/kubernetes/pki/ca.key 根证书
    # 
    tree /etc/kubernetes/pki
    	/etc/kubernetes/pki
    	├── apiserver.crt
    	├── apiserver-etcd-client.crt
    	├── apiserver-etcd-client.key
    	├── apiserver.key
    	├── apiserver-kubelet-client.crt
    	├── apiserver-kubelet-client.key
    	├── ca.crt
    	├── ca.key
    	├── etcd
    	│   ├── ca.crt
    	│   ├── ca.key
    	│   ├── healthcheck-client.crt
    	│   ├── healthcheck-client.key
    	│   ├── peer.crt
    	│   ├── peer.key
    	│   ├── server.crt
    	│   └── server.key
    	├── front-proxy-ca.crt
    	├── front-proxy-ca.key
    	├── front-proxy-client.crt
    	├── front-proxy-client.key
    	├── sa.key
    	└── sa.pub

    备份

    1
    2
    3
    4
    # 备份原有证书
    cp -rp /etc/kubernetes /etc/kubernetes.bak
    # 备份etcd数据目录
    cp -r /var/lib/etcd /var/lib/etcd.bak

    更新证书

    生成集群配置的yaml文件

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    kubeadm config view > /root/kubeadm.yaml
    cat /root/kubeadm.yaml
    apiServer:
      extraArgs:
        authorization-mode: Node,RBAC
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta2
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    controllerManager: {}
    dns:
      type: CoreDNS
    etcd:
      local:
        dataDir: /var/lib/etcd
    imageRepository: k8s.gcr.io
    kind: ClusterConfiguration
    kubernetesVersion: v1.17.0
    networking:
      dnsDomain: cluster.local
      podSubnet: 10.244.0.0/16
      serviceSubnet: 10.96.0.0/12
    scheduler: {}

    证书更新使用帮助

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    kubeadm alpha certs renew --help
    Usage:
      kubeadm alpha certs renew [flags]
      kubeadm alpha certs renew [command]

    Available Commands:
      admin.conf               Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
      all                      Renew all available certificates
      apiserver                Renew the certificate for serving the Kubernetes API
      apiserver-etcd-client    Renew the certificate the apiserver uses to access etcd
      apiserver-kubelet-client Renew the certificate for the API server to connect to kubelet
      controller-manager.conf  Renew the certificate embedded in the kubeconfig file for the controller manager to use
      etcd-healthcheck-client  Renew the certificate for liveness probes to healthcheck etcd
      etcd-peer                Renew the certificate for etcd nodes to communicate with each other
      etcd-server              Renew the certificate for serving etcd
      front-proxy-client       Renew the certificate for the front proxy client
      scheduler.conf           Renew the certificate embedded in the kubeconfig file for the scheduler manager to use

    更新证书操作

    每个master节点都需要执行的, 切记切记
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    # 更新所有服务的证书,如果不确定可以先更新一个看下结果用检查证书的命令, 上面的用法上有指定单独服务的名称
    kubeadm alpha certs renew all --config=/root/kubeadm.yaml
    [renew] Reading configuration from the cluster...
    [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

    certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
    certificate for serving the Kubernetes API renewed
    certificate the apiserver uses to access etcd renewed
    certificate for the API server to connect to kubelet renewed
    certificate embedded in the kubeconfig file for the controller manager to use renewed
    certificate for liveness probes to healthcheck etcd renewed
    certificate for etcd nodes to communicate with each other renewed
    certificate for serving etcd renewed
    certificate for the front proxy client renewed
    certificate embedded in the kubeconfig file for the scheduler manager to use renewed

    再次查询证书期限

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    root @ master ➜  pki  kubeadm alpha certs check-expiration
    [check-expiration] Reading configuration from the cluster...
    [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

    CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
    admin.conf                 Jan 04, 2022 07:55 UTC   364d                                    no
    apiserver                  Jan 04, 2022 07:55 UTC   364d            ca                      no
    apiserver-etcd-client      Jan 04, 2022 07:55 UTC   364d            etcd-ca                 no
    apiserver-kubelet-client   Jan 04, 2022 07:55 UTC   364d            ca                      no
    controller-manager.conf    Jan 04, 2022 07:55 UTC   364d                                    no
    etcd-healthcheck-client    Jan 04, 2022 07:55 UTC   364d            etcd-ca                 no
    etcd-peer                  Jan 04, 2022 07:55 UTC   364d            etcd-ca                 no
    etcd-server                Jan 04, 2022 07:55 UTC   364d            etcd-ca                 no
    front-proxy-client         Jan 04, 2022 07:55 UTC   364d            front-proxy-ca          no
    scheduler.conf             Jan 04, 2022 07:55 UTC   364d                                    no

    CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
    ca                      Dec 23, 2029 11:49 UTC   8y              no
    etcd-ca                 Dec 23, 2029 11:49 UTC   8y              no
    front-proxy-ca          Dec 23, 2029 11:49 UTC   8y              no

    重启服务

    如果上述操作执行之后集群就恢复了,可以不执行如下操作, 但是没有的话,尝试下如下的命令.
    命令的作用是直接重启下和证书相关的应用的容器,重新加载证书.
    1
    docker ps |grep -E 'k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd' | awk -F ' ' '{print $1}' |xargs docker restart