k8s TLS bootstrap解析-k8s TLS bootstrap流程分析

当k8s集群开启了TLS认证后,每个节点的kubelet组件都要使用由kube-apiserver的CA签发的有效证书才能与kube-apiserver通信;当节点非常多的时候,为每个节点都单独签署证书是一件非常繁琐而又耗时的事情。此时k8s TLS bootstrap功能应运而生。k8s TLS bootstrap功能就是让kubelet先使用一个预先商定好的低权限的bootstrap token连接到kube-apiserver,向kube-apiserver申请证书,然后kube-controller-manager给kubelet动态签署证书,后续kubelet都将通过动态签署的证书与kube-apiserver通信。

k8s TLS bootstrap解析-k8s TLS bootstrap流程分析

概述

当k8s集群开启了TLS认证后,每个节点的kubelet组件都要使用由kube-apiserver的CA签发的有效证书才能与kube-apiserver通信;当节点非常多的时候,为每个节点都单独签署证书是一件非常繁琐而又耗时的事情。

此时k8s TLS bootstrap功能应运而生。

k8s TLS bootstrap功能就是让kubelet先使用一个预先商定好的低权限的bootstrap token连接到kube-apiserver,向kube-apiserver申请证书,然后kube-controller-manager给kubelet动态签署证书,后续kubelet都将通过动态签署的证书与kube-apiserver通信。

TLS bootstrap涉及组件相关参数

1.kube-apiserver

(1)--client-ca-file:认证客户端证书的CA证书;

(2)--enable-bootstrap-token-auth:设置为true则代表开启TLS bootstrap特性;

2.kube-controller-manager

(1)--cluster-signing-cert-file--cluster-signing-key-file:用来签发kubelet证书的CA证书和私钥,这里的kubelet证书指的是用来跟kube-apiserver通信,kube-apiserver认证kubelet身份的证书,所以–cluster-signing-cert-file指定的值与kube-apiserver的–client-ca-file指定值一致,而私钥则也是对应的私钥;

(2)--cluster-signing-duration:签发给kubelet的证书有效期;

3.kubelet

(1)--bootstrap-kubeconfig:TLS bootstrap的配置文件,文件中一般包含bootstrap token和master url等信息;

(2)--kubeconfig:在kubelet的CSR被批复并被kubelet取回时,一个引用所生成的密钥和所获得证书的kubeconfig文件会被写入到通过 –kubeconfig所指定的文件路径下,而证书和密钥文件会被放到–cert-dir所指定的目录中;

(3)--rotate-certificates:开启证书轮换,kubelet在其现有证书即将过期时通过创建新的CSR来轮换其客户端证书。

详细流程解析

下面以kubeadm使用k8s TLS bootstrap将一个node节点加入已有的master为例,对TLS bootstrap部分进行详细流程解析。

1.RBAC相关操作

(1)生成bootstrap token,创建bootstrap token secret;

bootstrap token secret模板:

apiVersion: v1
data:
  auth-extra-groups: system:bootstrappers:kubeadm:default-node-token
  expiration: 2022-04-03T11:13:09+08:00
  token-id: {token-id}
  token-secret: {token-secret}
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
kind: Secret
metadata:
  name: bootstrap-token-{token-id}
  namespace: kube-system
type: bootstrap.kubernetes.io/token

关于bootstrap token secret相关的格式说明:

secret的name格式为bootstrap-token-{token-id}的格式;
secret的type固定为bootstrap.kubernetes.io/token
secret data中的token-id为6位数字字母组合字符串,token-secret为16位数字字母组合字符串;
secret data中的auth-extra-groups定义了bootstrap token所代表用户所属的的group,kubeadm使用了system:bootstrappers:kubeadm:default-node-token
secret所对应的bootstrap token为{token-id}.{token-secret}

bootstrap token secret示例:

apiVersion: v1
data:
  auth-extra-groups: system:bootstrappers:kubeadm:default-node-token
  expiration: 2022-04-03T11:13:09+08:00
  token-id: abcdef
  token-secret: 0123456789abcdef
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
kind: Secret
metadata:
  name: bootstrap-token-abcdef
  namespace: kube-system
type: bootstrap.kubernetes.io/token

上述secret示例中,kubeadm生成的bootstrap token为abcdef.0123456789abcdef,其代表的用户所在用户组为system:bootstrappers:kubeadm:default-node-token

(2)授予bootstrap token创建CSR证书签名请求的权限,即授予kubelet创建CSR证书签名请求的权限;

即创建ClusterRoleBinding — kubeadm:kubelet-bootstrap

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:kubelet-bootstrap
  ...
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token

kubeadm生成的bootstrap token所代表的用户所在用户组为system:bootstrappers:kubeadm:default-node-token,所以这里绑定权限的时候将权限绑定给了用户组system:bootstrappers:kubeadm:default-node-token

接下来看下被授予的权限ClusterRole — system:node-bootstrapper

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:node-bootstrapper
  ...
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - create
  - get
  - list
  - watch

(3)授予bootstrap token权限,让kube-controller-manager可以自动审批其发起的CSR;

即创建ClusterRoleBinding — kubeadm:node-autoapprove-bootstrap

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:node-autoapprove-bootstrap
  ...
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token

kubeadm生成的bootstrap token所代表的用户所在用户组为system:bootstrappers:kubeadm:default-node-token,所以这里绑定权限的时候将权限绑定给了用户组system:bootstrappers:kubeadm:default-node-token

接下来看下被授予的权限ClusterRole — system:certificates.k8s.io:certificatesigningrequests:nodeclient

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
  ...
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/nodeclient
  verbs:
  - create

(4)授予kubelet权限,让kube-controller-manager自动批复kubelet的证书轮换请求;

即创建ClusterRoleBinding — kubeadm:node-autoapprove-certificate-rotation

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:node-autoapprove-certificate-rotation
  ...
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes

kubelet创建的CSR用户名格式为system:node:<name>,用户组为system:nodes,所以kube-controller-manager为kubelet生成的证书所代表的用户所在用户组为system:nodes,所以这里绑定权限的时候将权限绑定给了用户组system:nodes

接下来看下被授予的权限ClusterRole — system:certificates.k8s.io:certificatesigningrequests:selfnodeclient

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  ...
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/selfnodeclient
  verbs:
  - create

2.启动kubelet,开始TLS bootstrap

(0)根据bootstrap token以及master url等信息生成bootstrap-kubeconfig文件;

(1)启动kubelet,配置了kubeconfig文件目录,但kubeconfig文件为空,再指定bootstrap-kubeconfig文件为上述步骤生成的bootstrap-kubeconfig文件;

(2)kubelet发现配置的kubeconfig文件为空,则加载bootstrap-kubeconfig文件,读取其中的bootstrap token以及master url;

(3)kubelet使用bootstrap token与apiserver通信,创建CSR证书签名请求;

(4)kube-controller-manager批复CSR证书签名请求,为其签发相关证书;

(5)kubelet取回kube-controller-manager生成的相关证书,默认存放在/var/lib/kubelet/pki 目录下,然后根据证书生成kubeconfig文件,后续kubelet将使用该kubeconfig文件与kube-apiserver进行认证通信;

# ls /var/lib/kubelet/pki/
kubelet-client-2022-03-18-14-29-00.pem	kubelet-client-current.pem  kubelet.crt  kubelet.key

kubelet-client-current.pem是个软链,指向kubelet-client-2022-03-18-14-29-00.pem文件,kubelet-client-2022-03-18-14-29-00.pem文件名记录的是证书创建时间,后续kubelet将会根据证书过期时间,在证书临过期前向kube-apiserver重新申请证书,然后自动轮换该证书;

# cat /etc/kubernetes/kubelet.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0F...
    server: https://192.168.1.10:6443
  name: test-cluster
contexts:
- context:
    cluster: test-cluster
    user: system:node:test-cluster-node-1
  name: system:node:test-cluster-node-1
current-context: system:node:test-cluster-node-1
kind: Config
preferences: {}
users:
- name: system:node:test-cluster-node-1
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

3.kubelet自动轮换证书

(1)kubelet在证书接近于过期时自动向kube-apiserver请求更新证书;

(2)kube-controller-manager自动批复,为其签发新的证书;

(3)kubelet取回kube-controller-manager生成的相关证书,替换掉本地的旧证书,后续kubelet将使用新证书来与kube-apiserver进行认证通信;

总结

最后以一幅图来总结一下k8s TLS bootstrap的整个流程。

页面下部广告