在Kubernetes(k8s)环境中,证书的管理至关重要,因为集群的各个组件高度依赖于它们来确保安全的通信。然而,Kubernetes默认的证书管理策略存在一些潜在的风险。具体来说:
- CA证书有效期长:默认情况下,集群的CA(证书颁发机构)证书有效期长达十年。这意味着CA证书本身在长时间内是有效的,减少了因CA证书过期而引发的紧急更新需求。
- 其他证书有效期短:与CA证书形成鲜明对比的是,集群中的其他证书(如API服务器证书、kubelet客户端证书等)默认有效期仅为一年。这种较短的有效期设计初衷是为了提高安全性,通过定期更新证书来降低潜在的安全风险。
然而,这种默认的证书管理策略在实际生产环境中可能会引发严重问题。一旦这些短期有效的证书过期,集群中的许多组件将无法正常通信,从而导致整个集群的功能受到影响。这包括但不限于API服务器与kubelet之间的通信、kubectl命令行工具与API服务器之间的通信等。
因此,在生产环境下,必须高度重视Kubernetes中的证书管理问题。为了避免因证书过期而导致的集群故障,建议采取以下措施:
- 定期监控证书有效期:使用专门的工具或脚本来监控集群中所有证书的有效期,并在证书即将过期时提前进行更新。
- 自动化证书更新流程:通过配置证书管理工具和Kubernetes的集成,实现证书的自动化更新和续签,以减少人为干预和潜在的错误。
- 合理规划证书有效期:根据集群的实际需求和安全性考虑,合理规划各类证书的有效期,确保在不影响集群稳定性的前提下提高安全性。
综上所述,Kubernetes中的证书管理是一个复杂而关键的任务。通过采取有效的监控、自动化更新和合理规划等措施,可以确保集群在各种情况下都能保持稳定的运行。
方法一
[root@k8s-master1 ~]# cd /etc/kubernetes/pki
[root@k8s-master1 pki]# for i in $(ls *.crt); do echo "====================== $i ========"; openssl x509 -in $i -text -noout | grep -A 3 'Validity' ; done
====================== apiserver.crt ========
Validity
Not Before: Sep 25 12:52:10 2023 GMT
Not After : Sep 23 13:22:12 2025 GMT
Subject: CN=kube-apiserver
====================== apiserver-etcd-client.crt ========
Validity
Not Before: Sep 25 12:52:11 2023 GMT
Not After : Sep 23 13:22:13 2025 GMT
Subject: O=system:masters, CN=kube-apiserver-etcd-client
====================== apiserver-kubelet-client.crt ========
Validity
Not Before: Sep 25 12:52:10 2023 GMT
Not After : Sep 23 13:22:13 2025 GMT
Subject: O=system:masters, CN=kube-apiserver-kubelet-client
====================== ca.crt ========
Validity
Not Before: Sep 25 12:52:10 2023 GMT
Not After : Sep 22 12:52:10 2033 GMT
Subject: CN=kubernetes
====================== front-proxy-ca.crt ========
Validity
Not Before: Sep 25 12:52:10 2023 GMT
Not After : Sep 22 12:52:10 2033 GMT
Subject: CN=front-proxy-ca
====================== front-proxy-client.crt ========
Validity
Not Before: Sep 25 12:52:10 2023 GMT
Not After : Sep 23 13:22:13 2025 GMT
Subject: CN=front-proxy-client
方法二
[root@k8s-master1 pki]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster…
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Sep 23, 2025 13:22 UTC 309d ca no
apiserver Sep 23, 2025 13:22 UTC 309d ca no
apiserver-etcd-client Sep 23, 2025 13:22 UTC 309d etcd-ca no
apiserver-kubelet-client Sep 23, 2025 13:22 UTC 309d ca no
controller-manager.conf Sep 23, 2025 13:22 UTC 309d ca no
etcd-healthcheck-client Sep 23, 2025 13:22 UTC 309d etcd-ca no
etcd-peer Sep 23, 2025 13:22 UTC 309d etcd-ca no
etcd-server Sep 23, 2025 13:22 UTC 309d etcd-ca no
front-proxy-client Sep 23, 2025 13:22 UTC 309d front-proxy-ca no
scheduler.conf Sep 23, 2025 13:22 UTC 309d ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Sep 22, 2033 12:52 UTC 8y no
etcd-ca Sep 22, 2033 12:52 UTC 8y no
front-proxy-ca Sep 22, 2033 12:52 UTC 8y no
证书有效期修改
>>>>
安装go环境
下载
官网 https://go.dev/dl/
安装
tar xf go1.20.6.linux-amd64.tar.gz -C /usr/local/
添加环境变量
echo "export PATH=$PATH:/usr/local/go/bin" >>/etc/profile
source /etc/profile
验证
go version
go version go1.20.6 linux/amd64
>>>>
Kubernetes源码下载
官方的github上下载https://github.com/kubernetes/kubernetes/releases
查看版本
kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:20:54Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
下载
可以看搭配,我上面的版本是1.27.4
解压
tar xf kubernetes-1.27.4.tar.gz
cd kubernetes-1.27.4/
修改源文件
vim cmd/kubeadm/app/constants/constants.go
....
// CertificateValidity defines the validity for all the signed certificates generated by kubeadm
//CertificateValidity = time.Hour * 24 * 365
// 将1年改成100年
CertificateValidity = time.Hour * 24 * 365*100
...
修改 constants.go
文件
修改cert.go
文件
vim staging/src/k8s.io/client-go/util/cert/cert.go
.....
//NotAfter: now.Add(duration365d * 10).UTC(),
NotAfter: now.Add(duration365d * 100).UTC(),
.....
编译源代码
make WHAT=cmd/kubeadm GOFLAGS=-v
等待编译完成
编译完成后查看结果,就可以看到生成的kubeadm文件
ll _output/bin/
总用量 46M
-rwxr-xr-x 1 root root 46M 7月 23 18:40 kubeadm
>>>>
替换旧文件
更换kubeadm
备份原文件
cp /usr/bin/kubeadm /usr/bin/kubeadm.bak
替换
cp _output/bin/kubeadm /usr/bin/
备份证书
cd /etc/kubernetes
cp -R pki pki.bak
更新所有证书
kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
这里直接选择重启主机>>>>
查看证书
重启后查看新的证书,看结果除了ca以外的证书都变100年了,这是因为ca证书是不会被更新的,所以还是保持10年,不过对于一般的生产环境,十年绝对是够了。
kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jun 29, 2123 08:08 UTC 99y ca no
apiserver Jun 29, 2123 08:08 UTC 99y ca no
apiserver-etcd-client Jun 29, 2123 08:08 UTC 99y etcd-ca no
apiserver-kubelet-client Jun 29, 2123 08:08 UTC 99y ca no
controller-manager.conf Jun 29, 2123 08:08 UTC 99y ca no
etcd-healthcheck-client Jun 29, 2123 08:08 UTC 99y etcd-ca no
etcd-peer Jun 29, 2123 08:08 UTC 99y etcd-ca no
etcd-server Jun 29, 2123 08:08 UTC 99y etcd-ca no
front-proxy-client Jun 29, 2123 08:08 UTC 99y front-proxy-ca no
scheduler.conf Jun 29, 2123 08:08 UTC 99y ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jun 29, 2033 07:47 UTC 9y no
etcd-ca Jun 29, 2033 07:47 UTC 9y no
front-proxy-ca Jun 29, 2033 07:47 UTC 9y no
如果有其他master,就直接把编译好的kubeadm文件,从第一台更新过的scp过去,然后按上面步骤,重新生成新的证书文件就可以了。更新过需要重启一下服务或者服务器。
这就是修改过程。
补充
在生产环境中,考虑到更换CA证书可能带来的复杂性和服务中断风险,通常不建议轻易进行此类操作。为了确保系统的稳定性和连续性,一种更为稳妥的方法是,在初次部署Kubernetes(k8s)集群时,就直接配置根证书的有效期为100年。这样做的好处在于,可以长远地避免未来因证书过期而引发的一系列问题,同时也减少了管理和维护的复杂性。因此,推荐在部署阶段就设定好较长的证书有效期,以避免后续不必要的麻烦。
>>>>
编译源文件
同上面一样,最后编译完,获得kubeadm二进制文件,直接在安装完kubelet kubeadm kubectl 之后,就直接将编译好的文件替换掉kubeadm。
cp _output/bin/kubeadm /usr/bin/
同样也要复制到其他节点上,无论master还是work之后的操作就是正常部署就好了,而部署完了,证书就全都变成100年了。
kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jun 29, 2123 08:08 UTC 99y ca no
apiserver Jun 29, 2123 08:08 UTC 99y ca no
apiserver-etcd-client Jun 29, 2123 08:08 UTC 99y etcd-ca no
apiserver-kubelet-client Jun 29, 2123 08:08 UTC 99y ca no
controller-manager.conf Jun 29, 2123 08:08 UTC 99y ca no
etcd-healthcheck-client Jun 29, 2123 08:08 UTC 99y etcd-ca no
etcd-peer Jun 29, 2123 08:08 UTC 99y etcd-ca no
etcd-server Jun 29, 2123 08:08 UTC 99y etcd-ca no
front-proxy-client Jun 29, 2123 08:08 UTC 99y front-proxy-ca no
scheduler.conf Jun 29, 2123 08:08 UTC 99y ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jun 29, 2123 08:08 UTC 99y no
etcd-ca Jun 29, 2123 08:08 UTC 99y no
front-proxy-ca Jun 29, 2123 08:08 UTC 99y no