1、查看证书的有效期
[root@master ~]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep 'Not'
Not Before: Jan 17 13:34:36 2022 GMT #从2022-1-17开始
Not After : Jan 17 13:34:37 2023 GMT #到2023-1-17结束
2、查看各个证书的详细信息
[root@master ~]# kubeadm alpha certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Jan 17, 2023 13:34 UTC 364d no apiserver Jan 17, 2023 13:34 UTC 364d ca no apiserver-etcd-client Jan 17, 2023 13:34 UTC 364d etcd-ca no apiserver-kubelet-client Jan 17, 2023 13:34 UTC 364d ca no controller-manager.conf Jan 17, 2023 13:34 UTC 364d no etcd-healthcheck-client Jan 17, 2023 13:34 UTC 364d etcd-ca no etcd-peer Jan 17, 2023 13:34 UTC 364d etcd-ca no etcd-server Jan 17, 2023 13:34 UTC 364d etcd-ca no front-proxy-client Jan 17, 2023 13:34 UTC 364d front-proxy-ca no scheduler.conf Jan 17, 2023 13:34 UTC 364d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Jan 15, 2032 13:34 UTC 9y no etcd-ca Jan 15, 2032 13:34 UTC 9y no front-proxy-ca Jan 15, 2032 13:34 UTC 9y no
3、导出集群配置
[root@master ~]# kubeadm config view > kube-config.yaml
4、备份原理的证书
[root@master ~]# cp -r /etc/kubernetes/pki/ /etc/kubernetes/pki_backup
5、更新证书
[root@master ~]# kubeadm alpha certs renew all --config=kube-config.yaml W0117 22:07:55.114229 33581 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed certificate embedded in the kubeconfig file for the controller manager to use renewed certificate for liveness probes to healthcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed certificate embedded in the kubeconfig file for the scheduler manager to use renewed
6、再次查看证书的信息
[root@master ~]# kubeadm alpha certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Jan 17, 2023 14:07 UTC 364d no apiserver Jan 17, 2023 14:07 UTC 364d ca no apiserver-etcd-client Jan 17, 2023 14:07 UTC 364d etcd-ca no apiserver-kubelet-client Jan 17, 2023 14:07 UTC 364d ca no controller-manager.conf Jan 17, 2023 14:07 UTC 364d no etcd-healthcheck-client Jan 17, 2023 14:07 UTC 364d etcd-ca no etcd-peer Jan 17, 2023 14:07 UTC 364d etcd-ca no etcd-server Jan 17, 2023 14:07 UTC 364d etcd-ca no front-proxy-client Jan 17, 2023 14:07 UTC 364d front-proxy-ca no scheduler.conf Jan 17, 2023 14:07 UTC 364d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Jan 15, 2032 13:34 UTC 9y no etcd-ca Jan 15, 2032 13:34 UTC 9y no front-proxy-ca Jan 15, 2032 13:34 UTC 9y no
可以看日期会看不出来，因为我们做实验是同一天做的。但是我们可以看具体的时间，可以发现变更了
7、在更新完证书后需要重启四个容器：kube-apiserver、kube-controller、kube-scheduler、etcd
当然如果嫌弃费事，可以直接重启docker
docker restart `docker ps |grep kube-scheduler |awk '{print $1}'`
docker restart `docker ps |grep kube-controller |awk '{print $1}'`
docker restart `docker ps |grep etcd |awk '{print $1}'`
docker restart `docker ps |grep kube-apiserver |awk '{print $1}'`
#或者
systemctl restart docker
8、再次查看集群中所有的Pod
[root@master ~]# kubectl get pods --all-namespaces -o wide证书更新填坑
1、先查看证书的更新时间，可以发现是2023-1-17过期
[root@master ~]# kubeadm alpha certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Jan 17, 2023 14:07 UTC 364d no apiserver Jan 17, 2023 14:07 UTC 364d ca no apiserver-etcd-client Jan 17, 2023 14:07 UTC 364d etcd-ca no apiserver-kubelet-client Jan 17, 2023 14:07 UTC 364d ca no controller-manager.conf Jan 17, 2023 14:07 UTC 364d no etcd-healthcheck-client Jan 17, 2023 14:07 UTC 364d etcd-ca no etcd-peer Jan 17, 2023 14:07 UTC 364d etcd-ca no etcd-server Jan 17, 2023 14:07 UTC 364d etcd-ca no front-proxy-client Jan 17, 2023 14:07 UTC 364d front-proxy-ca no scheduler.conf Jan 17, 2023 14:07 UTC 364d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Jan 15, 2032 13:34 UTC 9y no etcd-ca Jan 15, 2032 13:34 UTC 9y no front-proxy-ca Jan 15, 2032 13:34 UTC 9y no
2、现在改变时间，来使证书过期
[root@master ~]# date -s 2023-2-2 Thu Feb 2 00:00:00 CST 2023 [root@master ~]# date Thu Feb 2 00:00:01 CST 2023 [root@master ~]# kubectl get pods Unable to connect to the server: x509: certificate has expired or is not yet valid #可以发现证书已经无效
3、更新证书
#注意区别上面写的，这里没有带”--config=kube-config.yaml “参数，那这里就代表更新全部证书 [root@master ~]# kubeadm alpha certs renew all [renew] Reading configuration from the cluster... [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [renew] Error reading configuration from the Cluster. Falling back to default configuration W0202 00:02:14.485812 52342 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed certificate embedded in the kubeconfig file for the controller manager to use renewed certificate for liveness probes to healthcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed certificate embedded in the kubeconfig file for the scheduler manager to use renewed
注意这里，这里看似是更新了全部证书，也就是说更新了/etc/kubernetes/pki/目录下的全部证书，但是没有更新
/etc/kubernetes/目录下的kubelet.conf文件信息。
所以下面就要删除原来的kubelet.conf，在生成新的kubelet.conf文件
4、生成新的kubelet.conf证书
[root@master ~]# mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old [root@master ~]# kubeadm init phase kubeconfig kubelet [root@master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@master ~]# systemctl restart kubelet [root@master ~]# systemctl status kubelet
5、重启上面的四个容器，这里直接重启docker
[root@master ~]# systemctl restart docker
注意：这个坑只有在kubelet重启才会发现，因为这个坑会导致kubelet重启失败
6、到这里坑是解决了，但是查看节点状态会发现，node1、node2已经NoReady了
[root@master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready master 380d v1.18.1 node1 NotReady380d v1.18.1 node2 NotReady 380d v1.18.1
这是因为我们重新生成了kubelet.conf文件，这就意味着node节点需要重新加入集群
7、测试，可以通过dashboard的状态来测试，如果正常则正常
也可以再次查看各个证书的详细信息
[root@master ~]# kubeadm alpha certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Feb 01, 2024 16:02 UTC 364d no apiserver Feb 01, 2024 16:02 UTC 364d ca no apiserver-etcd-client Feb 01, 2024 16:02 UTC 364d etcd-ca no apiserver-kubelet-client Feb 01, 2024 16:02 UTC 364d ca no controller-manager.conf Feb 01, 2024 16:02 UTC 364d no etcd-healthcheck-client Feb 01, 2024 16:02 UTC 364d etcd-ca no etcd-peer Feb 01, 2024 16:02 UTC 364d etcd-ca no etcd-server Feb 01, 2024 16:02 UTC 364d etcd-ca no front-proxy-client Feb 01, 2024 16:02 UTC 364d front-proxy-ca no scheduler.conf Feb 01, 2024 16:02 UTC 364d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Jan 15, 2032 13:34 UTC 8y no etcd-ca Jan 15, 2032 13:34 UTC 8y no front-proxy-ca Jan 15, 2032 13:34 UTC 8y no
可以看到成功续期一年
集群证书更新—源码包编译前提：部署好完整的k8s集群
实验环境：kubernetes版本：v1.18.1，
1、获取源码
访问：https://github.com/kubernetes/kubernetes/releases，下载特定版本源码
wget https://github.com/kubernetes/kubernetes/archive/v1.18.1.tar.gz tar -zxvf kubernetes-1.18.1.tar.gz mv kubernetes-1.18.1 kubernetes cd kubernetes
或使用git获取
yum install -y git #在官网中，是以https协议的网址，这样网络不行的会下载失败，这时将https替换为git，如下： git clone git://github.com/kubernetes/kubernetes.git #clone完毕会产生个kubernetes目录 [root@master ~]# du -sh kubernetes/ 1.2G kubernetes/ [root@master ~]# git checkout -b remotes/origin/release-1.18 v1.18.1关于git命令
查看远程所有分支
git branch -a [root@master kubernetes]# git branch -a * (detached from v1.18.0) master remotes/origin/release-1.18 remotes/origin/HEAD -> origin/master remotes/origin/feature-rate-limiting remotes/origin/feature-serverside-apply remotes/origin/feature-workload-ga remotes/origin/master remotes/origin/release-0.10 remotes/origin/release-0.12 remotes/origin/release-0.13 ... remotes/origin/release-1.6 remotes/origin/release-1.6.3 remotes/origin/release-1.7 remotes/origin/release-1.8 remotes/origin/release-1.9 #git branch不带参数,列出本地已经存在的分支，并且在当前分支的前面用*标记，加上-a参数可以查看所有分支列表，包括本地和远程，远程分支一般会用红色字体标记出来
其实我们上述使用git命令去clone源代码，是把所有的版本全部clone下来了，而我们现在的kubernetes版本是v1.18.1，我们只需要去v1.18.1版本的源代码进行操作即可，因此我们需要再次执行新的分支
git checkout -b remotes/origin/release-1.18 v1.18.1 #git checkout -b origin/远程分支名 本地分支名 #这个命令就是新建v1.18.1分支并切换到指定分支， #该命令可以将远程git仓库里的指定分支拉取到本地，这样就在本地新建了一个v1.18.1分支，并和指定的远程分支remotes/origin/release-1.18关联了起来。 #所以这个命令的作用就是过滤出和本k8s版本的源代码修改证书有效期
1、修改CA证书有效期为100 年（默认为 10 年）
// 这个方法里面 NotAfter: now.Add(duration365d * 10).UTC()
// 默认有效期就是 10 年，改成 100 年
// 输入 /NotAfter 查找，回车定位
// NewSelfSignedCACert creates a CA certificate
[root@master ~]# vi /root/kubernetes/staging/src/k8s.io/client-go/util/cert/cert.go
func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
now := time.Now()
tmpl := x509.Certificate{
SerialNumber: new(big.Int).SetInt64(0),
Subject: pkix.Name{
CommonName: cfg.CommonName,
Organization: cfg.Organization,
},
NotBefore: now.UTC(),
NotAfter: now.Add(duration365d * 100).UTC(), #将其改为‘*100’
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
BasicConstraintsValid: true,
IsCA: true,
}
certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key)
if err != nil {
return nil, err
}
return x509.ParseCertificate(certDERBytes)
}
2、修改证书有效期为 100 年（默认为 1 年）
// 就是这个常量定义 Certificatevalidity，改成 * 100 年
// 输入 /Certificatevalidity 查找，回车定位
[root@master ~]# vi /root/kubernetes/cmd/kubeadm/app/constants/constants.go
const (
// KubernetesDir is the directory Kubernetes owns for storing various configuration files
KubernetesDir = "/etc/kubernetes"
// ManifestsSubDirName defines directory name to store manifests
ManifestsSubDirName = "manifests"
// TempDirForKubeadm defines temporary directory for kubeadm
// should be joined with KubernetesDir.
TempDirForKubeadm = "tmp"
// Certificatevalidity defines the validity for all the signed certificates generated by kubeadm
Certificatevalidity = time.Hour * 24 * 365 * 100 #添加‘* 100’
// CACertAndKeybaseName defines certificate authority base name
CACertAndKeybaseName = "ca"
// CACertName defines certificate name
CACertName = "ca.crt"
// CAKeyName defines certificate name
CAKeyName = "ca.key"
编译
镜像编译
需要下载kube-cross镜像，其实这个镜像就是安装的Go语言的镜像
#查看k8sv1.18.1版本的源码包所需要Go语言的版本 [root@master ~]# cat /root/kubernetes/build/build-image/cross/VERSION v1.13.9-2 #可以看到需要Gov1.13.9-2的版本，那我们pull的kube-cross镜像就需要高于或等于这个v1.13.9-2版本的 #pull镜像 [root@master ~]# docker pull wzshiming/kube-cross:v1.15.5-1 [root@master ~]# docker run --rm -v /root/kubernetes:/go/src/k8s.io/kubernetes -it wzshiming/kube-cross:v1.15.5-1 bash #--rm 的意思就是，在容器退出时，删除容器，并且删除容器的匿名卷，所以执行docker run --rm命令，等价于容器退出后，执行docker rm -v # 编译 kubeadm, 这里主要编译 kubeadm 即可 make all WHAT=cmd/kubeadm GOFLAGS=-v # 编译 kubelet # make all WHAT=cmd/kubelet GOFLAGS=-v # 编译 kubectl # make all WHAT=cmd/kubectl GOFLAGS=-v # 退出容器 exit
编译完产物在 _output/bin/kubeadm 目录下，
cp /root/kubernetes/_output/local/bin/linux/amd64/kubeadm /usr/bin/kubeadm
chmod +x /usr/bin/kubeadm
# 验证版本
kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.1-dirty", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"dirty", BuildDate:"2022-01-18T06:00:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
本机编译
1、安装编译包
#安装编译包 yum install gcc make -y yum install rsync jq -y #jq需要epel源
2、查看 kube-cross 的 TAG 版本号
[root@master ~]# cat /root/kubernetes/build/build-image/cross/VERSION v1.13.9-2
3、安装Go环境
wget https://dl.google.com/go/go1.13.9.linux-amd64.tar.gz tar zxvf go1.13.9.linux-amd64.tar.gz -C /usr/local vi /etc/profile 文件添加如下： #go setting export GOROOT=/usr/local/go export GOPATH=/usr/local/gopath export PATH=$PATH:$GOROOT/bin #生效 source /etc/profile # 这里一次性编译，直接执行如下命令即可 export PATH=$PATH:/usr/local/go/bin #版本验证 go version go version go1.13.9 linux/amd64
6、编译
# 编译 kubeadm, 这里主要编译 kubeadm 即可 make all WHAT=cmd/kubeadm GOFLAGS=-v # 编译 kubelet # make all WHAT=cmd/kubelet GOFLAGS=-v # 编译 kubectl # make all WHAT=cmd/kubectl GOFLAGS=-v #编译完产物在 _output/bin/kubeadm 目录下， #其中 bin 是使用了软连接 #真实路径是_output/local/bin/linux/amd64/kubeadm mv /usr/bin/kubeadm /usr/bin/kubeadm_backup cp _output/local/bin/linux/amd64/kubeadm /usr/bin/kubeadm chmod +x /usr/bin/kubeadm
kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.1-dirty", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"dirty", BuildDate:"2022-01-18T06:00:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
如果master有多个节点
scp /usr/bin/kubeadm root@master2:/usr/bin/执行命令更新证书
可以先备份证书，证书在 /etc/kubernetes/pki # 早期版本 (1.19 及之前版本) 命令如下 kubeadm alpha certs check-expiration #1.19以后 kubeadm certs check-expiration
kubeadm alpha certs 命令 1.20 开始废弃。
kubeadm alpha 命令 1.21 开始彻底废弃
1、续订全部证书
kubeadm alpha certs renew all
2、查看全部证书的详细信息，可以看到99y，加上今年就是100y
[root@master ~]# kubeadm alpha certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Dec 25, 2121 06:04 UTC 99y no apiserver Dec 25, 2121 06:04 UTC 99y ca no apiserver-etcd-client Dec 25, 2121 06:04 UTC 99y etcd-ca no apiserver-kubelet-client Dec 25, 2121 06:04 UTC 99y ca no controller-manager.conf Dec 25, 2121 06:04 UTC 99y no etcd-healthcheck-client Dec 25, 2121 06:04 UTC 99y etcd-ca no etcd-peer Dec 25, 2121 06:04 UTC 99y etcd-ca no etcd-server Dec 25, 2121 06:04 UTC 99y etcd-ca no front-proxy-client Dec 25, 2121 06:04 UTC 99y front-proxy-ca no scheduler.conf Dec 25, 2121 06:04 UTC 99y no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Jan 15, 2032 13:34 UTC 9y no etcd-ca Jan 15, 2032 13:34 UTC 9y no front-proxy-ca Jan 15, 2032 13:34 UTC 9y no
