链接: K8S社区 | K8S中文社区 | K8S安装文档
环境要求:
- 3 台 Linux 主机：基于 Debian 或 Red Hat 的发行版
- 2C2G 及以上主机配置
- 集群节点之间网络畅通（公网或内网网络均可）
- 唯一的主机名、MAC 地址、product_uuid
- 指定端口无占用
- 关闭 swap，以保障 kubelet 正常工作
- 安装containerd 的目录使用 ext4 文件系统（xfs 文件系统，会出现无法创建 snapshot 的问题）
集群规划（常规）:
| 操作系统 | 主机 IP | 主机名 | 主机配置 | 角色 |
|---|---|---|---|---|
| CentOS 8.2 | 10.206.0.7 | k8s-node-01 | 4C8G64GB | master |
| CentOS 8.2 | 10.206.32.12 | k8s-node-02 | 4C8G64GB | work |
| CentOS 8.2 | 10.206.32.16 | k8s-node-03 | 4C8G64GB | work |
集群规划（高可用）:
| 操作系统 | 主机 IP | 主机名 | 主机配置 | 角色 |
|---|---|---|---|---|
| CentOS 8.2 | 10.206.0.7 | k8s-node-01 | 4C8G64GB | worker |
| CentOS 8.2 | 10.206.32.12 | k8s-node-02 | 4C8G64GB | master |
| CentOS 8.2 | 10.206.32.16 | k8s-node-03 | 4C8G64GB | master |
说明 1：查看系统版本信息
$ hostnamectl
说明 2：查看 CPU 信息
$ lscpu
说明 3：查看内存信息
$ cat /proc/meminfo
说明 4：查询主机发行版的方式
$ cat /etc/redhat-release
说明 5：获取主机名、MAC 地址、product_uuid 的方式
# 获取主机名 $ hostname # 获取 MAC 地址 $ ifconfig # 获取产品 UUID $ cat /sys/class/dmi/id/product_uuid
说明 6：检测端口是否占用
$ nc ip port
说明 7：判断目录的文件系统
$ df -T
说明 8：K8S占用端口列表
| 端口范围 | 协议 | 组件 |
|---|---|---|
| 6443 | TCP | api-server |
| 2379-2380 | TCP | etcd |
| 10250 | TCP | kubelet |
| 10259 | TCP | kube-scheduler |
| 10257 | TCP | kube-controller-manager |
| 30000-32767 | TCP | kubelet |
| 179 | TCP | calico |
2.1. 主机配置所有节点均需要执行
1）关闭防火墙
$ systemctl stop firewalld $ systemctl disable firewalld
2）关闭 selinux
# 临时关闭 $ setenforce 0 # 永久关闭 $ sed -i 's/enforcing/disabled/' /etc/selinux/config
3）关闭 swap
# 临时 $ swapoff -a # 永久关闭 $ sed -ri 's/.*swap.*/#&/' /etc/fstab
4）修改主机名
# 根据规划设置主机名【node-01节点上操作】 $ hostnamectl set-hostname k8s-node-01 # 根据规划设置主机名【node-02节点操作】 $ hostnamectl set-hostname k8s-node-02 # 根据规划设置主机名【node-03节点操作】 $ hostnamectl set-hostname k8s-node-03
5）配置 Host
# 在所有节点添加hosts $ cat >> /etc/hosts << EOF 10.206.0.7 k8s-node-01 10.206.32.12 k8s-node-02 10.206.32.16 k8s-node-03 EOF
6）时间同步
$ yum install chrony -y $ systemctl start chronyd $ systemctl enable chronyd $ chronyc sources
7）设置内核参数
# 设置内核参数，注意和/etc/sysctl.conf等文件中的重复项冲突 $ cat > /etc/sysctl.d/99-kubernetes-cri.conf <8）设置网络参数
# 加载 overlay 模块 $ modprobe overlay # 加载 br_netfilter 模块 $ modprobe br_netfilter9）安装 ipvsadm
# 安装ipvsadm,并设置ipvs模块自启 yum install ipvsadm -y $ cat > /etc/sysconfig/modules/ipvs.modules << EOF /sbin/modinfo -F filename ip_vs > /dev/null 2>&1 if [ $? -eq 0 ];then /sbin/modprobe ip_vs fi EOF $ systemctl daemon-reload $ systemctl enable ipvsadm $ systemctl restart ipvsam2.2. 安装 containerdcontainerd 简介
containerd 是一个工业级标准的容器运行时，它强调简单性、健壮性和可移植性。containerd 可以在宿主机中管理完整的容器生命周期，包括容器镜像的传输和存储、容器的执行和管理、存储和网络等。
docker vs containerd
containerd 是从 docker 中分离出来的一个项目，可以作为一个底层容器运行时，现在它成了 Kubernete 容器运行时更好的选择。从 1.24 开始，K8S 已取消 docker 作为其默认运行时。
K8S 为什么要放弃使用 docker 作为容器运行时
使用 docker 作为 K8S 容器运行时的话，kubelet 需要先要通过 dockershim 去调用 docker，再通过 docker 去调用 containerd。
使用 containerd 作为 K8S 容器运行时的话，由于 containerd 内置了 CRI （Container Runtime Interface：容器运行时接口）插件，kubelet 可以直接调用 containerd。
1）安装依赖
$ yum install yum-utils device-mapper-persistent-data lvm2 iproute-tc -y2）添加 yum 源
$ yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo3）安装 containerd
$ yum install containerd.io-1.6.6-3.1.el8.x86_64 -y4）生成配置文件
containerd config default > /etc/containerd/config.toml5）修改配置文件
# 修改 cgroups 为 systemd sed -i 's#SystemdCgroup = false#SystemdCgroup = true#' /etc/containerd/config.toml # 修改 pause 镜像地址 sed -i 's#k8s.gcr.io#registry.aliyuncs.com/google_containers#' /etc/containerd/config.toml # 修改容器存储路径到空间比较充裕的路径 root = "/var/lib/containerd"注意：一定要修改 sandbox_image镜像地址，否则会由于防火墙问题，导致 kubelet 启动时无法加载镜像
6）配置 crictl
# 配置文件地址 /etc/crictl.yaml，修改 sock 地址 $ cat </etc/crictl.yaml runtime-endpoint: unix:///run/containerd/containerd.sock image-endpoint: unix:///run/containerd/containerd.sock timeout: 10 debug: false EOF 7）启动服务
$ systemctl enable containerd $ systemctl daemon-reload $ systemctl restart containerd8）查看状态
$ systemctl status containerd
说明 1：使用 ctr 管理容器运行时
COMMANDS: attach Attach to a running container create Create a new container exec Run a command in a running container version Display runtime version information images, image, img List images inspect Display the status of one or more containers inspecti Return the status of one or more images imagefsinfo Return image filesystem info inspectp Display the status of one or more pods logs Fetch the logs of a container port-forward Forward local port to a pod ps List containers pull Pull an image from a registry run Run a new container inside a sandbox runp Run a new pod rm Remove one or more containers rmi Remove one or more images rmp Remove one or more pods pods List pods start Start one or more created containers info Display information of the container runtime stop Stop one or more running containers stopp Stop one or more running pods update Update one or more running containers config Get and set crictl client configuration options stats List container(s) resource usage statistics completion Output shell completion code help, h Shows a list of commands or help for one command说明 2：使用 crictl 管理容器运行时
COMMANDS: attach Attach to a running container create Create a new container exec Run a command in a running container version Display runtime version information images, image, img List images inspect Display the status of one or more containers inspecti Return the status of one or more images imagefsinfo Return image filesystem info inspectp Display the status of one or more pods logs Fetch the logs of a container port-forward Forward local port to a pod ps List containers pull Pull an image from a registry run Run a new container inside a sandbox runp Run a new pod rm Remove one or more containers rmi Remove one or more images rmp Remove one or more pods pods List pods start Start one or more created containers info Display information of the container runtime stop Stop one or more running containers stopp Stop one or more running pods update Update one or more running containers config Get and set crictl client configuration options stats List container(s) resource usage statistics completion Output shell completion code help, h Shows a list of commands or help for one command2.3. 安装 kubeadm、kubelet、kubectl
- kubeadm：用来初始化集群
- kubeadm init 启动引导一个 Kubernetes 主节点
- kubeadm join 启动引导一个 Kubernetes 工作节点并且将其加入到集群
- kubeadm token 使用 kubeadm join 来管理令牌
- kubeadm reset 还原之前使用 kubeadm init 或者 kubeadm join 对节点所作改变
- kubelet：在集群中的每个节点上用来启动 Pod 和容器等
- kubectl：用来与集群通信的命令行工具
1）配置 yum 源
$ cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF2）安装
# 安装组件 $ yum install -y iproute-tc $ yum install kubectl-1.24.1-0.x86_64 kubelet-1.24.1-0.x86_64 kubeadm-1.24.1-0.x86_64 -y # 设置开机启动 $ systemctl enable kubelet $ systemctl daemon-reload $ systemctl restart kubelet3. 安装集群 3.1. 安装 K8S 3.1.1. 常规集群（单 Master）3.1.1.1. 初始化 Master 节点常规集群：单 Master 节点，资源占用少，有单点风险，不建议生产使用
在 Master 节点执行
- 方法一：通过配置文件初始化，不推荐采用该方案
1）生成并修改配置文件
# 生成配置文件 $ mkdir /app $ kubeadm config print init-defaults > kubeadm.yaml修改如下配置：
- advertiseAddress：为控制切面地址，（ Master 主机 IP ）
- criSocket：为 containerd 的 socket 文件地址
- imageRepository：阿里云镜像代理地址，否则拉取镜像会失败
- kubernetesVersion：为 k8s 版本
注意：一定要配置镜像代理，否则会由于防火墙问题导致集群安装失败
修改后配置文件如下：
apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: # 控制切面的IP地址 advertiseAddress: 10.206.0.7 bindPort: 6443 nodeRegistration: # 容器运行时 socket 文件地址 criSocket: unix:///var/run/containerd/containerd.sock imagePullPolicy: IfNotPresent # 控制面主机名，可省略 name: k8s-node-01 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: {} etcd: local: dataDir: /var/lib/etcd # 镜像服务地址 imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration # K8S 版本 kubernetesVersion: 1.24.1 networking: dnsDomain: cluster.local # service 的网段 serviceSubnet: 10.96.0.0/16 scheduler: {} --- kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 cgroupDriver: systemd2）初始化 K8S 集群
# 查看所需镜像列表 $ kubeadm config images list --config /app/kubeadm.yml # 拉取镜像 $ kubeadm config images pull --config /app/kubeadm.yml # 根据配置文件启动 kubeadm 初始化 k8s $ kubeadm init --config=/app/kubeadm.yml --upload-certs --v=6
- 方法二：通过命令初始化
参数 作用 缺省 配置 –apiserver-advertise-address apiserver 绑定的 IP Master 主机 IP –apiserver-bind-port apiserver 监听的端口 6443 6443 –cri-socket CRI socket 文件路径 “unix:///var/run/containerd/containerd.sock” –control-plane-endpoint 控制面地址 master_vip:6440 –image-repository 选择拉取镜像的仓库 k8s.gcr.io registry.aliyuncs.com/google_containers –kubernetes-version 选择K8S版本 stable-1 1.24.1 –pod-network-cidr 指定 pod 的网络 10.96.0.0/16 –service-cidr 指定service 的IP 范围 10.96.0.0/12 10.95.0.0/16 注意1：单 master 节点时不需要配置 control-plane-endpoint，多 master 节点时必须配置
注意2：一定要配置镜像代理，否则会由于防火墙问题导致集群安装失败命令执行
kubeadm init --apiserver-advertise-address=10.206.0.7 --apiserver-bind-port=6443 --cri-socket="unix:///var/run/containerd/containerd.sock" --image-repository registry.aliyuncs.com/google_containers --kubernetes-version=1.24.1 --pod-network-cidr=10.96.0.0/16 --service-cidr=10.95.0.0/16执行结果
Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 10.206.32.12:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:68d48107177ec63c7e36276e2087b00767884c56304fff8ab013efbe498cd61b
说明 1：加入指令可以通过如下命令重复获取
$ kubeadm token create --print-join-command说明 2：初始化失败可以用 reset 指令重置，解决问题后重新初始化
$ kubeadm reset说明 3：查看日志
$ journalctl -f -u kubelet说明 4：token 失效处理
# 重新生成token $ kubeadm token create abcdef.0123456789abcdef # 获取ca证书sha256编码hash值 $ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' b615fccddcd4e80fc6f9c5e477bfc7a053b017660b73fdeccf89c559739664d7 # 将新的node节点加入到k8s集群中 kubeadm join node主机ip地址:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:b615fccddcd4e80fc6f9c5e477bfc7a053b017660b73fdeccf89c559739664d73.1.1.2. 添加 Worker 节点在 Worker 节点执行
$ kubeadm join 10.206.32.12:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:68d48107177ec63c7e36276e2087b00767884c56304fff8ab013efbe498cd61b执行结果
[preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
说明 1：如果初始化过程不顺利，可以添加日志输出
$ kubeadm join 10.206.32.12:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:68d48107177ec63c7e36276e2087b00767884c56304fff8ab013efbe498cd61b --v=63.1.2. 高可用集群（多 Master）3.1.2.1. 安装配置 VIP+Keepalived+Haproxy高可用集群：有多个 Master 节点，避免单点风险
1）配置 VIP 绑定到所有 Master 节点（云厂商提供 VIP 配置界面，也可以选用成本更高，性能更好的硬件 F5）
2）在所有 Master 节点上安装 Keepalived 服务yum install keepalived -y systemctl daemon-reload systemctl enable keepalived修改配置文件：
vim /etc/keepalived/keepalived.conf # 替换为如下配置 global_defs { router_id GA_K8S } vrrp_instance VI_1 { # 节点类型（master或 backup，1 个集群只有 1 个 master角色，其它为 backup 角色） state MASTER # 绑定的网卡 interface eth0 virtual_router_id 200 # 优先级，值越大，优先级越高 priority 100 nopreempt advert_int 1 authentication { auth_type PASS auth_pass jdvm2vm8weiyhlwa } virtual_ipaddress { # VIP 地址 10.206.32.18 } } systemctl restart keepalived3）安装 Haproxy
yum install -y haproxy systemctl daemon-reload systemctl enable haproxy修改配置文件：
vim vim /etc/haproxy/haproxy.cfg # 替换为如下配置 #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 listen stats mode http bind *:7000 stats enable stats uri #--------------------------------------------------------------------- # k8s-api frontend which proxys to the backends #--------------------------------------------------------------------- frontend k8s-api # 本机 ha 服务暴露地址，不同节点，配置 IP 不一样 bind 10.206.32.16:6443 bind 127.0.0.1:6443 mode tcp option tcplog default_backend k8s-api #--------------------------------------------------------------------- # round robin balancing between the various backends #--------------------------------------------------------------------- backend k8s-api mode tcp option tcplog option tcp-check balance roundrobin # k8s 集群，api-server 的地址列表 server k8s-api-1 10.206.0.7:6443 check server k8s-api-2 10.206.32.12:6443 check server k8s-api-3 10.206.32.16:6443 check systemctl restart haproxy
说明 1：查看 Keepalived 的服务状态
systemctl status keepalived说明 2：查看 haproxy 的服务状态
systemctl status haproxy3.1.2.2. 安装配置 Haproxy（单节点模拟 VIP）本次由于没有 VIP，演示高可用集群效果，因此采用单节点 Haproxy 模拟 VIP ，生产不建议这样用，这种方式又会引入 HA 的单点风险
1）安装 Haproxy
$ yum install -y haproxy $ systemctl enable haproxy $ systemctl daemon-reload2）配置 Haproxy
$ vim /etc/haproxy/haproxy.conf # 添加如下配置 frontend k8s-api # 本机 ha 服务暴露地址，不同节点，配置 IP 不一样 bind 10.206.32.12:6440 bind 127.0.0.1:6440 mode tcp option tcplog default_backend k8s-api backend k8s-api mode tcp option tcplog option tcp-check balance roundrobin # k8s 集群，api-server 的地址列表 server k8s-api-1 10.206.0.7:6443 check server k8s-api-2 10.206.32.12:6443 check server k8s-api-3 10.206.32.16:6443 check3）查看状态
$ systemctl status haproxy3.1.2.3. 初始化 Master 节点在初始 Master 节点执行
命令执行
kubeadm init --apiserver-advertise-address=10.206.32.12 --apiserver-bind-port=6443 --cri-socket="unix:///var/run/containerd/containerd.sock" --control-plane-endpoint=10.206.32.12:6440 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version=1.24.1 --pod-network-cidr=10.96.0.0/16 --service-cidr=10.95.0.0/16 --upload-certs执行结果
Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join 10.206.32.12:6440 --token rrm7vb.u5qceu1p8gwzgptu --discovery-token-ca-cert-hash sha256:a612e48dd14d5a19eba3cf30531c7134b4c83fe2098cb32c6ca6196cb289de8b --control-plane --certificate-key b65ed82bec66c20342e3a5e013408520f23c0f66f413729139e0c6e6f978c88a Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join 10.206.32.12:6440 --token rrm7vb.u5qceu1p8gwzgptu --discovery-token-ca-cert-hash sha256:a612e48dd14d5a19eba3cf30531c7134b4c83fe2098cb32c6ca6196cb289de8b3.1.2.4. 加入 Master 节点执行命令
$ kubeadm join 10.206.32.12:6440 --token rrm7vb.u5qceu1p8gwzgptu --discovery-token-ca-cert-hash sha256:a612e48dd14d5a19eba3cf30531c7134b4c83fe2098cb32c6ca6196cb289de8b --control-plane --certificate-key b65ed82bec66c20342e3a5e013408520f23c0f66f413729139e0c6e6f978c88a执行结果
[preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.3.1.2.5. 加入 Worker 节点执行命令
$ kubeadm join 10.206.32.12:6440 --token rrm7vb.u5qceu1p8gwzgptu --discovery-token-ca-cert-hash sha256:a612e48dd14d5a19eba3cf30531c7134b4c83fe2098cb32c6ca6196cb289de8b执行结果
[preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.3.1.2.6. 集群效果$ kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-74586cf9b6-cwhmr 0/1 Pending 0 18mkube-system coredns-74586cf9b6-jdzdw 0/1 Pending 0 18m kube-system etcd-k8s-node-02 1/1 Running 4 19m 10.206.32.12 k8s-node-02 kube-system etcd-k8s-node-03 1/1 Running 0 10m 10.206.32.16 k8s-node-03 kube-system kube-apiserver-k8s-node-02 1/1 Running 4 19m 10.206.32.12 k8s-node-02 kube-system kube-apiserver-k8s-node-03 1/1 Running 1 (10m ago) 10m 10.206.32.16 k8s-node-03 kube-system kube-controller-manager-k8s-node-02 1/1 Running 5 19m 10.206.32.12 k8s-node-02 kube-system kube-controller-manager-k8s-node-03 1/1 Running 0 9m47s 10.206.32.16 k8s-node-03 kube-system kube-proxy-nwsxm 1/1 Running 0 10m 10.206.32.16 k8s-node-03 kube-system kube-proxy-pw9b5 1/1 Running 0 5m49s 10.206.0.7 k8s-node-01 kube-system kube-proxy-t4jn6 1/1 Running 0 18m 10.206.32.12 k8s-node-02 kube-system kube-scheduler-k8s-node-02 1/1 Running 5 19m 10.206.32.12 k8s-node-02 kube-system kube-scheduler-k8s-node-03 1/1 Running 0 10m 10.206.32.16 k8s-node-03 可以看到关键服务已经做到双活，集群创建成功，但是集群尚未就绪，需要接下来安装网络组件
3.2. 配置 kubectl配置 master 节点
$ rm -rf ~/.kube/config $ mkdir ~/,kube $ cp -i /etc/kubernetes/admin.conf ~/.kube/config $ kubectl cluster-info3.3. 安装 calico在 Master 节点执行
1）下载安装 yaml
$ curl -o /app/calico.yaml https://docs.projectcalico.org/manifests/calico.yaml2）修改calico.yaml里的pod网段。
$ vim /app/calico.yaml # 把calico.yaml里pod所在网段改成kubeadm init时所指定的网段， # 原配置 # no effect. This should fall within `--cluster-cidr`. # - name: CALICO_IPV4POOL_CIDR # value: "192.168.0.0/16" # Disable file logging so `kubectl logs` works. - name: CALICO_DISABLE_FILE_LOGGING value: "true" # 修改后的配置 # no effect. This should fall within `--cluster-cidr`. - name: CALICO_IPV4POOL_CIDR value: "10.96.0.0/16" # Disable file logging so `kubectl logs` works. - name: CALICO_DISABLE_FILE_LOGGING value: "true"注意：修改时注意缩进
3）安装
$ kubectl apply -f /app/calico.yaml4）验证
结果（Node Ready）$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-node-01 Ready9m42s v1.24.1 10.206.0.7 CentOS Linux 8 4.18.0-348.7.1.el8_5.x86_64 containerd://1.6.6 k8s-node-02 Ready control-plane 22m v1.24.1 10.206.32.12 CentOS Linux 8 (Core) 4.18.0-348.7.1.el8_5.x86_64 containerd://1.6.6 k8s-node-03 Ready control-plane 14m v1.24.1 10.206.32.16 CentOS Linux 8 (Core) 4.18.0-348.7.1.el8_5.x86_64 containerd://1.6.6 结果（Pod Running)
$ kubectl get po -A -o wide | grep calico kube-system calico-kube-controllers-56cdb7c587-prjs5 1/1 Running 0 65s 10.96.154.193 k8s-node-013.4. 管理集群kube-system calico-node-mgcjk 1/1 Running 0 65s 10.206.32.12 k8s-node-02 kube-system calico-node-qc6fx 1/1 Running 0 65s 10.206.0.7 k8s-node-01 kube-system calico-node-rpfz4 1/1 Running 0 65s 10.206.32.16 k8s-node-03 为节点打标签：便于管理
# 查询 $ kubectl describe node k8s-node-02 # 添加 $ kubectl label node k8s-node-01 node-role.kubernetes.io/worker=true $ kubectl label node k8s-node-02 node-role.kubernetes.io/master=true $ kubectl label node k8s-node-03 node-role.kubernetes.io/master=true # 删除 $ kubectl label node k8s-node-03 node-role.kubernetes.io/master- # 效果 $ kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-node-01 Ready worker 45m v1.24.1 k8s-node-02 Ready control-plane,master 58m v1.24.1 k8s-node-03 Ready control-plane,master 50m v1.24.1为节点加污点：便于隔离
# 查看 kubectl describe node k8s-node-02 # 添加 kubectl taint node k8s-node-02 node-role.kubernetes.io/master=true:NoSchedule # 删除 kubectl taint node k8snode01 node-role.kubernetes.io/master=true:NoSchedule- #删除污点3.5. 测试集群1）部署 nginx 服务
$ kubectl create deployment nginx --image=nginx2）查看状态
$ kubectl get po -A | grep nginx default nginx-8f458dc5b-2rdjx 1/1 Running 0 53s3）暴露端口
kubectl expose deployment nginx --port=80 --type=NodePort4）查看端口
$ kubectl get svc | grep nginx nginx NodePort 10.111.193.22480:30493/TCP 45s 能够看到，我们已经成功暴露了 80 端口 到 30493 上
5）访问服务
$ curl http://k8s-node-01:30493/3.6. 网络工具Welcome to nginx! Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.Thank you for using nginx.
由于常用应用镜像里面缺少各种网络工具，难以定位问题，因此部署 busybox 服务，便于定位节点网络问题
1）创建 yaml 文件
$ cat </app/busybox.yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: busybox-test namespace: default spec: selector: matchLabels: name: busybox-test template: metadata: labels: name: busybox-test spec: containers: - name: app image: busybox command: - sleep - "3600" imagePullPolicy: IfNotPresent terminationGracePeriodSeconds: 30 EOF 2）启动服务
$ kubectl apply -f /app/busybox.yaml3）查看服务
$ kubectl get po -A -o wide | grep busybox default busybox-test-9lcbw 1/1 Running 29 (13m ago) 29h 10.103.175.1 k8s-worker-01default busybox-test-nz85n 1/1 Running 29 (32m ago) 29h 10.101.40.5 k8s-worker-02 4）进入 pod，定位节点问题
$ kubectl exec -ti busybox-test-9lcbw sh -n default4. 安装 Dashbord在 Master 节点安装可视化控制台，方便日常监控使用
开源项目地址：点击跳转
1）下载配置文件
curl -o /app/dashboard.yaml https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.0/aio/deploy/recommended.yaml2）暴露服务
# dashboard.yaml 添加如下配置 kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 30443 selector: k8s-app: kubernetes-dashboard3）创建用户，获取 token
$ vim /app/admin-user.yaml apiVersion: v1 kind: ServiceAccount metadata: name: admin-user namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: admin-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: admin-user namespace: kubernetes-dashboard $ kubectl apply -f /app/admin-user.yaml4）生成 token
$ kubectl -n kubernetes-dashboard create token admin-user eyJhbGciOiJSUzI1NiIsImtpZCI6IllZNFoyRzJHakRYRUw3WnRFeFcxb2lOY0EtWkZiYWljajJUR2RaaUdZNzAifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjU0OTg1NTgwLCJpYXQiOjE2NTQ5ODE5ODAsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiNDBmNDZiYjUtN2EzZi00MDdiLThmNTYtNWM2YWQwZjU1ZWU4In19LCJuYmYiOjE2NTQ5ODE5ODAsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbi11c2VyIn0.VXZHV8_1ELUnGFBAqloAuxFDUVyLWszDZ76Ykp7U_vRPfibl547EC5smjF8jPkGoKFxJY8KyCEsLZDAtDpaObazpECXxg7T7wAqV0TbYvTi7ZZDRDZ0x32OL_9gpefKvXZhCFxxwXSYCC2RBAC85PP5XRCSi1NqD8hZjPZptV_DVtnverq5fITi0hSx6_HBY-f0iqqW0hc6YM-KOdyalcYhfCFXlt5VHV06FuBVI6AaBstddwGAoA48C_kAekkvJQtFqGWl53zmpI1r7yS7vuYtiVBWYWzitEMzbQ-pCha4LQy1B4cfyIW2t4aAkCXqy-viMaGFN16eCSwwky_Pzjg5）配置 token
5. 集群卸载
- 配置入口
- 访问界面
sudo kubeadm reset -f sudo systemctl disable kubelet sudo systemctl disable containerd yum -y remove kubelet kubeadm kubectl containerd sudo rm -rvf ~/.kube/ sudo rm -rvf /etc/kubernetes/ sudo rm -rvf /usr/bin/kube* sudo rm -rvf /etc/cni sudo rm -rvf /etc/containers sudo rm -rvf /etc/containerd sudo rm -rvf /etc/crictl.yml sudo rm -rvf /opt/cni sudo rm -rvf /var/lib/etcd sudo rm -rvf /var/etcd sudo systemctl daemon-reloadK8S 会创建虚拟网卡和 iptables 规则，如无必要，不建议清理
6. 常见问题 6.1. Pod 已启动，但是 Pod 内无法 Ping 外网域名故障现象：Pod 内无法 ping 通外网域名，访问外网 IP、K8S 内部域名或者IP均正常
问题原因：CoreDNS 解析问题
解决方案：删除 CoreDNS，重新创建 CoreDNS
具体步骤：
1）删除已有 Coredns 服务：$ kubectl delete --namespace=kube-system deployment coredns2）重新安装
$ wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed $ wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh $ chmod +x /app/deploy.sh $ /app/deploy.sh | kubectl apply -f -备注：上述资源文件有防火墙
总结通过本博客，基本掌握 K8S 普通集群和高可用集群的从 0 到 1 的搭建，在搭建过程中，需要注意的是由于防火墙问题导致的集群部署失败，一定要注意文中标注的几个镜像仓库配置步骤，避免浪费时间。
接下来的分享计划：
- 多种容器运行时的异同和原理：Dokcer、Containerd、Podman、CRI-O 等
- 多种容器网络组件的异同和原理：Flannel、Calico、CNI 等
- K8S 的资源对象及使用
欢迎大家交流，WX公众号：源圈
