采用单 master 方式安装 Kubesphere 3.2.1 以后, 启动了集群 gateway, 但是 ingress 总是不能正常工作，排查了很久，终于解决问题
问题1: ingress controller 镜像版本过低，启动失败。-
kubesphere 3.2.1 默认的 ingress controller 是 kubesphere/nginx-ingress-controller:v0.48.1。我在安装 kubesphere 时，选择安装的 kubernetes 1.23.0, 因此存在兼容性问题。需要修改默认的 nginx ingress controller 版本，查看 docekrhub 上 kubesphere 的最新 nginx ingress controller 版本是 v1.1.0。
-
通过 修改 configmap ks-router-config 中的 nginx ingress controller 的 image 版本号为 v1.1.0，再重启 kubersphere controller，可以解决该问题。
-
这个地方我走了很多弯路，发现 ks-config 的配置需要在重启 kubesphere controller 才会生效。
-
kubesphere 采用 helm 方式安装额 nginx ingress controller, 但是该 helm 是包含在 kubesphere controller 镜像中，我尝试修改 helm 包的 values.yaml，替换新的镜像文件，实际上也不能生效，
- 查看 nginx ingress controller 的 pod 日志，发现是无法完成 ingress controller 选举，提示没有权限更新 ingress-controller-leader-kubesphere-router-kubesphere-system
- 通过增加 可以 update 该 configmap 的 clusterrole, 绑定到 sa kubesphere-router-kubesphere-system, 问题解决。
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: ingress-controller-configmap-update rules: - apiGroups: [""] resources: ["configmaps"] resourceNames: ["ingress-controller-leader-kubesphere-router-kubesphere-system"] verbs: ["update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubesphere-router-kubesphere-system-update-configmap roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-controller-configmap-update subjects: - kind: ServiceAccount name: kubesphere-router-kubesphere-system namespace: kubesphere-controls-system排查权限过程如下
################################################################################################################################
[root@ks-master ~]# kcs get clusterrolebindings |grep kubesphere-router-kubesphere-system
kubesphere-router-kubesphere-system ClusterRole/kubesphere-router-kubesphere-system 6m48s
################################################################################################################################
[root@ks-master ~]# kubectl describe clusterrolebindings kubesphere-router-kubesphere-system
Name: kubesphere-router-kubesphere-system
Labels: app.kubernetes.io/instance=kubesphere-router-kubesphere-system-ingress
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=0.48.1
helm.sh/chart=ingress-nginx-3.35.0
Annotations: meta.helm.sh/release-name: kubesphere-router-kubesphere-system-ingress
meta.helm.sh/release-namespace: kubesphere-controls-system
operator-sdk/primary-resource: kubesphere-controls-system/kubesphere-router-kubesphere-system-ingress
operator-sdk/primary-resource-type: Nginx.gateway.kubesphere.io
Role:
Kind: ClusterRole
Name: kubesphere-router-kubesphere-system
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount kubesphere-router-kubesphere-system kubesphere-controls-system
################################################################################################################################
[root@ks-master ~]# kcs describe clusterrole kubesphere-router-kubesphere-system
Name: kubesphere-router-kubesphere-system
Labels: app.kubernetes.io/instance=kubesphere-router-kubesphere-system-ingress
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=0.48.1
helm.sh/chart=ingress-nginx-3.35.0
Annotations: meta.helm.sh/release-name: kubesphere-router-kubesphere-system-ingress
meta.helm.sh/release-namespace: kubesphere-controls-system
operator-sdk/primary-resource: kubesphere-controls-system/kubesphere-router-kubesphere-system-ingress
operator-sdk/primary-resource-type: Nginx.gateway.kubesphere.io
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
events [] [] [create patch]
services [] [] [get list watch]
ingresses.extensions [] [] [get list watch]
ingressclasses.networking.k8s.io [] [] [get list watch]
ingresses.networking.k8s.io [] [] [get list watch]
nodes [] [] [list watch get]
configmaps [] [] [list watch]
endpoints [] [] [list watch]
pods [] [] [list watch]
secrets [] [] [list watch]
ingresses.extensions/status [] [] [update]
ingresses.networking.k8s.io/status [] [] [update]
################################################################################################################################
################################################################################################################################
[root@ks-master ~]# kcs get rolebindings
NAME ROLE AGE
kubesphere-router-kubesphere-system Role/kubesphere-router-kubesphere-system 7m50s
nginx-ingress-role-nisa-binding Role/system:kubesphere-router-role 11d
################################################################################################################################
[root@ks-master ~]# kcs describe role kubesphere-router-kubesphere-system
Name: kubesphere-router-kubesphere-system
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=kubesphere-router-kubesphere-system-ingress
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=0.48.1
helm.sh/chart=ingress-nginx-3.35.0
Annotations: meta.helm.sh/release-name: kubesphere-router-kubesphere-system-ingress
meta.helm.sh/release-namespace: kubesphere-controls-system
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
events [] [] [create patch]
configmaps [] [] [get list watch create]
endpoints [] [] [get list watch]
pods [] [] [get list watch]
secrets [] [] [get list watch]
services [] [] [get list watch]
ingresses.extensions [] [] [get list watch]
ingressclasses.networking.k8s.io [] [] [get list watch]
ingresses.networking.k8s.io [] [] [get list watch]
configmaps [] [ingress-controller-leader-kubesphere-router-kubesphere-system-nginx] [get update]
namespaces [] [] [get]
ingresses.extensions/status [] [] [update]
ingresses.networking.k8s.io/status [] [] [update]
################################################################################################################################
[root@ks-master ~]# kcs describe role system:kubesphere-router-role
Name: system:kubesphere-router-role
Labels: app.kubernetes.io/managed-by=Helm
Annotations: kubernetes.io/created-by: kubesphere.io/ks-router
meta.helm.sh/release-name: ks-core
meta.helm.sh/release-namespace: kubesphere-system
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
configmaps [] [] [get create]
configmaps [] [ingress-controller-leader-nginx] [get update]
endpoints [] [] [get]
namespaces [] [] [get]
pods [] [] [get]
secrets [] [] [get]
