k8s部分问题总结

对应节点的kubelet日志显示：

Get https://registry-1.docker.io/v2/: x509: certificate signed by unknown authority

手动pull：

docker pull natsio/nats-server-config-reloader:0.6.2

提示一样的

原因应该是ca证书加载有问题，kubectl不明原因重启后（journalctl -u kubelet --since “1 day ago” | grep “Stopping kubelet”）问题解决，应该是kubelet启动会重新加载根证书。根据日志可能是/etc/kubernetes/pki/ca.crt

Jun 01 01:26:45 globalrsyncid-2 kubelet[12340]: I0601 01:26:45.912330   12340 topology_manager.go:221] [topologymanager] RemoveContainer - Container ID: 8c49c81e6a05a7d5d59b712b0b20ff648f661641e285ccb67ab8e2b413ed8928
Jun 01 01:26:45 globalrsyncid-2 kubelet[12340]: I0601 01:26:45.924174   12340 topology_manager.go:221] [topologymanager] RemoveContainer - Container ID: 8c49c81e6a05a7d5d59b712b0b20ff648f661641e285ccb67ab8e2b413ed8928
Jun 01 01:26:45 globalrsyncid-2 kubelet[12340]: E0601 01:26:45.927884   12340 remote_runtime.go:291] RemoveContainer "8c49c81e6a05a7d5d59b712b0b20ff648f661641e285ccb67ab8e2b413ed8928" from runtime service failed: rpc error: code = Unknown desc = failed to remove container "8c49c81e6a05a7d5d59b712b0b20ff648f661641e285ccb67ab8e2b413ed8928": Error response from daemon: removal of container 8c49c81e6a05a7d5d59b712b0b20ff648f661641e285ccb67ab8e2b413ed8928 is already in progress
Jun 01 01:26:45 globalrsyncid-2 kubelet[12340]: E0601 01:26:45.927938   12340 kuberuntime_container.go:704] failed to remove pod init container "istio-init": rpc error: code = Unknown desc = failed to remove container "8c49c81e6a05a7d5d59b712b0b20ff648f661641e285ccb67ab8e2b413ed8928": Error response from daemon: removal of container 8c49c81e6a05a7d5d59b712b0b20ff648f661641e285ccb67ab8e2b413ed8928 is already in progress; Skipping pod "ssi-54fb84ddc8-585fc_synccontroller(9fbdce45-d0f1-457f-95a7-a06fa21ffe3b)"
Jun 01 01:26:45 globalrsyncid-2 kubelet[12340]: E0601 01:26:45.928328   12340 pod_workers.go:191] Error syncing pod 9fbdce45-d0f1-457f-95a7-a06fa21ffe3b ("ssi-54fb84ddc8-585fc_synccontroller(9fbdce45-d0f1-457f-95a7-a06fa21ffe3b)"), skipping: failed to "StartContainer" for "istio-init" with CrashLoopBackOff: "back-off 5m0s restarting failed container=istio-init pod=ssi-54fb84ddc8-585fc_synccontroller(9fbdce45-d0f1-457f-95a7-a06fa21ffe3b)"

kubectl logs podname -n xxx istio-proxy发现有异常，因为本namespace不需要istio，所以可以去掉：

kubectl label namespace synccontroller istio-injection=disabled --overwrite=true

3、etcd helm反亲和

antiAffinity: true
    affinity:
       podAntiAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           - labelSelector:
               matchExpressions:
                 - key: etcd_cluster
                   operator: In
                   values:
                     - etcd-cluster
             topologyKey: kubernetes.io/hostname