Version Choise
.ref https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
- kubeadm version: &version.Info{Major:“1”, Minor:“26”, GitVersion:“v1.26.1”, GitCommit:“8f94681cd294aa8cfd3407b8191f6c70214973a4”, GitTreeState:“clean”, BuildDate:“2023-01-18T15:56:50Z”, GoVersion:“go1.19.5”, Compiler:“gc”, Platform:“linux/amd64”}
- docker Version: 20.10.12
- containerd: 1.6.18
Prepare
1# check swap off with free -m .ref https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
2swapoff -a
3
4
5# check port is available
6nc 127.0.0.1 6443
7
8# some host dependences
9apt install socat conntrack
10
11# Kubernetes package repositories
12sudo apt-get update
13sudo apt-get install -y apt-transport-https ca-certificates curl
14
15curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
16echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
17
18sudo apt-get update
19sudo apt-get install -y kubelet kubeadm kubectl
20sudo apt-mark hold kubelet kubeadm kubectl
21
22# In China, user can use sources of aliyun
23curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
24cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
25deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
26EOF
27apt-get update
28apt-get install -y kubelet kubeadm kubectl
29
30# Install crictl
31VERSION="v1.26.0"
32wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
33sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
34rm -f crictl-$VERSION-linux-amd64.tar.gz
35
36# Prepare config file for kubelet
37RELEASE_VERSION="v0.4.0"
38curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service
39sudo mkdir -p /etc/systemd/system/kubelet.service.d
40curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Install
1kubeadm init --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers --service-cidr=10.96.0.0/16 --pod-network-cidr=10.244.0.0/16 --v=5
2
3# if Your Kubernetes control-plane has initialized successfully!
4kubectl get node
5NAME STATUS ROLES AGE VERSION
6bvc-smt-004 Ready control-plane 30m v1.26.1
7
8crictl ps|grep kube or docker ps|grep kub
9
10# The join node token needs to be regenerated after it expires in 24h.
11# openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -pubkey | openssl rsa -pubin -outform DER 2>/dev/null | sha256sum | cut -d' ' -f1
12kubeadm token create --print-join-command
13kubeadm join 10.23.34.45:6443 --token xxx --discovery-token-ca-cert-hash shar:123
14
15# Test pod
16cat hello.yaml
17apiVersion: v1
18kind: Pod
19metadata:
20 name: hello-pod
21 labels:
22 app.kubernetes.io/name: MyApp
23spec:
24 containers:
25 - name: hello-pod
26 image: busybox:1.28
27 command: ['sh', '-c', 'echo The app is running! && sleep 20']
28
29kubectl apply -f hello.yaml
30
31# Install CNI
32calico https://docs.tigera.io/calico/latest/getting-started/kubernetes/quickstart
33
Troubleshooting & Resolution
there are many init fail when check kubelet status
systemctl status kubelet
journalctl -xu kubelet
crictl ps check contained if is ready
if the error log is about endpoint, try modify /etc/systemd/system/kubelet.service.d/10-kubeadm.conf（you can find template in k8s official website）
ExecStart=/usr/bin/kubelet –container-runtime-endpoint=unix:///run/containerd/containerd.sock $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
And in no-control-plane nodes, kubeadm reset -f before kubeadm join if you have executed kubeadm init
containerd 状态失败
runtime connect using default endpoint
crictl config runtime-endpoint unix:///run/containerd/containerd.sock
or update containerd
apt rm containerd ## apt install containerd && rm /etc/containerd/config.toml && containerd config default > /etc/containerd/config.toml && systemctl restart containerd
关闭交换区
swapoff -a
update recv-key
apt-key adv –keyserver keyserver.ubuntu.com –recv-keys 40976EAF437D05B5
install kubeadm，kubelet，kubectl offline
RELEASE=v1.26.1 ARCH=“amd64” curl -L –remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet,kubectl}
mv kube* /usr/local/bin && chmod +x /usr/local/bin/kube*
HTTP call equal to ‘curl -sSL http://localhost:10248/healthz’ failed with error:
added docker config file ‘/etc/docker/daemon.json’ and added below to the file. { “exec-opts”: [“native.cgroupdriver=systemd”] }
镜像仓库没有偷传到下面
resolve image pull fail with agent, kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers ## use containerd need tag crictl images, special pause ## ctr –namespace=k8s.io image tag registry.aliyuncs.com/google_containers/pause:3.9 registry.k8s.io/pause:3.9
or edit /etc/containerd/config.toml
Containerd harbor config
config harbor of hub.xxx.co with name: xxx
1[plugins."io.containerd.grpc.v1.cri".registry]
2
3 [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
4
5 [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
6
7 endpoint = ["https://registry-1.docker.io"]
8
9 [plugins."io.containerd.grpc.v1.cri".registry.mirrors."hub.xxx.co"]
10
11 endpoint = ["https://hub.xxx.co"]
12
13 [plugins."io.containerd.grpc.v1.cri".registry.configs]
14
15 [plugins."io.containerd.grpc.v1.cri".registry.configs."hub.xxx.co".tls]
16
17 insecure_skip_verify = true
18
19 #ca_file = "/etc/containerd/certs.d/registry.harbor.com/ca.crt" #ca证书
20
21 #cert_file = "/etc/containerd/certs.d/registry.harbor.com/registry.harbor.com.cert" #harbor证书
22
23 #key_file = "/etc/containerd/certs.d/registry.harbor.com/registry.harbor.com.key" #密钥
24
25 [plugins."io.containerd.grpc.v1.cri".registry.configs."hub.xxx.co".auth]
26
27 username = "xxx"
28
29 password = "xxxxxx"
Control-plane can schduler pod
kubectl taint node xxx node-role.kubernetes.io/control-plane-
rollback: kubectl taint node xxx node-role.kubernetes.io/control-plane:NoSchedule
couldn’t get current server API group list on node
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
Check sock
tr \\0 ' ' < /proc/"$(pgrep kubelet)"/cmdline
CNI
kubeadm reset systemctl stop kubelet rm -rf /var/lib/cni/ rm -rf /var/lib/kubelet/* rm -rf /etc/cni/ ifconfig cni0 down ifconfig flannel.1 down ip link delete cni0 ip link delete flannel.1 systemctl start kubelet
bin缺少 https://github.com/containernetworking/plugins/releases
install calico
DNS 问题 （条目不能超过3）
systemctl status systemd-resolved
cat /etc/systemd/resolved.conf
cat /etc/resolved.conf
plugin type=“flannel” failed (add): open /run/flannel/subnet.env: no such file or directory
try rm -f /etc/cni/net.d/*flannel*
dashboad baretoken
https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md
Kubeadmin join throws this error “/proc/sys/net/bridge/bridge-nf-call-iptables does not exist”
1modprobe br_netfilter
2echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
3echo 1 > /proc/sys/net/ipv4/ip_forward
