minikube 클러스터 삭제 후 kubeadm init 에러

minikube 클러스터 구축 실험 후 minikube를 삭제하고 kubeadm으로 다시 클러스터 구축하려고 init 했을 때 발생한 에러이다.

 

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s

[kubelet-check] Initial timeout of 40s passed.

[kubelet-check] It seems like the kubelet isn't running or healthy.

[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.

위와 같이 kubelet이 runninig중이지 않거나 health 상태가 아니라고 뜬다.

 

이 때, systemctl status kubelet로 kubelet 상태를 확인하면 activating 이고 CGroup이 /var/lib/minikube/binaries/v1.25.3/kubelet 을 잡고 있는 경우가 있다. 이럴 땐, journalctl -xeu kubelet로 서비스 로그를 확인하면 더더욱 확신이 간다.

Jan 10 13:41:32 DS-DEV-005 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: A start job for unit kubelet.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit kubelet.service has finished successfully.
--
-- The job identifier is 26326018.
Jan 10 13:41:32 DS-DEV-005 systemd[2832154]: kubelet.service: Failed to execute command: No such file or directory
Jan 10 13:41:32 DS-DEV-005 systemd[2832154]: kubelet.service: Failed at step EXEC spawning /var/lib/minikube/binaries/v1.21.7/kubelet>
-- Subject: Process /var/lib/minikube/binaries/v1.21.7/kubelet could not be executed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The process /var/lib/minikube/binaries/v1.21.7/kubelet could not be executed and failed.
--
-- The error number returned by this process is ERRNO.
Jan 10 13:41:32 DS-DEV-005 systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit kubelet.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 203.
Jan 10 13:41:32 DS-DEV-005 systemd[1]: kubelet.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit kubelet.service has entered the 'failed' state with result 'exit-code'.
lines 6546-6602/6602 (END)

 

 

해결

1. 도커 데몬 config 파일에 cgroup 추가

$ mkdir /etc/docker

$ cat <<EOF | sudo tee /etc/docker/daemon.json

{

  "exec-opts": ["native.cgroupdriver=systemd"],

  "log-driver": "json-file",

  "log-opts": {

    "max-size": "100m"

  },

  "storage-driver": "overlay2"

}

EOF

$ systemctl daemon-reload
$ systemctl restart docker
$ systemctl restart kubelet

 

 

2. kubeadm config 파일 수정

$ whereis kubelet
>> /usr/bin/kubelet

$ vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

[Unit]
Wants=docker.socket

[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.25.3/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf ~

[Install]

#########################
# /var/lib/minikube~ 이 부분을 /usr/bin/kubelet 으로 변경 후 저장해준다.



$ systemctl daemon-reload
$ systemctl restart kubelet