[Kubernetes] Probes

kubelet은 컨테이너가 시작되자마자 실제로 준비되지 않은 상황에서도 살아있고 준비된 상태라고 생각한다.
아직 활성화 되지 않은 부분이 활성화 된 것처럼 취급되면 안된다.

probe는 파드의 health 체크를 도와주는 도구일 뿐이다.
파드내 어플이 시작 되었는지, 살아있는지, 요청을 받을 준비가 되었는지를 확인한다

 

Probe Types

Terms

• Live => 파드가 살아있는지?
• Ready => 파드가 요청을 받을 준비가 되었는지?

Probes	Description	Action If Fails
startupProbe	컨테이너 내부 애플리케이션이 시작되었는지 확인	재시작
livenessProbe	애플리케이션이 여전히 살아 있는지 확인	재시작
readinessProbe	애플리케이션이 서비스로부터 요청을 처리할 준비가 되었는지 확인	서비스에서 제외

 

k8s는 위의 방법에 3가지 옵션을 제공한다.

• exec: 아무 커맨드를 체크하기 위해 실행한다 ex) cat /tmp/app.log
• httpGet: http 엔드포인트에 요청한다. ex) /health
• tcpSocket: 앱이 특정 포트에서 수신 대기 중인지 확인한다.

 

어떤 프로브에 어떤 옵션을 사용하던 상관없다

프로브 속성은 아래와 같다.

Properties	Default Values
initialDelaySeconds	0
periodSeconds	10
timeoutSeconds	1
successThreshold	1
failureThreshold	3

 

Startup Probe - httpGet

실패하는 프로브를 생성하고 관찰해보자

#01-startup-httpget.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: nginx
      image: nginx
      startupProbe:
        httpGet:
          path: /dummy
          port: 80
        periodSeconds: 1
        failureThreshold: 3





kubectl apply -f .

 

kubectl describe pod

# 결과
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  27s                default-scheduler  Successfully assigned default/my-pod to dev-cluster-worker2
  Normal   Pulled     27s                kubelet            Successfully pulled image "nginx" in 1.579s (1.579s including waiting). Image size: 72099501 bytes.
  Normal   Pulled     22s                kubelet            Successfully pulled image "nginx" in 1.519s (1.519s including waiting). Image size: 72099501 bytes.
  Normal   Pulling    18s (x3 over 27s)  kubelet            Pulling image "nginx"
  Normal   Created    17s (x3 over 27s)  kubelet            Created container nginx
  Normal   Pulled     17s                kubelet            Successfully pulled image "nginx" in 1.542s (1.542s including waiting). Image size: 72099501 bytes.
  Normal   Started    16s (x3 over 27s)  kubelet            Started container nginx
  Warning  Unhealthy  14s (x9 over 26s)  kubelet            Startup probe failed: HTTP probe failed with statuscode: 404
  Normal   Killing    13s (x3 over 23s)  kubelet            Container nginx failed startup probe, will be restarted
  Warning  BackOff    13s                kubelet            Back-off restarting failed container nginx in pod my-pod_def

Startup probe가 실패한걸 확인할 수 있다.
아래 처럼 변경하고 다시 시작해보자

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: nginx
      image: nginx
      startupProbe:
        httpGet:
          path: /
          port: 80
        periodSeconds: 1
        failureThreshold: 3



kubectl apply -f .

 

Startup Probe - tcpSocket

마찬가지로 의도적으로 실패하는 코드를 만들어보자

# 02-startup-tcpsocket.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: mongo
      image: mongo
      startupProbe:
        tcpSocket:
          port: 27018
        periodSeconds: 1
        failureThreshold: 5

적용후 살펴보면 startup probe 가 실패한걸 확인할 수 있다.

kubectl apply -f 02-startup-tcpsocket.yaml
kubectl describe pod

# 결과
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  22s               default-scheduler  Successfully assigned default/my-pod to dev-cluster-worker2
  Normal   Pulled     11s               kubelet            Successfully pulled image "mongo" in 12.796s (12.796s including waiting). Image size: 274465658 bytes.
  Normal   Pulling    6s (x2 over 22s)  kubelet            Pulling image "mongo"
  Normal   Killing    6s                kubelet            Container mongo failed startup probe, will be restarted
  Normal   Created    4s (x2 over 11s)  kubelet            Created container mongo
  Normal   Started    4s (x2 over 11s)  kubelet            Started container mongo
  Normal   Pulled     4s                kubelet            Successfully pulled image "mongo" in 1.536s (1.536s including waiting). Image size: 274465658 bytes.
  Warning  Unhealthy  0s (x9 over 10s)  kubelet            Startup probe failed: dial tcp 10.244.1.7:27018: connect: connection refused

이 후 27017로 포트번호를 변경하고 적용하면 생성되는걸 확인할 수 있다.

 

Startup Probe - exec

#03-startup-exec.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: nginx
      image: nginx
      startupProbe:
        exec:
          command:
            - "cat"
            - "dummy.txt"
        periodSeconds: 1
        failureThreshold: 3

일반적으로 httpGet이나 tcpSocket을 더 선호하며 exec는 둘 다 안될 경우를 위한다.

kubectl apply -f 03-startup-exec.yaml
kubectl describe pod
# 결과
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  13s               default-scheduler  Successfully assigned default/my-pod to dev-cluster-worker2
  Normal   Pulled     12s               kubelet            Successfully pulled image "nginx" in 1.506s (1.506s including waiting). Image size: 72099501 bytes.
  Normal   Pulled     6s                kubelet            Successfully pulled image "nginx" in 1.618s (1.618s including waiting). Image size: 72099501 bytes.
  Normal   Pulling    3s (x3 over 13s)  kubelet            Pulling image "nginx"
  Normal   Killing    3s (x2 over 8s)   kubelet            Container nginx failed startup probe, will be restarted
  Normal   Created    2s (x3 over 12s)  kubelet            Created container nginx
  Normal   Pulled     2s                kubelet            Successfully pulled image "nginx" in 1.568s (1.568s including waiting). Image size: 72099501 bytes.
  Normal   Started    1s (x3 over 12s)  kubelet            Started container nginx
  Warning  Unhealthy  1s (x7 over 11s)  kubelet            Startup probe failed: cat: dummy.txt: No such file or directory

 

아래처럼 파일을 변경하고 삭제 후 재실행하면

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: nginx
      image: nginx
      startupProbe:
        exec:
          command:
            - "cat"
            - "/usr/share/nginx/html/index.html"
        periodSeconds: 1
        failureThreshold: 3

 

Liveness Probe

#04-liveness-probe.yaml
# 실제로는 nignx를 사용한다
# nginx가 10초 되에 시작될 것이다.
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  terminationGracePeriodSeconds: 1
  containers:
    - name: probe-demo
      image: vinsdocker/k8s-probe-demo
      startupProbe:
        httpGet:
          path: /
          port: 80
        periodSeconds: 1
        failureThreshold: 3

이렇게 실행하면 실제로 10초 뒤에 nginx가 실행되는 docker 이기 때문에 실패한다 failureThreshold를 30으로 바꿔주자

 

# 실제로는 nignx를 사용한다
# nginx가 10초 되에 시작될 것이다.
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  terminationGracePeriodSeconds: 1
  containers:
    - name: probe-demo
      image: vinsdocker/k8s-probe-demo
      startupProbe:
        httpGet:
          path: /
          port: 80
        periodSeconds: 1
        failureThreshold: 30
      livenessProbe:
        httpGet:
          path: /live.html
          port: 80
        periodSeconds: 1
        failureThreshold: 3

 

적용하고 컨테이너가 실행중인것을 확인한 후 파드로 들어가보자

kubectl apply -f 04-liveness-probe.yaml
kubectl exec -it my-pod -- bash
root@my-pod:/# cd /usr/share/nginx/html
root@my-pod:/usr/share/nginx/html# ls
50x.html  index.html  live.html  ready.html

이제 여기서 live.html을 지워보자

root@my-pod:/usr/share/nginx/html# rm live.html
root@my-pod:/usr/share/nginx/html# command terminated with exit code 137

liveness probe가 문제가 있다고 생각하고 파드를 재시작했다.

 

Readiness Probe

# 05-readiness-probe.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  terminationGracePeriodSeconds: 1
  containers:
    - name: probe-demo
      image: vinsdocker/k8s-probe-demo
      startupProbe:
        httpGet:
          path: /
          port: 80
        periodSeconds: 1
        failureThreshold: 30
      livenessProbe:
        httpGet:
          path: /live.html
          port: 80
        periodSeconds: 1
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /ready.html
          port: 80
        periodSeconds: 1
        failureThreshold: 1

파드를 준비하고 실행하자 그리고 내부로 접속해서 파일을 삭제해보자

kubectl exec -it my-pod -- bash
root@my-pod:/# cd /usr/share/nginx/html/
root@my-pod:/usr/share/nginx/html# ls
50x.html  index.html  live.html  ready.html
root@my-pod:/usr/share/nginx/html# rm ready.html

 

그럼 컨테이너에 문제가 없기 때문에 파드는 재실행되지 않고 Ready 상태가 0인걸 확인할 수 있다.
touch ready.html 을 통해 다시 파일을 만들어주면 ready 상태가 1/1 로 바뀐것을 확인할 수 있다.

 

Readiness Probe with service

#05-readiness-probe.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  labels:
    app: my-app
spec:
  terminationGracePeriodSeconds: 1
  containers:
    - name: probe-demo
      image: vinsdocker/k8s-probe-demo
      startupProbe:
        httpGet:
          path: /
          port: 80
        periodSeconds: 1
        failureThreshold: 30
      livenessProbe:
        httpGet:
          path: /live.html
          port: 80
        periodSeconds: 1
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /ready.html
          port: 80
        periodSeconds: 1
        failureThreshold: 1
---
apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 80

 

파일을 적용하고 살펴보자

#적용
kubectl apply -f 05-readiness-probe.yaml
#감시
watch -t -x kubectl describe svc my-app

 

ready.html 파일을 삭제하면 Endpoint가 사라지지만 k8s 1.31에서는 그런 현상은 일어나지 않는다.