在 Kubernetes 資源是怎麼被回收的？

是由 kubernetes master node 上的 kube-controller-manger 中的 garbage-collection controller 進行回收管理的。

我們先不管 kubernetes 底層是怎麼回收這些垃圾物件的，先把焦點放在我們要怎麼設定yaml檔，畢竟我們是yaml工程師嘛(笑)，本篇文章會用進行幾個實驗來觀察各種 kubernetes 回收策略是如何進行的。

當我們刪除物件時，可以指定該物件底下關聯的子物件是否也跟著自動刪除。

自動刪除附屬的行為也稱為級聯刪除（Cascading Deletion）

Kubernetes 中有兩種 Cascading Deletion (聯集) 刪除分別是：

後台（Background）模式
前台（Foreground）模式。

Foreground

條件

物件的 metadata.finalizers 被設定為 foregroundDeletion
物件處於 deletion in progress 狀態（deletionTimestamp被建立）

行為

需要等到物件所有的關聯的子物件被删除完之後，才可以删除該物件
如何確定關聯的子物件的父親是誰？
- 透過子物件的 ownerReferences 來確定

Background

kubernetes 立刻 馬上 删除物件， garbage-collection controller 會在後台(背景)删除該物件的子物件。

除了在背景刪除子物件的行為外還有一種是不刪除子物件，讓子物件變成孤兒(Orphan)。

實驗 propagation Policy (Foreground)

deploy

部署測試的nginx deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
EOF
deployment.apps/nginx-deployment created

狀態

取的deployment ReplicaSet 以及pod的狀態

1
2
3
4
5
6
7
8
9



kubectl get deploy,pod
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   3/3     3            3           4m44s

NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-6b474476c4-nbtkw   1/1     Running   0          4m44s
pod/nginx-deployment-6b474476c4-nkbrb   1/1     Running   0          4m44s
pod/nginx-deployment-6b474476c4-zh5g7   1/1     Running   0          4m44s

取得ReplicaSet與pod的ownerReferences 用來確定物件之間的關係

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


kubectl get rs nginx-deployment-6b474476c4 -o go-template --template={{.metadata.ownerReferences}}
[map[
apiVersion:apps/v1 
blockOwnerDeletion:true 
controller:true 
kind:Deployment 
name:nginx-deployment 
uid:597d36f5-968a-4025-8621-b24f17f7f3a6]]

kubectl get pod nginx-deployment-6b474476c4-nbtkw  -o go-template --template={{.metadata.ownerReferences}}
[map[
apiVersion:apps/v1 
blockOwnerDeletion:true 
controller:true 
kind:ReplicaSet 
name:nginx-deployment-6b474476c4 
uid:97fb6974-6882-4459-b6b0-b39357a7650b]]

destroy Foreground

透過指定的刪除模式來刪除物件，這裡是透過Foreground 的方式刪除物件。

curl -X DELETE curl -X DELETE localhost:8080/apis/apps/v1/namespaces/default/deployments/nginx-deployment \
  -d '{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Foreground"}' \
  -H "Content-Type: application/json"

狀態

取的deployment 以及pod的狀態觀察刪除狀態。

從這個狀態可以看到所有的 pod 都在 Terminating 的狀態， ReplicaSet 以及 deployment 都沒有先移除。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


kubectl get pod,deploy,rs
NAME                                    READY   STATUS        RESTARTS   AGE
pod/nginx-deployment-6b474476c4-nbtkw   0/1     Terminating   0          133m
pod/nginx-deployment-6b474476c4-nkbrb   0/1     Terminating   0          133m
pod/nginx-deployment-6b474476c4-zh5g7   0/1     Terminating   0          133m

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   0/3     0            0           133m

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-6b474476c4   3         0         0       133m

比較差異

比較刪除前後的差異

Deployment

Deployment狀態的差異

從 diff 的狀態可以看出來，在 metadata 的部分修改了 finalizers 並且指定了 foregroundDeletion 的模式表示使用 foreground 刪除模式，另外新增了 deletionTimestamp 的時間確定了物件的刪除時間。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


--- a/dpeloy.yaml
+++ b/dpeloy.yaml
-  generation: 1
+  deletionGracePeriodSeconds: 0
+  deletionTimestamp: "2020-08-10T08:22:55Z"
+  finalizers:
+  - foregroundDeletion
+  generation: 2
   namespace: default
-  resourceVersion: "1480766"
+  resourceVersion: "1499040"

...
 status:
-  availableReplicas: 3
   conditions:
-  - lastTransitionTime: "2020-08-10T06:10:18Z"
-    lastUpdateTime: "2020-08-10T06:10:18Z"
-    message: Deployment has minimum availability.
-    reason: MinimumReplicasAvailable
-    status: "True"
-    type: Available

...
-  observedGeneration: 1
-  readyReplicas: 3
-  replicas: 3
-  updatedReplicas: 3
+  - lastTransitionTime: "2020-08-10T08:22:55Z"
+    lastUpdateTime: "2020-08-10T08:22:55Z"
+    message: Deployment does not have minimum availability.
+    reason: MinimumReplicasUnavailable
+    status: "False"
+    type: Available
+  observedGeneration: 2
+  unavailableReplicas: 3

ReplicaSet

觀察ReplicaSet的變化也是確定了刪除的時間 deletionTimestamp 以及修改時間 time ，以及是透過 foregroundDeletion 的策略進行刪除。

在狀態欄的地方也能看出現在狀態replicas的數量被縮減為0個。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


--- a/rs.yaml
+++ b/rs.yaml
   creationTimestamp: "2020-08-10T06:09:12Z"
-  generation: 1
+  deletionGracePeriodSeconds: 0
+  deletionTimestamp: "2020-08-10T08:22:55Z"
+  finalizers:
+  - foregroundDeletion
+  generation: 2
...
-    time: "2020-08-10T06:10:18Z"
+    time: "2020-08-10T08:22:55Z"
...
   ownerReferences:
@@ -86,7 +87,7 @@ metadata:
     kind: Deployment
     name: nginx-deployment
     uid: 597d36f5-968a-4025-8621-b24f17f7f3a6
-  resourceVersion: "1480765"
+  resourceVersion: "1499039"
...
       terminationGracePeriodSeconds: 30
 status:
-  availableReplicas: 3
-  fullyLabeledReplicas: 3
-  observedGeneration: 1
-  readyReplicas: 3
-  replicas: 3
+  observedGeneration: 2
+  replicas: 0

Pod

pod 的部分也想當的簡單，因為沒有子物件所以只要確定刪除時間 deletionTimestamp 以及修改時間 time 。

pod的狀態會被修改成 pending 以及相關的資源都會被移除例如: pod ip。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53


--- a/pod.yaml
+++ b/pod.yaml
   creationTimestamp: "2020-08-10T06:09:12Z"
+  deletionGracePeriodSeconds: 30
+  deletionTimestamp: "2020-08-10T08:23:25Z"
...
-    time: "2020-08-10T06:10:16Z"
+    time: "2020-08-10T08:22:57Z"
     uid: 97fb6974-6882-4459-b6b0-b39357a7650b
-  resourceVersion: "1480754"
+  resourceVersion: "1499048"
status: "True"
     type: Initialized
   - lastProbeTime: null
-    lastTransitionTime: "2020-08-10T06:10:16Z"
-    status: "True"
+    lastTransitionTime: "2020-08-10T08:22:57Z"
+    message: 'containers with unready status: [nginx]'
+    reason: ContainersNotReady
+    status: "False"
     type: Ready
   - lastProbeTime: null
-    lastTransitionTime: "2020-08-10T06:10:16Z"
-    status: "True"
+    lastTransitionTime: "2020-08-10T08:22:57Z"
+    message: 'containers with unready status: [nginx]'
+    reason: ContainersNotReady
+    status: "False"
...
   containerStatuses:
-  - containerID: docker://f20bbce2b58ac42426b61fc21e3f1a61938a51a4dc30277f50e9ac7aea88aa3d
-    image: nginx:1.14.2
-    imageID: docker-pullable://nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
+  - image: nginx:1.14.2
+    imageID: ""
     lastState: {}
     name: nginx
-    ready: true
+    ready: false
     restartCount: 0
-    started: true
+    started: false
     state:
-      running:
-        startedAt: "2020-08-10T06:10:16Z"
+      waiting:
+        reason: ContainerCreating
   hostIP: 172.18.0.5
-  phase: Running
-  podIP: 10.32.0.5
-  podIPs:
-  - ip: 10.32.0.5
+  phase: Pending

實驗propagationPolicy (Background)

deploy

部署測試的nginx deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
EOF
deployment.apps/nginx-deployment created

狀態

取的Deployment ReplicaSet 以及Pod的狀態

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


kubectl get deploy,rs,pod
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   3/3     3            3           55s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-6b474476c4   3         3         3       55s

NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-6b474476c4-4hjx8   1/1     Running   0          55s
pod/nginx-deployment-6b474476c4-6qvvg   1/1     Running   0          55s
pod/nginx-deployment-6b474476c4-plsn7   1/1     Running   0          55s

取得ReplicaSet與pod的ownerReferences用來確定物件之間的關係

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


kubectl get replicaset.apps/nginx-deployment-6b474476c4  -o go-template --template={{.metadata.ownerReferences}}
[map[
apiVersion:apps/v1
blockOwnerDeletion:true
controller:true
kind:Deployment 
name:nginx-deployment 
uid:8a2be904-c306-426c-881e-c6914415c5fe]]


kubectl get pod nginx-deployment-6b474476c4-6qvvg  -o go-template --template={{.metadata.ownerReferences}}
[map[
apiVersion:apps/v1 
blockOwnerDeletion:true 
controller:true
kind:ReplicaSet 
name:nginx-deployment-6b474476c4
uid:565540ad-fbf1-4d74-8841-f0475e12a200]]

destroy background

1
2
3


curl -X DELETE localhost:8080/apis/apps/v1/namespaces/default/deployments/nginx-deployment \
  -d '{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Background"}' \
  -H "Content-Type: application/json"

狀態

取的deployment 以及pod的狀態

從這個狀態可以看到所有的 pod 都在Terminating 的狀態，但是replicaset 以及 deployment 都先被移除。

1
2
3
4
5


kubectl get pod,deploy,rs
NAME                                    READY   STATUS        RESTARTS   AGE
pod/nginx-deployment-6b474476c4-4hjx8   0/1     Terminating   0          31m
pod/nginx-deployment-6b474476c4-6qvvg   0/1     Terminating   0          31m
pod/nginx-deployment-6b474476c4-plsn7   0/1     Terminating   0          31m

比較差異

deployment

deployment狀態的差異

可以看到deployment 直接被殺掉了。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


--- a/dpeloy.yaml
+++ b/dpeloy.yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  annotations:
-    deployment.kubernetes.io/revision: "1"
-    kubectl.kubernetes.io/last-applied-configuration: |
-      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"nginx"},"name":"nginx-deployment","namespace":"default"},"spec":{"replicas":3,"selector":{"matchLabels":{"app":"nginx"}},"template":{"metadata":{"labels":{"app":"nginx"}},"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx","ports":[{"containerPort":80}]}]}}}}
-  creationTimestamp: "2020-08-10T10:05:25Z"
-  generation: 1
-  labels:
-    app: nginx
-  managedFields:
-  - apiVersion: apps/v1
...
...

replicaset

可以看到 replicaset 也是直接被殺掉了。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


--- a/rs.yaml
+++ b/rs.yaml
-apiVersion: apps/v1
-kind: ReplicaSet
-metadata:
-  annotations:
-    deployment.kubernetes.io/desired-replicas: "3"
-    deployment.kubernetes.io/max-replicas: "4"
-    deployment.kubernetes.io/revision: "1"
-  creationTimestamp: "2020-08-10T10:05:25Z"
-  generation: 1
-  labels:
-    app: nginx
-    pod-template-hash: 6b474476c4
-  managedFields:
-  - apiVersion: apps/v1
-    fieldsType: FieldsV1
-    fieldsV1:
-      f:metadata:
-        f:annotations:
-          .: {}
-          f:deployment.kubernetes.io/desired-replicas: {}
-          f:deployment.kubernetes.io/max-replicas: {}
-          f:deployment.kubernetes.io/revision: {}
...
...

pod

可以看到 pod 是緩慢的回收，可以看到被設定了移除的時間以及相關狀態。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


--- a/pod.yaml
+++ b/pod.yaml
-    time: "2020-08-10T10:05:25Z"
+    time: "2020-08-10T10:08:20Z"

+  deletionGracePeriodSeconds: 30
+  deletionTimestamp: "2020-08-10T10:08:20Z"
   generateName: nginx-deployment-6b474476c4-
-  resourceVersion: "1513382"
+  resourceVersion: "1513717"
...
-    lastTransitionTime: "2020-08-10T10:08:20Z"
-    status: "True"
+    lastTransitionTime: "2020-08-10T10:08:20Z"
+    message: 'containers with unready status: [nginx]'
+    reason: ContainersNotReady
+    status: "False"
     type: Ready
   - lastProbeTime: null
-    lastTransitionTime: "2020-08-10T10:08:20Z"
-    status: "True"
+    lastTransitionTime: "2020-08-10T10:08:20Z"
+    message: 'containers with unready status: [nginx]'
+    reason: ContainersNotReady
+    status: "False"
-  - containerID: docker://bb34d6af8dbe1c72c423fece3d9d797ec8a5a0b62fd82f1f46bfcf5d67157be1
-    image: nginx:1.14.2
-    imageID: docker-pullable://nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
+  - image: nginx:1.14.2
+    imageID: ""
     lastState: {}
     name: nginx
-    ready: true
+    ready: false
     restartCount: 0
-    started: true
+    started: false
     state:
-      running:
-        startedAt: "2020-08-10T10:08:20Z"
+      waiting:
+        reason: ContainerCreating
   hostIP: 172.18.0.5
-  phase: Running
-  podIP: 10.32.0.6
-  podIPs:
-  - ip: 10.32.0.6
+  phase: Pending
...

實驗 propagation Policy (Orphan)

deploy

部署測試的nginx deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
EOF
deployment.apps/nginx-deployment created

狀態

取的deployment replicaset 以及pod的狀態

1
2
3
4
5
6
7
8
9



kubectl get deploy,pod
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   3/3     3            3           4m44s

NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-6b474476c4-nbtkw   1/1     Running   0          4m44s
pod/nginx-deployment-6b474476c4-nkbrb   1/1     Running   0          4m44s
pod/nginx-deployment-6b474476c4-zh5g7   1/1     Running   0          4m44s

取得replicaset與pod的ownerReferences確定物件之間的關係

1
2
3
4
5
6
7


kubectl get rs nginx-deployment-6b474476c4 -o go-template --template={{.metadata.ownerReferences}}

[map[apiVersion:apps/v1 blockOwnerDeletion:true controller:true kind:Deployment name:nginx-deployment uid:b1d1a61d-8b51-4511-8c91-de44aaa2cdd0]]

kubectl get pod nginx-deployment-6b474476c4-nbtkw  -o go-template --template={{.metadata.ownerReferences}}

[map[apiVersion:apps/v1 blockOwnerDeletion:true controller:true kind:ReplicaSet name:nginx-deployment-6b474476c4 uid:b052f80f-d72a-4e32-a4c4-f598274f2b07]]

destroy Orphan

1
2
3


curl -X DELETE localhost:8080/apis/apps/v1/namespaces/default/deployments/nginx-deployment \
  -d '{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Orphan"}' \
  -H "Content-Type: application/json"

狀態

取的deployment 以及pod的狀態

從這個狀態可以看到所有的 pod 都在 Running 的狀態，ReplicaSet 以及 Pod 都沒有被移除，只有 Deployment 被殺掉而已。

1
2
3
4
5
6
7
8
9


kubectl get pod,deploy,rs

NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-6b474476c4-d6wns   1/1     Running   0          12m
pod/nginx-deployment-6b474476c4-d7dtf   1/1     Running   0          12m
pod/nginx-deployment-6b474476c4-m7rbz   1/1     Running   0          12m

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-6b474476c4   3         3         3       12m

小結

從上面三個實驗可以看到不同的移除方式會有不同的結果

Front Ground
需要等到關聯的子物件被刪除後才會進行清理的動作(打上deletionTimestamp)
BackGround
先刪除物件(打上deletionTimestamp)，再慢慢回收子物件(打上deletionTimestamp)

3.Orphan
直接把物件刪除（打上deletionTimestamp），所有子物件不做任何動作。

學習Kubernetes Garbage Collection機制

在 Kubernetes 資源是怎麼被回收的？

Foreground

條件

行為

Background

實驗 propagation Policy (Foreground)

deploy

狀態

destroy Foreground

狀態

比較差異

Deployment

ReplicaSet

Pod

實驗propagationPolicy (Background)

deploy

狀態

destroy background

狀態

比較差異

deployment

replicaset

pod

實驗 propagation Policy (Orphan)

deploy

狀態

destroy Orphan

狀態

小結