簡單理解 Operator Lifecycle Manager - 2 (OLM)

OpenShift Container Platform 4，使用了大量的 Operator 管理 Kubernetes Resource ，或許這跟 redhat 的推行Kubernetes政策有關吧 xD。

但是這邊會延伸出蛋生雞雞生蛋的問題，那 Operator 要怎麼被管理呢？

redhat 給出了一個解答那就是 Operator Lifecycle Manager 又稱 OLM，使用者透過 Declarative 的方式告訴 OLM ，要建立什麼樣子的 Operator 。

:::info
本篇文章的重點在於 OLM，是用來管理 Operator。
朝著這方面去思考一且都會變得比較簡單！
:::

接續著上篇提到觀念本篇會示範如何在 cluster 中從 CatalogSources 下載一個 PackageManifest 並且安裝到環境。

GUI 操作

可以從圖片看到在 administration 模式底下可以從左邊選取 Operators ，裡面有兩個分頁分別是 OperatorHub 以及 install Operators。

我們可以透過 OCP Portal 直接下載想要的 Operator ，可以在右手邊看到有許多種類提供我們做選擇。

這個部份留給有興趣的朋友，今天著重於 CLI 的操作。

CRD - OperatorGroup

從 CatalogSources 下載一個 PackageManifest 之前我們要先建立一個project(namespaces)，讓操作都在這個 project 下。

1

oc new-project playolm

建立完成 project 後我們還需要建立一個 OperatorGroup 確認之後安裝的 operator 只有操作這個 namespaces 的權限，對於 OperatorGroup 想要多瞭解一下可以參考olm book。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


cat <<EOF | oc apply -f -
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: prometheus-operatorgroup
  namespace: playolm
spec:
  targetNamespaces:
    - playolm
EOF
operatorgroup.operators.coreos.com/prometheus-operatorgroup created

透過指令確認一下剛剛部署上去的 operatorgroup

1
2
3


oc get operatorgroup prometheus-operatorgroup
NAME                   AGE
prometheus-operatorgroup   114s

CRD - Subscription

剛剛設定好operatorgroup，現在要到 CatalogSources 下載一個 PackageManifest 還記得上一個章節有拿一個 community-operators 的 prometheus-exporter-operator yaml 簡單的看一下裡面的內容嗎？

這邊在執行一次看一下裡面的 yaml 檔

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124


oc get packagemanifests  prometheus-exporter-operator -o yaml
apiVersion: packages.operators.coreos.com/v1
kind: PackageManifest
metadata:
 creationTimestamp: "2020-06-20T07:23:23Z"
 labels:
   catalog: community-operators
   catalog-namespace: openshift-marketplace
   olm-visibility: hidden
   openshift-marketplace: "true"
   operatorframework.io/arch.amd64: supported
   operatorframework.io/os.linux: supported
   opsrc-datastore: "true"
   opsrc-owner-name: community-operators
   opsrc-owner-namespace: openshift-marketplace
   opsrc-provider: community
   provider: Red Hat
   provider-url: ""
 name: prometheus-exporter-operator
 namespace: default
 selfLink: /apis/packages.operators.coreos.com/v1/namespaces/default/packagemanifests/prometheus-exporter-operator
spec: {}
status:
 catalogSource: community-operators
 catalogSourceDisplayName: Community Operators
 catalogSourceNamespace: openshift-marketplace
 catalogSourcePublisher: Red Hat
 channels:
 - currentCSV: prometheus-exporter-operator.v0.2.0
   currentCSVDesc:
     annotations:
       alm-examples: |-
         [
           {
             "apiVersion": "monitoring.3scale.net/v1alpha1",
             "kind": "PrometheusExporter",
             "metadata": {
               "name": "example-memcached"
             },
             "spec": {
               "dbHost": "your-memcached-host",
               "dbPort": 11211,
               "grafanaDashboard": {
                 "label": {
                   "key": "autodiscovery",
                   "value": "enabled"
                 }
               },
               "type": "memcached"
             }
           }
         ]
       capabilities: Deep Insights
       categories: Monitoring
       certified: "false"
       containerImage: quay.io/3scale/prometheus-exporter-operator:v0.2.0
       createdAt: "2020-06-08 00:00:00"
       description: Operator to setup 3rd party prometheus exporters, with a collection
         of grafana dashboards
       repository: https://github.com/3scale/prometheus-exporter-operator
       support: Red Hat, Inc.
     apiservicedefinitions: {}
     customresourcedefinitions:
       owned:
       - description: Configures a prometheus exporter to monitor a memcached instance
         displayName: PrometheusExporter
         kind: PrometheusExporter
         name: prometheusexporters.monitoring.3scale.net
         version: v1alpha1
     description: |
       A Kubernetes Operator based on the Operator SDK to centralize the setup of 3rd party prometheus exporters on **Kubernetes/OpenShift**, with a collection of grafana dashboards.

       You can setup different prometheus exporters to monitor the internals from different databases, or even any available cloudwatch metric from any AWS Service, by just providing a few parameters like **dbHost** or **dbPort** (operator manages the container image, port, argument, command, volumes... and also prometheus **ServiceMonitor** and **GrafanaDashboard** k8s objects).

       Current prometheus exporters types supported, managed by same prometheus-exporter-operator:
       * memcached
       * redis
       * mysql
       * postgresql
       * sphinx
       * es (elasticsearch)
       * cloudwatch

       The operator manages the lifecycle of the following objects:
       * Deployment (one per CR)
       * Service (one per CR)
       * ServiceMonitor (optional, one per CR)
       * GrafanaDashboard (optional, one per Namespace)

       ### Documentation
       Documentation can be found on our [website](https://github.com/3scale/prometheus-exporter-operator#documentation).

       ### Getting help
       If you encounter any issues while using operator, you can create an issue on our [website](https://github.com/3scale/prometheus-exporter-operator) for bugs, enhancements, or other requests.

       ### Contributing
       You can contribute by:
       * Raising any issues you find using Prometheus Exporter Operator
       * Fixing issues by opening [Pull Requests](https://github.com/3scale/prometheus-exporter-operator/pulls)
       * Submitting a patch or opening a PR
       * Improving [documentation](https://github.com/3scale/prometheus-exporter-operator)
       * Talking about Prometheus Exporter Operator
       All bugs, tasks or enhancements are tracked as [GitHub issues](https://github.com/3scale/prometheus-exporter-operator/issues).

       ### License
       Prometheus Exporter Operator is licensed under the [Apache 2.0 license](https://github.com/3scale/prometheus-exporter-operator/blob/master/LICENSE)
     displayName: Prometheus Exporter Operator
     installModes:
     - supported: true
       type: OwnNamespace
     - supported: true
       type: SingleNamespace
     - supported: false
       type: MultiNamespace
     - supported: true
       type: AllNamespaces
     provider:
       name: Red Hat
     version: 0.2.0
   name: alpha
 defaultChannel: alpha
 packageName: prometheus-exporter-operator
 provider:
   name: Red Hat 

裡面的內容很多看起來心煩意亂的，不過我們真正要關心的只有以下四點。

name: prometheus-exporter-operator
defaultChannel: alpha
catalogSource: community-operators
catalogSourceNamespace: openshift-marketplace

有這幾個參數就可以告訴olm說我們要用的是哪個catalogsources以及是哪一個packagemanifests。
從defaultChannel 我們可以看到我們要用到是 alpha 版，alpha 對應到的是operator prometheus-exporter-operator.v0.2.0 版。

進入這個 CRD Subscription 的重頭戲，宣告我要用哪一個catalogsources以及是哪一個packagemanifests。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: prometheus-exporter-operator
  namespace: playolm
spec:
  channel: alpha
  installPlanApproval: Manual
  name: prometheus-exporter-operator
  source: community-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Manual  
EOF
subscription.operators.coreos.com/prometheus-exporter-operator created

這時候可以透過指令看看這個project訂閱了哪些subscription。

1
2
3


$ oc get subscription
NAME                           PACKAGE                        SOURCE                CHANNEL
prometheus-exporter-operator   prometheus-exporter-operator   community-operators   alpha

這邊的 subscription 可以把它當成如果packagemanifests 有更新我們會收到更新的概念，說白了就是你訂閱youtuber有開啟小鈴鐺他就會通知你xD。

CRD - Installplan

可以透過另外一個CRD看到目前 project 底下有訂閱（安裝？抱歉找不到一個適當的中文）哪些 Operator 他的版本是多少，現在安裝了嗎等資訊。

1
2
3


oc get installplan
NAME            CSV                                   APPROVAL   APPROVED
install-b5tmz   prometheus-exporter-operator.v0.2.0   Manual     false

從上署指令的結果來看這個 project 訂閱了prometheus-exporter-operator.v0.2.0 現在還沒有被 approved 所以還沒安裝到環境中。

現在我們要手動上一個 patch 讓這個安裝包被 approved ，再來觀察環境的變化。

1
2


oc patch installplan install-b5tmz --type='json' -p '[{"op": "replace", "path": "/spec/approved", "value":true}]'
installplan.operators.coreos.com/install-b5tmz patched

再透過指令看現在prometheus-exporter-operator.v0.2.0是否被approved了。

1
2
3


oc get installplan
NAME            CSV                                   APPROVAL   APPROVED
install-b5tmz   prometheus-exporter-operator.v0.2.0   Manual     true

CRD - ClusterServiceVersion (CSV)

可以透過 ClusterServiceVersion 觀察目前在 cluster 執行的 Operator 是哪一個版本，以及有哪些 operator 正在執行。
:::warning
這邊要注意，只有被approved了的installplan才會被clusterserviceversion紀錄。
:::

1
2
3


oc get clusterserviceversion
NAME                                  DISPLAY                        VERSION   REPLACES   PHASE
prometheus-exporter-operator.v0.2.0   Prometheus Exporter Operator   0.2.0                Succeeded

看起來都有成功在執行，現在可以觀察 Kubernetes Cluster 裡面是否有安裝 prometheus 的相關元件。

觀察prometheus CRD有沒有成功安裝

1
2
3
4


oc get crd | grep prometheus
prometheuses.monitoring.coreos.com                          2020-06-20T07:24:12Z
prometheusexporters.monitoring.3scale.net                   2020-07-24T03:20:46Z
prometheusrules.monitoring.coreos.com                       2020-06-20T07:24:13Z

觀察 Prometheus CRD有沒有成功安裝

1
2
3
4


oc get crd | grep prometheus
prometheuses.monitoring.coreos.com                          2020-06-20T07:24:12Z
prometheusexporters.monitoring.3scale.net                   2020-07-24T03:20:46Z
prometheusrules.monitoring.coreos.com                       2020-06-20T07:24:13Z

觀察 Prometheus ServiceAccount 有沒有成功安裝

1
2


oc get sa | grep prometheus
prometheus-exporter-operator   2         7m45s

觀察 Prometheus role rolebinding 有沒有成功安裝

1
2
3


oc get role,rolebinding | grep prometheus
role.rbac.authorization.k8s.io/prometheus-exporter-operator.v0.2.0-2nb8s   8m40s
rolebinding.rbac.authorization.k8s.io/prometheus-exporter-operator.v0.2.0-2nb8s-prometheus-exporfkbqh   8m40s

最後看看 Prometheus Operator 有沒有成功安裝到 Kubernetes 上

oc get deployments
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-exporter-operator   1/1     1            1           9m38s

看起來一切都很正常，但魔鬼藏在細節裡是不是 prometheus-exporter-operator Deployment版本如我們預期的一樣

1
2


oc get deployment prometheus-exporter-operator -o go-template --template '{{range .spec.template.spec.containers}} {{.image}}{{end}}'
 quay.io/3scale/prometheus-exporter-operator:v0.2.0

從上述輸出的結果來看版本跟 packagemanifests 所定義的是一樣的。

結語

雖然感覺得出來 RedHat 立意良好，但是操作那麼多 CRD 情況真的有點麻煩，另外我在測試這幾個 CRD 的時候發現當刪除 ClusterServiceVersion 的時候，Operator的相關資源都會被刪除，感覺這個邏輯不怎麼正確。

我個人覺得是不是 installplan 刪除或是 DISAPPROVED 才會刪除 ClusterServiceVersion ，使用者就算 誤刪 了也應該偵測到這件事情並重新生成一個 ClusterServiceVersion 才對。

以上純屬個人見解，希望能與大家討論交流。