2023-11-14  351f894b1d34bff04ef49c3cd6cd4146 99+   29 分钟   4.3 k0次访问

K8s Pod 创建埋点处理(Mutating Admission Webhook)

对每个人而言，真正的职责只有一个：找到自我。然后在心中坚守其一生，全心全意，永不停息。所有其它的路都是不完整的，是人的逃避方式，是对大众理想的懦弱回归，是随波逐流，是对内心的恐惧 ——赫尔曼·黑塞《德米安》

写在前面

  • 工作中涉及到相关的知识
  • 在实际的生产中，我们可以有需求对 创建的资源做一些类似 埋点 相关的操作，比如添加一些 Pod 创建时的自定义验证逻辑，类似表单提交验证那样，或者希望对创建的资源对象进行加工，在比如给资源对象添加对应的 zone 标签，涉及到 SC 相关的标签，或者根据命名空间动态织入亲和性和拓扑相关约束,添加一些 卷
  • 上面的这些需求我们可以通过 k8s Admission Webhook来完成，博文为查阅资料整理笔记，内容为涉及
  • 一个大佬写好的 自定义准入控制器 Demo 学习
  • 理解不足小伙伴帮忙指正

对每个人而言，真正的职责只有一个：找到自我。然后在心中坚守其一生，全心全意，永不停息。所有其它的路都是不完整的，是人的逃避方式，是对大众理想的懦弱回归，是随波逐流，是对内心的恐惧 ——赫尔曼·黑塞《德米安》

学习之前，建议看看这两篇官网的文章：

准入控制器参考:

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

Kubernetes 准入控制器指南:

https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

学习的项目文章：

https://didil.medium.com/building-a-kubernetes-mutating-admission-webhook-7e48729523ed

翻译版本：

https://cloudnative.to/blog/mutating-admission-webhook/

通过 Admission Webhook 为每个创建的 Pod(打了指定标签) 添加一个 CM ，以卷的方式挂载，类似为 每个命名空间 中的 Pod 自动挂载当前命名默认生成 SA 的 token 一样。

克隆一下作者的项目：

https://github.com/didil/k8s-hello-mutating-webhook

1
2
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook]
└─$git clone  https://github.com/didil/k8s-hello-mutating-webhook.git

确认一下 对应的 Webhook 钩子对应的镜像能不能使用

1
2
3
4
5
6
7
8
9
10
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$docker pull quay.io/didil/hello-webhook:0.1.9
0.1.9: Pulling from didil/hello-webhook
df20fa9351a1: Pull complete
c5ef763ec908: Pull complete
242c22b4c8b8: Pull complete
f201599daacc: Pull complete
Digest: sha256:8c6ef414e4df46f15b9b63f83bd24f7c6a2cc5b5b937c53f0811fd509969ad05
Status: Downloaded newer image for quay.io/didil/hello-webhook:0.1.9
quay.io/didil/hello-webhook:0.1.9

学习的环境

1
2
3
4
5
6
7
8
9
10
11
┌──[root@vms100.liruilongs.github.io]-[~]
└─$kubectl get node
NAME                          STATUS   ROLES           AGE    VERSION
vms100.liruilongs.github.io   Ready    control-plane   292d   v1.25.1
vms101.liruilongs.github.io   Ready    control-plane   292d   v1.25.1
vms102.liruilongs.github.io   Ready    control-plane   292d   v1.25.1
vms103.liruilongs.github.io   Ready    <none>          292d   v1.25.1
vms105.liruilongs.github.io   Ready    <none>          292d   v1.25.1
vms106.liruilongs.github.io   Ready    <none>          292d   v1.25.1
┌──[root@vms100.liruilongs.github.io]-[~]
└─$

看下作者的 yaml 文件，可以看到，当前作者使用 kustomization 来管理 k8s yaml 资源文件, kustomize 是一个 生成 k8s 资源 yaml 文件的插件，支持继承组合等一些面向对象思维方面的 yaml 文件生成

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$ls
csr  deployment  other
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$tree .
.
├── csr
│   ├── csr-job.yaml
│   ├── csr-rolebinding.yaml
│   ├── csr-role.yaml
│   ├── csr-sa.yaml
│   └── kustomization.yaml
├── deployment
│   ├── deployment.yaml
│   └── kustomization.yaml
└── other
    ├── configmap.yaml
    ├── kustomization.yaml
    ├── service.yaml
    └── webhookconf.yaml

3 directories, 11 files
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$

需要安装 kustomize ,嫌麻烦当然也可以直接看

1
2
3
4
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$kustomize version
{Version:kustomize/v4.5.7 GitCommit:56d82a8378dfc8dc3b3b1085e5a6e67b82966bd7 BuildDate:2022-08-02T16:35:54Z GoOs:linux GoArch:amd64}

这里分别看下作者的三个目录里面放了什么

deployment 目录主要为注入逻辑的工作负载hello-webhook，这里需要注意的是挂载了一个证书文件，由于 Webhook 必须通过 HTTPS 提供，因此我们需要为服务器提供适当的自签名 CA 签名。证书的公用名 （CN） 必须与 Kubernetes API 服务器使用的服务器名称匹配。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$ls
deployment.yaml  kustomization.yaml
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$kubectl kustomize ./
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hello-webhook
  name: hello-webhook-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-webhook
  template:
    metadata:
      labels:
        app: hello-webhook
    spec:
      containers:
      - image: quay.io/didil/hello-webhook:0.1.9
        name: hello-webhook
        ports:
        - containerPort: 8000
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
        volumeMounts:
        - mountPath: /tls
          name: hello-tls-secret
          readOnly: true
      volumes:
      - name: hello-tls-secret
        secret:
          secretName: hello-tls-secret
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$

下面为 webhook 的 核心逻辑,用于对 Pod 添加对应的 CM 卷 ,添加对应的操作记录，返回修改后的对象

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
func (app *App) HandleMutate(w http.ResponseWriter, r *http.Request) {
 admissionReview := &admissionv1.AdmissionReview{}

 // read the AdmissionReview from the request json body
 err := readJSON(r, admissionReview)
 if err != nil {
  app.HandleError(w, r, err)
  return
 }

 // unmarshal the pod from the AdmissionRequest
 pod := &corev1.Pod{}
 if err := json.Unmarshal(admissionReview.Request.Object.Raw, pod); err != nil {
  app.HandleError(w, r, fmt.Errorf("unmarshal to pod: %v", err))
  return
 }

 // add the volume to the pod
 pod.Spec.Volumes = append(pod.Spec.Volumes, corev1.Volume{
  Name: "hello-volume",
  VolumeSource: corev1.VolumeSource{
   ConfigMap: &corev1.ConfigMapVolumeSource{
    LocalObjectReference: corev1.LocalObjectReference{
     Name: "hello-configmap",
    },
   },
  },
 })

 // add volume mount to all containers in the pod
 for i := 0; i < len(pod.Spec.Containers); i++ {
  pod.Spec.Containers[i].VolumeMounts = append(pod.Spec.Containers[i].VolumeMounts, corev1.VolumeMount{
   Name:      "hello-volume",
   MountPath: "/etc/config",
  })
 }

 containersBytes, err := json.Marshal(&pod.Spec.Containers)
 if err != nil {
  app.HandleError(w, r, fmt.Errorf("marshall containers: %v", err))
  return
 }

 volumesBytes, err := json.Marshal(&pod.Spec.Volumes)
 if err != nil {
  app.HandleError(w, r, fmt.Errorf("marshall volumes: %v", err))
  return
 }

 // build json patch
 patch := []JSONPatchEntry{
  JSONPatchEntry{
   OP:    "add",
   Path:  "/metadata/labels/hello-added",
   Value: []byte(`"OK"`),
  },
  JSONPatchEntry{
   OP:    "replace",
   Path:  "/spec/containers",
   Value: containersBytes,
  },
  JSONPatchEntry{
   OP:    "replace",
   Path:  "/spec/volumes",
   Value: volumesBytes,
  },
 }

 patchBytes, err := json.Marshal(&patch)
 if err != nil {
  app.HandleError(w, r, fmt.Errorf("marshall jsonpatch: %v", err))
  return
 }

 patchType := admissionv1.PatchTypeJSONPatch

 // build admission response
 admissionResponse := &admissionv1.AdmissionResponse{
  UID:       admissionReview.Request.UID,
  Allowed:   true,
  Patch:     patchBytes,
  PatchType: &patchType,
 }

 respAdmissionReview := &admissionv1.AdmissionReview{
  TypeMeta: metav1.TypeMeta{
   Kind:       "AdmissionReview",
   APIVersion: "admission.k8s.io/v1",
  },
  Response: admissionResponse,
 }

 jsonOk(w, &respAdmissionReview)
}

上述代码主要做了如下事情：

  • 将来自 Http 请求中的 AdmissionReview json 输入反序列化。
  • 读取 Pod 的 spec 信息。
  • 将 hello-configmap 作为数据源，添加 hello-volume 卷到 Pod。
  • 挂载卷至 Pod 容器中。
  • 以 JSON PATCH 的形式记录变更信息，包括卷的变更，卷挂载信息的变更。顺道为容器添加一个“hello-added=true”的标签。
  • 构建 json 格式的响应结果，结果中包含了这次请求中的被修改的部分。

other 目录主要放了 需要动态织入的 CM，以卷的方式使用，以及 MutatingWebhookConfiguration 准入控制器的定义，以及 webhook 的 SVC(SVC 名字太长，没办法生成证书，需要修改一下)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/other]
└─$kubectl kustomize .
apiVersion: v1
data:
  hello.txt: "\n /$$$$$$$$ /$$   /$$ /$$$$$$  /$$$$$$        /$$$$$$  /$$$$$$        /$$
    \  /$$  /$$$$$$   /$$$$$$ \n|__  $$__/| $$  | $$|_  $$_/ /$$__  $$      |_  $$_/
    /$$__  $$      | $$  /$$/ /$$__  $$ /$$__  $$\n   | $$   | $$  | $$  | $$  | $$
    \ \\__/        | $$  | $$  \\__/      | $$ /$$/ | $$  \\ $$| $$  \\__/\n   | $$
    \  | $$$$$$$$  | $$  |  $$$$$$         | $$  |  $$$$$$       | $$$$$/  |  $$$$$$/|
    \ $$$$$$ \n   | $$   | $$__  $$  | $$   \\____  $$        | $$   \\____  $$      |
    $$  $$   >$$__  $$ \\____  $$\n   | $$   | $$  | $$  | $$   /$$  \\ $$        |
    $$   /$$  \\ $$      | $$\\  $$ | $$  \\ $$ /$$  \\ $$\n   | $$   | $$  | $$ /$$$$$$|
    \ $$$$$$/       /$$$$$$|  $$$$$$/      | $$ \\  $$|  $$$$$$/|  $$$$$$/\n   |__/
    \  |__/  |__/|______/ \\______/       |______/ \\______/       |__/  \\__/ \\______/
    \ \\______/ \n                                                                                                  \n
    \                                                                                                 \n
    \                                                                                                 \n"
kind: ConfigMap
metadata:
  name: hello-configmap
---
apiVersion: v1
kind: Service
metadata:
  name: hello-webhook-service
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 8000
  selector:
    app: hello-webhook
  type: ClusterIP
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: hello-webhook.leclouddev.com
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    service:
      name: hello-webhook-service
      namespace: default
      path: /mutate
  name: hello-webhook.leclouddev.com
  objectSelector:
    matchLabels:
      hello: "true"
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: Namespaced
  sideEffects: None
  timeoutSeconds: 10
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/other]
└─$

简单分析一下 MutatingWebhookConfiguration ,主要的配置 webhooks 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: hello-webhook.leclouddev.com
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig: #指定了客户端配置，用于指定 Webhook 的服务信息。
    service:
      name: hello-webhook-service
      namespace: default
      path: /mutate
  name: hello-webhook.leclouddev.com #指定了 Webhook 的名称
  objectSelector: # 指定了对象选择器，用于选择要应用 Webhook 的对象。
    matchLabels:
      hello: "true"
  rules: #指定了 Webhook 的规则
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: Namespaced
  sideEffects: None
  timeoutSeconds: 10 #指定了 Webhook 的超时时间，此处为 10 秒。

这里可以看到当前准入控制器 webhook 只处理 打了标签 hello=true 的 pod

csr 目录为权限，生成证书的 Job，SA 以及 通过 SA 添加对应的集群权限

生成的相关证书,用于 K8s Webhook 通信，相关项目地址：

https://github.com/didil/k8s-webhook-cert-manager

当前作者的 csr-job.yaml 封装的 pod 比较旧，shell 脚本需要调整，所有我们不需要，直接本地生成 证书和对应的秘密

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$ls
csr-job.yaml  csr-rolebinding.yaml  csr-role.yaml  csr-sa.yaml  kustomization.yaml
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl kustomize  ./
apiVersion: v1
kind: ServiceAccount
metadata:
  name: webhook-cert-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: webhook-cert-cluster-role
rules:
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  verbs:
  - get
  - create
  - patch
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - create
  - get
  - delete
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/approval
  verbs:
  - update
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - get
  - patch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: webhook-cert-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: webhook-cert-cluster-role
subjects:
- kind: ServiceAccount
  name: webhook-cert-sa
  namespace: default
---
apiVersion: batch/v1
kind: Job
metadata:
  name: webhook-cert-setup
spec:
  backoffLimit: 3
  template:
    spec:
      containers:
      - args:
        - --service
        - hello-webhook-service
        - --webhook
        - hello-webhook.leclouddev.com
        - --secret
        - hello-tls-secret
        - --namespace
        - default
        command:
        - ./generate_certificate.sh
        image: quay.io/didil/k8s-webhook-cert-manager:0.13.19-1-a
        name: webhook-cert-setup
      restartPolicy: OnFailure
      serviceAccountName: webhook-cert-sa
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$

部署项目

在部署之前，我们需要改一下命名空间,创建一个新的命名空间，mutating-webhook

1
2
3
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl create  ns mutating-webhook
namespace/mutating-webhook created

然后修改一下每个目录的 kustomization.yaml 文件,类似下面这样，添加 namespace: mutating-webhook

1
2
3
4
5
6
7
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/other]
└─$cat kustomization.yaml
namespace: mutating-webhook
resources:
- configmap.yaml
- service.yaml
- webhookconf.yaml

然后还需要修改一下证书生成的的命名空间

1
./generate_certificate.sh  --service hello-webhook-service --webhook hello-webhook.leclouddev.com --secret hello-tls-secret --namespace mutating-webhook

之后就可以部署了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl apply  -k ./
serviceaccount/webhook-cert-sa created
clusterrole.rbac.authorization.k8s.io/webhook-cert-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/webhook-cert-cluster-role-binding created
job.batch/webhook-cert-setup created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl apply  -k ../other/
configmap/hello-configmap created
service/hello-webhook-service created
mutatingwebhookconfiguration.admissionregistration.k8s.io/hello-webhook.leclouddev.com created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl apply  -k ../deployment/
deployment.apps/hello-webhook-deployment created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl config  set-context --current --namespace mutating-webhook
Context "kubernetes-admin@kubernetes" modified.

webhook 一直没办法正常运行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl get all
NAME                                            READY   STATUS              RESTARTS      AGE
pod/hello-webhook-deployment-7f599b95c4-pg9w5   0/1     ContainerCreating   0             55s
pod/webhook-cert-setup-v62rt                    0/1     CrashLoopBackOff    2 (15s ago)   79s

NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/hello-webhook-service   ClusterIP   10.103.24.30   <none>        443/TCP   59s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-webhook-deployment   0/1     1            0           55s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-webhook-deployment-7f599b95c4   1         1         0       55s

NAME                           COMPLETIONS   DURATION   AGE
job.batch/webhook-cert-setup   0/1           79s        79s
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$

这里发现作者生成证书的 job 中脚本使用的 镜像中的 kubectl 版本太低了，没办法正常执行，所以我这么直接在的本地生成 证书，创建对应的 secret ,任然使用作者的 脚本generate_certificate.sh

作者原来的证书签名请求对应的 API 资源对象使用的是bata 版本，现在已经更新为正式版本，需要替换一下,其他部分逻辑也需要调整，下面为调整后的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$cat generate_certificate.sh
#!/usr/bin/env sh

set -e

usage() {
  cat <<EOF
Generate certificate suitable for use with any Kubernetes Mutating Webhook.
This script uses k8s' CertificateSigningRequest API to a generate a
certificate signed by k8s CA suitable for use with any Kubernetes Mutating Webhook service pod.
This requires permissions to create and approve CSR. See
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
detailed explantion and additional instructions.
The server key/cert k8s CA cert are stored in a k8s secret.
usage: ${0} [OPTIONS]
The following flags are required.
    --service          Service name of webhook.
    --webhook          Webhook config name.
    --namespace        Namespace where webhook service and secret reside.
    --secret           Secret name for CA certificate and server certificate/key pair.
The following flags are optional.
    --webhook-kind     Webhook kind, either MutatingWebhookConfiguration or
                       ValidatingWebhookConfiguration (defaults to MutatingWebhookConfiguration)
EOF
  exit 1
}

while [ $# -gt 0 ]; do
  case ${1} in
      --service)
          service="$2"
          shift
          ;;
      --webhook)
          webhook="$2"
          shift
          ;;
      --secret)
          secret="$2"
          shift
          ;;
      --namespace)
          namespace="$2"
          shift
          ;;
      --webhook-kind)
          kind="$2"
          shift
          ;;
      *)
          usage
          ;;
  esac
  shift
done

[ -z "${service}" ] && echo "ERROR: --service flag is required" && exit 1
[ -z "${webhook}" ] && echo "ERROR: --webhook flag is required" && exit 1
[ -z "${secret}" ] && echo "ERROR: --secret flag is required" && exit 1
[ -z "${namespace}" ] && echo "ERROR: --namespace flag is required" && exit 1

fullServiceDomain="${service}.${namespace}.svc"

# THE CN has a limit of 64 characters. We could remove the namespace and svc
# and rely on the Subject Alternative Name (SAN), but there is a bug in EKS
# that discards the SAN when signing the certificates.
#
# https://github.com/awslabs/amazon-eks-ami/issues/341
if [ ${#fullServiceDomain} -gt 64 ] ; then
  echo "ERROR: common name exceeds the 64 character limit: ${fullServiceDomain}"
  exit 1
fi

if [ ! -x "$(command -v openssl)" ]; then
  echo "ERROR: openssl not found"
  exit 1
fi

csrName=${service}.${namespace}
tmpdir=$(mktemp -d)
echo "creating certs in tmpdir ${tmpdir} "

cat <<EOF >> "${tmpdir}/csr.conf"
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${fullServiceDomain}
DNS.4 = ${fullServiceDomain}.cluster.local
EOF
echo "/CN=${fullServiceDomain}"
openssl genrsa -out "${tmpdir}/server-key.pem" 2048
#openssl req -new -key "${tmpdir}/server-key.pem" -subj "/CN=${fullServiceDomain}" -out "${tmpdir}/server.csr" -config "${tmpdir}/csr.conf"
openssl req -new -key "${tmpdir}/server-key.pem" -subj "/CN=system:node:${fullServiceDomain};/O=system:nodes" -out "${tmpdir}/server.csr" -config "${tmpdir}/csr.conf"
set +e
# clean-up any previously created CSR for our service. Ignore errors if not present.
if kubectl delete csr "${csrName}"; then
    echo "WARN: Previous CSR was found and removed."
fi
set -e

# create server cert/key CSR and send it to k8s api
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: ${csrName}
spec:
  #signerName: kubernetes.io/kube-apiserver-client
  signerName: kubernetes.io/kubelet-serving
  groups:
  - system:authenticated
  request: $(base64 < "${tmpdir}/server.csr" | tr -d '\n')
  usages:
  - server auth
  - digital signature
  - key encipherment
EOF

set +e
# verify CSR has been created
while true; do
  if kubectl get csr "${csrName}"; then
      echo "CertificateSigningRequest create succsee"
      break
  fi
done
set -e

# approve and fetch the signed certificate . !! not working with k8s 1.19.1, running the command separately outside of the container / node
set +e
while true; do
  if kubectl certificate approve "${csrName}"; then
     echo "${csrName} certificate approve"
     break
  fi
done

set -e

set +e
# verify certificate has been signed
i=1
while [ "$i" -ne 10 ]
do
  serverCert=$(kubectl get csr "${csrName}" -o jsonpath='{.status.certificate}')
  if [ "${serverCert}" != '' ]; then
      break
  fi
  sleep 5
  i=$((i + 1))
done

set -e
if [ "${serverCert}" = '' ]; then
  echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
  exit 1
fi

echo "${serverCert}" | openssl base64 -d -A -out "${tmpdir}/server-cert.pem"

# create the secret with CA cert and server cert/key
kubectl create secret tls "${secret}" \
      --key="${tmpdir}/server-key.pem" \
      --cert="${tmpdir}/server-cert.pem" \
      --dry-run -o yaml |
  kubectl -n "${namespace}" apply -f -

#caBundle=$(base64 < /run/secrets/kubernetes.io/serviceaccount/ca.crt  | tr -d '\n')
caBundle=$(cat ${tmpdir}/server-cert.pem)
set +e
# Patch the webhook adding the caBundle. It uses an `add` operation to avoid errors in OpenShift because it doesn't set
# a default value of empty string like Kubernetes. Instead, it doesn't create the caBundle key.
# As the webhook is not created yet (the process should be done manually right after this job is created),
# the job will not end until the webhook is patched.
while true; do
  echo "INFO: Trying to patch webhook adding the caBundle."
  if kubectl patch "${kind:-mutatingwebhookconfiguration}" "${webhook}" --type='json' -p "[{'op': 'add', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${serverCert}'}]"; then
      break
  fi
  echo "INFO: webhook not patched. Retrying in 5s..."
  sleep 5
done
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$

生成证书，并且 创建 csr secret ，同时更新 mutatingwebhookconfiguration 的 caBundle 字段

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$./generate_certificate.sh  --service webhook-svc --webhook hello-webhook.leclouddev.com --secret hello-tls-secret --namespace mutating-webhook
creating certs in tmpdir /tmp/tmp.Di367dgyMz
/CN=webhook-svc.mutating-webhook.svc
Generating RSA private key, 2048 bit long modulus
........................................+++
.........+++
e is 65537 (0x10001)
Error from server (NotFound): certificatesigningrequests.certificates.k8s.io "webhook-svc.mutating-webhook" not found
certificatesigningrequest.certificates.k8s.io/webhook-svc.mutating-webhook created
NAME                           AGE   SIGNERNAME                      REQUESTOR          REQUESTEDDURATION   CONDITION
webhook-svc.mutating-webhook   0s    kubernetes.io/kubelet-serving   kubernetes-admin   <none>              Pending
CertificateSigningRequest create succsee
certificatesigningrequest.certificates.k8s.io/webhook-svc.mutating-webhook approved
webhook-svc.mutating-webhook certificate approve
W1115 17:25:40.403890   66211 helpers.go:663] --dry-run is deprecated and can be replaced with --dry-run=client.
secret/hello-tls-secret created
INFO: Trying to patch webhook adding the caBundle.
mutatingwebhookconfiguration.admissionregistration.k8s.io/hello-webhook.leclouddev.com patched

如果 命名空间或者 svc 名字太长的话，会报下面的错，需要调整短一点

1
2
3
4
5
6
7
8
9
10
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$./generate_certificate.sh  --service hello-webhook-service --webhook hello-webhook.leclouddev.com --secret hello-tls-secret --namespace k8s-hello-mutating-webhook
creating certs in tmpdir /tmp/tmp.GcNh5TwKXP
/CN=hello-webhook-service.k8s-hello-mutating-webhook.svc
Generating RSA private key, 2048 bit long modulus
....+++
...................................................+++
e is 65537 (0x10001)
problems making Certificate Request
140682165290896:error:0D07A097:asn1 encoding routines:ASN1_mbstring_ncopy:string too long:a_mbstr.c:158:maxsize=64

测试结果

创建指定标签的 Pod ,自动挂载 CM

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl run busybox-1 --image=busybox  --restart=Never -l=app=busybox,hello=true -- sleep 3600
pod/busybox-1 created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl exec busybox-1 -it -- sh -c "ls /etc/config/hello.txt"
/etc/config/hello.txt
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl exec busybox-1 -it -- sh -c "cat /etc/config/hello.txt"

 /$$$$$$$$ /$$   /$$ /$$$$$$  /$$$$$$        /$$$$$$  /$$$$$$        /$$   /$$  /$$$$$$   /$$$$$$
|__  $$__/| $$  | $$|_  $$_/ /$$__  $$      |_  $$_/ /$$__  $$      | $$  /$$/ /$$__  $$ /$$__  $$
   | $$   | $$  | $$  | $$  | $$  \__/        | $$  | $$  \__/      | $$ /$$/ | $$  \ $$| $$  \__/
   | $$   | $$$$$$$$  | $$  |  $$$$$$         | $$  |  $$$$$$       | $$$$$/  |  $$$$$$/|  $$$$$$
   | $$   | $$__  $$  | $$   \____  $$        | $$   \____  $$      | $$  $$   >$$__  $$ \____  $$
   | $$   | $$  | $$  | $$   /$$  \ $$        | $$   /$$  \ $$      | $$\  $$ | $$  \ $$ /$$  \ $$
   | $$   | $$  | $$ /$$$$$$|  $$$$$$/       /$$$$$$|  $$$$$$/      | $$ \  $$|  $$$$$$/|  $$$$$$/
   |__/   |__/  |__/|______/ \______/       |______/ \______/       |__/  \__/ \______/  \______/


没有标签的没有自动挂载

1
2
3
4
5
6
7
8
9
10
11
12
13
14
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl run busybox-2 --image=busybox --restart=Never -l=app=busybox -- sleep 3600
pod/busybox-2 created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl get pod -l=app=busybox -L=hello-added
NAME        READY   STATUS    RESTARTS   AGE   HELLO-ADDED
busybox-1   1/1     Running   0          30m   OK
busybox-2   1/1     Running   0          28s
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl exec busybox-2 -it -- sh -c "ls /etc/config/hello.txt"
ls: /etc/config/hello.txt: No such file or directory
command terminated with exit code 1
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$

博文部分内容参考

© 文中涉及参考链接内容版权归原作者所有，如有侵权请告知，这是一个开源项目，如果你认可它，不要吝啬星星哦 :)

https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#service-reference

https://kubernetes.io/zh-cn/docs/reference/kubernetes-api/extend-resources/mutating-webhook-configuration-v1/

https://cloudnative.to/blog/mutating-admission-webhook/

https://github.com/didil/k8s-hello-mutating-webhook

https://didil.medium.com/building-a-kubernetes-mutating-admission-webhook-7e48729523ed

© 2018-2023 liruilonger@gmail.com, All rights reserved. 保持署名-非商用-相同方式共享(CC BY-NC-SA 4.0)

K8s Pod 创建埋点处理(Mutating Admission Webhook)

https://liruilongs.github.io/2023/11/14/待发布/K8s Pod 创建埋点处理(Mutating Admission Webhook)/

发布于

2023-11-14

更新于

2024-01-13

许可协议

# 相关文章
  1.K8s 镜像缓存管理 kube-fledged 认知
  2.K8s集群故障(The connection to the server <host>:<port> was refused - did you specify the right host or port)解决
  3.关于 Kubernetes中Admission Controllers(准入控制器) 认知的一些笔记
  4.关于AI(深度学习)相关项目 K8s 部署的一些思考
  5.K8s Pod 安全认知：从openshift SCC 到 PSP 弃用以及现在的 PSA
  6.K8s：通过 PSA(Pod Security Admission) 定义K8s 集群安全基线
  7.Kubernetes 触发 OOMKilled(内存杀手)如何排除故障
  8.K8s：K8s 20个常用命令汇总
# 推荐文章
  1.《Kubernetes权威指南:从Docker到Kubernetes实践全接触》读书笔记
  2.关于 Git 重写历史的一些笔记
  3.关于Git分支变基操作的一些笔记
  4.关于Git分支基础知识的一些笔记
  5.关于Git分支工作流的一些笔记
  6.关于Git分支高级合并的一些笔记整理

喜欢这篇文章？打赏一下作者吧

评论

目录

关注我

链接

最新评论

加载中，最新评论有1分钟缓存...

最新文章

分类

归档

标签

Kubernetes94
Linux42
Ansible25
Python14
Ceph11
Docker8
JAVA8
华为云8
Git7
Go6
OpenShift6
DNS5
OKD5
程序人生5
selenium4
容器4
网络4
CKA3
GO3
ISCSI3
查看全部>>

订阅更新

Your browser is out-of-date!

Update your browser to view this website correctly.&npsb;Update my browser now

×

×