对每个人而言，真正的职责只有一个：找到自我。然后在心中坚守其一生，全心全意，永不停息。所有其它的路都是不完整的，是人的逃避方式，是对大众理想的懦弱回归，是随波逐流，是对内心的恐惧 ——赫尔曼·黑塞《德米安》
写在前面
生产环境中的 k8s 集群安全不可忽略，即使是内网环境
容器化的应用部署虽然本质上没有变化，始终是机器上的一个进程
但是提高了安全问题的处理的复杂性
分享一个开源的 k8s 集群安全合规检查/漏洞扫描 工具 kubescape
博文内容涉及：
kubescape 简介介绍kubescape 命令行工具安装，扫描运行的集群kubescape 在集群下安装，通过 kubescape Clound 可视化查看扫描信息
理解不足小伙伴帮忙指正
需要有科学上网环境
对每个人而言，真正的职责只有一个：找到自我。然后在心中坚守其一生，全心全意，永不停息。所有其它的路都是不完整的，是人的逃避方式，是对大众理想的懦弱回归，是随波逐流，是对内心的恐惧 ——赫尔曼·黑塞《德米安》
简单介绍 k8s 安全问题不可忽略 镜像的不可变性使我们能够方便地部署、测试和发布镜像到其他环境，这是一个很大的优势，但也带来了潜在的风险：镜像及其依赖在过时或者被弃用时，无法自动更新或打新的安全补丁，尤其一些基础镜像，重新做镜像，需要依赖一些厂商，或者开源项目 team 来维护。
RedHat 在 2022 年 Kubernetes 安全报告中对 300 多名 DevOps、工程和安全专业人士进行了调查，发现对 容器安全威胁的担忧以及对容器安全投资的缺乏 是 31% 的受访者对容器策略最普遍的担忧。
支持这些担忧的是 93% 的受访者在过去 12 个月内在其 Kubernetes 环境中经历过至少一次安全事件，这些事件有时会导致收入或客户流失。超过一半的受访者 (55%) 在过去一年中还因为安全问题而不得不推迟应用程序的推出。
尽管媒体广泛关注网络攻击，但该报告强调，实际上是错误配置让 IT 专业人员彻夜难眠。Kubernetes 是高度可定制的，具有可以影响应用程序安全状况的各种配置选项。因此，受访者最担心的是容器和 Kubernetes 环境中的错误配置导致的风险暴露 (46%) —— 几乎是对攻击的担忧程度 (16%) 的三倍。尽可能自动化配置管理有助于缓解这些问题，因此安全工具 —— 而不是人类 —— 提供了帮助开发人员和 DevOps 团队更安全地配置容器和 Kubernetes 的护栏。
以上内容来自：Redhat Blog(The State of Kubernetes Security in 2022)
Kubescape 简单介绍 Kubescape 是一个开源的 Kubernetes 安全平台。它的功能包括 风险分析、安全合规性 和 错误配置扫描。针对 DevSecOps 从业者或平台工程师，提供易于使用的 CLI 界面、灵活的输出格式和自动扫描功能。同时对于小集群提供了免费的 在线 面板工具，它为 Kubernetes 用户和管理员节省了宝贵的时间、精力和资源。
Kubescape 可以扫描运行的集群、静态 YAML 文件和 本地 Helm Charts。它根据多个框架(包括 NSA-CISA、MITRE ATT&CK®和CIS Benchmark)检测错误配置。
Kubescape 由 ARMO 创建，是一个 Cloud Native Computing Foundation (CNCF) 沙盒项目。
Kubescape 如果小伙伴觉得重的话， kubectl 有一个类似的插件，个人十分推荐，叫做 kube-score ,很轻量，也可以做一些简单的合规性扫描。但是只有扫描合规性的提示，没有规范出处。
Kubescape 安装 当前的集群信息
1 2 3 4 5 6 7 8 9 10 11 12 13 ┌──[root@vms100.liruilongs.github.io]-[~] └─$kubectl get nodes NAME STATUS ROLES AGE VERSION vms100.liruilongs.github.io Ready control-plane 8d v1.25.1 vms101.liruilongs.github.io Ready control-plane 8d v1.25.1 vms102.liruilongs.github.io Ready control-plane 8d v1.25.1 vms103.liruilongs.github.io Ready <none> 8d v1.25.1 vms105.liruilongs.github.io Ready <none> 8d v1.25.1 vms106.liruilongs.github.io Ready <none> 8d v1.25.1 vms107.liruilongs.github.io Ready <none> 8d v1.25.1 vms108.liruilongs.github.io Ready <none> 8d v1.25.1 ┌──[root@vms100.liruilongs.github.io]-[~] └─$
集群节点信息
1 2 3 4 5 6 7 8 9 10 11 12 13 14 ┌──[root@vms100.liruilongs.github.io]-[~] └─$hostnamectl Static hostname: vms100.liruilongs.github.io Icon name: computer-vm Chassis: vm Machine ID: e93ae3f6cb354f3ba509eeb73568087e Boot ID: a1150b6d97dc4afbb81dae58f131a487 Virtualization: vmware Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 5.4.230-1.el7.elrepo.x86_64 Architecture: x86-64 ┌──[root@vms100.liruilongs.github.io]-[~] └─$
Kubescape 命令行工具安装
Kubescape CLI 安装 通过下面的方式自动安装
1 curl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash
如果没有科学上网，找一台可以访问的集群，下载 install.sh 文件，按照下面方式修改，获取 curl 命令，自行下载。
1 2 3 4 5 6 7 8 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$vim install.sh ....... echo "curl --progress-bar -L $DOWNLOAD_URL -o $OUTPUT " exit 1...
运行 shell，获取下载命令，找有网的机器下载
1 2 3 4 5 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$sh install.sh Installing Kubescape... curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescape
上传到指定位置
1 2 3 4 5 6 7 8 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$ls install.sh kubescape-ubuntu-latest ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$mv kubescape-ubuntu-latest /root/.kubescape/kubescape mv：是否覆盖"/root/.kubescape/kubescape" ？ y ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$
修改脚本，再次运行。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$vim install.sh ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$sh install.sh Installing Kubescape... curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescape Finished Installation. Your current version is: v2.0.183 [git enabled in build: true ] Usage: $ kubescape scan --enable-host-scan
到这里 ，kubescape 命令行工具即安装成功，扫描当前运行的集群可以运行如下命令。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubescape scan --enable-host-scan --format html --output results.html --verbose [info] Kubescape scanner starting [info] Installing host scanner [info] Downloading/Loading policy definitions [success] Downloaded/Loaded policy [info] Accessing Kubernetes objects [success] Accessed to Kubernetes objects [info] Requesting images vulnerabilities results [success] Requested images vulnerabilities results [info] Requesting Host scanner data [info] Host scanner version : v1.0.39 ◑[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-xshr6; error: the server could not find the requested resource (get pods http:host-scanner-xshr6:7888) ◒[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-4tgnp; error: the server could not find the requested resource (get pods http:host-scanner-4tgnp:7888) ....... [success] Done scanning. Cluster: kubernetes-admin-kubernetes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ApiVersion: hostdata.kubescape.cloud/v1beta0 Kind: KubeletInfo Name: vms105.liruilongs.github.io Controls: 21 (Failed: 6, Excluded: 0) +----------+--------------------------------+------------------------------------+------------------------+ | SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION | +----------+--------------------------------+------------------------------------+------------------------+ | High | CIS-4.1.7 Ensure that the | https://hub.armosec.io/docs/c-0168 | | | | certificate authorities file | | | | | permissions are set to 600 or | | | | | more restrictive | | | + +--------------------------------+------------------------------------+------------------------+ | | CIS-4.1.9 If the kubelet | https://hub.armosec.io/docs/c-0170 | | | | config.yaml configuration | | | | | file is being used validate | | | | | permissions set to 600 or more | | | | | restrictive | | | +----------+--------------------------------+------------------------------------+------------------------+ ........ +----------+--------------------------------+------------------------------------+------------------------+ | Low | CIS-4.2.6 Ensure that the | https://hub.armosec.io/docs/c-0177 | protectKernelDefaults | | | --protect-kernel-defaults | | | | | argument is set to true | | | + +--------------------------------+------------------------------------+------------------------+ | | CIS-4.2.7 Ensure that the | https://hub.armosec.io/docs/c-0178 | makeIPTablesUtilChains | | | --make-iptables-util-chains | | | | | argument is set to true | | | +----------+--------------------------------+------------------------------------+------------------------+ ApiVersion: v1 Kind: Namespace Name: kubescape Controls: 18 (Failed: 17, Excluded: 0) +----------+--------------------------------+------------------------------------+----------------------------------------------------------------+ | SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION | +----------+--------------------------------+------------------------------------+----------------------------------------------------------------+ | High | CIS-5.2.11 Minimize the | https://hub.armosec.io/docs/c-0202 | metadata.labels[pod-security.kubernetes.io/enforce]=baseline | | | admission of Windows | | | | | HostProcess Containers | | | + +--------------------------------+------------------------------------+ + | | CIS-5.2.2 Minimize the | https://hub.armosec.io/docs/c-0193 | | | | admission of privileged | | | | | containers | | | +----------+--------------------------------+------------------------------------+----------------------------------------------------------------+ | Medium | CIS-5.2.1 Ensure that | https://hub.armosec.io/docs/c-0192 | metadata.labels[pod-security.kubernetes.io/enforce]=YOUR_VALUE | | | the cluster has at least | | | | | one active policy control | | | | | mechanism in place | | | + +--------------------------------+------------------------------------+----------------------------------------------------------------+ ....... .......... ApiVersion: hostdata.kubescape.cloud/v1beta0 Kind: ControlPlaneInfo Name: vms102.liruilongs.github.io Controls: 25 (Failed: 1, Excluded: 0) +----------+--------------------------------+------------------------------------+-----------------------+ | SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION | +----------+--------------------------------+------------------------------------+-----------------------+ | High | CIS-1.1.20 Ensure that the | https://hub.armosec.io/docs/c-0111 | | | | Kubernetes PKI certificate | | | | | file permissions are set to | | | | | 600 or more restrictive | | | +----------+--------------------------------+------------------------------------+-----------------------+ ApiVersion: apps/v1 Kind: Deployment Name: local-path-provisioner Namespace: local-path-storage Controls: 35 (Failed: 18, Excluded: 0) ... ........ ......
输出的合规信息和漏洞信息。
异常问题
我在 kubescape 多次次扫描中，集群因部分节点因为端口问题，无法发生调度，节点上的 kubescape 对于的 pod 无法自行删除，如果上一次扫描的 pod 或则 ns 没有删除，那么下一次的扫描无法进行，之前创建的 pod 和 ns 状态一直为 Terminating, 解决办法需要 对命名空间进行彻底删除。
这里执行完命令会进入阻塞状态。
1 2 3 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubevirt] └─$kubectl delete ns kubescape-host-scanner namespace "kubescape-host-scanner" deleted
运行脚本删除
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s_shell_secript] └─$cat delete_namespace.sh coproc kubectl proxy --port=30990 & if [ $# -eq 0 ] ; then echo "后面加上你所要删除的ns." exit 1 fi kubectl get namespace $1 -o json > logging.json sed -i '/"finalizers"/{n;d}' logging.json curl -k -H "Content-Type: application/json" -X PUT --data-binary @logging.json http://127.0.0.1:30990/api/v1/namespaces/${1} /finalize kill %1┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s_shell_secript] └─$sh delete_namespace.sh kubescape-host-scanner
也可以离线运行Kubescape，时间网络原因，这里不做分享，有需要的小伙伴可以到 github 上的项目地址查看详细信息。
一些其他的用法 扫描正在运行的 Kubernetes 集群：
1 kubescape scan --enable-host-scan --verbose
使用替代的 kubeconfig 文件：
1 kubescape scan --kubeconfig cluster.conf
扫描特定的命名空间：
1 kubescape scan --include-namespaces development,staging,production
排除某些命名空间：
1 kubescape scan --exclude-namespaces kube-system,kube-public
部署前扫描本地 YAML/JSON 文件：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/helm] └─$kubescape scan --enable-host-scan kube-prometheus-stack.yaml --format html --output resout.html [info] Kubescape scanner starting [warning] in setContextMetadata. case : git-url; error: repository host 'gitee.com' not supported [info] Downloading/Loading policy definitions [success] Downloaded/Loaded policy [info] Accessing local objects [success] Done accessing local objects [info] Scanning GitLocal [success] Done scanning GitLocal ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Controls: 55 (Failed: 31, Excluded: 0, Skipped: 0) Failed Resources by Severity: Critical — 0, High — 34, Medium — 151, Low — 24 +----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+ | SEVERITY | CONTROL NAME | FAILED RESOURCES | EXCLUDED RESOURCES | ALL RESOURCES | % RISK-SCORE | +----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+ | High | Resources memory limit and request | 7 | 0 | 7 | 100% | | High | Resource limits | 7 | 0 | 7 | 100% | | High | List Kubernetes secrets | 4 | 0 | 7 | 57% | 。。。。。。。。。。。。。。。。。。。。。 | Medium | Allow privilege escalation | 6 | 0 | 7 | 86% | | Medium | Ingress and Egress blocked | 7 | 0 | 7 | 100% | 。。。。。。。。。。。。。。。。。。。。。 | Medium | CIS-5.4.1 Prefer using secrets as files over secrets as environment variables | 1 | 0 | 7 | 14% | | Medium | CIS-5.7.2 Ensure that the seccomp profile is set to docker/default in your pod definitions | 7 | 0 | 7 | 100% | | Medium | CIS-5.7.4 The default namespace should not be used | 56 | 0 | 61 | 92% | | Low | Immutable container filesystem | 6 | 0 | 7 | 86% | | Low | Configured readiness probe | 5 | 0 | 7 | 71% | | Low | Malicious admission controller (validating) | 1 | 0 | 1 | 100% | | Low | Pods in default namespace | 7 | 0 | 7 | 100% | | Low | Naked PODs | 1 | 0 | 1 | 100% | | Low | Label usage for resources | 3 | 0 | 7 | 43% | | Low | K8s common labels usage | 1 | 0 | 7 | 14% | +----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+ | | RESOURCE SUMMARY | 66 | 0 | 72 | 43.26% | +----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+ FRAMEWORKS: ArmoBest (risk: 33.51), cis-v1.23-t1.0.1 (risk: 62.75), DevOpsBest (risk: 61.72), AllControls (risk: 36.46), MITRE (risk: 11.79), NSA (risk: 33.74) [success] Scan results saved. filename: resout.html ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Scan results have not been submitted: run kubescape with the '--account' flag For more details: https://hub.armosec.io/docs/installing-kubescape?utm_campaign=Submit&utm_medium=CLI&utm_source=GitHub ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Run with '--verbose' /'-v' flag for detailed resources view ┌──[root@vms100.liruilongs.github.io]-[~/ansible/helm] └─$l
从 Git 存储库扫描 Kubernetes 清单文件：
1 kubescape scan https://github.com/kubescape/kubescape
扫描 Helm 图表,kubescape 将加载默认的 VALUES 文件。
1 kubescape scan </path/to/directory>
扫描 Kustomize 目录,Kubescape 将使用 kustomize 文件生成 Kubernetes YAML 对象并扫描它们以确保安全。
1 kubescape scan </path/to/directory>
使用 NSA 框架扫描正在运行的 Kubernetes 集群：
1 kubescape scan framework nsa
使用 MITRE ATT&CK® 框架扫描正在运行的 Kubernetes 集群：
1 kubescape scan framework mitre
使用控件名称或控件 ID 扫描特定控件。请参阅控件列表。
1 kubescape scan control "Privileged container"
指定报告输出格式 JSON：
1 kubescape scan --format json --format-version v2 --output results.json
XML：
1 kubescape scan --format junit --output results.xml
PDF:
1 kubescape scan --format pdf --output results.pdf
普罗米修斯指标：
1 kubescape scan --format prometheus
HTML
1 kubescape scan --format html --output results.html
显示所有扫描到的资源(包括通过的资源)：
1 kubescape scan --verbose
Kubescape 在集群下安装 Kubescape 也可以在集群下安装，通过集群安装，可以在 cloud 里的 UI 界面查看具体扫描信息，集群中安装 Kubescape，先决条件
确保您拥有 Kubescape Cloud 帐户——如果没有，请在此处注册
您需要拥有对集群的安装权限(您应该能够创建 Deployments、CronJobs、ConfigMaps 和 Secrets)
你必须有 Kubectl 和 Helm
集群要求
Kubescape 运算符组件至少需要 400Mib RAM 和 400m CPU
具体的安装可以参考下的教程： 需要注册 Kubescape Cloud，并且 只有的工作节点小于 10 个的时候才免费 。
https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster
注册登录
这里登录完会弹出一个安装部署，安装安装部署安装即可
添加 Helm 源，并更新
1 2 3 4 5 6 7 8 9 10 11 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$helm repo add kubescape https://kubescape.github.io/helm-charts/ "kubescape" has been added to your repositories┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "rancher-stable" chart repository ...Successfully got an update from the "botkube" chart repository ...Successfully got an update from the "awx-operator" chart repository ...Successfully got an update from the "kubescape" chart repository Update Complete. ⎈Happy Helming!⎈
运行 charts
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$helm upgrade --install kubescape kubescape/kubescape-cloud-operator -n kubescape --create-namespace --set account=97f09924-0c06-42e4-bdad-5b333321af77 --set clusterName=`kubectl config current-context` Release "kubescape" does not exist. Installing it now. ......... ...................... NAME: kubescape LAST DEPLOYED: Sat Feb 4 10:03:36 2023 NAMESPACE: kubescape STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Thank you for installing kubescape-cloud-operator version 1.9.5. In a few minutes your scan results will be available in the following link: https://cloud.armosec.io/config-scanning/kubernetes-admin-kubernetes You can see and change the values of your's recurring configurations daily scan in the following link: https://cloud.armosec.io/settings/assets/clusters/scheduled-scans?cluster=kubernetes-admin-kubernetes > kubectl -n kubescape get cj kubescape-scheduler -o=jsonpath=' {.metadata.name}{"\t" }{.spec.schedule}{"\n" }' You can see and change the values of your' s recurring images daily scan in the following link:https://cloud.armosec.io/settings/assets/images > kubectl -n kubescape get cj kubevuln-scheduler -o=jsonpath='{.metadata.name}{"\t"}{.spec.schedule}{"\n"}' See you!!! ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$
运行完成，安装提示操作，点击刚才注册生成的页面。
验证 Kubescape 在集群中运行状态
1 2 3 4 5 6 7 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$kubectl -n kubescape get deployments.apps NAME READY UP-TO-DATE AVAILABLE AGE gateway 1/1 1 1 19m kubescape 1/1 1 1 19m kubevuln 1/1 1 1 19m operator 1/1 1 1 19m
kubescape 会定期扫描
1 2 3 4 5 6 7 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$kubectl get cronjobs.batch -n kubescape NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE kubescape-scheduler 21 3 * * * False 0 <none> 20m kubevuln-scheduler 13 22 * * * False 0 <none> 20m ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$
在登录的主页中可以看到集群和第一次扫描结果
安全合规性扫描信息
可以通过不同的维度来看
合规性的维度查看
集群 Pod/Deploy 的维度查看
具体的 合规配置信息查看
漏洞扫描
关于 kubescape 和小伙伴分享到这里，时间关系，没有深入太多。偶尔听大佬谈到，所以研究一下，有需要的小伙伴快去尝试吧。
博文部分内容参考 文中涉及参考链接内容版权归原作者所有，如有侵权请告知
https://github.com/kubescape/kubescape
https://www.redhat.com/en/blog/state-kubernetes-security-2022-1
https://betterprogramming.pub/image-vulnerability-scanning-for-optimal-kubernetes-security-c3ba933190ef
https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster
© 2018-至今 liruilonger@gmail.com , All rights reserved. 保持署名-非商用-相同方式共享(CC BY-NC-SA 4.0)
