Kubernetes DANM 电脑版发表于:2022/10/31 18:16 ![](https://img.tnblog.net/arcimg/hb/58f541af8f8848ffb4e1d30cea3f3d19.png) >#Kubernetes DANM [TOC] tn>只支持Kubernetes 1.17.X - 1.19.X的版本。 ## DANM 简介 tn2>DANM是诺基亚将电信级网络管理引入Kubernetes集群的解决方案!DANM在公司已有4年多的历史,目前已投入生产,最终在GitHub上为所有人提供。<br/> 其主要功能是为Pod提供多个网络接口与网卡的关联,这样使得更多CNI插件可以共同管理与关联,且互不影响。并且所链接的网络模式可以是L3,L2等模式。 ## 安装DANM tn2>首先克隆DANM到本地,并执行`./build_danm.sh`脚本完成安装(国内用户还不可以)。 ```bash git clone https://github.com/nokia/danm.git cd danm ./build_danm.sh ``` tn2>要想启动它需要有4个容器镜像: | 镜像 | 描述 | | ------------ | ------------ | | danm-cni-plugins | 此图包含核心CNI插件(danm、fakeipam)。稍后,它将被部署为一个DaemonSet,将这些二进制文件放置在每个Kubernetes节点中。 | | netwatcher | 镜像将由netwatcher守护程序集使用 | | webhook | 此图像将由webhook部署使用 | | svcwatcher | 如果您选择安装,svcwatcher守护程序集将使用此镜像。 | ## 国内手动安装安装 tn2>首先先将项目克隆到本地。 ```bash git clone https://github.com/nokia/danm.git cd danm ``` >### 扩展 Kubernetes API tn2>有两种可选的安装方式:<br/> 1.轻量级:通过从项目根目录执行以下命令,使用DanmNet和DanmEp CRD对象扩展Kubernetes API,以简化网络管理体验: ```bash kubectl create -f integration/crds/lightweight ``` tn2>2.生产:通过从项目根目录执行以下命令,使用TenantNetwork、ClusterNetwork、TenantConfig和DanmEp CRD对象扩展Kubernetes API,以获得支持多租户的生产级网络管理体验: ```bash kubectl create -f integration/crds/production ``` tn2>这里我们选择轻量级的即可。 >### 创建 DANM CNI 服务帐户 tn2>为了完成其工作,DANM需要一个服务帐户来访问集群,并为该帐户提供必要的RBAC角色。 我们还需要提取此服务帐户的令牌,因为下一步将需要它: ```bash kubectl create --namespace kube-system serviceaccount danm vim danm-secret.yaml ``` ```yaml apiVersion: v1 kind: Secret metadata: name: danm namespace: kube-system annotations: kubernetes.io/service-account.name: danm type: kubernetes.io/service-account-token ``` ```bash kubectl create -f danm-secret.yaml SERVICEACCOUNT_TOKEN=$(kubectl get --namespace kube-system secrets danm -o jsonpath='{.data.token}' | base64 -d) ``` >### 创建有效的CNI配置文件 tn2>根据以下ecxample配置,将有效的CNI配置文件放入所有kubelet节点的CNI目录(默认为/etc/cni/net.d/00-danm.conf): ```bash { "cniVersion": "0.3.1", "name": "meta_cni", "name_comment": "Mandatory parameter, but can be anything", "type": "danm", "type_comment": "Mandatory parameter according to CNI spec, MUST be set to danm", "kubeconfig": "/etc/kubernetes/kubeconfig/danmc.yml", "kubeconfig_comment": "Mandatory parameter, must point to a valid kubeconfig file containing the necessary RBAC setting for DANM's service account", "cniDir": "/etc/cni/net.d", "cniDir_comment": "Optional parameter, if defined CNI config files for static delegates are searched here. Default value is /etc/cni/net.d", "namingScheme": "awesome", "namingScheme_comment": "Optional parameter, if it is set to legacy container network interface names are set exactly to DanmNet.Spec.Options.container_prefix, otherwise prefix simply behaves as a prefix and is suffixed with a sequence ID. Default value is empty (e.g. not legacy)" } ``` tn2>在创建时,`kubeconfig`参数可以指定DANM的配置,但创建kubeconfig之前我们还需要创建其中运用到的环境变量:集群名称、集群服务器、集群证书。 ```bash CLUSTER_NAME=$(kubectl config view -o jsonpath='{.clusters[0].name}') CLUSTER_SERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}') CLUSTER_CA_CERTIFICATE=$(kubectl config view --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}') ``` tn2>然后我们创建`kubeconfig`文件。 ```bash mkdir /etc/kubernetes/kubeconfig/ -p vim /etc/kubernetes/kubeconfig/danmc.yml ``` ```yaml apiVersion: v1 kind: Config current-context: default clusters: - cluster: certificate-authority-data: ${CLUSTER_CA_CERTIFICATE} server: ${CLUSTER_SERVER} name: ${CLUSTER_NAME} contexts: - context: cluster: ${CLUSTER_NAME} user: danm name: default users: - name: danm user: token: ${SERVICEACCOUNT_TOKEN} preferences: {} ``` ![](https://img.tnblog.net/arcimg/hb/1e81ca8df6cc485098a81ead34ddfda5.png) tn2>还需要一定的RBAC权限,方便DANM正常运行 ```bash kubectl create -f integration/cni_config/danm_rbac.yaml ``` ![](https://img.tnblog.net/arcimg/hb/d93d2e036efd43948653e04d3eb0fffc.png) >### 导入镜像 tn2>docker的方式 ```bash docker pull danmcni/damn-installer:latest docker pull danmcni/danm-cni-plugins:latest docker pull danmcni/webhook:latest docker pull danmcni/svcwatcher:latest docker pull danmcni/netwatcher:latest # 转本地 docker tag danmcni/damn-installer:latest damn-installer:latest docker tag danmcni/danm-cni-plugins:latest danm-cni-plugins:latest docker tag danmcni/webhook:latest webhook:latest docker tag danmcni/svcwatcher:latest svcwatcher:latest docker tag danmcni/netwatcher:latest netwatcher:latest ``` >### 创建CNI插件DaemonSet tn2>生成工具 ```yaml cd cmd/danm/ go build danm.go cd ../../ cd cmd/fakeipam/ go build fakeipam.go ``` ![](https://img.tnblog.net/arcimg/hb/07e920ae90834427929d4701bb76c318.png) ![](https://img.tnblog.net/arcimg/hb/b6d8f190b8c94d619e17a4fd48eeeadd.png) tn2>在创建CNI插件之前我们需要将DANM的cni工具(danm与fakeipam)放到我们的`/opt/cni/bin`目录下面。 ```bash cd danm chmod +x cmd/danm/danm cmd/fakeipam/fakeipam cp cmd/danm/danm /opt/cni/bin cp cmd/fakeipam/fakeipam /opt/cni/bin ls /opt/cni/bin/ ``` tn2>安装到其他节点上 ```bash scp cmd/danm/danm node01:/opt/cni/bin/ scp cmd/fakeipam/fakeipam node01:/opt/cni/bin/ ``` ![](https://img.tnblog.net/arcimg/hb/12100d12a85c49b7956766ec84a0bd17.png) tn2>修改`cni_plugins`目录下的`cni_plugins_ds.yaml` ![](https://img.tnblog.net/arcimg/hb/f7c3a42dc8c84128b3c64a5755cc0d28.png) ```bash yum install -y jq kubectl create -f integration/manifests/cni_plugins kubectl get pod -n kube-system ``` ![](https://img.tnblog.net/arcimg/hb/d574ebd8cb4047d7a339df3e43d2ce6f.png) ![](https://img.tnblog.net/arcimg/hb/a57369c3b20e4f57901165c561bf5ad9.png) >### 创建 NetWatcher tn2>修改NetWatcher文件中的镜像改为`danmcni/netwatcher`. ```bash vim integration/manifests/netwatcher/netwatcher_ds.yaml ``` ![](https://img.tnblog.net/arcimg/hb/f7ce90cdeb86486c810689e3678118e0.png) tn2>通过从项目的根目录执行以下命令创建netwatcher守护程序集: ```bash kubectl create -f integration/manifests/netwatcher/ ``` ![](https://img.tnblog.net/arcimg/hb/6d51ee65d957442e851c1c92d27034b9.png) >### 部署 Webhook tn2>通过从项目的根目录执行以下命令,创建webhookDeployment并为其提供证书: 以下脚本需要jq工具和openssl;请确保安装了它们。 ```bash sudo chmod +x ./integration/manifests/webhook/webhook-create-signed-cert.sh sudo chmod +x ./integration/manifests/webhook/webhook-patch-ca-bundle.sh sed -i 's/v1beta1/v1/g' ./integration/manifests/webhook/webhook-create-signed-cert.sh vim ./integration/manifests/webhook/webhook-create-signed-cert.sh # 添加signerName # spec.signerName: kubernetes.io/legacy-unknown ./integration/manifests/webhook/webhook-create-signed-cert.sh cat ./integration/manifests/webhook/webhook.yaml | \ ./integration/manifests/webhook/webhook-patch-ca-bundle.sh > \ ./integration/manifests/webhook/webhook-ca-bundle.yaml vim ./integration/manifests/webhook/webhook-ca-bundle.yaml ``` ![](https://img.tnblog.net/arcimg/hb/31892ea83d494c19bdbcf1f96613d037.png) ![](https://img.tnblog.net/arcimg/hb/c9b7404e931c42218152ca26c5de77b2.png) ```bash kubectl create -f integration/manifests/webhook/webhook-ca-bundle.yaml ``` tn2>安装Svcwatcher,需要替换一下镜像名 ```bash kubectl create -f integration/manifests/svcwatcher/ ``` 未完待续。。。