Integrating Kubernetes with HashiCorp Vault for secrets management

Kubernetes also known as K8s is the is an open-source container-orchestration system for automating deployment, scaling and management of containerized applications. Kubernetes has become a leading  tool set to build a cloud agnostic , scalable application platform as it is supported by major cloud provides , AWS (EKS) , Google (GKE) and Azure (AKS) as well as on-premise and hybrid architecture.

Before this cloud and elastic computing revolution the secrets used to reside on the servers with few administrators  have access to read and modify it manually or some automated scripts. With infrastructure as code and  highly elastic and scalable workload , it was become extremely difficult and dangerous to manage secrets this way .Here comes Vault to solve this problem.

Vault is an open-source tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.

Kubernetes and Vault are the important tools if you want to build a cloud agnostic application platform . If you are reading this page you might have already chosen to build the cloud agnostic platform using these tools.

How Kubernetes vault integration works.

The kubernetes auth method (since Vault 0.8.3) can be used to authenticate with Vault using a Kubernetes Service Account Token. This method of authentication makes it easy to introduce a Vault token into a Kubernetes Pod. The authentication is role based and the role is bound to a service account name and a namespace.

A Kubernetes Service Account provides an identity for processes that run in a Pod.

When you (a human) access the cluster (for example, using kubectl), you are authenticated by the apiserver as a particular User Account (currently this is usually admin, unless your cluster administrator has customized your cluster). Processes in containers inside pods can also contact the apiserver. When they do, they are authenticated as a particular Service Account (for example, default).

A file containing a JWT token for a pod’s service account is automatically mounted at /var/run/secrets/kubernetes.io/serviceaccount/token.

Note : You should create service account for each application so that you can control access to each resource using least privileged rule.

Steps for Integration.

Step 1. Kubernetes - Set up service account for Vault token review

a)     Create the service account vault-tokenreview:

               kubectl -n default create serviceaccount vault-tokenreview

b)    Create the ClusterRoleBinding for the vault-tokenreview service account to access the Kubernetes TokenReview API:

               kubectl -n default create -f example/k8s_auth/vault-tokenreview-binding.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata:   name: vault-tokenreview-binding   namespace: default roleRef:   apiGroup: rbac.authorization.k8s.io   kind: ClusterRole   name: system:auth-delegator subjects: - kind: ServiceAccount   name: vault-tokenreview   namespace: default

c)     Fetch the token for the vault-tokenreview service account:

SECRET_NAME=$(kubectl -n default get serviceaccount vault-tokenreview -o jsonpath='{.secrets[0].name}')

TR_ACCOUNT_TOKEN=$(kubectl -n default get secret ${SECRET_NAME} -o jsonpath='{.data.token}' | base64 --decode)

Step 2. Vault - Enable and configure the backend

a)     Enable the Kubernetes auth backend:

vault auth-enable kubernetes

Note: It is a good practice to use path for kubernetes to support multiple clusters.

vault auth-enable --path="kube-cluster-A" kubernetes

b)    Configure the backend with the Kubernetes master server URL and certificate-authority-data.

vault write auth/kube-cluster-A/config kubernetes_host= kubernetes_ca_cert=@ca.crt token_reviewer_jwt=$TR_ACCOUNT_TOKEN

Vault and Kubernetes auth setup

Step 3. Kubernetes -Create a service account specific to your application

kubectl create -f - <<EOF apiVersion: v1 kind: ServiceAccount metadata:   name: myapp EOF

Step 4. Vault -Create a policy and role

The Kubernetes backend authorizes a resource  by granting it a role mapped to a service account. A role is configured with policies which control the resource 's access to paths and operations in Vault.

a)    Create a new policy demo-policy using example policy file policy.hcl.

vault policy write myapp-policy -<<EOF # paths: path “secret/myapp*” {  capabilities = [“create”] } path “secret/myapp*” {  capabilities = [“read”] } --> EOF

b)    Create a new role myapp -role configured for the service account myapp and policy myapp -policy:

vault write auth/kubernetes/role/myapp-role \     bound_service_account_names= myapp  \     bound_service_account_namespaces=default \     policies= myapp -policy \     ttl=1h

Service account creation and Vault Role mapping Flow 

Step 5. Test the Integration

Now use the service account token to authenticate for the role myapp-role

a)     Fetch the token for the default service account:

               SECRET_NAME=$(kubectl -n default get serviceaccount myapp -o jsonpath='{.secrets[0].name}')

               MYAPP_ACCOUNT_TOKEN=$(kubectl -n default get secret ${SECRET_NAME} -o jsonpath='{.data.token}' | base64 --decode)

b)    Log in to the Kubernetes auth backend using the service account token:

               $ vault write auth/kube-cluster-A/login role=myapp-role jwt=${ MYAPP _ACCOUNT_TOKEN}

accessing vault token from kubernetes

Best Practices :

1.     Use an automated tool ci-cd pipeline to build Kuberntes Vault Integration. Step 1 and Step 2 can be included in a pipeline that build / update your Kubernetes cluster.

2.     Use path for kubernetes to support multiple clusters. vault auth-enable --path="kube-cluster-A" kubernetes , this helps when same vault server is used for multiple kubernetes clusters.

3.     You should create service account for each application so that you can control access to each resource using least privileged rule.

4.     Define a naming convention for your application name , service account name, policy name  , policy paths and role name  and create a template. This helps in defining clear access rules and make auditing easy. E.g.

application name=

service account name=

policy name=_policy

policy paths = secret/

role name=_role

5.     Use an automated ci-cd pipeline to create service account , vault policy and vault role and vault role to service account mapping  . This can be part of the same pipeline which provision your application infrastructure.
 You should add CIDR restriction to vault to only allow the kubernetes CIDR blocks in the production.