Vault on Red Hat Demo Platform (RHDP)
Red Hat's OpenShift is a distribution of the Kubernetes platform that provides a number of usability and security enhancements.
The Red Hat Demo Platform (RHDP) is a way for Red Hat employees and official Red Hat partners to deploy and demo Red Hat products for specific use cases alongside Red Hat partner technology. Visit the Red Hat website for demos and workshops.
In this tutorial, you will login to an OpenShift cluster and configure the authentication between Vault and the cluster. You will then deploy two web applications; One that authenticates and requests secrets directly from the Vault server. The other will utilize deployment annotations that enable it to remain Vault unaware.
Upon completion of this tutorial, you will gain a better understanding of how Vault is deployed on Red Hat OpenShift and how Vault integrates with applications running in OpenShift clusters.
Note
The RHDP system is only available for Red Hat/HashiCorp employees and official Red Hat partners. Please contact technologypartners@hashicorp.com for further information and access to the system.
Prerequisites
Please make sure your system and network can make external ssh connections.
Once the workshop admin spins up the cluster, you'd be provided with an access URL with username and password.
Upon login, users will receive credentials similar to what's shown below.
Example output:
HTPasswd Authentication is enabled on this cluster.Users user1 .. user30 are created with password openshiftUser opentlc-mgr with password <password_here> is cluster adUser opentlc-mgr with password <password_here> is clusterlearn-vault-kubernetes/openshift admin.You can access your bastion via SSH:ssh lab-user@bastion.r47nd.sandbox695.opentlc.com
Upon completion of the last step, you are connected to the bastion running RHEL. Confirm that OpenShift environment is running and the Vault namespace has been created and Vault pods are successfully running.
$ oc get ns vaultNAME STATUS AGEvault Active 1h
Output:
$ kubectl get pods -n vaultNAME READY STATUS RESTARTS AGEopenshift-helm-charts-vault-0 1/1 Running 0 22mopenshift-helm-charts-vault-agent-injector-777b86fbbd-cxrgb 1/1 Running 0 22m
Retrieve the web application and additional configuration by cloning the learn-vault-kubernetes repository from GitHub.
$ git clone https://github.com/hashicorp-education/learn-vault-kubernetes.git
Change the working directory to
learn-vault-kubernetes/openshift
.$ cd learn-vault-kubernetes/openshift
Note
This tutorial assumes that the remainder of commands are executed within this directory.
Configure the Vault Kubernetes authentication
Vault provides a Kubernetes authentication method that enables clients to authenticate with Vault within an OpenShift cluster. This authentication method configuration requires the location of the Kubernetes API, which is available in environment variables within the pod.
Set the current project to Vault
$ oc project vaultNow using project "vault" on server "https://api.cluster-<id>.<id>.sandbox<id>.opentlc.com:6443".
Your system prompt is replaced with a new prompt
/ #
. Commands issued at this prompt are executed on theopenshift-helm-charts-vault-0
container.Start an interactive shell session on the
openshift-helm-charts-vault-0
pod.$ oc exec -it openshift-helm-charts-vault-0 -- /bin/sh/ #
Your system prompt is replaced with a new prompt
/ #
. Commands issued at this prompt are executed on theopenshift-helm-charts-vault-0
container.Enable the Kubernetes authentication method.
$ vault auth enable kubernetesSuccess! Enabled kubernetes auth method at: kubernetes/
Configure the Kubernetes authentication method to use the location of the Kubernetes host. It will automatically use the pod's own identity to authenticate with Kubernetes when querying the token review API.
Note
For the best compatibility with recent Kubernetes versions, ensure you are using Vault v1.9.3 or greater.$ vault write auth/kubernetes/config \ kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
Output:
Success! Data written to: auth/kubernetes/config
The authentication method is now configured.
Exit the
openshift-helm-charts-vault-0
pod.$ exit
Deployment: Request secrets directly from Vault
Applications on pods can directly communicate with Vault to authenticate and request secrets. An application needs:
- a service account
- a Vault secret
- a Vault policy to read the secret
- a Kubernetes authentication role
Create the service account
Display the service account defined in
service-account-webapp.yml
.$ cat service-account-webapp.ymlapiVersion: v1kind: ServiceAccountmetadata:name: webapp
This definition of the service account creates the account with the name
webapp
.Apply the service account.
$ oc apply --filename service-account-webapp.ymlserviceaccount/webapp created
Get all the service accounts.
$ oc get serviceaccountsNAME SECRETS AGEbuilder 1 47mdefault 1 47mdeployer 1 47mopenshift-helm-charts-vault 1 31mopenshift-helm-charts-vault-agent-injector 1 31mwebapp 1 10s
The
webapp
service account is displayed.
Create the secret
Start an interactive shell session on the
openshift-helm-charts-vault-0
pod.$ oc exec -it openshift-helm-charts-vault-0 -- /bin/sh
Your system prompt is replaced with a new prompt
/ #
. Commands issued at this prompt are executed on theopenshift-helm-charts-vault-0
container.Create a secret at path
secret/webapp/config
with ausername
andpassword
.$ vault kv put secret/webapp/config username="static-user" \ password="static-password"
Output:
Key Value--- -----created_time 2020-07-14T20:20:04.315677226Zdeletion_time n/adestroyed falseversion 1
Read the secret at path
secret/webapp/config
.$ vault kv get secret/webapp/config====== Metadata ======Key Value--- -----created_time 2020-07-14T20:20:04.315677226Zdeletion_time n/adestroyed falseversion 1 ====== Data ======Key Value--- -----password static-passwordusername static-user
The secret with the username and password is displayed.
Define the read policy
Write out the policy named
webapp
that enables theread
capability for secrets at pathsecret/data/webapp/config
.$ vault policy write webapp - <<EOFpath "secret/data/webapp/config" {capabilities = ["read"]}EOF
Output:
Success! Uploaded policy: webapp
The policy
webapp
is used in the Kubernetes authentication role definition.
Create a Kubernetes authentication role
Create a Kubernetes authentication role, named
webapp
, that connects the Kubernetes service account name andwebapp
policy.$ vault write auth/kubernetes/role/webapp \ bound_service_account_names=webapp \ bound_service_account_namespaces=vault \ policies=webapp \ ttl=24h
Output:
Success! Data written to: auth/kubernetes/role/webapp
The role connects the Kubernetes service account,
webapp
, the namespace,vault
, with the Vault policy,webapp
. The tokens returned are valid for 24 hours.Exit the
openshift-helm-charts-vault-0
pod.$ exit
Deploy the application
Examine the webapp deployment defined in
deployment-webapp-rhdp.yml
.$ cat deployment-webapp-rhdp.yml
The deployment deploys a pod with a web application running under the
webapp
service account that talks directly to the Vault service created by the Vault Helm charthttp://openshift-helm-charts-vault:8200
.deployment-webapp-rhdp.yml
---apiVersion: apps/v1kind: Deploymentmetadata:name: webapplabels: app: webappspec:replicas: 1selector: matchLabels: app: webapptemplate: metadata: labels: app: webapp spec: serviceAccountName: webapp containers: - name: app image: hashieducation/simple-vault-client:latest imagePullPolicy: Always env: - name: VAULT_ADDR value: 'http://openshift-helm-charts-vault:8200' - name: JWT_PATH value: '/var/run/secrets/kubernetes.io/serviceaccount/token' - name: SERVICE_PORT value: '8080'
Apply the webapp deployment.
$ oc apply --filename deployment-webapp-rhdp.ymldeployment.apps/webapp created
Display all the pods within the vault namespace.
$ oc get podsNAME READY STATUS RESTARTS AGEopenshift-helm-charts-vault-0 1/1 Running 0 7m16sopenshift-helm-charts-vault-agent-injector-597d5f5569-h4gbp 1/1 Running 0 7m16swebapp-55c464ff57-grjbp 1/1 Running 0 2m55s
Wait until the
webapp
pod is running and ready (1/1
).This web application runs an HTTP service that listens on port 8080.
Perform a
curl
request athttp://localhost:8080
on thewebapp
pod.$ oc exec \ $(oc get pod --selector='app=webapp' --output='jsonpath={.items[0].metadata.name}') \ --container app -- curl -s http://localhost:8080 ; echo
Output:
username:static-user password:static-password
The web application running on port 8080 in the webapp pod:
- authenticates with the Kubernetes service account token
- receives a Vault token with the read capability at the
secret/data/webapp/config
path - retrieves the secrets from
secret/data/webapp/config
path - displays the secrets as JSON
Deployment: Secrets through Annotations
Applications on pods can remain Vault unaware if they provide deployment annotations that the Vault Agent Injector detects. This injector service leverages the Kubernetes mutating admission webhook to intercept pods that define specific annotations and inject a Vault Agent container to manage these secrets. An application needs:
- a service account
- a Vault secret
- a Vault policy to read the secret
- a Kubernetes authentication role
- a deployment with Vault Agent Injector annotations
Create the service account
Display the service account defined in
service-account-issues.yml
.$ cat service-account-issues.ymlapiVersion: v1kind: ServiceAccountmetadata:name: issues
This definition of the service account creates the account with the name
issues
.Apply the service account.
$ oc apply --filename service-account-issues.ymlserviceaccount/issues created
Get all the service accounts.
$ oc get serviceaccountsNAME SECRETS AGEbuilder 1 47mdefault 1 47mdeployer 1 47mopenshift-helm-charts-vault 1 31mopenshift-helm-charts-vault-agent-injector 1 31mwebapp 1 10s
The
issues
service account is displayed.
Create the secret
Start an interactive shell session on the
openshift-helm-charts-vault-0
pod.$ oc exec -it openshift-helm-charts-vault-0 -- /bin/sh
Your system prompt is replaced with a new prompt
/ #
. Commands issued at this prompt are executed on theopenshift-helm-charts-vault-0
container.Create a secret at path
secret/issues/config
with ausername
andpassword
.$ vault kv put secret/issues/config username="annotation-user" \ password="annotation-password"
Output:
Key Value--- -----created_time 2020-07-14T20:25:24.043709599Zdeletion_time n/adestroyed falseversion 1
Get the secret at path
secret/issues/config
.$ vault kv get secret/issues/config====== Metadata ======Key Value--- -----created_time 2020-07-14T20:25:24.043709599Zdeletion_time n/adestroyed falseversion 1 ====== Data ======Key Value--- -----password annotation-passwordusername annotation-user
The secret with the username and password is displayed.
Define the read policy
Write out the policy named
issues
that enables theread
capability for secrets at pathsecret/data/issues/config
.$ vault policy write issues - <<EOFpath "secret/data/issues/config" {capabilities = ["read"]}EOF
Output:
Success! Uploaded policy: issues
The policy
issues
is used in the Kubernetes authentication role definition.
Create a Kubernetes authentication role
Create a Kubernetes authentication role, named
issues
, that connects the Kubernetes service account name andissues
policy.$ vault write auth/kubernetes/role/issues \ bound_service_account_names=issues \ bound_service_account_namespaces=vault \ policies=issues \ ttl=24h
Output:
Success! Data written to: auth/kubernetes/role/issue
The role connects the Kubernetes service account,
issues
, the namespace,vault
, with the Vault policy,issues
. The tokens returned are valid for 24 hours.Exit the
openshift-helm-charts-vault-0
pod.$ exit
Deploy the application
Examine the issues deployment defined in
deployment-issues.yml
.$ cat deployment-issues.yml
The Vault Agent Injector service reads the metadata annotations prefixed with
vault.hashicorp.com
.agent-inject
enables the Vault Agent injector servicerole
is the Vault Kubernetes authentication roleagent-inject-secret-FILEPATH
prefixes the path of the file,issues-config.txt
written to the/vault/secrets
directory. The value is the path to the Vault secret.agent-inject-template-FILEPATH
formats the secret with a provided template.
deployment-issues.yaml
apiVersion: apps/v1kind: Deploymentmetadata:name: issueslabels: app: issuesspec:selector: matchLabels: app: issuesreplicas: 1template: metadata: annotations: vault.hashicorp.com/agent-inject: 'true' vault.hashicorp.com/role: 'issues' vault.hashicorp.com/agent-inject-secret-issues-config.txt: 'secret/data/issues/config' vault.hashicorp.com/agent-inject-template-issues-config.txt: | {{- with secret "secret/data/issues/config" -}} postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard {{- end -}} labels: app: issues spec: serviceAccountName: issues containers: - name: issues image: jweissig/app:0.0.1
Apply the issues deployment.
$ oc apply --filename deployment-issues.ymldeployment.apps/issues created
Display all the pods within the vault namespace.
$ oc get podsNAME READY STATUS RESTARTS AGEissues-75d794c744-x9gnf 2/2 Running 0 17sopenshift-helm-charts-vault-0 1/1 Running 0 9m53sopenshift-helm-charts-vault-agent-injector-597d5f5569-h4gbp 1/1 Running 0 9m53swebapp-55c464ff57-grjbp 1/1 Running 0 5m32s Wait until the `issues` pod is running and ready (`2/2`). This new pod launches two containers. The application container, named`issues`, and the Vault Agent container, named `vault-agent`.
Display the logs of the
vault-agent
container in theissues
pod.$ oc logs \ $(oc get pod -l app=issues -o jsonpath="{.items[0].metadata.name}") \ --container vault-agent
Output:
==> Vault server started! Log data will stream in below:==> Vault agent configuration: Cgo: disabled Log Level: info Version: Vault v1.4.2[INFO] sink.file: creating file sink[INFO] sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----[INFO] auth.handler: starting auth handler[INFO] auth.handler: authenticating[INFO] template.server: starting template server[INFO] (runner) creating new runner (dry: false, once: false)[INFO] (runner) creating watcher[INFO] sink.server: starting sink server[INFO] auth.handler: authentication successful, sending token to sinks[INFO] auth.handler: starting renewal process[INFO] template.server: template server received new token[INFO] (runner) stopping[INFO] (runner) creating new runner (dry: false, once: false)[INFO] (runner) creating watcher[INFO] (runner) starting[INFO] sink.file: token written: path=/home/vault/.vault-token[INFO] auth.handler: renewed auth token
Display the secret written to the
issues
container.$ oc exec \ $(oc get pod -l app=issues -o jsonpath="{.items[0].metadata.name}") \ --container issues -- cat /vault/secrets/issues-config.txt ; echo
Output:
postgresql://annotation-user:annotation-password@postgres:5432/wizard
The secrets are rendered in a PostgreSQL connection string is present on the container.
Conclusion
Congratulations on finishing the tutorial!
In this lesson, you learned how to:
- Access the RHDP system and log into the OpenShift cluster
- Verify Vault is running on the OpenShift cluster
- Enable and configure Kubernetes auth method
- Deploy a web application that reads secrets directly from Vault
- Leverage Vault Agent Injector to manage secrets while keeping the application Vault unaware
Next steps
Learn more about the Vault Helm chart by reading the documentation or exploring the project source code.
Additional suggested reading:
- The blog post announcing the Injecting Vault Secrets into Kubernetes Pods via a Sidecar
- Documentation for Vault Agent Injector service.
- Repo for vault-hello-world