AWS Secret Manager Tutorial

AWS Secret Manager Tutorial – A Step by Step Guide

Kubernetes CSI is a standard for storage providers. They can provide block storage to containerized applications running in Kubernetes. This standardized approach helped develop a huge market for third parties to provide plugins for various kinds of storage devices.

Kubernetes users and application developers have to install the driver from a storage provider to use that block device as a mountable storage volume. Check our premium elastic kubernetes service.

A wide variety of CSI drivers exists for diverse types of storage devices. This document introduces CSI drives for the AWS Secrets manager service.

AWS Secret Manager

AWS Secrets, a popular, secret management service. It stores secrets as a key/values database. All secrets, including passwords, security tokens, and sensitive data, can be stored in the secrets manager. Developers and Admins do not need to know any application or service password. When an organisation uses AWS Secrets Manager. The security administrators will set up passwords and password rotation policies to secure passwords from people who don’t need to know them.

Secrets can be created and retrieved by using AWS Command-line tools and programmatically. The secrets are always stored in encrypted manners that meet compliance requirements of many organisations, including HIPAA, SOC and DoD, to mention a few. Secrets can only be accessed and retrieved for properly authenticated and authorised clients that use a secure TLS communication channel.

As a managed service, AWS Secrets Manager is fully integrated with all other AWS services and tools that allow tracking of its usage proper usage and can alert unauthorized attempts. Such Capability itself is a requirement for most security-related standards and benchmarks. AWS Secrets manager supports secret rotation with other services like RDS and RedShift. Password Rotation is an effective strategy against password exploit attacks popular among hackers.

Folks at CNCF have provided a CSI driver for AWS Secret Manager called AWS Secrets & Configuration Provider (ASCP). ACSP works with many cloud secrets providers, including AWS Secrets Manager.. This document shows how to use ASCP with AWS Secrets Manager.

Figure above shows a pod that has a Storage Volume defined with specific storage class kind called “SecretsProviderClass”

When Pod is initialised, the CSI storage provider calls ACSP driver to pull the secrets from key defined in Secrets Provider Class. The retrieved secret is then communicated over to Kubelet, so a volume with a mount path representing the secret is mounted on a path. The pod can now retrieve decrypted password

How to set up AWS Secret & Configuration Provider?

This section shows how to set up AWS Secrets & Configuration Provider (ASCP) to work with the Secrets Store CSI driver on your Kubernetes clusters. The Secrets Store CSI driver allows Kubernetes to mount secrets stored in external secrets stores into the pods as volumes.

After the volumes are attached, the data is mounted into the container’s file system. In this example, the external secret store is Secrets Manager.

This solution includes the details for the following steps:

• Restrict access to your pods using IAM roles for service accounts
• Install the Kubernetes secrets store CSI driver
• Install the AWS Secrets & Configuration Provider
• Create and deploy the SecretProviderClass custom resource
• Configure and deploy the Pods to mount the volumes based on the configured secrets
• Load secrets and configurations from the volumes mounted to the container

Here are actual steps to create and mount a secret for a pod:

1. Install driver for CSI

helm repo add secrets-store-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts
helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver
Adding ---set  syncSecret.enabled=true enables alpha feature of synch aws secrets with pods mounted volume containing secret values

2. Install AWS provider

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

3. Set the AWS Region and the name of your cluster as shell variables so you can use them in bash commands. For <REGION>, enter the AWS Region where your Amazon EKS cluster runs. For <CLUSTERNAME>, enter the name of your cluster.

REGION=us-west-2
CLUSTERNAME=my-eks-my-demo

4. Create a test secret on Secret Manager.

aws --region "$REGION" secretsmanager  create-secret --name MySecret --secret-string '{"username":"db_user", "password":"It$ProbablyMe"}'

5. Create a resource policy for the pod that limits its access to the secret you created in the previous step. For <SECRETARN>, use the ARN of the secret. Save the policy ARN in a shell variable.

POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn --output text iam create-policy --policy-name nginx-deployment-policy --policy-document '{
 POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn --output text iam create-policy --policy-name nginx-deployment-policy --policy-document '{
     "Version": "2012-10-17",
     "Statement": [ {
         "Effect": "Allow",
         "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
         "Resource": ["arn:aws:secretsmanager:us-west-2:894870328975:secret:MySecret-RntYgP"]
     } ]
 }')

6. Create an IAM OIDC provider for the cluster if you don’t already have one.

eksctl utils associate-iam-oidc-provider --region="$REGION" --cluster="$CLUSTERNAME" --approve # Only run this once

7. Create the service account the pod uses and associate the resource policy you created in step 3 with that service account.

eksctl create iamserviceaccount --name nginx-deployment-sa --region="$REGION" --cluster "$CLUSTERNAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts

8. Create a secret provider class

cat <<EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:                    # provider-specific parameters
    objects:  |
      - objectName: "MySecret"
        objectType: "secretsmanager"
EOF

9. Deploy nginx

cat <<EOF | kubectl apply -f -
kind: Service
apiVersion: v1
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: nginx-deployment-sa
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "aws-secrets"
      containers:
      - name: nginx-deployment
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
EOF

10. Print and verify secrets

cat <<EOF | kubectl apply -f -
kind: Service
apiVersion: v1
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: nginx-deployment-sa
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "aws-secrets"
      containers:
      - name: nginx-deployment
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
EOF

This should print original secrets

Defaulting container name to nginx-deployment.
Use 'kubectl describe pod/nginx-deployment-9b87cc8c7-r2fk4 -n default' to see all of the containers in this pod.
{"username":"db_user", "password":"it$ProbablyMe"}

Change password in secrets manager and run below command

kubectl rollout restart deployment/nginx-deployment

Committed to delivering the best

Thousands of AWS and CNCF-certified Kubernetes solution partners have unique expertise and focus areas. Our focus is on best practices in security, automation, and excellence in Cloud-based operations.

Please reach out to us if you have any questions.