How to Encrypt Kubernetes Secrets?

Kubernetes Secrets are a mechanism for storing and managing sensitive information such as passwords, OAuth tokens, SSH keys, and other credentials needed by applications running in a Kubernetes cluster.

1. Keeps sensitive data separate from application code.
2. Managed via Kubernetes APIs for creating, updating, and handling secrets.
3. Configurable access policies to restrict secret access.
4. Exposed to pods as files in a volume or environment variables.

Importance of Encrypting Secrets :

1. Unencrypted secrets in etcd can be accessed if the database is compromised.
2. Risk of accidental exposure through misconfigured resources.
3. Accessible by administrators and users with storage access.

Benefits of Encryption :

1. Makes secrets unreadable without decryption keys.
2. Helps meet regulatory data protection requirements.
3. Reduces damage from breaches as encrypted data is useless without keys.
4. Prevents interception during network transmission.

Examples of Kubernetes Secrets:

1. Passwords: Database credentials, application login passwords, or any other form of user authentication passwords.

• Example: MySQL database password.

2. API Keys: Tokens required to authenticate and access external services and APIs.

• Example: Google Maps API key, Stripe API key.

3. SSH Keys: Keys used for secure shell access to servers.

• Example: Private SSH key for accessing a remote Git repository.

4. TLS Certificates: Certificates used for establishing secure HTTPS connections.

• Example: SSL/TLS certificate for a web server.

5. OAuth Tokens: Tokens used for authorization in OAuth flows.

• Example: Access token for a third-party API like GitHub or Google.

7. Docker Registry Credentials: Credentials for accessing private Docker registries.

• Example: Username and password for Docker Hub or other container registries.

8. Encryption Keys: Keys used for encrypting and decrypting data.

• Example: AES encryption key for encrypting sensitive data at rest.

Usage of Secrets in Kubernetes:

1. Environment Variables: Secrets can be exposed as environment variables within the container.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mycontainer
    image: myimage
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mysecret
          key: db_password

2. Volume Mounts: Secrets can be mounted as files within a container.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mycontainer
    image: myimage
    volumeMounts:
    - name: secret-volume
      mountPath: "/etc/secrets"
  volumes:
  - name: secret-volume
    secret:
      secretName: mysecret

Types of Encryption for Kubernetes Secrets:

Encryption at Rest: Protects secrets stored on disk using methods like KMS, AES-CBC, AES-GCM, and Secretbox.

Encryption in Transit: Protects secrets during transmission using TLS.

External Tools: Solutions like HashiCorp Vault offer advanced encryption and management options for Kubernetes secrets.

Create an Encryption Configuration File:

This file specifies the encryption providers and the keys used for encryption.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: <base64-encoded-secret>
    - identity: {}

Generate an Encryption Key:

Generate a base64-encoded 256-bit encryption key using OpenSSL, you can use the following command in your terminal:

openssl rand -base64 32

#   openssl rand: This command generates random data.
#   -base64: This option specifies that the output should be base64-encoded.
#   32: This specifies the number of bytes to generate. Since 256 bits equals 32 bytes, this will generate a 256-bit key.

#Sample Output:
 
eXbTsjM9JkYxO+ZxCnzG5kPhQVpJnxH5c8+cv8GBlxE=

This output is a base64-encoded 256-bit encryption key. You can use this key in various encryption algorithms and applications requiring a 256-bit key.

Apply the Encryption Configuration:

You need to modify the API server to use this encryption configuration file.

• Locate the API Server Manifest: Typically found in /etc/kubernetes/manifests/kube-apiserver.yaml.
• Modify the API Server Manifest: Add the --encryption-provider-config flag to the API server manifest.

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - name: kube-apiserver
    command:
    - kube-apiserver
    # Other flags...
    - --encryption-provider-config=/path/to/encryption-config.yaml

Ensure the encryption configuration file is accessible at the specified path on all control plane nodes.

Restart the API Server:

The API server will pick up the new configuration and start encrypting secrets at rest.

Verify Encryption:

To verify that secrets are being encrypted:

• Create a Test Secret:

kubectl create secret generic test-secret --from-literal=key1=supersecret

• Check etcd: Access the etcd data directly (be cautious as this involves directly querying etcd, which should generally be avoided in production). Use the etcdctl tool to check if the data is encrypted.

ETCDCTL_API=3 etcdctl get /registry/secrets/default/test-secret --prefix --key-file=<path-to-key-file> --cert-file=<path-to-cert-file> --cacert=<path-to-ca-cert>

Rotating Encryption Keys:

Periodically rotate encryption keys for enhanced security.

• Add a New Key: Update the encryption configuration file with a new key at the top of the list.
• Re-encrypt Secrets: Re-encrypt all secrets with the new key.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

• Remove the Old Key: Once all secrets are re-encrypted, remove the old key from the configuration.

Encrypting Kubernetes Secrets is crucial for securing sensitive data in your cluster. Here are the various methods to encrypt Kubernetes Secrets:

1. Encrypting Secrets at Rest Using the Built-in Mechanism

Kubernetes provides built-in support for encrypting secrets at rest. This is the most straightforward method and involves configuring the API server with encryption providers. Here’s how to do it:

Steps:

1. Create an Encryption Configuration File:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-key>
  - identity: {}

2. Specify the Encryption Configuration in the API Server: Edit the kube-apiserver manifest (typically located in /etc/kubernetes/manifests/kube-apiserver.yaml).

--encryption-provider-config=/path/to/encryption-config.yaml

3. Restart the API Server: After updating the manifest, the kube-apiserver will automatically restart and begin encrypting secrets.

2. Using External Key Management Services (KMS)

For higher security, Kubernetes can integrate with external Key Management Services like AWS KMS, Google Cloud KMS, or HashiCorp Vault. This method allows Kubernetes to use external systems for key management.

Steps:

1. Configure KMS Provider:

• For AWS KMS: Use the AWS KMS provider plugin and configure the encryption configuration file accordingly.
• For Google Cloud KMS: Use the GCP KMS provider plugin and configure the encryption configuration file.
• For HashiCorp Vault: Configure Vault to manage keys and set up the Vault provider.

2. Update the Encryption Configuration File:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - kms:
      name: <provider-name>
      endpoint: <kms-endpoint>
      cachesize: 1000
  - identity: {}

3. Update the API Server Configuration:

--encryption-provider-config=/path/to/encryption-config.yaml

4. Restart the API Server: Ensure the API server is using the new configuration.

3. Encrypting Secrets Using Custom Encryption Providers

If you need more control over the encryption process, you can implement a custom encryption provider. This approach involves writing and deploying a custom encryption plugin.

Steps:

1. Develop the Custom Provider: Implement the encryption and decryption logic based on your requirements.
2. Deploy the Custom Provider: Ensure your custom provider is accessible to the API server.
3. Configure the Encryption Configuration:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - custom:
      name: <custom-provider-name>
      endpoint: <custom-provider-endpoint>
  - identity: {}

4. Update the API Server:

--encryption-provider-config=/path/to/encryption-config.yaml

5. Restart the API Server: Apply the configuration changes.

4. Encrypting Secrets at the Application Level

In addition to or instead of encrypting secrets at rest, you can encrypt data at the application level before storing it in Kubernetes Secrets. This method requires the application to handle encryption and decryption.

Steps:

1. Implement Encryption in the Application: Use libraries or tools to encrypt sensitive data before creating a Kubernetes Secret.
2. Store the Encrypted Data as a Kubernetes Secret: The data stored in the Secret is already encrypted.
3. Decrypt Data in the Application: When the application retrieves the Secret, it will need to decrypt the data before using it.

5. Using Sealed Secrets

Sealed Secrets is a project by Bitnami that allows you to store encrypted secrets in Git repositories.

Steps:

1. Install kubeseal: Install the kubeseal CLI tool.
2. Encrypt the Secret: Use kubeseal to create a SealedSecret from a Kubernetes Secret.

kubectl create secret generic mysecret --from-literal=username=myuser --from-literal=password=mypass -o yaml --dry-run=client > mysecret.yaml
kubeseal < mysecret.yaml > mysealedsecret.yaml

3. Apply the SealedSecret: Apply the SealedSecret manifest to your cluster.

kubectl apply -f mysealedsecret.yaml

4. Controller Decrypts at Runtime: The Sealed Secrets controller in the cluster decrypts the secret and creates the actual Secret resource.

6. Using SOPS (Secrets OPerationS)

SOPS is a tool that can be used to encrypt Kubernetes Secret manifests.

Steps:

1. Install SOPS: Install the SOPS CLI tool.
2. Encrypt the Secret Manifest: Create a Kubernetes Secret manifest and encrypt it using SOPS.

sops --encrypt --kms arn:aws:kms:region:account-id:key/key-id secret.yaml > encrypted-secret.yaml

3. Apply the Encrypted Secret: Apply the encrypted manifest to your cluster.

kubectl apply -f encrypted-secret.yaml

4. Decrypt at Runtime: Use SOPS to decrypt the secrets at runtime within your CI/CD pipeline or application logic.

These methods provide varying levels of security and flexibility depending on your infrastructure and security requirements. Implementing encryption for Kubernetes Secrets helps ensure that sensitive data remains protected throughout its lifecycle in your cluster.

If you found this guide helpful then do click on 👏 the button.

Follow for more Learning like this 😊

If there’s a specific topic you’re curious about, feel free to drop a personal note or comment. I’m here to help you explore whatever interests you!

Thanks for spending your valuable time learning to enhance your knowledge!