Amazon EKS

Amazon Elastic Kubernetes Services (Amazon EKS) is a popular managed Kubernetes service for deploying and managing containerized applications. Amazon EKS users are challenged with ensuring security and governance across their entire fleet of clusters. Kyverno, an open source policy engine, enables policy-based security, governance and compliance for Amazon EKS clusters. Using Kyverno, enterprise platform teams can deploy the necessary guardrails to ensure that the deployed applications are secure, compliant and follow the Amazon EKS Best Practices.

Deploying the Enterprise Kyverno on Amazon EKS cluster

Prerequisites

Subscribe to Nirmata Enterprise Kyverno on AWS Marketplace at: https://aws.amazon.com/marketplace/pp/prodview-nj5p5k32xgkxw
Install the following tools: kubectl, AWS CLI, and optionally eksctl
You have access to an Amazon EKS cluster

Deploy Enterprise Kyverno as an Amazon EKS add-on

After subscribing to Nirmata Enterprise Kyverno on AWS Marketplace and following the on-screen instructions successfully, you are redirected to Amazon EKS console.

Create the IAM role for service account. This assumes that you created the IAM OIDC provider with your cluster. To learn more about this step, see the AWS IAM role for service account documentation.

Note: Remember to replace $CLUSTER_NAME & $CLUSTER_REGION with the actual cluster name and the region where the cluster is located. Also you can specify the name for the role to avoid conflicts across clusters. In the example below, the role name is AmazonEKS-kyverno-role.

eksctl create iamserviceaccount \
    --name kyverno \
    --namespace kyverno \
    --cluster $CLUSTER_NAME \
    --attach-policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage \
    --attach-policy-arn arn:aws:iam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy \
    --approve \
    --override-existing-serviceaccounts \
    --region $CLUSTER_REGION \
    --role-only
    --role-name AmazonEKS-kyverno-role

Using AWS Console

Go to your existing Amazon EKS cluster and in the Add-ons tab, select Get more add-ons to find Kyverno add-on and select it. Click Next.

Select add-on

On the next page, select the latest version and select the IAM role, AmazonEKS-kyverno-role (or the IAM role you created earlier). Click on Next to review the configuration and the select Create. This should deploy the add-on to your cluster.

Configure add-on

Using AWS CLI

On your workspace, run the following command to enable Kyverno add-on for your Amazon EKS cluster:

Note: The IAM role for service account should already be created. Also you can specify the name for the role to avoid conflicts across clusters. In the example below, the role name is AmazonEKS-kyverno-role.

export SERVICE_ACCOUNT_ROLE_ARN=$(aws iam get-role --role-name AmazonEKS-kyverno-role --output json | jq -r '.Role.Arn')

aws eks create-addon --cluster-name $CLUSTER_NAME  \
    --region $CLUSTER_REGION \
    --addon-name nirmata_kyverno \
    --addon-version v1.8.5-eksbuild.0 \
    --service-account-role-arn $SERVICE_ACCOUNT_ROLE_ARN \
    --resolve-conflicts OVERWRITE

Verify add-on installation

Once the add-on has been installed, verify that the Kyverno pod is running.

kubectl get pods -n kyverno

Next, you can deploy Kyverno policies to secure your cluster. You can find supported and validated policy sets here.

Deploy Enterprise Kyverno on an existing Amazon EKS cluster using the Helm Chart

After subscribing to Nirmata Enterprise Kyverno on AWS Marketplace and following the on-screen instructions successfully, you are redirected to Amazon EKS console.

Use the following command to create an AWS IAM role and Kubernetes service account. This command will create the Kyverno namespace and also the service account.

Note: Remember to replace $CLUSTER_NAME & $CLUSTER_REGION with the actual cluster name and the region where the cluster is located.

kubectl create namespace kyverno
            
eksctl create iamserviceaccount \
    --name kyverno \
    --namespace kyverno \
    --cluster $CLUSTER_NAME \
    --attach-policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringFullAccess \
    --attach-policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage \
    --attach-policy-arn arn:aws:iam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy \
    --approve \
    --override-existing-serviceaccounts\
    --region=$CLUSTER_REGION \
    --role-name AmazonEKS_Kyverno_LicenseManagerRole

export SERVICE_ACCOUNT_ROLE_ARN=$(aws iam get-role --role-name AmazonEKS_Kyverno_LicenseManagerRole --output json | jq -r '.Role.Arn')

Use the following commands to launch this software by installing a Helm chart on your Amazon EKS cluster. The Helm CLI version in your launch environment must be 3.7.1.

export HELM_EXPERIMENTAL_OCI=1

aws ecr get-login-password \
    --region us-east-1 | helm registry login \
    --username AWS \
    --password-stdin 709825985650.dkr.ecr.us-east-1.amazonaws.com

mkdir awsmp-chart && cd awsmp-chart

helm pull oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/nirmata/kyverno-chart --version v1.10.0-06-aws

tar xf $(pwd)/* && find $(pwd) -maxdepth 1 -type f -delete

helm install kyverno \
    --namespace kyverno ./* \
    --set createnamespace=false \
    --set rbac.serviceAccount.create=false \
    --set rbac.serviceAccount.name=kyverno 
    --set rbac.serviceAccount.annotations."eks\.amazonaws\.com\/role-arn"=$SERVICE_ACCOUNT_ROLE_ARN \
    --set cluster.region=$CLUSTER_REGION

Next, verify that Kyverno is installed and the Kyverno pod is running.

kubectl get pods -n kyverno

Now you should be able to deploy Kyverno policies to secure your cluster. You can find supported and validated policy sets here.

Troubleshooting

Kyverno fails to start

If Kyverno pod status show Init:Error or Init:CrashLoopBackOff, there was an issue starting the init container. Check the init container logs as follows.

kubectl logs <kyverno pod name> -n kyverno awslm-init

Based on the error message you should be able to identify the issue:

Error: No Entitlements Allowed
Problem: No entitlements are available in your license.
Solution: Increase the number of licensed nodes on the marketplace subscription.

Error: failed to create LicenseCredentialsProvider
Problem: IAM role is not configured correctly.
Solution: Check the eks.amazonaws.com/role-arn annotation on the kyverno service account and make sure it is configured correctly.

Error: CheckoutLicense because no identity-based policy.
Problem: IAM role for service account is missing.
Solution: Follow the steps above to configure the IAM role for service account and verify that the correct role ARN is added to the the eks.amazonaws.com/role-arn annotation on the kyverno service account.