Migration Guide

When the Reports Server is enabled, migration from etcd occurs automatically under the following conditions:

  • The cluster has Kyverno already installed.
  • The cluster has policy reports crds already installed.

Migration from Kyverno 1.12+

For users running older versions of Kyverno, the recommended migration path is -

  1. Upgarde to the latest N4K version
  2. Migrate from K8s etcd to Reports Server

Optionally verify the policyreports stored in etcd.

etcdctl get "/registry/wgpolicyk8s.io/policyreports" --prefix --keys-only

Since N4K is already running in the cluster, run the following commands to install reports-server:

helm repo add reports-server https://nirmata.github.io/reports-server/
helm repo update reports-server
helm install reports-server reports-server/reports-server -n kyverno --set apiServicesManagement.migrateReportsServer.enabled=true

Wait for reports-server pod to come up and check for apiservices.

kubectl get pods -n kyverno
kubectl get apiservices

NOTE: Existing policy reports from K8s etcd should be manually cleaned up.

When reports-server is introduced in an active cluster (reports exist in k8s etcd), the reports are copied to the offloaded datastore (etcd or postgres). But these reports are not automatically deleted from k8s etcd. Users have to manually delete those reports.

First check reports are properly copied over to reports-server etcd. We can confirm it by looking at reports-server pod logs using:

kubectl logs reports-server-969f45d4b-jj9r5 -n kyverno

Manually delete reports from k8s etcd:

kubectl exec -it etcd-test-rs-control-plane -n kube-system -- sh

Inside the pod, etcdctl is usually installed. Export the following variables:

export ETCDCTL_API=3
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key

Run the following commands for deleting policy reports:

etcdctl del "/registry/wgpolicyk8s.io/policyreports" --prefix
etcdctl get "/registry/wgpolicyk8s.io/policyreports" --prefix --keys-only

Rollback from Reports Server

When reports-server is uninstalled, kyverno controllers will start throwing errors for missing report CRDs.

  1. Manually install the required report CRDs.
  2. Uninstall reports-server.
  3. Verify if apiservices are removed and reports-server is deleted.
  4. Reports will be lost, so manually scale down and up the background controller to trigger reconciliation and recreate reports.
  5. Verify if policyreports are regenerated.

NOTE: If you are using Postgres for offloading, note that policy reports will have to be manually cleaned up from the database.