How to Restore Nirmata Private Edition

Utilize this guide If Nirmata PE encounters an error and requires recovery.

Scale Down Services

First, scale down services by running the Scale Services Down Command.

Scale Services Down Command:

./nadm scale --nirmata --replicas=0

In a mongo_db pd, run the kubectl Command.

Kubectl Command:

kubectl -n nirmata exec -ti mongodb-2 sh

After running the kubectl command, enter # mongo and then # rs.status() (check status of all nodes)

Exit from the pod.

Backup the Database

Perform a database backup by running the Backup Database Command.

Backup Database Command:

./nadm backup

After backing up the database, scale down mongodb, kafka, zookeeper, kafka-es, and elasticsearch.

./nadm scale -s mongodb --replicas=0
./nadm scale -s kafka --replicas=0
./nadm scale -s zk --replicas=0
./nadm scale -s kafka-es-connector  --replicas=0
./nadm scale -s elasticsearch --replicas=0

Monitor all of the services and verify that each has come down using the Verify Status Command.

Verify Status Command:

Nadm status -w (confirm on namespace)

Wait for each service to have a status of READY and RUNNING before moving to the next step. After all services are READY and RUNNING, bring up mongodb using the Bring Up MongoDB Command.

Check the status of each replica using the Check Status Command and ensure that each is running.

Bring Up MongoDB Command:

./nadm scale -s mongodb --replicas=3

Check Status Command:

./nadm status -w (wait for all three)

Verify that the database running correctly by executing into mongodb shel and mongocli. Confirm the rs.status(). Find the primary pod and execute the pod and the shell into mongo; then check the database using the Check Database Command to verify that all of the data is present.

Check Database Command:

show dbs

If there is a problem with the database and not all data is present, run the nadm Restore Command.

nadm Restore Command:

Nadm restor -d .../dir/backup/file

When the database is restored, bring up zookeeper using the Start Zookeeper Command.

Start Zookeeper Command:

./nadm scale -s zk --replicas=3

Check the status of each replica using the Check Status Command and ensure that each is running.

Check Status Command:

./nadm status -w (wait for all three)

After starting zookeeper, execute into zookeeper and check the status of all three replicas using the Status Confirmation Command.

Status Confirmation Command:

./opt/zookeeper/bin/zkServer.sh status 

Confirm at least one leader and one follower across cluster.

After verifying zookeeper, bring up kafka using the Start kafka Command.

Start kafka Command:

./nadm scale -s kafka --replicas=3

Check the status of each replica using the Check Status Command and ensure that each is running.

Check Status Command:

./nadm status -w (wait for all three)

After starting kafka, status execute into zookeeper run the Status Execute Command.

Status Execute Command:

./opt/zookeeper/bin/zkCli.sh  at prompt run ls / to view brokers

When kafka is started, bring up kafka-es using the Start kafka-es Command.

Start kafka-es Command:

./nadm scale -s kafka-es-connector --replicas=1

After bringing up kafka-es, start elasticsearch using the Start elasticsearch Command.

Start elasticsearch Command:

./nadm scale -s elasticsearch --replicas=3

Check the status of each replica using the Check Status Command and ensure that each is running.

Check Status Command:

./nadm status -w (wait for all three)

Finally, bring up Nirmata Services using the Start Nirmata Command.

Start Nirmata Command:

./nadm scale --nirmata --replicas=3

(select no for different namespace)

Check the status the Check Status Command.

Check Status Command:

./nadm status -w 

Login to the user interface and verify that all services are running.