Identity-Based Microsegmentation Guide
LAST UPDATED: October 8, 2021

Back up

Enabling automatic backups

About automatic backups

If you have deployed the multi-container Microsegmentation Console, you can enable automatic snapshots of the following volumes.

  • mongodb
  • victoriametrics
  • redis

The auto-backup pods are run as cron jobs, using an auto-backup service account. You must associate the auto-backup service account with a workload identity to give the service account permissions to use the cloud provider’s snapshot API.

We only support automatic backups in the following environments at this time.

  • AWS
  • GCP

If you require an alternate cloud provider, contact us for assistance. Refer to the /opt/aporeto/voila/flavors/k8s/helpers/auto-backup/README.md file for more details.

Enabling backups

If you are upgrading or did not enable backups when you created your Voila environment, use the following command to enable them.

set_value global.backup.enabled true override

Then run doit to deploy the backup jobs.

Configuring backups on AWS

Create an IAM role with the required permissions. An example follows.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSnapshot",
                "ec2:CreateSnapshots",
                "ec2:DeleteSnapshot",
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:EnableFastSnapshotRestores",
                "ec2:DescribeFastSnapshotRestores",
                "ec2:DisableFastSnapshotRestores",
                "ec2:CopySnapshot",
                "ec2:ModifySnapshotAttribute",
                "ec2:DescribeSnapshotAttribute"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": "arn:aws:ec2:*::snapshot/*"
        }
    ]
}

Refer to the AWS documentation for more information. Once you have added the required role, annotate the auto-backup service account to use this role.

k annotate serviceaccount auto-backup \
eks.amazonaws.com/role-arn=arn:aws:iam::[PROJECT_ID]:role/[ROLE_NAME]

Configuring backups on Google Cloud

Create a service account with a role that allows it to manage storage and snapshot. Make sure the cluster is leveraging workload identity. Then bind the auto-backup service account to the role.

gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:[PROJECTID].svc.id.goog[default/auto-backup]" [SERVICE_ACCOUNT]@[PROJECTID].iam.gserviceaccount.com

Annotate the service account.

k annotate serviceaccount auto-backup \
iam.gke.io/gcp-service-account=[SERVICE_ACCOUNT]@[PROJECTID].iam.gserviceaccount.com

Refer to the GKE documentation for more information.

Modifying auto-backup defaults

The auto-backup jobs use the following default configuration.

  • Runs at 11AM UTC each day
  • Two snapshots retained
  • No disaster recovery region set

You can modify these settings from within Voila.

auto-backup job --auto --keep 3 --copy-to us-west --copy-keep 4 | kubectl apply -f -
auto-backup job|run --auto --keep <N> --copy-to <REGION> --copy-keep <N> --copy-kind partial|full

Run auto-backup -h for more details.

Restoring a snapshot

The rewind tool allows you to create new set of persistent volume claims for a running service from current snapshotted volumes. Simply call rewind --service redis for instance and follow the instruction written on the output.

To cleanup dangling resources and their disk you can use rewind --cleanup.

You can also check the file /opt/aporeto/voila/flavors/k8s/helpers/rewind/README.md for more details.

Supported providers:

  • AWS: Elastic Block Store (EBS)
  • GCP: persistent disk (PD)

Restoring mongo in another namespace

You can also restore a snapshot in a different namespace to take a look at it or for a selective restoration. We will leverage rewind to restore a mongodb cluster on another namespace to restore or explore a snapshot. This can be useful to restore only a few documents instead of rolling back the entire cluster.

Create a temporary namespace and create persistent volume claims from snapshots for the primary node only.

k create ns test
rewind --target-namespace test --service mongo --pvc "mongodb-shard-(config-0|data-[0-2]-0)" --last

Should end up with:

VOILA_K8S_NAMESPACE=test k get pvc
NAME                          STATUS   VOLUME                                      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
snap-mongodb-shard-config-0   Bound    sandbox-mongodb-shard-config-0-1610653775   50Gi       RWO            fast-xfs       2m9s
snap-mongodb-shard-data-0-0   Bound    sandbox-mongodb-shard-data-0-0-1610653775   1000Gi     RWO            fast-xfs       2m1s
snap-mongodb-shard-data-1-0   Bound    sandbox-mongodb-shard-data-1-0-1610653775   1000Gi     RWO            fast-xfs       89s
snap-mongodb-shard-data-2-0   Bound    sandbox-mongodb-shard-data-2-0-1610653775   1000Gi     RWO            fast-xfs       57s

Deploy mongo to that namespace with the following options:

VOILA_K8S_NAMESPACE=test deploy i mongodb-shard \
--set global.resources.enabled=false \
--set nodeAffinity.operator=NotIn \
--set nodeAffinity.value=mongodb \
--set volumeClaimTemplateName=snap \
--set 'shard.shards[0].replicas=1' \
--set 'shard.shards[1].replicas=0' \
--set config.replicas=1 \
--set router.replicas=1 \
--set useNamespacedFQDN=false \
--set recoveryMode=true

NOTE

Because we are using disk-based snapshots that are using the XFS file system we cannot mount those on the same node that are running the snapshotted disk. There is a conflict in XFS UID (the pod stays at container creating and barks a about bad mount option…). You might need to pop more nodes like the MongoDB ones.

Should end up as:

VOILA_K8S_NAMESPACE=test k get pod
NAME                     READY   STATUS    RESTARTS   AGE
mongodb-shard-config-0   2/2     Running   0          26s
mongodb-shard-data-0-0   2/2     Running   0          26s
mongodb-shard-data-1-0   2/2     Running   0          26s
mongodb-shard-data-2-0   2/2     Running   0          26s
mongodb-shard-router-0   2/2     Running   0          26s

NOTE

If there is some error related to the volume mount try VOILA_K8S_NAMESPACE=test restart -s mongodb-shard-data-2 for instance.

Make sure it uses the correct disk:

VOILA_K8S_NAMESPACE=test k get pvc
NAME                          STATUS   VOLUME                                      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
snap-mongodb-shard-config-0   Bound    sandbox-mongodb-shard-config-0-1610653775   50Gi       RWO            fast-xfs       3h16m
snap-mongodb-shard-data-0-0   Bound    sandbox-mongodb-shard-data-0-0-1610653775   1000Gi     RWO            fast-xfs       3h16m
snap-mongodb-shard-data-1-0   Bound    sandbox-mongodb-shard-data-1-0-1610653775   1000Gi     RWO            fast-xfs       3h15m
snap-mongodb-shard-data-2-0   Bound    sandbox-mongodb-shard-data-2-0-1610653775   1000Gi     RWO            fast-xfs       3h15m

No new persistent volume claim created means that we are using the one based on the snapshots.

Check that the recovery worked by checking the logs. It should end up with:

[RECOVERY] Recovery operation completed you can update the services to regular mode.

Now we need to update the deployment to start as a cluster:

VOILA_K8S_NAMESPACE=test deploy u mongodb-shard \
--set global.resources.enabled=false \
--set nodeAffinity.operator=NotIn \
--set nodeAffinity.value=mongodb \
--set volumeClaimTemplateName=snap \
--set 'shard.shards[0].replicas=1' \
--set 'shard.shards[1].replicas=0' \
--set config.replicas=1 \
--set router.replicas=1 \
--set useNamespacedFQDN=false \
--set recoveryMode=false

After few seconds check the state of the pods and the state of the cluster with:

VOILA_K8S_NAMESPACE=test mgos status
MongoDB status

* Sharding status:

Shard shard-z0-2 tagged as [z0] members
 - mongodb-shard-data-2-0.mongodb-shard-data-2:27018

Shard reports-z1-0 tagged as [z1] members
 - mongodb-shard-reports-0-0.mongodb-shard-reports-0:27018

Shard shard-z0-1 tagged as [z0] members
 - mongodb-shard-data-1-0.mongodb-shard-data-1:27018

Shard shard-z0-0 tagged as [z0] members
 - mongodb-shard-data-0-0.mongodb-shard-data-0:27018

* Config node replicaset:

mongodb-shard-config-0.mongodb-shard-config:27019 PRIMARY

* Data shard 0 mongodb-shard-data node replicaset:

mongodb-shard-data-0-0.mongodb-shard-data-0:27018 PRIMARY

* Data shard 1 mongodb-shard-data node replicaset:

mongodb-shard-data-1-0.mongodb-shard-data-1:27018 PRIMARY

* Data shard 2 mongodb-shard-data node replicaset:

mongodb-shard-data-2-0.mongodb-shard-data-2:27018 PRIMARY

* Data shard 0 mongodb-shard-reports node replicaset:

mongodb-shard-reports-0-0.mongodb-shard-reports-0:27018 PRIMARY

You are ready to explore data with:

VOILA_K8S_NAMESPACE=test mgos

To stream data from this cluster to the main one:

# stream data!
mgos stream listen
Listening for data stream on:
10.64.20.54:42000
Use the above line as streamer target

# send data to the listener
VOILA_K8S_NAMESPACE=test mgos stream to 10.64.20.54:42000 --db squall --collection=policy
Streaming data to 10.64.20.54:42000...
 with args --db squall --collection=policy
2021-01-20T19:48:47.307+0000	writing squall.policy to archive on stdout
2021-01-20T19:48:47.604+0000	done dumping squall.policy (4567 documents)
Transfert is complete

Backing up the Voila environment

A Voila environment is set with a version control system (git).

Each time you exit an activated Voila environment, it will commit its changes and encrypt private information like:

  • certificates in ./certs folder
  • aporeto.yaml

This means that if the Voila environment is not activated, you cannot edit those files.

Once activated, the files are decrypted and you can edit them either from the Voila shell, or from the machine that is mounting the Voila environment.

The Voila environment MUST be saved as its your entry point to manage Microsegmentation Console upgrades and operations.

The recommended way is to push the Voila environment to a git remote repository like GitHub or GitLab. If you don’t have a git remote repository infrastructure, just make sure to backup the Voila environment to a safe place each time you are done using it.