Identity-Based Microsegmentation Guide
LAST UPDATED: November 8, 2021

Kubernetes/OpenShift clusters

Overview

Microsegmentation provides a close integration with Kubernetes and OpenShift to make it easy to control and monitor clusters composed of Linux hosts. You can use either of the following methods to deploy the enforcer DaemonSet.

TIP

Either procedure deploys the enforcer to a cluster in approximately five minutes.

Using a YAML file

Prerequisites

Before you begin

The enforcer requires a CNI plugin. Most Kubernetes and OpenShift clusters use CNI plugins by default, but GKE and AKS do not. GKE and AKS default to kubenet. Before deploying the enforcer to a GKE or AKS cluster, configure them to use CNI as follows.

  • GKE: pass the --enable-network-policy flag to use CNI.

    gcloud container clusters update $CLUSTER-NAME --update-addons=NetworkPolicy=ENABLED
    gcloud container clusters update $CLUSTER-NAME --enable-network-policy
    
    gcloud container clusters create $CLUSTER-NAME --enable-network-policy
    

  • AKS: pass the --network-plugin azure flag at creation to use CNI.

    az aks create --name $CLUSTER-NAME --resource-group $RESOURCE_GROUP --generate-ssh-keys --network-plugin azure
    

TIP

If you’re not sure if the cluster uses a CNI plug-in or not, exec into one of your nodes and run cat /etc/cni/net.d. This file, if present, contains your CNI configuration. You can also check the pods in kube-system and see if a CNI plugin is listed. On EKS, you can use the following command to discover the CNI details kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

Deploying the enforcers

  1. Set a CLUSTER_NS environment variable identifying the Microsegmentation namespace for this cluster. This should be a grandchild namespace.

    export CLUSTER_NS=/acme/aws-dev/k8s
    
  2. To generate a YAML file that deploys the enforcers to your current context, use one of the following commands.

    apoctl enforcer install kubernetes --installation-mode yaml \
                                       --cluster-type eks \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns
    
    apoctl enforcer install kubernetes --installation-mode yaml \
                                       --cluster-type gke \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns 
    
    apoctl enforcer install kubernetes --installation-mode yaml \
                                       --cluster-type aks \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns 
    
    apoctl enforcer install kubernetes --installation-mode yaml \
                                       --cluster-type custom \
                                       --custom-cni-chained \
                                       --custom-cni-bin-dir /opt/cni/bin \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns 
    
    apoctl enforcer install kubernetes --installation-mode yaml \
                                       --cluster-type custom \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns 
    
    apoctl enforcer install kubernetes --installation-mode yaml \
                                       --cluster-type ocp4 \
                                       --enable-openshift \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns 
    

    TIP

    We detail the apoctl enforcer install kubernetes command further in the reference documentation. You can also run apoctl enforcer install kubernetes -h to review its flags. To learn how to customize your deployment, refer to Enforcer configuration options at the bottom of this page.

  3. Review the generated YAML.

    cat enforcerd.yaml
    
  4. Apply the YAML to your cluster.

    kubectl apply -f enforcerd.yaml
    
  5. To confirm your deployment, issue the following command.

    watch kubectl get pods --all-namespaces
    

    Wait until all of the pods have a status of Running or Completed.

    NOTE

    The above command uses watch, which is not installed by default on macOS. While we recommend installing it, you can also omit the watch portion of the command and repeatedly issue the command until the enforcer pods achieve the necessary status.

  6. Press CTRL+C to exit the watch command. Issue the following apoctl command to check the enforcers.

    apoctl api list enforcers --namespace $CLUSTER_NS \
                              -o table \
                              -c ID \
                              -c name \
                              -c namespace \
                              -c operationalStatus
    
  7. apoctl should return a list of the enforcers deployed. You should see an enforcer instance on each agent node. An example for a three-node GKE cluster follows.

                 ID            |                    name                   |    namespace                     | operationalStatus  
    ---------------------------+-------------------------------------------+----------------------------------+--------------------
      5f74d837f0fe170703c10d6b | gke-aws-dev-01-default-pool-cf284cf1-5bqn | /acme/aws-dev/k8s | Connected          
      5f74d836f0fe170703c10d6a | gke-aws-dev-01-default-pool-cf284cf1-5pjs | /acme/aws-dev/k8s | Connected          
      5f74d836f0fe170703c10d69 | gke-aws-dev-01-default-pool-cf284cf1-cqrd | /acme/aws-dev/k8s | Connected  
    

    All enforcer instances should have an operationalStatus of Connected.

  8. Open the Microsegmentation Console web interface, navigate to the enforcer’s namespace, and select Enforcers under Manage. You should find your enforcers listed with the status connected. Click the enforcers to review their Microsegmentation tags.

  9. Select Platform in the side navigation menu. If your cluster contains pods outside of the kube-system namespace, you should see them with dashed green lines to a Somewhere external network. Your cluster is in discovery mode. Refer to Securing a Kubernetes namespace to learn how to allow the desired traffic and disable discovery mode.

    TIP

    To see the pods and their traffic in the Platform pane, you may need to toggle Recursive to on.

Using a Helm chart

Prerequisites

Before you begin

The enforcer requires a CNI plugin. Most Kubernetes and OpenShift clusters use CNI plugins by default, but GKE and AKS do not. GKE and AKS default to kubenet. Before deploying the enforcer to a GKE or AKS cluster, configure them to use CNI as follows.

  • GKE: pass the --enable-network-policy flag to use CNI.

    gcloud container clusters update $CLUSTER-NAME --update-addons=NetworkPolicy=ENABLED
    gcloud container clusters update $CLUSTER-NAME --enable-network-policy
    
    gcloud container clusters create $CLUSTER-NAME --enable-network-policy
    

  • AKS: pass the --network-plugin azure flag at creation to use CNI.

    az aks create --name $CLUSTER-NAME --resource-group $RESOURCE_GROUP --generate-ssh-keys --network-plugin azure
    

TIP

If you’re not sure if the cluster uses a CNI plug-in or not, exec into one of your nodes and run cat /etc/cni/net.d. This file, if present, contains your CNI configuration. You can also check the pods in kube-system and see if a CNI plugin is listed. On EKS, you can use the following command to discover the CNI details kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

Deploying the enforcers

  1. Set a CLUSTER_NS environment variable identifying the Microsegmentation namespace for this cluster. This should be a grandchild namespace.

    export CLUSTER_NS=/acme/aws-dev/k8s
    
  2. To generate a Helm chart that deploys the enforcers to your current context, use one of the following commands.

    apoctl enforcer install kubernetes --installation-mode helm \
                                       --cluster-type eks \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns
    
    apoctl enforcer install kubernetes --installation-mode helm \
                                       --cluster-type gke \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns 
    
    apoctl enforcer install kubernetes --installation-mode helm \
                                       --cluster-type aks \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns 
    
    apoctl enforcer install kubernetes --installation-mode helm \
                                       --cluster-type custom \
                                       --custom-cni-chained \
                                       --custom-cni-bin-dir /opt/cni/bin \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns 
    
    apoctl enforcer install kubernetes --installation-mode helm \
                                       --cluster-type custom \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns 
    
    apoctl enforcer install kubernetes --installation-mode helm \
                                       --cluster-type ocp4 \
                                       --enable-openshift \
                                       --namespace $CLUSTER_NS \
                                       --api $MICROSEG_API \
                                       --repo https://charts.aporeto.com/releases/release-5.0.8/clients \
                                       --set imageRegistry=gcr.io/prismacloud-cns 
    

    TIP

    We detail the apoctl enforcer install kubernetes command further in the reference documentation. You can also run apoctl enforcer install kubernetes -h to review its flags. To learn how to customize your deployment, refer to Enforcer configuration options at the bottom of this page.

  3. Confirm the Helm chart creation.

    ls enforcerd
    
  4. Create an aporeto namespace.

    kubectl create namespace aporeto
    
  5. Use the Helm chart to deploy the enforcers to your cluster.

    helm install enforcerd ./enforcerd --namespace aporeto
    
  6. To confirm your deployment, issue the following command.

    watch kubectl get pods --all-namespaces
    

    Wait until all of the pods have a status of Running or Completed.

    NOTE

    The above command uses watch, which is not installed by default on macOS. While we recommend installing it, you can also omit the watch portion of the command and repeatedly issue the command until the enforcer pods achieve the necessary status.

  7. Press CTRL+C to exit the watch command. Issue the following apoctl command to check the enforcers.

    apoctl api list enforcers --namespace $CLUSTER_NS \
                              -o table \
                              -c ID \
                              -c name \
                              -c namespace \
                              -c operationalStatus
    
  8. apoctl should return a list of the enforcers deployed. You should see an enforcer instance on each agent node. An example for a three-node GKE cluster follows.

                 ID            |                    name                   |    namespace                     | operationalStatus  
    ---------------------------+-------------------------------------------+----------------------------------+--------------------
      5f74d837f0fe170703c10d6b | gke-aws-dev-01-default-pool-cf284cf1-5bqn | /acme/aws-dev/k8s | Connected          
      5f74d836f0fe170703c10d6a | gke-aws-dev-01-default-pool-cf284cf1-5pjs | /acme/aws-dev/k8s | Connected          
      5f74d836f0fe170703c10d69 | gke-aws-dev-01-default-pool-cf284cf1-cqrd | /acme/aws-dev/k8s | Connected  
    

    All enforcer instances should have an operationalStatus of Connected.

  9. Open the Microsegmentation Console web interface, navigate to the enforcer’s namespace, and select Enforcers under Manage. You should find your enforcers listed with the status connected. Click the enforcers to review their Microsegmentation tags.

  10. Select Platform in the side navigation menu. If your cluster contains pods outside of the kube-system namespace, you should see them with dashed green lines to a Somewhere external network. Your cluster is in discovery mode. Refer to Securing a Kubernetes namespace to learn how to allow the desired traffic and disable discovery mode.

    TIP

    To see the pods and their traffic in the Platform pane, you may need to toggle Recursive to on.

Enforcer configuration options

The enforcer exposes the following configuration options. You can pass these to the apoctl enforcer install command using the --raw-flags flag. Example: --raw-flags "--log-level=debug --log-format=human --log-to-console=true" You can also modify the enforcer’s configuration after install via kubectl edit daemonset enforcerd -n aporeto, adding the flags as arguments to the container. The enforcer pods will restart. An example follows.

...
    spec:
      containers:
      - args:
      - --log-level=debug
      - --log-format=human
...
Flag Description
--activate-control-plane-pus Pass this flag if you wish to recognize the Microsegmentation Console as a processing unit, allowing its communications to be monitored and controlled. By default, the enforcer ignores them.
--activate-kube-system-pus Pass this flag if you wish to recognize containers in the kube-system namespace as processing units, allowing their communications to be monitored and controlled. By default, the enforcer ignores them.
--activate-openshift-pus Pass this flag if you wish to recognize containers in Kubernetes namespaces starting with openshift- as processing units, allowing their communications to be monitored and controlled. By default, the enforcer ignores them.
--api The URL of the Microsegmentation Console API.
--api-cacert Path to CA certificate.
--api-skip-verify Disables check on certificate signature as trusted.
--appcreds Path to application credentials.
--application-proxy-port Start of the port range for ports used by the enforcer application proxy. Defaults to 20992. You may adjust this if you experience conflicts.
--cloud-probe-timeout The enforcer can determine if it is running in a cloud environment, such as AWS, GCP, or Azure. This is the maximum amount of time to wait for these internal probes to complete. Default is two seconds.
--disable-dns-proxy Pass this flag to disable the enforcer DNS proxy, which allows policies to be written based on FQDN, in cases where an exact IP address may be unpredictable.
--dns-server-address DNS server address or CIDR that is observed by the enforcer DNS proxy. Defaults to 0.0.0.0/0.
--enable-ebpf (Beta) Pass this flag to gain performance improvements by using extended Berkeley Packet Filter (eBPF) on systems that support it.
--enable-ipv6 The enforcer ignores IPv6 communications by default. If you have IPv6 enabled and wish to monitor and control these connections, pass this flag.
--log-level Quantity of logs that the enforcer should generate. Defaults to info. Alternatively, you can set it to debug, trace, or warn.
--log-to-console Controls whether the enforcer’s logs are written to stdout. Boolean that defaults to false.
--namespace The Microsegmentation namespace the enforcer should register in.
--tag Microsegmentation tag for this enforcer. Note: to modify after the enforcer has started, you must shut down the enforcer, delete the enforcer object in the Microsegmentation Console, and perform a fresh install.
--token Microsegmentation token for the enforcer to use to register to the Microsegmentation Console.
--working-dir A persistent working directory with write, read, and execute permissions. Files such as logs are stored here. Defaults to /var/lib/enforcerd