IDENTITY-BASED MICROSEGMENTATION DOCUMENTATION

Securing a Kubernetes namespace

About securing a Kubernetes namespace

We deploy enforcers in discovery mode, a very permissive initial configuration. This allows your pods to function as they did before you deployed the enforcer, with no impact to their accustomed communications.

We recommend allowing your namespace to run in discovery mode for some time, perhaps a week. During this interval, the IP addresses, protocols, and ports of its communications collect in the Microsegmentation Console. This helps you compose a comprehensive list of the minimum allowed communications, ensuring a seamless experience when you disable discovery mode. After disabling discovery mode, enforcers reject any traffic not explicitly allowed.

After letting the enforcer collect flow logs for some time, complete this section in sequence. First allow the desired traffic, then disable discovery mode.

Before you begin, we recommend reviewing basic network policy concepts.

We use Google’s Hipster Shop for our examples. If you don’t have a Kubernetes namespace with pods running, complete the following steps to deploy the Hipster Shop so you can follow along.

Deploy the Hipster Shop (optional)

If you already have pods in your namespace, you can skip to the next section.

  1. Ensure that your target cluster has at least five nodes. If you have a GKE cluster, you can resize it and enable autoscaling as follows.

    gcloud container clusters resize <cluster-name> --num-nodes=5
    gcloud container clusters update <cluster-name> --enable-autoscaling --min-nodes=5 --max-nodes=10 
    
  2. Create a Kubernetes namespace for the Hipster Shop.

    kubectl create namespace hipster-shop
    
  3. Deploy the Hipster Shop.

    kubectl apply -n hipster-shop \
                  -f "https://github.com/GoogleCloudPlatform/microservices-demo/blob/master/release/kubernetes-manifests.yaml?raw=true"
    
  4. Issue the following command to observe the progress of the deployment.

    watch kubectl get pods -n hipster-shop 
    

    TIP

    You can also use kubectl’s --watch flag, but the regular watch command works better. If you don’t have it installed, we recommend doing so.

  5. Once all of the pods achieve Running status, press CTRL+C to exit the watch interface.

    TIP

    Be patient. Some pods may go into a CrashLoopBackOff or restart a few times but should still reach Running status. If not, you may need to add more nodes to your cluster.

  6. Obtain the public IP address of the front end service.

    kubectl get service/frontend-external -n hipster-shop 
    
  7. Paste the IP address shown under EXTERNAL-IP into your browser prefixed with http://. Example: http://35.203.153.201

  8. Exercise the application by making a purchase.

Allow domain name system communications

After allowing the flow logs to collect, you will likely observe multiple green dashed lines to the any-udp external network. Clicking the lines for more details, you’ll probably see that they are mostly to the same IP address and use port 53. Researching the IP address in question, you’ll soon realize that these are domain name system (DNS) queries. In the following procedure we describe how to enable access to the DNS server of your Kubernetes cluster.

TIP

If you do not see green dashed lines to any-udp, exec into one of your pods and run curl google.com.

  1. Set a MICROSEG_NS environment variable containing the Microsegmentation namespace of your application. This should be a great-grandchild namespace. The fourth segment is your Kubernetes namespace. In our example, we have a Hipster Shop microservice application in the hipster-shop Kubernetes namespace.

    export MICROSEG_NS=/acme/team-a/dev/hipster-shop
    
  2. Set a DNS_SERVER_IP environment variable containing the IP address of the DNS server.

    export DNS_SERVER_IP=10.39.240.10
    
  3. Use the following command to create an external network representing the DNS server.

    cat <<EOF | apoctl api create externalnetwork -n $MICROSEG_NS -f -
    name: DNS
    entries:
    - $DNS_SERVER_IP
    servicePorts:
    - 'udp/53'
    associatedTags:
    - ext:name=dns
    EOF
    
  4. Use the following command to create a network policy explicitly allowing all processing units in the Kubernetes namespace to initiate connections with the DNS external network.

    cat <<EOF | apoctl api create networkaccesspolicy -n $MICROSEG_NS -f -
    name: allow-dns
    action: Allow
    applyPolicyMode: OutgoingTraffic
    logsEnabled: true
    subject:
    - - \$identity=processingunit
      - \$namespace=$MICROSEG_NS
    object:
    - - ext:name=dns
    EOF
    
  5. Open the Microsegmentation section of the Prisma Cloud web interface and click Platform. You should see an external network named DNS appear with solid green lines. When you click the line for more details, you’ll see that your policy was applied. Compare the time stamps for any-udp and DNS. The DNS time stamps should be newer. The DNS external network now contains all of your DNS flows.

    TIP

    You may need to wait for a few minutes for the DNS queries to occur.

    DNS OK

Allow pods in namespace

Next, let’s allow the pods in this namespace to communicate with each other.

  1. Use the following command to allow these communications.

    cat <<EOF | apoctl api create networkaccesspolicy -n $MICROSEG_NS -f -
    name: allow-pods-in-namespace
    action: Allow
    applyPolicyMode: Bidirectional
    logsEnabled: true
    subject:
    - - \$namespace=$MICROSEG_NS
      - \$identity=processingunit
    object:
    - - \$namespace=$MICROSEG_NS
      - \$identity=processingunit
    EOF
    
  2. Exercise the application to create new flows.

  3. In the Microsegmentation section of the Prisma Cloud web interface, select Platform. You should see solid green lines between the pods. Click one of them and check the policy that was applied.

    TIP

    You may need to exercise the application, wait a little while, and shorten the time window to see the solid green lines.

    Pods allowed

Allow metadata service

When using a managed cloud provider, you may notice connections to their instance metadata endpoint. GCP, AWS, and Azure use the link-local address 169.254.169.254. Let’s go ahead and allow these connections.

  1. Set a METADATA_IP environment variable containing the IP address of your metadata service.

    export METADATA_IP=169.254.169.254
    
  2. Create an external network to represent the metadata service.

    cat <<EOF | apoctl api create externalnetwork -n $MICROSEG_NS -f -
    name: metadata
    entries:
    - 169.254.169.254
    servicePorts:
    - 'tcp/80'
    associatedTags:
    - ext:name=metadata
    EOF
    
  3. Create a network policy allowing any pod in your namespace to initiate connections to the metadata external network.

    cat <<EOF | apoctl api create networkaccesspolicy -n $MICROSEG_NS -f -
    name: allow-metadata
    action: Allow
    applyPolicyMode: OutgoingTraffic
    logsEnabled: true
    subject:
    - - \$namespace=$MICROSEG_NS
      - \$identity=processingunit
    object:
    - - ext:name=metadata
    EOF
    
  4. Exercise the application to generate new traffic. You should see a new metadata external network appear with solid green lines to requesting pods.

    TIP

    You may need to exercise the application, wait a little while, and shorten the time window to see the solid green lines.

    Metadata ok

Allow external services

Often, pods may connect to services outside of the cluster. For example, the Hipster Shop microservice application connects to a variety of Google APIs. We recommend using DNS records to allow the connections whenever possible, as IP addresses can change.

  1. In the Microsegmentation section of the Prisma Cloud web interface, expand Monitor, select Logs, then select DNS Lookup Logs.

  2. Review the domain names listed under Resolved Name.

  3. Create an external network to represent the external service. In the following example, we use a wildcard * to represent any subdomain of googleapis.com over port 443.

    cat <<EOF | apoctl api create externalnetwork -n $MICROSEG_NS -f -
    name: googleapis
    entries:
    - '*.googleapis.com'
    servicePorts:
    - 'tcp/443'
    associatedTags:
    - ext:name=googleapis
    EOF
    
  4. Create a network policy to allow the traffic.

    cat <<EOF | apoctl api create networkaccesspolicy -n $MICROSEG_NS -f -
    name: allow-googleapis
    action: Allow
    applyPolicyMode: OutgoingTraffic
    logsEnabled: true
    subject:
    - - \$namespace=$MICROSEG_NS
      - \$identity=processingunit
    object:
    - - ext:name=googleapis
    EOF
    
  5. Return to the Microsegmentation section of the Prisma Cloud web interface and select Platform. You should see the external network you just created appear with solid green lines to requesting pods.

    TIP

    You may need to exercise the application, wait a little while, and shorten the time window to see the solid green lines.

    Google APIs ok

Allow web server

If your namespace includes a web server, complete the following steps to allow access.

  1. If you’re running the Hipster Shop, use the following command to extract the ports from the service definition (requires jq).

    export TARGET_PORT=$(kubectl -n hipster-shop get service frontend-external -o json | jq '.spec.ports | .[].targetPort')
    echo $TARGET_PORT
    export NODE_PORT=$(kubectl -n hipster-shop get service frontend-external -o json | jq '.spec.ports | .[].nodePort')
    echo $NODE_PORT
    export PORT=$(kubectl -n hipster-shop get service frontend-external -o json | jq '.spec.ports | .[].port')
    echo $PORT
    
  2. Create an external network to represent all of the ports and protocols that the web server accepts connections on. You can modify entries if you wish to restrict access to certain IP addresses.

    NOTE

    If you’re not running the Hipster Shop, you may need to add your port numbers manually under ports. You can view the ports in the service definition using kubectl get service <service-name> -o yaml. Ensure that you add all of the ports under ports in the service definition.

    cat <<EOF | apoctl api create externalnetwork -n $MICROSEG_NS -f -
    name: internet
    entries:
    - 0.0.0.0/0
    servicePorts:
    - "tcp/$NODE_PORT"
    - "tcp/$TARGET_PORT"
    - "tcp/$PORT"
    associatedTags:
    - ext:name=internet
    EOF
    
  3. In the Microsegmentation section of the Prisma Cloud web interface, click the processing unit representing the web server and examine its tags. Look for a tag that uniquely identifies the web server and persists through time. Kubernetes labels work well.

    Examine tags

  4. Set a WEBSERVER_TAG environment variable containing the Microsegmentation tag you wish to use to identify it. In our Hipster Shop example, the frontend pod has a Kubernetes label app=frontend.

    export WEBSERVER_TAG=app=frontend
    
  5. Create a network policy allowing connections to the frontend pod.

    cat <<EOF | apoctl api create networkaccesspolicy -n $MICROSEG_NS -f -
    name: allow-web-server
    action: Allow
    applyPolicyMode: IncomingTraffic
    logsEnabled: true
    subject:
    - - ext:name=internet
    object:
    - - $WEBSERVER_TAG
      - \$identity=processingunit
    EOF
    
  6. Visit the website again, either by refreshing your existing browser connection or initiating a new one.

  7. You should see a new internet external network with a solid green line to the frontend pod.

    TIP

    You may need to exercise the application, wait a little while, and shorten the time window to see the solid green lines.

    Web server ok

Allow kubelet communications

If the application uses health checks and liveness/readiness probes, you may see dashed green lines to your pods from any-tcp and any-icmp. The source IP address is a cluster IP but you cannot locate any pod, endpoint, service, or node with this IP. It may be the bridge IP address of your kubelet. The method of discovering the bridge IP address of your kubelet varies according to your provider or deployment mechanism. We provide instructions for GKE below.

  1. Use the following command to determine your pod CIDRs.

    kubectl get node -o yaml | grep " podCIDR:"
    

    It should return something like the following.

    podCIDR: 10.36.3.0/24
    podCIDR: 10.36.1.0/24
    podCIDR: 10.36.4.0/24
    podCIDR: 10.36.2.0/24
    podCIDR: 10.36.0.0/24
    
  2. Replace the last digit of the IP addresses with 1 to derive the kubelet’s IPs. Using our example above, the kubelet’s IP addresses would be as follows.

    10.36.3.1
    10.36.1.1
    10.36.4.1
    10.36.2.1
    10.36.0.1
    
  3. Set KUBELET_IP environment variables containing the IP addresses of your kubelets. Below, we set five. Adjust this according to the number of nodes and kubelets in your cluster.

    export KUBELET_IP1=10.36.3.1
    export KUBELET_IP2=10.36.1.1
    export KUBELET_IP3=10.36.4.1
    export KUBELET_IP4=10.36.2.1
    export KUBELET_IP5=10.36.0.1
    
  4. Create an external network to represent your kubelets.

    cat <<EOF | apoctl api create externalnetwork -n $MICROSEG_NS -f -
    name: kubelet
    entries:
    - $KUBELET_IP1
    - $KUBELET_IP2
    - $KUBELET_IP3
    - $KUBELET_IP4
    - $KUBELET_IP5
    servicePorts:
    - tcp/1:65535
    - icmp/8/0
    - icmp/0/0
    - icmp/11/0
    - icmp/3/4
    associatedTags:
    - ext:name=kubelet
    EOF
    
  5. Create a network policy to allow all of your pods to communicate with any of the kubelets in your cluster.

    cat <<EOF | apoctl api create networkaccesspolicy -n $MICROSEG_NS -f -
    name: allow-kubelets
    action: Allow
    applyPolicyMode: IncomingTraffic
    logsEnabled: true
    subject:
    - - ext:name=kubelet
    object:
    - - \$namespace=$MICROSEG_NS
      - \$identity=processingunit
    EOF
    
  6. After some time, depending on the frequency of the probes, you should see a new kubelet external network with solid green line to the pods with health checks or liveness/readiness probes.

    TIP

    You may need to shorten the time window to see the solid green lines.

    kubelet ok

Disable discovery mode

  1. In the Microsegmentation section of the Prisma Cloud web interface, select Platform, and review any dashed green flows. As soon as you disable discovery mode, these connections will be blocked. Take a few moments to allow any desired traffic before continuing to the next step.

  2. Toggle discovery mode off, using the button in the top right corner.

    Discovery off

  3. You may see a new external network named Somewhere with red flows or red flows between pods. If you click on the red lines you can see that the connections were denied due to Microsegmentation’s default Reject all policy.

    Congratulations! You have secured your Kubernetes namespace. Microsegmentation denies any traffic not explicitly allowed by policy.