Documentation

Working with CRDs

About working with Kubernetes custom resource definitions (CRDs)

One of the big advantages of integrating a Kubernetes cluster with Aporeto is that one can make use of existing Kubernetes tooling to program Aporeto’s Control Plane API. This is achieved by making use of Kubernetes custom resources. The required custom resource definitions are already installed if you followed one of our Kubernetes installation guides and you are ready to make use of them.

Explore the custom resource definitions

The easiest way to explore the installed custom resource definitions in a Kubernetes cluster is by listing all of the API resources of a cluster.

If you run the following command, you are going to get a filtered list of all the available custom resources that have been installed by Aporeto.

kubectl api-resources | grep aporeto
externalnetworks                  extnet,extnets,apoextnet,apoextnets                                             api.aporeto.io                 true         ExternalNetwork
httpresourcespecs                 apohttpresourcespec,apohttpresourcespecs                                        api.aporeto.io                 true         HTTPResourceSpec
namespacemappingpolicies          nspolicy,nspolicies,nsmap,nsmaps,aponspolicy,aponspolicies,aponsmap,aponsmaps   api.aporeto.io                 true         NamespaceMappingPolicy
namespaces                        apons                                                                           api.aporeto.io                 true         Namespace
networkaccesspolicies             aponetpol,aponetpols                                                            api.aporeto.io                 true         NetworkAccessPolicy
servicedependencies               srvdep,srvdeps,aposrvdep,aposrvdeps                                             api.aporeto.io                 true         ServiceDependency
services                          srv,aposrv                                                                      api.aporeto.io                 true         Service
tokenscopepolicies                tsp,tsps,apotsp,apotsps                                                         api.aporeto.io                 true         TokenScopePolicy
podinjectorselectors              podsel,podsels,apopodsel,apopodsels                                             k8s.aporeto.io                 true         PodInjectorSelector
servicemappings                   srvmap,srvmaps,aposrvmap,aposrvmaps                                             k8s.aporeto.io                 true         ServiceMapping

From the above output you can see that there are two different API groups on which custom resources are defined: api.aporeto.io and k8s.aporeto.io. Custom resource definitions that are defined on the api.aporeto.io group refer one-to-one to API object definitions on Aporeto’s control plane API. Custom resource definitions that are defined on the k8s.aporeto.io group are dedicated Kubernetes integration components and are discussed separately below.

Before we learn how to manually create our own custom resources, we can explore already existing ones in our cluster. The Aporeto operator has several controllers that create and manage Aporeto custom resources. Most prominently it will watch Kubernetes namespaces and create/delete Aporeto namespaces accordingly. If you have not done so yet, you should take some time to understand how namespaces are mapped by default. You will then understand that the Aporeto custom resource objects that represent the Aporeto namespaces must live in the base namespace in Kubernetes - which by default is the aporeto Kubernetes namespace. Going forward it is assumed that the base namespace of the cluster in this example is /acme/kubernetes.

So let us inspect the aporeto Kubernetes namespace and list all of the generated Aporeto namespace objects there:

kubectl -n aporeto get apons
NAME                STATUS
default-ms5lh       Synchronized
kube-public-xqf2j   Synchronized
kube-system-qbn4s   Synchronized

We can see by the names of the Kubernetes objects that they are most likely referring to the default, kube-public and kube-system Kubernetes namespaces. The Synchronized status in the second column is an indicator which tells us that this object has been synchronized successfully with Aporeto’s control plane API. As with every Kubernetes object the random characters at the end of the name indicate that the objects were stored making use of the generateName field in the metadata section of the object. All controllers usually make use of that to prevent name collisions - and it is generally a good indicator that an object has been generated by a controller rather than manually by a user.

If we inspect one of the namespace objects in more detail, we are going to reveal the true structure of them:

kubectl -n aporeto get apons default-ms5lh -o yaml
apiVersion: api.aporeto.io/v1beta1
kind: Namespace
metadata:
  creationTimestamp: "2019-05-30T01:23:07Z"
  generateName: default-
  generation: 77
  labels:
    aporeto.io/namespace: default
    aporeto.io/namespace-controller-created: ""
  name: default-ms5lh
  namespace: aporeto
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Namespace
    name: default
    uid: 733562b3-826e-11e9-84c3-42010a80017b
  resourceVersion: "3560809"
  selfLink: /apis/api.aporeto.io/v1beta1/namespaces/aporeto/namespaces/default-ms5lh
  uid: 7b1493d5-8279-11e9-84c3-42010a80017b
spec:
  SSHCA: ""
  SSHCAEnabled: false
  annotations: {}
  associatedSSHCAID: ""
  associatedTags:
  - k8s:label:aporeto.io/namespace=default
  - k8s:label:aporeto.io/namespace-controller-created=<empty>
  description: ""
  localCA: ""
  localCAEnabled: false
  metadata: []
  name: default
  namespace: ""
  networkAccessPolicyTags: []
  normalizedTags: []
  protected: true
  serviceCertificateValidity: 1h
status:
  aporetoStatus: Synchronized

If we compare the spec section of this custom resource object with the definition of an Aporeto namespace object:

apoctl api describe namespace
Object: namespace

Aliases:
  ns

A Namespace represents the core organizational unit of the system. All objects
always exists in a single namespace. A Namespace can also have child namespaces.
They can be used to split the system into organizations, business units,
applications, services or any combination you like.

Example Data:

{
  "ID": "",
  "SSHCA": "",
  "SSHCAEnabled": false,
  "annotations": {},
  "associatedSSHCAID": "",
  "associatedTags": [],
  "createTime": "0001-01-01T00:00:00Z",
  "description": "",
  "localCA": "",
  "localCAEnabled": false,
  "metadata": [],
  "name": "",
  "namespace": "",
  "networkAccessPolicyTags": [],
  "normalizedTags": [],
  "protected": false,
  "serviceCertificateValidity": "1h",
  "updateTime": "0001-01-01T00:00:00Z"
}

Use --full to see the complete description of the properties.

We can now see that the fields of the Kubernetes custom resource spec match one-to-one with the fields of the Aporeto namespace object. In the metadata section of the Kubernetes custom resource we can furthermore see that the actual Kubernetes namespace object owns this custom resource. This ensures that the Aporeto namespace is going to get deleted when the Kubernetes namespace gets deleted.

Let us also explore the existing namespace mapping policies in the base namespace of a cluster:

kubectl -n aporeto get aponsmap
NAME                SUBJECT                              MAPPEDNAMESPACE                STATUS
default-xn96l       [[@app:k8s:namespace=default]]       /acme/kubernetes/default       Synchronized
kube-public-w74k6   [[@app:k8s:namespace=kube-public]]   /acme/kubernetes/kube-public   Synchronized
kube-system-w8s9r   [[@app:k8s:namespace=kube-system]]   /acme/kubernetes/kube-system   Synchronized

We now have listed all the generated components to understand what the Aporeto operator does to closely integrate a Kubernetes cluster into Aporeto.

The controller is watching Kubernetes namespace objects and is generating Aporeto namespaces through custom resources in the base namespace of the cluster. This facilitates the synchronization of Kubernetes namespaces to Aporeto namespaces.

However, the enforcer is always going to create processing units in the same namespace where it has registered itself. This would mean that all processing units that will be created from pods would still show up in the base namespace of the cluster. In the above example they would therefore show up under /acme/kubernetes instead of as expected under /acme/kubernetes/default.

So the controller is furthermore also generating namespace mapping policies through custom resources in the base namespace of the cluster. All of these mapping policies refer to a common attribute on processing units: @app:k8s:namespace. This attribute is going to be part of all processing units that are being generated from pods and refers to the Kubernetes namespace of the pods of course. It is then setting the mappedNamespace property of the namespace mapping policy to the Aporeto namespace that has been generated in the first step. This facilitates that the processing units created from Kubernetes pods in a Kubernetes namespace will show up in the expected Aporeto namespace.

Create custom resources with kubectl

The most simple example for creating a custom resource is to create Aporeto namespaces. Create a file called apons.yaml with the following contents:

apiVersion: api.aporeto.io/v1beta1
kind: Namespace
metadata:
  name: databases
spec: {}
---
apiVersion: api.aporeto.io/v1beta1
kind: Namespace
metadata:
  name: master
spec:
  namespace: databases
  description: A child namespace within a child namespace

You can then create the Aporeto namespaces with your common Kubernetes tooling. Run the following command to create them:

kubectl create -f apons.yaml

Notes about api.aporeto.io objects

As we have learned in Explore the custom resource definitions all custom resource definitions in the api.aporeto.io group are one-to-one mappings of objects in Aporeto’s control plane API. Furthermore we learned that the spec section of the custom resources map one-to-one to properties of Aporeto’s control plane API object. However, there are a couple of exceptions for some of the common properties, and usage of the custom resources is a bit different from using them with apoctl.

  • You cannot set the ID property in a custom resource: it will fail object validation. You cannot retrieve it through the custom resource. If you need the value of this property you have to use apoctl.

  • You cannot set the createTime property in a custom resource: it will fail object validation. This is the same behavior as with apoctl. However, you are also not going to be able to retrieve this property through the custom resource. If you need the value of this property you have to use apoctl.

  • The same rules as for the createTime property apply for the updateTime property.

  • You can set the description property in a custom resource, however, it is not going to propagate to Aporeto’s control plane API. Every object that is going to get created through a custom resource in a Kubernetes cluster is going to propagate a dedicated description which explains that this resource is managed by the Aporeto operator. Here is an example: Managed by aporeto-operator from resource /apis/api.aporeto.io/v1beta1/namespaces/default/namespaces/databases

  • You usually do not need to set the name property in the spec of a custom resource. The Aporeto operator is going to use the Kubernetes name of an object (the .metadata.name field) if the property is not set or left empty. Furthermore it is going to enforce that the Kubernetes name and the name property of the spec always match. However, technically - with the exception of Aporeto namespaces - Aporeto’s control plane API allows to have multiple objects with the same name, the Kubernetes API though has a constraint on the object name. To work around this restriction you are allowed to use the generateName field on the Kubernetes object. The value needs to match the name property of the spec though; you are allowed to use an additional - character as a suffix in the generateName though.

  • You usually do not need to set the namespace property in the spec of a custom resource. Depending on your mapNamespaces setting of the Aporeto operator all custom resources are already being created in the correct namespace as expected. With mapNamespaces enabled - which is also the default - the Kubernetes namespace is taking into consideration and all custom resources will be created in the mapped Kubernetes namespace. However, if you create child namespaces through custom resources and you want to create further Aporeto objects inside of these namespaces, you can refer to these namespaces inside of the namespaces property here. Refer to the detailed explanation on how namespaces are mapped in the reference documentation.

  • You cannot disable object protection through the protected property. Every object created through a Kubernetes custom resources is going to have protected set to true. This is for a very good reason: the Kubernetes custom resources are managed through Kubernetes, and the Kubernetes API is the source of truth for the state of these objects. Therefore the synchronization between custom resources and Aporeto’s control plane API is a one-way sync. To prevent that objects are being modified from outside of the Kubernetes cluster we set the protected property to true on every custom resource.

Translate Kubernetes services to Aporeto services

NOTE

This section assumes that you are already familiar with Aporeto API services. If you only want to use Aporeto network policies and encryption within one cluster, you do not need Aporeto services. However, if you want to use Aporeto for user authorization, policies on your API endpoints or full mutual TLS encryption between any TCP or HTTP service, then this section is going to explain how the Aporeto operator can make the configuration of Aporeto significantly easier by introducing the concepts of service mappings.

One of the bigger challenges in a Kubernetes cluster is to manage Aporeto API services. For Aporeto services to work as expected you have to manage the DNS names and in some cases also the IP addresses of the service. This is also under the assumption that all access to this service is performed directly over the service IPs or service DNS names. Some applications are actually only looking up the endpoints that belong to a service and are making connections to the pod IPs directly. Now that means that the Aporeto service needs to be updated every time a pod goes away or joins a service so that it can update the list of IPs of the Aporeto service.

In order to manage this efficiently, automated and with the least amount of effort for the end user the Aporeto operator provides a custom resource definition which lets you map a Kubernetes service to an Aporeto service. It is the ServiceMapping custom resource definition within the k8s.aporeto.io API group. It will keep the Aporeto service up to date with any changes to the Kubernetes service as well as any changes to its own template.

Let’s take a look at an example service mapping object and let us define it in a file called myapp-aporeto-sm.yaml:

apiVersion: k8s.aporeto.io/v1beta1
kind: ServiceMapping
metadata:
  name:  myapp
  namespace: default
spec:
  mapping:
    portName: http
    publicPortName: https
    serviceName: myapp
  options:
    discoverEndpoints: true
    discoverNodes: false
  template:
    metadata:
      labels:
        app: myapp
    spec:
      type: HTTP
      TLSType: Aporeto
      authorizationType: JWT
      JWTSigningCertificate: |
        -----BEGIN PUBLIC KEY-----
        ...
        -----END PUBLIC KEY-----
      IPs:
        - 203.0.113.42
      hosts:
        - alternate.domain.example.com
      exposedAPIs:
        - - api=kubernetes

We can go ahead now and apply this service mapping with our usual kubectl commands:

kubectl apply -f myapp-aporeto-sm.yaml

If we do not have a Kubernetes service myapp installed into our default Kubernetes namespace, then nothing will actually happen. All that we have told Kubernetes now, is that for a corresponding Kubernetes service myapp a matching Aporeto service will be created. We can confirm this by running the following command:

kubectl get aposrvmap

It will reveal some of the details about the service mapping, and show in the status section that there is currently no Kubernetes service defined.

NAME    SERVICENAME   PUBLICPORTNAME   PORTNAME   TYPE   AUTHORIZATIONTYPE   STATUS
myapp   myapp         https            http       HTTP   JWT                 UndefinedService

However, let’s examine the object a bit closer before creating the matching Kubernetes service.

First of all, the main work of the mapping is done in the mapping section. The serviceName refers to the name of the Kubernetes service that should be mapped which must be within the same namespace. Note that as mentioned above your Kubernetes service does not need to exist for you to be able to create the mapping. The portName refers to a port name within the Kubernetes service definition that should be mapped to the Aporeto service port field. This must be the main port where your application is actually going to be listening on. If you want to use user authorization, you also need to define a public port in the Aporeto service. This is going to be mapped from the Kubernetes service using the publicPortName.

Using just this mapping information the Aporeto operator can already infer most of the values that it needs to populate an Aporeto service object. It is going to automatically take all Kubernetes DNS names for a service and use them for Aporeto service host names. It is also going to extract all service IPs - including the load balancer IP - and adds them to the list of IPs of the Aporeto service object.

You might find yourself in a situation though where you need to access your service just through pod IPs or through the node port of a service. In this case you can either activate the discoverEndpoints or the discoverNodes options from the options section of the service mapping specification. Discovering endpoints means that all endpoint IPs of a Kubernetes service are going to be added to the list of IPs of the Aporeto service. They are also going to be kept up to date with any changes that occur to the endpoint which are usually all pod IPs. Discovering nodes means that all internal and external IPs of all nodes, as well as all their FQDNs and host names are going to be added to the list of host names of your Aporeto service. As with discovering endpoints any changes of host names, IP addresses, or generally nodes being added to or removed from the cluster are going to update the mapped Aporeto service.

NOTE

Both discoverEndpoints and discoverNodes are optional as they are going to update the Aporeto service potentially very often. Use these options only if you really need them.

Last but not least, there is the template section. It works exactly like a pod template in a deployment or a replica set. Here you pass in your Aporeto service template that you would like to get merged into the resulting Aporeto service custom resource object. All entries for the IPs as well as hosts are going to be appended to what is auto discovered from the Kubernetes service. Ports are going to be derived from the Kubernetes service as explained above. All other Aporeto service properties will just be taken from the template as is. The metadata field in the template is Kubernetes metadata that works the same way as for a pod template. Feel free to add additional metadata like labels here and they will be added to the Aporeto service custom resource object once created.

Next we are going to define a Kubernetes service in a file called myapp-svc.yaml.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: myapp
  name: myapp
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app: myapp
  type: LoadBalancer

We can go ahead now and apply this service with our usual kubectl commands:

kubectl apply -f myapp-svc.yaml

Now, as the Kubernetes service name myapp matches the serviceName that was used in the service mapping definition, the Aporeto operator will go ahead and create an Aporeto service custom resource within the same default Kubernetes namespace.

Examining the service mapping again using kubectl get aposrvmap, it will reveal that the status of the service mapping is running.

NAME    SERVICENAME   PUBLICPORTNAME   PORTNAME   TYPE   AUTHORIZATIONTYPE   STATUS
myapp   myapp         https            http       HTTP   JWT                 Running

When a service mapping is running, it means that it is managing an Aporeto service object now. We can examine the Aporeto service custom resources by running the following command:

kubectl get aposrv

It will show that there is a generated Aporeto service object now, that will look similar to the following:

NAME          TYPE   TLSTYPE   AUTHORIZATIONTYPE   PORT   STATUS
myapp-d4q6r   HTTP   Aporeto   JWT                 8080   Synchronized

A synchronized status denotes that the custom resource has been successfully synchronized with Aporeto’s control plane API. The name myapp-d4q6r is a generated name using the service mapping name and the generateName property of the Kubernetes metadata. The name will vary on every system.

Looking at the complete output of the generated Aporeto service will put together the puzzle of auto-discovered IPs and hosts and values taken from the template:

kubectl get aposrv myapp-d4q6r -o yaml

It will show output similar to the following:

apiVersion: api.aporeto.io/v1beta1
kind: Service
metadata:
  finalizers:
  - api.aporeto.io/control-plane-sync
  generateName: myapp-
  generation: 2
  labels:
    aporeto.io/servicemapping-discover-endpoints: ""
    aporeto.io/servicemapping-selected-service: myapp
    aporeto.io/servicemapping-selected-service-port: http
    aporeto.io/servicemapping-uid: a74ad566-c604-11e9-8cfd-42010a80001d
    app: myapp
  name: myapp-d4q6r
  namespace: default
  ownerReferences:
  - apiVersion: k8s.aporeto.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ServiceMapping
    name: myapp
    uid: a74ad566-c604-11e9-8cfd-42010a80001d
  resourceVersion: "21822863"
  selfLink: /apis/api.aporeto.io/v1beta1/namespaces/default/services/myapp-d4q6r
  uid: 5f2073b8-c605-11e9-8cfd-42010a80001d
spec:
  IPs:
  - 203.0.113.42
  - 10.27.252.84
  - 34.66.31.164
  JWTSigningCertificate: |
    -----BEGIN PUBLIC KEY-----
    ...
    -----END PUBLIC KEY-----
  TLSType: Aporeto
  authorizationType: JWT
  exposedAPIs:
  - - api=kubernetes
  exposedPort: 80
  hosts:
  - alternate.domain.example.com
  - myapp
  - myapp.default
  - myapp.default.svc.cluster.local
  port: 8080
  protected: true
  publicApplicationPort: 8080
  selectors:
  - - app=myapp
  type: HTTP
status:
  aporetoStatus: Synchronized

The IP 203.0.113.42 and the host alternate.domain.example.com are coming from the service mapping template. However, the IP 10.27.252.84 is the cluster IP of the Kubernetes service, and 34.66.31.164 is the IP of the load balancer that was assigned to this service. The host names myapp.default.svc.cluster.local, myapp.default and myapp all refer to valid in-cluster DNS names of the Kubernetes service myapp.

However, the service mapping also activated the discover endpoints option. So why are there no pod IPs? The answer to this is simple: just because we created a service mapping and a Kubernetes service does not necessarily mean that there are pods running which match the service. List the endpoints of the service with the following command:

kubectl get ep myapp

It will be clearly shown that there are no endpoints in this Kubernetes service yet.

NAME    ENDPOINTS   AGE
myapp   <none>      17m

So in the next step let’s create a sample application that will match the Kubernetes service with this or a similar command:

kubectl run myapp --image=gcr.io/google-samples/hello-app:2.0 -l app=myapp --generator=run-pod/v1

Retrieving the details of the Aporeto service again (by running kubectl get aposrv myapp-d4q6r -o yaml) will show that the pod IP has been added to the service.

apiVersion: api.aporeto.io/v1beta1
kind: Service
metadata:
  finalizers:
  - api.aporeto.io/control-plane-sync
  generateName: myapp-
  generation: 5
  labels:
    aporeto.io/servicemapping-discover-endpoints: ""
    aporeto.io/servicemapping-selected-service: myapp
    aporeto.io/servicemapping-selected-service-port: http
    aporeto.io/servicemapping-uid: a74ad566-c604-11e9-8cfd-42010a80001d
    app: myapp
  name: myapp-d4q6r
  namespace: default
  ownerReferences:
  - apiVersion: k8s.aporeto.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ServiceMapping
    name: myapp
    uid: a74ad566-c604-11e9-8cfd-42010a80001d
  resourceVersion: "21828835"
  selfLink: /apis/api.aporeto.io/v1beta1/namespaces/default/services/myapp-d4q6r
  uid: 5f2073b8-c605-11e9-8cfd-42010a80001d
spec:
  IPs:
  - 203.0.113.42
  - 10.27.252.84
  - 34.66.31.164
  - 10.24.1.34
  JWTSigningCertificate: |
    -----BEGIN PUBLIC KEY-----
    ...
    -----END PUBLIC KEY-----
  TLSType: Aporeto
  authorizationType: JWT
  exposedAPIs:
  - - api=kubernetes
  exposedPort: 80
  hosts:
  - alternate.domain.example.com
  - myapp
  - myapp.default
  - myapp.default.svc.cluster.local
  port: 8080
  protected: true
  publicApplicationPort: 8080
  selectors:
  - - app=myapp
  type: HTTP
status:
  aporetoStatus: Synchronized

In summary, Aporeto service mappings are able to update service definitions dynamically depending on the state of your Kubernetes cluster.

Use pod injector selectors for more security

The enforcer has a very lightweight integration model. The benefit of this is that all you need to do in order to integrate your Kubernetes cluster with Aporeto, you can install the enforcer as a DaemonSet. When you want to uninstall the Aporeto enforcer, all you need to do is to delete the DaemonSet. You do not need to reconfigure CNI on your Kubernetes nodes, you do not need to reconfigure the kubelet on the nodes, you do not need to perform any other configuration that would directly affect the configuration or installation of Kubernetes itself.

NOTE

The uninstallation procedure for the Aporeto operator is more involved. Please follow the uninstall instructions very carefully to fully uninstall the Aporeto operator.

However, this also has a direct consequence to our processing unit activation model. In Kubernetes - compared to every other activation model - we can only start a processing unit after the first container of a pod has started. This means that a Kubernetes pod could be potentially unprotected for a fraction of time.

To solve this problem, the Aporeto operator provides a custom resource definition called PodInjectorSelector which resides in the k8s.aporeto.io API group. Its specification has a pod selector which injects an init container into every Kubernetes pod which matches the selector. The init container will then wait indefinitely until the processing unit is up and running and is considered healthy - which means that the pod is then fully protected by Aporeto. Internally the init container is basically calling the health check API of the enforcer until it receives a successful health check from that API.

The following example is going to create a pod injector selector which is going to inject the Aporeto init container into every Kubernetes pod which has the label app=secretsd set. Create a file called secretsd-pis.yaml with the following contents:

apiVersion: k8s.aporeto.io/v1beta1
kind: PodInjectorSelector
metadata:
  name: secretsd
spec:
  selector:
    matchLabels:
      app: secretsd
  #options:
  #  image: gcr.io/aporetodev/aporeto-init-waitforpu
  #  imagePullPolicy: Always
  #  warnOnly: true
  #  warnOnlyTimeoutSeconds: 3

Apply the manifest to your Kubernetes cluster with the following kubectl command:

kubectl apply -f secretsd-pis.yaml

Existing pods that match the label selector in your pod injector selector definition are not going to be touched: if you already have pods with the secretsd application running, they are not going to be restarted. However, every new pod that is going to match the label selector is going to get the Aporeto init container injected.

There are also a couple of options that can be passed through to the init container from the pod injector selector. They are commented out in the example above because they are usually not required. If you are in a more restricted environment and cannot just get your images from docker hub, you can push the init container to an alternative location and use the image property on the options field within the pod injector selector specifications to use this alternate image instead. The accompanying imagePullPolicy works like in any other container definition for a Kubernetes pod.

If you want to ensure that your Kubernetes pods have activated Aporeto processing units before your applications come up, but you also do not want to wait forever in the case the processing unit cannot get activated (e.g. because you might not be able to afford that the application is gone forever), you can enable the warnOnly option. The init container is going to have a grace period then of warnOnlyTimeoutSeconds that it is going to wait for the processing unit to be started before it will log a warning to the console. It is going to allow the startup of the pod to continue regardless. If you do not use the warnOnly option, the Aporeto init container is going to wait indefinitely until the corresponding processing unit has been activated. The warnOnly option essentially provides a “best effort” model for ensuring activation of processing units.

To demonstrate and explore what the injected init container is going to look like, let us create a pod which will match the label selector from the previously defined pod injector selector.

kubectl run secretsd --image=gcr.io/google-samples/hello-app:2.0 -l app=secretsd --generator=run-pod/v1

Now let us explore what happened to the running Kubernetes pod. Get the full manifest of the created Kubernetes pod by running:

kubectl get pod secretsd -o yaml

You will see output which is going to be similar to the following. Note that for clarity most of the output below has been removed.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    aporeto.io/inject: ""
    aporeto.io/inject-image: aporeto/aporeto-init-waitforpu:release-3.11.0
    aporeto.io/inject-image-pull-policy: IfNotPresent
    aporeto.io/inject-podinjectorselector-name: secretsd
    aporeto.io/inject-warn-only: "false"
    aporeto.io/inject-warn-only-timeout-sec: "3"
  labels:
    app: secretsd
  name: secretsd
spec:
  containers:
  - image: gcr.io/google-samples/hello-app:2.0
    imagePullPolicy: IfNotPresent
    name: secretsd
  initContainers:
  - image: aporeto/aporeto-init-waitforpu:release-3.11.0
    imagePullPolicy: IfNotPresent
    name: aporetoinit

You can observe that there is a whole new sections of annotations now on the pod. The annotations are essentially responsible for injecting the Aporeto container to begin with. If you do not want to work with the pod injector selector custom resources, you can work directly with the annotations inside of your Kubernetes pod templates to inject the init container. Check out the reference documentation on all the names and usage of these annotations. If you do not want anything to get injected into your pods at runtime, you can even make use of this feature by defining the init container directly inside of your pod templates.

IMPORTANT

The Aporeto init container is always going to be injected as the first init container into the pod. If you already have other init containers defined in the pod, they are going to be executed after a successful run of the Aporeto init container.

Refer to Kubernetes secrets in custom resources

Sometimes when using custom resources, you would actually like to retrieve values for the custom resources from other ConfigMap resources or from Kubernetes secrets. Let us review a slightly modified version of the service mapping example:

apiVersion: k8s.aporeto.io/v1beta1
kind: ServiceMapping
metadata:
  name:  myapp
  namespace: default
spec:
  mapping:
    portName: http
    publicPortName: https
    serviceName: myapp
  options:
    discoverEndpoints: true
    discoverNodes: false
  template:
    metadata:
      labels:
        app: myapp
    spec:
      type: HTTP
      exposedAPIs:
        - - api=kubernetes
      publicApplicationPort: 443
      exposedServiceIsTLS: false
      authorizationType: OIDC
      OIDCClientSecret: |-
        my revealing secret
      TLSType: External
      TLSCertificateKey: |-
        -----BEGIN PRIVATE KEY-----
        ...
        -----END PRIVATE KEY-----
      TLSCertificate: |-
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      # more fields for a real configuration of a
      # working OIDC servime mapping are required

In the above example, let us assume that we want to get exposedAPIs, publicApplicationPort and exposedServiceIsTLS from a Kubernetes ConfigMap resource; and that we want to get OIDCClientSecret, TLSCertificateKey and TLSCertificate from a Kubernetes secret.

To solve this problem the Aporeto operator makes use of the annotations property that is part of every Aporeto object. Its dedicated purpose is to help third party integrations or other extended use cases that go beyond the original purpose of the object (like here).

In the following example below we are going to take values from a ConfigMap resource as well as two different Kubernetes secrets. The values are going to be updated in Aporeto’s control plane API only, and will not be visible in the Kubernetes custom resource.

Before we change the definition of the service mapping, let us create dedicated Kubernetes ConfigMap resources and secrets.

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: myapp
  namespace: default
data:
  exposedAPIs: |
    - - api=kubernetes
  publicApplicationPort: "443"
  exposedServiceIsTLS: "false"
---
kind: Secret
apiVersion: v1
metadata:
  name: oidc
  namespace: default
stringData:
  clientSecret: |-
    my revealing secret
---
kind: Secret
apiVersion: v1
metadata:
  name: server-tls
  namespace: default
stringData:
  tls.key: |-
    -----BEGIN PRIVATE KEY-----
    ...
    -----END PRIVATE KEY-----
  tls.crt: |-
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

Now finally, let us adjust the service mapping to retrieve the values for the above mentioned fields from the Kubernetes ConfigMap resources and secrets directly.

apiVersion: k8s.aporeto.io/v1beta1
kind: ServiceMapping
metadata:
  name:  myapp
  namespace: default
spec:
  mapping:
    portName: http
    publicPortName: https
    serviceName: myapp
  options:
    discoverEndpoints: true
    discoverNodes: false
  template:
    metadata:
      labels:
        app: myapp
    spec:
      annotations:
        exposedAPIs:
          - type=KubernetesConfigMap
          - name=myapp
          - namespace=default
          - field=exposedAPIs
        publicApplicationPort:
          - type=KubernetesConfigMap
          - name=myapp
          - namespace=default
          - field=publicApplicationPort
        exposedServiceIsTLS:
          - type=KubernetesConfigMap
          - name=myapp
          - namespace=default
          - field=exposedServiceIsTLS
        OIDCClientSecret:
          - type=KubernetesSecret
          - name=oidc
          - namespace=default
          - field=clientSecret
        TLSCertificateKey:
          - type=KubernetesSecret
          - name=server-tls
          - namespace=default
          - field=tls.key
        TLSCertificate:
          - type=KubernetesSecret
          - name=server-tls
          - namespace=default
          - field=tls.crt
      type: HTTP
      authorizationType: OIDC
      TLSType: External
      # more fields for a real configuration of a
      # working OIDC servime mapping are required

IMPORTANT

Aporeto custom resources are currently not receiving updates from ConfigMap resources or secrets. If you are changing values inside of a ConfigMap resource or a secret that you are using in an Aporeto custom resource, you have to trigger a change to that custom resource yourself. For example, add or a remove an annotation or a label so that the Aporeto operator is going to reconcile the state of the object and is going to update Aporeto’s control plane API with the new values.

Last but not least, there are RBAC requirements for the Aporeto operator. If you are just getting started and are comfortable to give the Aporeto operator access to all secrets and ConfigMap resources, you can use the clusterwideSecretsAccess and clusterwideConfigMapsAccess options of the operator Helm charts. However, for every real production installation you should not hand out permissions lightly and narrow down access for the service account of the Aporeto operator as much as possible. You can either write your own RBAC policies using the aporeto-operator service account of the aporeto-operator namespace (assuming installation defaults), or you can specify specific access requirements at installation time of the Aporeto operator with the secretsAccess and configmapsAccess options of the Helm charts.