What is an Operator?

An Operator is a method of packaging, deploying and managing a Kubernetes application. A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl/oc tooling. You can think of Operators as the runtime that manages this type of application on Kubernetes.

Operators fall into a few different categories, based on the type of applications they run:

1. Service Operators (MongoDB, Spark, Nginx)

2. Platform Operators (aggregated logging, security scanning, namespace management)

3. Resource Operators (CNI operators, Hardware management operators, Telco/Cellular radios)

4. Full Solution (combination of above operators)

Conceptually, an Operator takes human operational knowledge and encodes it into software that is more easily packaged and shared with consumers. Think of an Operator as an extension of the software vendor’s engineering team that watches over a Kubernetes environment and uses its current state to make decisions in milliseconds. Advanced Operators are designed to handle upgrades seamlessly, react to failures automatically, and not take shortcuts, like skipping a software backup process to save time.

Advantages of Operators

• Pod startup ordering: Kubernetes does not provide guarantees of container- or pod-startup ordering without sophisticated use of concepts like init containers.

• Familiar experience on any infrastructure: Your customers have the same deployment and day 2 experience regardless of infrastructure provider - anywhere Kubernetes runs, your Operator should run.

• Provider maintained: De-couple your OSR from the release cadence of Kubernetes/OpenShift itself, and deliver features, bug- and security-fixes to your customers as they become available.

• Ease of installation: We have heard from vendors providing out-of-tree kernel modules that installations (DKMS or manual rpmbuild / similar) end up in a poor experience (even for sophisticated customers). Operators provide a one-click method for installation and operations of your hardware and software stack.

• Standardized operational model and reduce support burden: Operators provide a way for you to standardize deployments, upgrades and configuration to only what you support. This avoids “snowflake” deployments and ensures smooth upgrades.

• Upgrades: Operators allow you to encode best-practices (e.g. straight from your documentation), reducing configuration drift in customer deployments, increasing deployment success ratio and thereby reducing support costs.

• Network adapter integrations “White list” of supported SR-IOV card/driver combinations for the device and CNI Multus plugins

This guide will cover how to build and test your operator. Before you go through the guide you will need to setup your system and environments.

You will need 2 environments.

1. Local Build Environment

2. 1. NOTE: We recommend using Fedora 31+, you can either install these packages locally or use a Fedora VM to build your operator. In this section we will only go over how to install these packages in Fedora, but you can use another OS as well.

The following are prerequisites for your Local Build Environment:

The Dockerfile can be found in the main directory of your operator project. For Certified Operator Images Dockerfile requirements are as follows:

1. You must configure the required labels (name, maintainer, vendor, version, release, summary)

Below is an example Dockerfile for a Helm Operator which includes the aforementioned requirements:

# Build the manager binary
FROM registry.redhat.io/openshift4/ose-helm-operator:v4.7

### Required OpenShift Labels
LABEL name="Wordpress Operator" \
      vendor="Bitnami" \
      version="v0.0.1" \
      release="1" \
      summary="This is an example of a wordpress helm operator." \
      description="This operator will deploy wordpress to the cluster."

# Required Licenses
COPY licenses /licenses

ENV HOME=/opt/helm
COPY watches.yaml ${HOME}/watches.yaml
COPY helm-charts  ${HOME}/helm-charts
WORKDIR ${HOME}

A few things to note about the Dockerfile above:

• The default FROM line produced by the SDK needs to be replaced with the line listed above.

• This Dockerfile contains all of the required labels. These labels must be manually added (name, vendor, version, release, summary, and description).

• This Dockerfile also references a licenses/ directory, which needs to be manually added to the root of the project. This directory must include the software license(s) of your project.

Your project directory structure should look similar to the hierarchy below. Note the location of the licenses directory.

.
├── charts
│   └── mariadb
│       ├── Chart.yaml
│       ├── files
│       │   └── docker-entrypoint-initdb.d
│       │       └── README.md
│       ├── OWNERS
│       ├── README.md
│       ├── templates
│       │   ├── _helpers.tpl
│       │   ├── initialization-configmap.yaml
│       │   ├── master-configmap.yaml
│       │   ├── master-pdb.yaml
│       │   ├── master-statefulset.yaml
│       │   ├── master-svc.yaml
│       │   ├── NOTES.txt
│       │   ├── rolebinding.yaml
│       │   ├── role.yaml
│       │   ├── secrets.yaml
│       │   ├── serviceaccount.yaml
│       │   ├── servicemonitor.yaml
│       │   ├── slave-configmap.yaml
│       │   ├── slave-pdb.yaml
│       │   ├── slave-statefulset.yaml
│       │   ├── slave-svc.yaml
│       │   ├── test-runner.yaml
│       │   └── tests.yaml
│       ├── values-production.yaml
│       ├── values.schema.json
│       └── values.yaml
├── Chart.yaml
├── config
│   ├── crd
│   │   ├── bases
│   │   │   └── example.com_wordpresses.yaml
│   │   └── kustomization.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   └── manager_auth_proxy_patch.yaml
│   ├── manager
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── role.yaml
│   │   ├── wordpress_editor_role.yaml
│   │   └── wordpress_viewer_role.yaml
│   ├── samples
│   │   ├── example_v1alpha1_wordpress.yaml
│   │   └── kustomization.yaml
│   └── scorecard
│       ├── bases
│       │   └── config.yaml
│       ├── kustomization.yaml
│       └── patches
│           ├── basic.config.yaml
│           └── olm.config.yaml
├── Dockerfile
├── helm-charts
│   └── wordpress
│       ├── charts
│       ├── Chart.yaml
│       ├── templates
│       │   ├── deployment.yaml
│       │   ├── _helpers.tpl
│       │   ├── hpa.yaml
│       │   ├── ingress.yaml
│       │   ├── NOTES.txt
│       │   ├── serviceaccount.yaml
│       │   ├── service.yaml
│       │   └── tests
│       │       └── test-connection.yaml
│       └── values.yaml
├── licenses
│   └── license.txt
├── Makefile
├── PROJECT
├── README.md
├── requirements.lock
├── requirements.yaml
├── templates
│   ├── deployment.yaml
│   ├── externaldb-secrets.yaml
│   ├── _helpers.tpl
│   ├── ingress.yaml
│   ├── NOTES.txt
│   ├── pvc.yaml
│   ├── secrets.yaml
│   ├── servicemonitor.yaml
│   ├── svc.yaml
│   ├── tests
│   │   └── test-mariadb-connection.yaml
│   └── tls-secrets.yaml
├── values.schema.json
├── values.yaml
└── watches.yaml

You can think of the Operator Capability Level model as a visual sizing guide as to which toolkit you should use to build your operator. Once you've decided which phases (or feature sets) are being targeted, this model helps you decide which framework(s) you can use to achieve that goal.

Use Cases ideal for operator development

• stateful applications (such as databases)

• clustered or high-availability applications (clustered databases, key-value stores such as etcd, in-memory cache clusters such as Redis)

• multiservice applications (an application which needs dependent services to come online first)

• microservices (numerous small components that together make up a cohesive product or app)

Use cases less ideal for operator development

• stateless apps (most web apps or front-ends)

• infrastructure/host agents (monitoring agents or log forwarders)

Operators are intended for use cases where there are more complex or stateful applications. Kubernetes already handles stateless applications (with pods and deployments) and host agents (Daemonsets) rather well. With that being said, those "less than ideal" use cases can still benefit from an operator. The ideal toolkit for building an operator for stateless applications and host agents is either Helm or Ansible, both of which allow for a quick, simple development cycle using the operator-sdk. This provides you with the ability to leverage the marketplace that's built into OpenShift 4.x and provide end users with a one click install experience on OpenShift.

By default, config/default/manager_auth_proxy_patch.yaml uses an uncertified kubebuilder image. In order to pass certification, it'll need to be changed to the certified image.

Edit the controller manager

vi config/default/manager_auth_proxy_patch.yaml

Replace the image with the certified version, registry.redhat.io/openshift4/ose-kube-rbac-proxy

# This patch inject a sidecar container which is a HTTP proxy for the
# controller manager, it performs RBAC authorization against the Kubernetes API using SubjectAccessReviews.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller-manager
  namespace: system
spec:
  template:
    spec:
      containers:
      - name: kube-rbac-proxy
        #use this certified image
        image: registry.redhat.io/openshift4/ose-kube-rbac-proxy:latest

In this section we will be going over the changes needed to build a Certified Helm Operator. This section will also cover the beginning of testing the operator.

Why Use Ansible in an Operator?

Ansible is useful for DevOps and infrastructure teams who are already familiar with the language. It also uses a similar pattern as Kubernetes in that playbooks/roles are written in declarative YAML. Jinja2 templating in Ansible is also easy to use, and is nearly identical in syntax to the templating engine used in Helm. This means that Helm templates can easily be converted or migrated into Ansible tasks, with only minor editing required.

Ansible Operator at a Glance

An Ansible Operator is a Golang operator that uses Ansible for the reconciliation logic. This logic (contained in roles or playbooks) gets mapped to a particular Custom Resource by a watches.yaml file inside the operator image. The operator go code checks watches.yaml to see which playbook or roles to execute, and then launches the playbook/role using Ansible Runner. This is illustrated and contrasted with golang operators in the following diagram (key points are in orange):

The Ansible Operator's point of user interaction is through a Kubernetes Custom Resource. A user creates a Custom Resource yaml (or json) file and passes this to Operator using the command line oc (OpenShift) or kubectl (upstream K8s) commands, or the web console (Operator Lifecycle Manager). The variables defined in the Custom Resource object get passed to Ansible Runner as --extra-vars, and Ansible Runner invokes the playbook or role containing your customized logic using these variables.

The next section outlines the basic steps for updating an Ansible Operator with certification changes. For more information on how to create an Ansible Role from a Helm Chart you can visit:

The first place we will need to make changes is within the values.yaml file referenced by the Helm Chart. Any image that is referenced with multiple image variables needs to be commented out and a new single image variable needs to be created.

vi values.yaml

###omitted for brevity#

##​image: 
##  registry: docker.io
##  repository: bitnami/wordpress
##  tag: 5.3.2-debian-10-r0
  image: docker.io/bitnami/wordpress:5.3.2-debian-10-r0  
## Specify a imagePullPolicy  
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'  
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images  
  pullPolicy: IfNotPresent​

###omitted for brevity###​

As you can see we condensed the image from being listed as 3 sub-variables to one single image variable. This must be done for all images listed within the values.yaml file. In this example this is the only image listed, therefore the only image that needs to have been changed. Next, we will need to identify any references to the image we modified in the templates/ directory.

cd templates/

grep "image:" *

This will list out all of the templates files that reference the image. Using wordpress as an example you can see that the deployment.yaml is the only file we need to change. Within this file we now need to list the image as shown below.

### omitted for brevity ###​

     containers:
       - name: wordpress        
         image: {{ .Values.image.image }}        
         pullPolicy: {{ .Values.image.pullPolicy }}​
         
### omitted for brevity ###

Next we need to edit the watches.yaml file to add the overrideValues: field

vi watches.yaml

---
- version: v1alpha1  
  group: apigw.wordpress.com  
  kind: Wordpress  
  chart: helm-charts/wordpress  
  overrideValues:    
    image.image: $RELATED_IMAGE_WORDPRESS

Finally before we build the ClusterServiceVersion we need to edit the manager.yaml.

vi config/manager/manager.yaml

apiVersion: v1
kind: Namespace
metadata:
  labels:
    control-plane: controller-manager
  name: system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: wordpress-operator
  template:
    metadata:
      labels:
        name: wordpress-operator
    spec:
      serviceAccountName: wordpress-operator
      containers:
        - name: wordpress-operator
          # Replace this with the built image name
          image: PUBLISHED IMAGE IN CONNECT (registry.redhat.connect/etc)
          imagePullPolicy: Always
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "wordpress-operator"
            - name: RELATED_IMAGE_WORDPRESS
              value: docker.io/bitnami/wordpress:5.3.2-debian-10-10

Adding the image location and the last environment variable to the manager.yaml will allow you to build your CSV with these changes automatically being generated. If you have already built the CSV you will need to adjust it as well.

Setup Quay.io account (for testing only)

Before we build our operator image we will want to setup an account on Quay.io. We will need to push this image to quay.io so that we can test the Metadata Files discussed in the next section. To setup an account on quay.io:

2. Click on the SIGN IN button located on the upper right side

3. You can either Sign In with your GitHub account or click Create Account

4. Once logged into Quay, click CREATE REPOSITORY to create a place to push your operator image for testing (example name: wordpress-operator)

Build and Push Operator Image

Now that your project is setup with Labels and Licenses, and your quay.io account is setup we can build the operator image. In this example, replace rhc4tp with your quay.io account username:

sudo make docker-build docker-push IMG=quay.io/rhc4tp/wordpress-operator:v0.0.1

Now that you pushed your image to quay.io you need to edit the config/manager/manager.yaml file and change the image field to reflect your image repository. (Note: line 19)

apiVersion: v1
kind: Namespace
metadata:
  labels:
    control-plane: controller-manager
  name: system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: wordpress-operator
  template:
    metadata:
      labels:
        name: wordpress-operator
    spec:
      serviceAccountName: wordpress-operator
      containers:
        - name: wordpress-operator
          # Replace this with the built image name
          image: quay.io/rhc4tp/wordpress-operator:v0.0.1
          imagePullPolicy: Always
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "wordpress-operator"
            - name: RELATED_IMAGE_WORDPRESS
              value: docker.io/bitnami/wordpress:5.3.2-debian-10-10

You must create metadata for your operator as part of the build process. These metadata files are the packaging format used by the Operator Lifecycle Manager (OLM) to deploy your operator onto OpenShift (OLM comes pre-installed in OpenShift 4.x). A basic metadata bundle for the operator contains the following YAML files listed below. This metadata will be hosted inside of its own container image called the bundle image.

Custom Resource Definition

This file gets used to register the Custom Resource(s) being used by the operator with the Kubernetes cluster. There can be more than one CRD included in a bundle, depending on how many Custom Resources get managed by the operator. There is usually only one CRD to start, but there can be more as features get added to the operator over time.

The CRD(s) that you will need to add to your metadata bundle are located within the config/crd/bases directory of the operator project.

Cluster Service Version

The ClusterServiceVersion or CSV is the main packaging component of the operator. It contains all of the resource manifests (such as the operator deployment and any custom resource templates) and RBAC rules required to deploy the operator onto a Kubernetes cluster.

It's also used to populate user interfaces with information like your logo, product description, version and so forth.

There can be multiple CSV files in a bundle, each tied to a specific version of an operator image.

The Dockerfile can be found in the root directory of your operator. For Certified Operator Image Dockerfile requirements are as follows:

1. You must configure the required labels (name, maintainer, vendor, version, release, summary)

Below is an example Dockerfile for a Ansible Operator which includes the aforementioned requirements:

A few things to note about the Dockerfile above:

• The default FROM line produced by the SDK needs to be replaced with the line listed above.

• This Dockerfile contains all of the required labels. These labels must be manually added (name, vendor, version, release, summary, and description).

• If you are planning to use a playbook, that file will also need to be copied.

• Lastly, this Dockerfile also references a licenses/ directory, which needs to be manually added to the root of the project. This directory must include the software license(s) of your project.

Your project directory structure should look similar to the hierarchy below. Note the location of the licenses directory.

The com.redhat.openshift.versions field, part of the metadata in the operator bundle, is used to determine whether an operator is included in the certified catalog for a given OpenShift version. You must use it to indicate the version(s) of OpenShift supported by your operator.

The value is set through a LABEL in the bundle.Dockerfile. Note that the letter 'v' must be used before the version, and spaces are not allowed.

The syntax is as follows:

• A single version indicates that the operator is supported on that version of OpenShift or later. The operator will be automatically added to the certified catalog for all subsequent OpenShift releases.

• A single version preceded by '=' indicates that the operator is supported ONLY on that version of OpenShift. For example, using "=v4.6" will add the operator to the certified catalog for OpenShift 4.6, but not for later OpenShift releases.

• A range can be used to indicate support only for OpenShift versions within that range. For example, using "v4.5-v4.7" will add the operator to the certified catalog for OpenShift 4.5, 4.6 and 4.7 but not for OpenShift 4.8.

The following table provides some examples of whether an operator is included or not in the catalog for different OpenShift versions, depending on the value of com.redhat.openshift.versions.

Some testing can be preformed prior to deployment to verify that the Metadata Bundle files are properly configured and formatted.

Validating the CSV

Verifying your Bundle with Operator-SDK

To check your metadata files, run the following command on the root directory of your project:

This will check your metadata files to see if there are any required fields missing. It will produce an error message for each missing component. You can also run the command below for a more detail validation.

Previewing your CSV on OperatorHub.io

You can copy and paste your CSV contents into the "Enter your operator's CSV YAML:" section as shown below. Then click the blue submit button to see how your information will render.

Work with your project manager to make sure your icon and information display correctly in the preview. You can go in and edit the fields of your CSV, some of which we added earlier, to change how the information is displayed.

Here is a look at how the embedded OperatorHub on OpenShift displays the information of certified partners.

OpenShift Container Platform 4

OpenShift Container Platform 4 (OCP 4) is the enterprise-ready Kubernetes platform that’s ready for deployments on these infrastructure providers:

• Cloud: AWS, Azure, Google Cloud

• Virtualization: VMware vSphere, Red Hat OpenStack Platform

• Bare Metal

Requirements

Deploying to a public cloud provider requires an account with the provider, and will incur hourly use charges for the utilized infrastructure.

Deploying to virtual platforms and bare metal will require a paid OpenShift subscription or a partner NFR subscription for OpenShift, in addition to providing the platform and/or infrastructure for the installation.

Applications with high or specialized resource requirements are best suited for an OCP cluster. For testing application scaling or other operational tasks, an OCP cluster is recommended. Infrastructure scalability inherent in these infrastructure providers affords OpenShift this ability in an OCP deployment.

Installation

CodeReady Containers

CodeReady Containers provides a minimal OpenShift 4 cluster to developers, free of cost. It consists of a single OpenShift node running as a virtual machine for offline development and testing on a laptop or desktop.

Requirements

CodeReady Containers runs on Windows, Mac, and Linux and requires the following minimum system requirements:

• 4 virtual CPUs (vCPUs)

• 8 GB of memory

• 35 GB of storage space

As described previously, applications will be limited to the resources provided by the host. If you desire to run resource intensive applications in CodeReady Containers, you can increase the size of the VM (CPU & RAM) at runtime.

Not all OpenShift features are available within the CodeReady Containers deployment. The following limitations exist:

• Operators for machine-config and monitoring are disabled by default, so the corresponding functions of the web console are non-functioning.

• Due to this, there is currently no upgrade path to newer versions of OpenShift.

• External networking and ingress does not function identically to an OCP 4 cluster due to running in a local virtual machine.

• Operational tasks, such as rebooting the CodeReady Containers VM are disruptive to applications. Such high availability is not achievable with a single node cluster.

• Custom network operators cannot be installed because the installation is not user configurable.

• Each binary release of CodeReady Containers expires after 30 days. You will then get certificate errors and must update the local crc binary, and then reinstall the cluster.

With these limitations in mind, it is suggested that final testing of containers and operators developed in CodeReady Containers should occur in OpenShift Container Platform prior to certification.

Installation

The next step is to extract the crc binary from the downloaded archive somewhere into your $PATH:

The crc command should now be accessible from the command line. You can now install the cluster:

Finally, start the cluster VM:

Instead of breaking up the image into <registry>/<image>:<tag> or some variation, use a single image variable for all three parts.

If you change the role for this, make sure you also update the defaults/main.yaml and vars/main.yaml to reflect the new variable.

By default, config/default/manager_auth_proxy_patch.yaml uses an uncertified kubebuilder image. In order to pass certification, it'll need to be changed to the certified image.‌

Edit the controller manager

‌

Replace the image with the certified version, registry.redhat.io/openshift4/ose-kube-rbac-proxy

An Operator Bundle is built as a scratch (non-runnable) container image that contains operator manifests and specific metadata in designated directories inside the image. Then, it can be pushed and pulled from an OCI-compliant container registry. Ultimately, an operator bundle will be used by Operator Registry and OLM to install an operator in OLM-enabled clusters.

SDK projects are scaffolded with a Makefile containing the bundle recipe by default, which wraps generate kustomize manifests, generate bundle, and other related commands. First we want to run the following command from the root of your project:

This will prompt a series of questions asking you for information that will added into the generated files (specifically the CSV)

• Display name for the operator (required):

• Description for the operator (required):

• Provider's name for the operator (required):

• Any relevant URL for the provider name (optional):

• Comma-separated list of keywords for your operator (required):

• Comma-separated list of maintainers and their emails (e.g. 'name1:email1, name2:email2') (required):

By default make bundle will generate a CSV, copy CRDs, and generate metadata in the bundle format:

Bundle metadata in bundle/metadata/annotations.yaml contains information about a particular Operator version available in a registry. OLM uses this information to install specific Operator versions and resolve dependencies. In the bundle above you can see that within the manifests directory we have the cluster role, the CSV, and the CRD. You will also see the tests directory contains information for running the scorecard. We will get to that in a later section.

Setup Quay.io account

2. Click on the SIGN IN button located on the upper right side

3. You can either Sign In with your GitHub account or click Create Account

4. Once logged into Quay, click CREATE REPOSITORY to create a place to push your operator image for testing (example name: mongodb-operator)

Building and Pushing the Operator

Build the mongodb-operator image and push it to a registry:

Editing your Image and ImagePullPolicy

Now that you have built and pushed the image to quay.io it is necessary to edit config/manager/manager.yaml and replace the following fields:

• image - your new image on quay.io

• imagePullPolicy - Always

The operator-sdk uses CustomResourceDefinition v1 by default for all automatically generated CRD's. However, v1beta1 CRD's were required for operator certification as recently as Q2 CY21 and are being deprecated as per the above warning. Thus, if your operator was certified prior to this timeframe and you haven't yet switched, then you should do so as soon as possible to so that your operator will be listed in OpenShift 4.9.

Edit each of your CRs as follows:

Here's a sample CRD shown before and after conversion. The apiVersion is changed to v1, and the schema is now defined per CR version (in v1beta1, you could only define per-version schemas if they were different).

Before:

After:

Helm Operators

Update your Helm chart

Make sure your Helm chart only references the values file for images. Each image should be a single value (it can't be split into repository and tag, for example).

  containers:
  - name: {{ template "etcd.fullname" . }}
    image: "{{ .Values.image.image }}"
    imagePullPolicy: "{{ .Values.image.pullPolicy }}"

A great way to find all the parts of your helm chart that will need to be updated is to recursively search your project's template folder for "image:"

$ grep -r 'image:' ./helm-charts/etcd/templates/*
  ./helm-charts/etcd/templates/statefulset.yaml:        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

Here we see that the image is split into two values - repository and tag. This won't work because we need to have a single variable to override. Replace this line with a new, single variable:

$ grep -r 'image:' ./helm-charts/etcd/templates/*
  ./helm-charts/etcd/templates/statefulset.yaml:        image: "{{ .Values.image.image }}"

Don't forget to add this new value to your Values.yaml file!

Override the image variable

In the watches.yaml file, add a field for overrideValues. It should contain each variable from the values.yaml file that corresponds to an image, and should be set to the environment variable you want to use.

overrideValues:
  image.image: $RELATED_IMAGE_STATEFULSET

NOTE: the variable name MUST follow the pattern RELATED_IMAGE_. There is code looking for that string in your operator.

Ansible Operators

Update your Ansible role

Make sure your role is using environment variables for images instead of hard-coded or regular variables. If it's not, update it:

containers:
- name: mongodb
  image: "{{ 'dbImage' | quote }}"

Above you can see a reference to a image that's defined in the role's defaults/main.yaml file with an option to override it in values/main.yaml. Instead, use Ansible's lookup module to reference an environment variable and remove the variable from defaults/main.yaml to avoid confusion:

        containers:
        - name: mongodb
          image: "{{ lookup('env','RELATED_IMAGE_DB') | quote }}"

NOTE: Your environment variables need to follow the RELATED_IMAGE_ format, as there is code looking for this pattern.

Golang Operators

Make sure your code is using environment variables for any images your operator uses (any image except the operator itself).

func newPodsForCR(cr *noopv1alpha1.UBINoOp) []*corev1.Pod {
	ubiImg := os.Getenv("RELATED_IMAGE_UBI_MINIMAL")
	}

All Operators (Helm, Ansible or Golang)

Define the environment variables

In the CSV and operator.yaml files, declare the variable and set to a default value.

            spec:
              containers:
              - env:
                - name: WATCH_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.annotations['olm.targetNamespaces']
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name          
                - name: OPERATOR_NAME
                  value: ectd-helm
                  
              #This is the new environment variable
                - name: RELATED_IMAGE_STATEFULSET
                  value: k8s.gcr.io/etcd-amd64:3.2.26
              #The operator image itself doesn't change    
                image: quay.io/dhoover103/etcd-helm-operator:v0.0.1

Entitled Registry

Other things to watch for

If you're in the habit of using the "latest" tag for your images, make sure you specify it. Because of how the automation is written that picks up these images, we need a tag to be present.

Make sure you aren't overwriting the image in your CR (and by extension in the alm-examples field of the CSV). The best way to handle this is removing the field from the CR/alm-examples.

Editing the CSV

Below is a list of other changes you need to make to the CSV as the command will not automatically create these necessary bits.

Fields to add under metadata.annotations

• description - Short description of the operator

• containerImage - The full location (registry, repository, name and tag) of the operator image

• createdAt - A rough (to the day) timestamp of when the operator image was created

• support - Name of the supporting vendor (eg: ExampleCo)

• repository - URL of the operator's source code repository (this field is optional)

Fields to adjust under spec

The Operator-SDK will generate the following fields but you will need to adjust them for certification

• description - Long description of the operator's owned customresourcedefinitions in Markdown format. Usage instructions and relevant info for the user goes here

• icon.base64data - A base64 encoded PNG, JPEG or SVG image will need to be added

• icon.medatype - The corresponding MIME type of the image (eg: image/png)

Example CSV

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    alm-examples: |-
      [
        {
          "apiVersion": "example.com/v1alpha1",
          "kind": "Wordpress",
          "metadata            "name": "wordpress-sample"
          },
          "spec": {
            "affinity": {},
            "autoscaling": {
              "enabled": false,
        "maxReplicas": 100,
              "minReplicas": 1,
              "targetCPUUtilizationPercentage": 80
            },
            "fullnameOverride": "",
            "image": {
            "pullPolicy": "IfNotPresent",
              "repository": "nginx",
              "tag": ""
            },
            "imagePullSecrets": [],
            "ingress": {
        
      ]
    capabilities: Basic Install
    categories: "Database"
    description: A brief description about this Wordpress Operator 
    containerImage: quay.io/rhc4tp/wordpress-operator:v0.0.1
    createdAt: 2020-09-03T12:59:59Z
    support: Red Hat Connect Team
    repository: https://github.com/rch4tp/wordpress-operator
    operators.operatorframework.io/builder: operator-sdk-v1.0.0
    operators.operatorframework.io/project_layout: helm.sdk.operatorframework.io/v1
  name: wordpress.v0.0.1
  namespace: placeholder
spec:
  apiservicedefinitions: {}
  customresourcedefinitions:
    owned:
    - kind: Wordpress
      name: wordpresses.example.com
      version: v1alpha1
  description: Description ---NEEDS TO BE UPDATED---
  displayName: Wordpress-operator
  icon:
  - base64data: "iVBORw0KGgoAAAANSUhEUgAAASwAAAEsCAIAAAGBGCm0AAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAmFlJREFUeNpi/P//PwN9ARMD3QFdrZy56iCQZKRpwC7bejK3bfm7j18Z/gGtAqH/V2bTykqgTV9uH0qxYmFmYvj7j2HByT8hBswBc35+vzCbhRb22cV3TXF58lmUxWbiD0dV5lw7FqC3BDgZnx+eSJOA/fnrz8Ulhb//ogiyAb1mWmyqo0CT5MNvkIFmX93236aJkyH2AQGVA5ZZL+1QISeyyIE7//aum0GrTPL7z99aN3Q/1Gz+RcN8yaaf4arOjCa4a24hraxk1Ek9XMCBnpT+MLhaatHESibdtKMFHIwY4hMO/sZUTGnycU3tk/5z+3AeO9a8tvnyX+pY+fz1R//cKafP3F+bzdFgxojHkCBXIyxRQHxRoO5Tc+v6i025HCI8jP+I0AQsbrRjJpFpJTBpNHiyuaiRFvGu035+Oz+LnIBVtE47ks9BcgJmZPj++z859eWcluwlCexkxHfayl+bpuSQbKWkZZqmOOHAtOn8AWe/+gKm/jOYyTIZaMiSZiWLbvqaJOL8x4aURGf+4GRlzFrzK9mSZdbqQ8RauW7POUbt1EP5UJNYmBj6D/72m/0zaO7PyIU/zz7+x8vOyALTFzr/p4MarJD7D0obxj3fp4WC9LbM2Io9miEp1sQzHagOrdIBVugOU/78OD8dq85Lt56EFs249eDl4XxouQOskIHtDEgKZwAW5v/+21ipHl5Ujt3Kos5V/bN3HylHpMyslb8u7plJOGVqph4pAumymfDj/9XZxMQDNJPs27sXaB8wZQtwMFpP/OFub3JxTxpBzVzGWUD7mBgZrPp+XNnUSGxq/k8uuHzrSVBkOoNWin18F0kaGUdb6zS18vwWlPJ31+RcAs3wYmes4idWdgOlgGjv9GJkxUB0fFkHSvJZWuSEHMNoXOLB6XWT4GygNWimfX3/CkhCfalg7HLv9E4Ie+fEHGQHXtu/8sLWOceWtn1592JjS9SqSi9wMOTBFbx/dndLZ8KtI+vRfBzZs2dFuQec++TKUXYefkS+VLX03TejVMnUHcj++u65gXcqkPHn14+o3r0QBUD2jy8fgIyw9m1wU1aUuUMU+JQvWFsfrGYTiGbrvz/Qts/2vvT3T++Ed26HWvnn109RRR0g//axTe+f3g1qXPsTbPrNw+u0naOgRQYblirz398/cLaeewKWsokJGoqeRTMhQQJ0IkjozcOrEInTaye+uHUGyGDnEfj44gHQviOLmyFSQE9gmmibgChxTq+dgCa7pjYgsns39gLv8Pz60LbNQAY8GMFeXG8WWmgdUwNJnECpr+9eMLGwQkt8MENW1+bq3mUXt83l5BM2CwE1kZmYWFZWeDEyMkqoGoU0b4ApZgMKSmmYQswfGaUPQADR25dMwzZIV2w/TfPwBLbrbz8ElW0i3Ixvvv4X5OOilX3ARtqZUo5ff4CNHgY2Zob6Hb8aPdg+amYx0ciyIwUc6y/+tZoAatPuuP4v0YzVeeIPF0tN6vuPUSP1SDFKWQhs7fnP+fn65Czqj0Q4JHQfK+FA7oIBLfOay/jh5CyajHz8eH3n3382ZBFrWCOW+vYBm8VoH"
    mediatype: "img/png"
  install:
    spec:
      clusterPermissions:
      - rules:
        - apiGroups:
          - ""
          resources:
          - namespaces
          verbs:
          - get
        - apiGroups:
          - ""
          resources:
          - secrets
          verbs:
          - '*'
        - apiGroups:
          - ""
          resources:
          - events
          verbs:
          - create
        - apiGroups:
          - example.com
          resources:
          - wordpresses
          - wordpresses/status
          verbs:
          - create
          - delete
          - get
          - list
          - patch
          - update
          - watch
        - apiGroups:
          - ""
          resources:
          - pods
          - services
          - services/finalizers
          - endpoints
          - persistentvolumeclaims
          - events
          - configmaps
          - secrets
          verbs:
          - create
          - delete
          - get
          - list
          - patch
          - update
          - watch
        - apiGroups:
          - apps
          resources:
          - deployments
          - daemonsets
          - replicasets
          - statefulsets
          verbs:
          - create
          - delete
          - get
          - list
          - patch
          - update
          - watch
        - apiGroups:
          - authentication.k8s.io
          resources:
          - tokenreviews
          verbs:
          - create
        - apiGroups:
          - authorization.k8s.io
          resources:
          - subjectaccessreviews
          verbs:
          - create
        serviceAccountName: default
      deployments:
      - name: wordpress-controller-manager
        spec:
          replicas: 1
          selector:
            matchLabels:
              control-plane: controller-manager
          strategy: {}
          template:
            metadata:
              labels:
                control-plane: controller-manager
            spec:
              containers:
              - args:
                - --secure-listen-address=0.0.0.0:8443
                - --upstream=http://127.0.0.1:8080/
                - --logtostderr=true
                - --v=10
                image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
                name: kube-rbac-proxy
              - name: RELATED_IMAGE_WORDPRESS
                value: docker.io/bitnami/wordpress:5.3.2-debian-10-10
                ports:
                - containerPort: 8443
                  name: https
                resources: {}
              - args:
                - --metrics-addr=127.0.0.1:8080
                - --enable-leader-election
                - --leader-election-id=wordpress
                image: controller:latest
                name: manager
                resources:
                  limits:
                    cpu: 100m
                    memory: 90Mi
                  requests:
                    cpu: 100m
                    memory: 60Mi
              terminationGracePeriodSeconds: 10
      permissions:
      - rules:
        - apiGroups:
          - ""
          resources:
          - configmaps
          verbs:
          - get
          - list
          - watch
          - create
          - update
          - patch
          - delete
        - apiGroups:
          - ""
          resources:
          - events
          verbs:
          - create
          - patch
        serviceAccountName: default
    strategy: deployment
  installModes:
  - supported: true
    type: OwnNamespace
  - supported: true
    type: SingleNamespace
  - supported: false
    type: MultiNamespace
  - supported: true
    type: AllNamespaces
  keywords:
  - cool
  - fun
  - easy
  links:
  - name: Wordpress
    url: https://wordpress.domain
  maintainers:
  - email: nhartman@redhat.com
    name: nhartman
  maturity: alpha
  provider:
    name: Provider Name
  version: 0.0.1

For Ansible or Helm based operators, this is done simply by setting the image location in the K8s manifest templates contained within your Ansible playbook/role or Helm chart to RHCC. For golang operators, the Go struct(s) representing the Kubernetes manifest(s) of your application must have the image source set to RHCC. In the following examples, replace the example image source string with that of your published application image.

Ansible

To the RHCC image source in Ansible, we'll use an example tasks file from a role. Let's say this file is found at roles/example/tasks/main.yaml within the operator project, and contains the following:

You can see where the image is set on the next to last line in the yaml snippet above.

Helm

In this helm example, lets assume the following deployment template exists at helm_charts/example/templates/example-deployment.yaml:

The image is shown in the next to last line in the above gotpl snippet.

Go

Below example in Golang (snippet of pkg/apis/example/v1alpha1/types.go from a hypothetical operator-sdk project):

In the above, the Spec field contents of the operator CR are defined, with the image field of the manifest being set by theExampleSpec.Image field in the above example. Note that no value is actually set in the struct, though the field is shown to be user-configurable by setting the spec.image field when creating the CR. Ideally, the image source would not be configurable, since we want to guarantee that the certified application images get pulled from RHCC. The default value for ExampleSpec.Image gets set in pkg/apis/example/v1alpha1/defaults.go as in the below example:

You can see the image source value in the next to last line of code in the golang snippet above.

When we ran the make bundle command earlier, part of what was generated was the metadata bundle image.

[nhartman@fedora wordpress]$ ls
bin                charts      Dockerfile   Makefile   requirements.lock  values.schema.json
bundle             Chart.yaml  helm-charts  PROJECT    requirements.yaml  values.yaml
bundle.Dockerfile  config      licenses     README.md  templates          watches.yaml

Here you see the bundle.Dockerfile in the root directory of your operator project. This is your Dockerfile for your metadata bundle image.

There are up to 3 labels you will need to add to the Dockerfile:

• LABEL com.redhat.openshift.versions
• LABEL com.redhat.delivery.operator.bundle=true

  • This just needs to be there

• LABEL com.redhat.deliver.backport=true

  • This is used to indicate support for OpenShift versions before 4.6. If you don't specify this flag, your operator won't be listed in 4.5 or earlier.

FROM scratch

LABEL operators.operatorframework.io.bundle.mediatype.v1=registry+v1
LABEL operators.operatorframework.io.bundle.manifests.v1=manifests/
LABEL operators.operatorframework.io.bundle.metadata.v1=metadata/
LABEL operators.operatorframework.io.bundle.package.v1=wordpress
LABEL operators.operatorframework.io.bundle.channels.v1=alpha
LABEL operators.operatorframework.io.bundle.channel.default.v1=
LABEL operators.operatorframework.io.metrics.builder=operator-sdk-v1.0.0
LABEL operators.operatorframework.io.metrics.mediatype.v1=metrics+v1
LABEL operators.operatorframework.io.metrics.project_layout=helm.sdk.operatorframework.io/v1
LABEL operators.operatorframework.io.test.config.v1=tests/scorecard/
LABEL operators.operatorframework.io.test.mediatype.v1=scorecard+v1

#Add these labels
LABEL com.redhat.openshift.versions="v4.6"
LABEL com.redhat.delivery.operator.bundle=true
LABEL com.redhat.delivery.backport=true

COPY bundle/manifests /manifests/
COPY bundle/metadata /metadata/
COPY bundle/tests/scorecard /tests/scorecard/

Now lets build the bundle image from the bundle.Dockerfile. You can use an existing public container registry of your choice, though we recommend using quay.io. When you create a new repository make sure its publicly available.

The next step is to build the bundle-dockerfile locally and push it to quay.io for testing which we will cover in the Deploying onto OpenShift section

podman build -t quay.io/<namespace>/wordpress-operator:v0.0.1 -f bundle.Dockerfile

podman push quay.io/<namespace>/wordpress-operator:v0.0.1

Adding an SCC to the Operator Metadata

SCC's must be applied to the service account which will run the application/operand pods that get managed by the operator. This is done by editing the CSV yaml file from the metadata bundle of your community operator.

Below is an example SCC applied to a named service account in a hypothetical CSV yaml file:

Managing SCCs for Multiple Service Accounts

It's worth noting that the clusterPermissions field is an array, so you can list multiple service accounts with a corresponding SCC (or SCCs) applied to each service account. See the below example:

Before getting started

Creating an index image

You can create an index image using the opm CLI.

1. Start a new index:

   Copy

   $ opm index add \
       --bundles quay.io/<namespace>/wordpress-operator:v0.0.1 \
       --tag quay.io/<namespace>/wordpress-operator-index:latest 

2. Push the index image to a registry:

   Copy

   $ podman push quay.io/<namespace>/wordpress-operator-index:latest

Create the CatalogSource

1. In your OpenShift environment create a catalog source in the openshift-marketplace namespace.

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: my-test-operators
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/<namespace>/wordpress-operator-index:latest
  displayName: Test Operators
  publisher: Red Hat Partner
  updateStrategy:
    registryPoll:
      interval: 5m

$ oc create -f test-operator-catalogsource.yaml 

2. You can verify the catalogsource has been created by running the following command:

$ oc -n openshift-marketplace get catalogsource

You should see the following output:

NAME                  DISPLAY               TYPE   PUBLISHER   AGE
certified-operators   Certified Operators   grpc   Red Hat     12d
community-operators   Community Operators   grpc   Red Hat     12d
my-test-operators     Test Operators        grpc   Red Hat     30s
redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     12d
redhat-operators      Red Hat Operators     grpc   Red Hat     12d

3. Once you have confirmed the catalogsource exists, run the following command to verify it's associated pod is successfully running. This verifies that the index image has been successfully pulled down from the repository.

$ oc -n openshift-marketplace get pods

You should see the following output:

NAME                                    READY   STATUS    RESTARTS   AGE
certified-operators-59b6bc7fbc-rzzwq    1/1     Running   0          12d
community-operators-5f89b4787d-ljvfb    1/1     Running   0          12d
marketplace-operator-5c84994668-927b6   1/1     Running   0          12d
my-test-operators-##########-#####      1/1     Running   0          12d
redhat-marketplace-86cc645bb6-8crdl     1/1     Running   0          12d
redhat-operators-5877478c4f-4ffs2       1/1     Running   0          12d

4. Verify that the operator package has been successfully added:

$ oc get packagemanifests | grep “Test Operators”

Create OperatorGroup

1. Create a namespace for your project

 $ oc new-project test-operator

2. Create the OperatorGroup

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: my-group
  namespace: test-operator
spec:
  targetNamespaces:
    - test-operator

$ oc create -f test-operatorgroup.yaml

3. Verify you have a working operatorgroup within the namespace you created:

$ oc get og

You should see the following output:

NAMESPACE                              NAME                           AGE
test-operator                          my-group                       30s 

Create a Subscription

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: test-subscription
spec:
  channel: alpha
  installPlanApproval: Automatic
  name: wordpress-operator
  source: my-test-operators
  sourceNamespace: openshift-marketplace

$ oc create -f test-subscription.yaml

Verify the subscription is created within your namespace:

$ oc get sub -n test-operator

You should see the following output:

NAMESPACE             NAME                              PACKAGE                           SOURCE                CHANNEL
test-operator         test-subscription                 wordpress-operator                test-operators        stable

The creation of the subscription should trigger the creation of the InstallPlan and CSV.

$ oc get installplan -n test-operator

$ oc get csv -n test-operator

You should see something similar to this output:

NAMESPACE                                          NAME                             DISPLAY                                 VERSION   REPLACES   PHASE
test-operator                                      wordpress-operator.v0.0.1        Wordpress Operator                      0.0.1                Succeeded

Verify your operator is running in the namespace:

$ oc get pods -n test-operator

Updating an existing index image

You can update an existing index image with a new operator bundle version using the opm CLI.

1. Update the existing index:

   Copy

   $ opm index add \
       --bundles quay.io/<namespace>/wordpress-operator:v0.0.2 \
       --from-index quay.io/<namespace>/wordpress-operator-index:latest \
       --tag quay.io/<namespace>/wordpress-operator-index:latest 

2. Push the updated index image:

   Copy

   $ podman push quay.io/<namespace>/wordpress-operator-index:latest

3. After OLM polls the index image at its specified interval, we should eventually see a new installplan for our new operator bundle version

   Copy

   $ oc get installplan -n test-operator

Run the Scorecard test

The Operator Scorecard is a testing utility included in the operator-sdk binary that guides users towards operator best practices by checking the correctness of their operators and CSVs. With your operator deployed via OLM, you can run the operator-sdk scorecard utility to create a CR which will trigger the operator and monitor what occurs. In order to certify you must pass the first two Basic Tests: Spec Block Exists, and Status Block Exists. Passing the third basic test (Writing into CRs has an effect) requires adding the scorecard-proxy container to the operator deployment, which is not desired in a production operator and is therefore not required for certification.

The operator certification pipeline, whether using the local test or hosted offering, performs a series of tests all of which must pass before you can successfully publish your operator.

Each of the tests performed by the certification pipeline is outlined below.

Operator Metadata Linting

In order to pass this test both of the commands below need to be run and yield no errors.

operator-sdk bundle validate ./bundle

Operator OLM Deployment

In order to pass this test, the operator must deploy successfully from the Operator Lifecycle Manager. Keep in mind that even when an operator is successfully deployed, there could still be non-fatal errors hidden in the operator's pod log, so prior testing on OpenShift is essential.

Operator Scorecard Tests

===== Test: operator-scorecard-tests =====

 Basic Tests:
	Spec Block Exists: 1/1
	Status Block Exists: 1/1
	Writing into CRs has an effect: 0/1
OLM Tests:
	Provided APIs have validation: 0/0
	Owned CRDs have resources listed: 2/2
	CRs have at least 1 example: 1/1
	Spec fields with descriptors: 8/8
	Status fields with descriptors: 1/1

Total Score: 74%

In order to certify you must pass the first two Basic Tests: Spec Block Exists, and Status Block Exists. Passing the third basic test (Writing into CRs has an effect) requires adding the scorecard-proxy container to the operator deployment, which is not desired in a production operator and is therefore not required for certification.

Passing the OLM Tests portion of the scorecard is not a requirement for certification, as currently the CR Spec and Status Descriptor fields don't populate the OpenShift UI.

Operator Image Source

Below is an example tree layout of a "broken" metadata bundle for an operator named example:

└── example
    ├── example.crd.yaml
    ├── example-operator.v0.1.0.clusterserviceversion.yaml
    ├── example-operator.v0.2.0.clusterserviceversion.yaml
    ├── example.package.yaml
    ├── some-extraneous-file.txt
    └── somedir
        └── some-other-file.yaml

You can see in the above that there are the usual operator metadata files (1 CRD, 2 CSVs and 1 package.yaml) with a couple of extra files and a directory thrown in. Before submitting an operator metadata bundle to Red Hat Connect, you need to archive only the relevant files that are part of the operator metadata (any CRDs, CSVs, and package.yaml files), omitting any extraneous files or directories. One caveat is that you can't just zip up the bundle directory itself and upload that, since the yaml files would be nested in a subdirectory within the archive.

Using the example bundle directory above, a valid metadata archive can be created using the linux zip utility as follows:

$ cd /path/to/bundle/example
$ zip example-metadata *.yaml

$ unzip -l example-metadata.zip
Archive:  example-metadata.zip
  inflating: example-operator-v0.1.0.clusterserviceversion.yaml  
  inflating: example-operator.v0.2.0.clusterserviceversion.yaml
  inflating: example.crd.yaml
  inflating: example.package.yaml

What are Security Context Constraints?

A Security Context Constraint is an OpenShift extension to the Kubernetes security model, that restricts which Security Contexts can be applied to a pod. This both prevents unprivileged (non cluster-admin) users from being able to apply Security Contexts that would allow a pod definition to forcefully override which UID (possibly as root / UID 0), GID, or fsGroup that a pod can run with, as well as which host volumes and node networking interfaces can be accessed by a pod.

To summarize, SCCs prevent unprivileged users from being able to run privileged containers on an OpenShift cluster, by restricting which user and/or group that a pod is able to run as on the worker nodes in the cluster. SCC's also prevent pods from gaining access to local resources on the nodes. SCCs restrict all of these things by default, unless explicitly overridden by a cluster-admin user or service account.

By default, OpenShift runs all pods with the restricted SCC. This causes pods to run with a randomized UID in a very high numerical range (100000+) and disregards the USER or UID specified in the container image Dockerfile (unless explicitly set to root, in which the pod will be prevented from running at all). The next section will describe alternative SCCs and when to use them.

Identifying Which SCC to Use

Given the nature of the restricted SCC, it's quite commonplace that a pod might need to run with a certain UID (as with many databases), or sometimes even as root, when the use case justifies doing so (as with host monitoring agents). This sort of classifies container workloads into three cases:

• Containers that are completely agnostic as to what UID or GID they run with (most operators)

• Containers that simply need to run as a certain UID (even root)

• Containers that require privileged, host-level access in order to run

The first case is handled easily by the default, restricted SCC. The second case requires the anyuid SCC, which will allow the container to run with the USER specified in the Dockerfile (even root). The third, and most special case is one that absolutely requires not only running as root, but access to node resources at the host level, and is handled by the privileged SCC. As such, the privileged SCC should be used with care and where it's justified, as mentioned previously. OpenShift does provide several other SCCs which allow more granular access, but SCC's aren't stackable, therefore you can't make a concoction of, say anyuid, hostaccess, and hostnetwork SCCs to grant each set of permissions to your pod. You can assign multiple SCC's to a service account, but the SCC with the least amount of privileges will take precedence over all the others. This makes the other SCC's less useful in common practice.

It's not common for an operator to require a non-default SCC, since they only make API calls back to the K8s environment where they're being run. However, it's quite common for an operand (pod or pods deployed and managed by an operator) to require a non-default SCC.

Apply an SCC to your Operator or Operand

If required, the last step is to apply either the anyuid or the privileged SCC to your operator or operand. This is done by modifying the set of RBAC rules being shipped with the operator. This will soon be managed directly by shipping a ClusterRole manifest in a containerized operator bundle, but today this requires modifying the ClusterServiceVersion from your metadata bundle to add a clusterPermissions field with the necessary RBAC rules.

For example, to add the anyuid SCC to a service account named my-operand , the following block would get added to the ClusterServiceVersion of your operator bundle:

You can see where we define the anyuid SCC under resourceNames, and we set the serviceAccountName that this ClusterRole will apply to as my-operand. To set the privileged SCC instead, set the field under resourceNames to privileged. Note that hyphens are required in the specified places, as these data types are enforced by the K8s API as arrays, which are denoted by a hyphen in the YAML syntax.

Certification of a Community Operator

The following changes must be made to the operator as well as the operator metadata (CSV) prior to submission (each item is covered in depth further below):

• The operator must consume the certified application image(s) aka operands from RHCC.

• The service account of your operand, (eg: the service account used to run your application's pods) must have the proper Security Context Constraint (SCC) applied in order to run containers either as root or a specific UID in OpenShift. This applies specifically to OperatorHub.io/Vanilla K8s operators

• Set the metadata.annotations.certified field to true

• The packageName of the metadata bundle in package.yamlmust be distinct from that of your community operator.

• The metadata bundle must be flattened and contain no sub-directories or extraneous files

Package name uniqueness (or lack of) can be a tripping point when certifying a community operator. Let's take a metadata bundle from a hypothetical community operator and examine the package.yaml file contents:

packageName: example
channels:
    - name: alpha
      currentCSV: example-operator-0.1.0

The typical convention is simply to add -certified as a suffix to the packageName like so:

packageName: example-certified
channels:
    - name: alpha
      currentCSV: example-operator-0.1.0

Now that your CSV and package yaml files are updated, you can proceed with bundling your metadata.

$ helmExport export <role> --helm-chart=<location of chart> --workspace=<output location> --generatreFilters=true --emitKeysSnakeCase=true

This will mostly give you an Ansible role that works the same as you Helm chart with the templates in /<your role>/templates/ . However, there are a few things that will need to be modified. Helm template syntax is a bit different at time from Jinja2 that Ansible uses. Some common things to watch out for are:

• The "toYaml" filter in Helm is "to_yaml" in jinja2.

• "toJSON" likewise needs to be changed to "to_json"

• The "b64enc" filter needs to be changed to "b64encode"

• Any varibles generated by your _helpers.tpl file are not available. You can either define them in the role's defaut/main.yml, hardcode them, or remove them from the templates.

• Values that come from the Chart directly instead of the Values.yaml file (ex. .Chart.Name) are also not available

Check your templates for these syntax errors and go ahead and fix them.

If you create new variables, remember to add them to the roles/<your role>/defaults/main.yml file (this is what replaces the Values.yaml file, and works exactly the same).

Helm templates tend to leave room for users to add their own annotations. Ansible considers this a security risk. While technically optional, you should find these open-ended annotations in your templates and remove them. Here's a sample of what they look like in a template:

metadata:
  name: {{ template "memcached.fullname" . }}
  labels:
    app: {{ template "memcached.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
  annotations:
{{ toYaml .Values.serviceAnnotations | indent 4 }}

In the above example, we'd take out lines 8 and 9 (line 8 would stay in if we had some annotations we wanted). The same bit of yaml in an Ansible role would look like this:

metadata:
  name: {{ name }} #The "name" variable was added to defaults/main.yml
  labels:
    app: {{ name }}
    chart: "{{ name }}-{{ version }}" #The "version" variable was also added to defaults/main.yml
    #The annotations field was removed entirely, since we don't have any for this object.

After making these changes, it is highly recommended to test the ansible role. It's impossible to account for all the filters with different syntax between Helm and Ansible, so you'll want to verify the role will run before moving on with your operator. To test the role, create a playbook.yml in the operator's root directory that looks like this:

- hosts: localhost
  roles:
  - <your role>

To run the playbook, run

$ ansible-playbook playbook.yml

while logged in to your testing cluster. If everything comes back fine, you're all set. Otherwise, you'll need to look at the error message to see what bit of syntax is causing the problem, and change it in your templates. To test each template one at a time, go to the roles/<your role>/tasks/main.yml and comment out the templates you aren't testing yet.

Network Operators are a special breed because they are required to be functional very early on during installation. OpenShift 4 has a facility for injecting custom objects at install time. In this case, we will use it to install a compliant network operator.

Network operators also need to consume and update certain special objects. This is how they inform cluster components of the current network status.

Requirements for OpenShift-compliant network operator

2. Publish the network status to downstream consumers. Cluster installation will fail to progress until this happens.

   1. Determine the currently-deployed ClusterNetwork, ServiceNetwork, and pod-to-pod MTU
3. Optional but recommended: React to network configuration changes

   1. Set up a watch on Network.config.openshift.io/v1 cluster

   2. Reconcile any changes to Spec to the running state of the network

   3. Publish the current state to the Status field

   4. Deployment strategy should be set to RollingUpdate.

Steps to install third party networking operator

Add network-operator to install payload.

Make the work directory

mkdir mycluster

Create install-config

openshift-install create install-config --dir=mycluster

1. Update the Network Type in the install-config

   a) Edit mycluster/install-config.yaml

   b) Replace OpenShiftSDN with the name of your network plugin. The value doesn’t matter. You should set it something meaningful to you and not to the “Cluster Network Operator” (CNO).

2. Create OpenShift manifests

openshift-install create manifests --dir=mycluster

Add your operator’s manifests to the installer

Create cluster:

openshift-install create cluster --dir=mycluster

This will deploy your cluster and apply the manifests of your CNI operator, leaving the Operator running but unmanaged.

Transition your operator to OLM ownership.

3. Verify that a ClusterServiceVersion object referring to your Operator is created

4. Verify that the resources now have owner references to OLM

OpenShift 4 now comes with an installer making it easier and simpler to setup an OpenShift Cluster. Before you run the installer you must first Configure AWS CLI locally and Configure your AWS account.

Configuring AWS CLI

1. Creating Access Key for an IAM user

   2. In the navigation pane, choose Users.

   3. Choose the name of the user whose access keys you want to create, and then choose the Security credentials tab.

   4. In the Access keys section, choose Create access key.

   5. To view the new access key pair, choose Show. You will not have access to the secret access key again after this dialog box closes. Your credentials will look something like this: Access key ID: AKIAIOSFODNN7EXAMPLE Secret access key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

   6. To download the key pair, choose Download .csv file. Store the keys in a secure location. You will not have access to the secret access key again after this dialog box closes.

   7. After you download the .csv file, choose Close. When you create an access key, the key pair is active by default, and you can use the pair right away.

2. Create the ~/.aws folder

   1. You should have 2 files in this folder, config and credentials. Notice your Access Key ID and Secrete Access Key will go in the credentials file. Example of these files: ~/.aws/config [default] region=us-west-2 output=json ~/.aws/credentials [default] aws_access_key_id=AKIAIOSFODNN7EXAMPLE aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Configuring your AWS account

Configuring Route53

1. Identify your domain, or subdomain, and registrar.

2. 1. Use an appropriate root domain, such as openshiftcorp.com, or subdomain, such as clusters.openshiftcorp.com

AWS Account limitation

By default, each cluster creates the following instances:

• One bootstrap machine, which is removed after installation

• Three master nodes

• Three worker nodes

Running the OpenShift Cluster Installer

With the adoption of OpenShift 4.5 we are introducing a new way to manage your metadata. Previously there was no standard way to associate and transmit operator manifests and metadata between clusters, or to associate a set of manifests with one or more runnable container images. The changes we are putting into place will remedy this situation and change the way a partner creates and maintains the metadata associated with your certified operator.

Updated Operator Certification Workflow:

We have sent you a zip file that contains your metadata and a Dockerfile for an image to hold it. That Dockerfile replaces the zip you used to upload for scanning. We have already submitted the bundle image to the scan test and everything has been successfully converted to use the new format. There are some changes to how you go about working with this new format and this section of the guide is designed to help you understand the new format and how to continue forward using this format. Below you can see an example of the new format of metdata that is shipped inside of the container image provided.

To create a new update for your operator, you'll need to create a new version directory, and place your crds and csv inside. You can make any updates to these files as normal.

Here's an example what the structure should look like when you're done:

Move into the bundle directory. You can now use opm to create the annotations.yaml and Dockerfile. Keep in mind that the new Dockerfile will be created in the directory you run the command, and it includes COPY commands. It'll also slightly re-scaffold the project.

Next, you'll need to add some LABELs to the Dockerfile.

When you're done, the finished Dockerfile should look something like this

Now you can build the new image and submit it to the pipeline. You'll have a new project in Connect for your bundle image that we created for you as part of migration with the same name as your operator project, but with "-bundle" on the end. Click "Push Image Manually" for instructions on uploading your metadata bundle for testing.

Each architecture-specific image for each component (operator or operand) must be scanned within the respective project and tagged according to the architecture. Take, for example an operator image named example-operator:v1.0.0 which has two associated architectures: one for Intel/AMD64 and another for IBM Z/s390x:

• Both the amd64 and s390x images should be scanned separately in the Operator Controller project

• Since the images can't share a tag name, they should be tagged according to the architecture.

• The amd64 image could be tagged as v1.0.0-amd64 or simply left as v1.0.0, assuming it's already been published

• Therefore, the Z/s390x image could be tagged as v1.0.0-s390x to distinguish it from the Intel/AMD64 image

Let's go through each of these steps to scan the s390x image for an externally-hosted operator controller.

Select the operator project (not the bundle project) from the list of projects, and click the Images header to view the associated images. Click Scan new image and you should see the following prompt:

Enter the pull spec containing the sha256 digest (not the tag) of the specific image that you wish to scan, along with the destination repository name and tag:

Wait for the scan to complete. Once completed, Publish the image by clicking the chevron > on the left side and then click Publish beside the corresponding tag.

What is the recommended version for Operator-SDK?

The recommended version is the 1.0.0+ this brings significant changes but it will allow you to build the bundle images using the new format.

What happens to the previous operator project? Do we just upload the actual operator image there?

That's correct. The previous operator project holds the operator container image. Not the bundle image. You will need to continue to update the operator image through that project, and the bundle metadata image to the new project.

How is a new version released? Just uploading and publishing the image in the bundle project is enough?

Once you update your operator and are ready for a new version, you will need to update the operator image and once that is published you can submit the bundle image. A new version is released when you push your metadata bundle image to the bundle project. Once you publish the Bundle Image it will take about 1-2 hours for the new version to show up on embedded OperatorHub.

What labels should be used for future OpenShift releases? Currently I see only 4.5 and 4.6

The label that says 4.5 implicitly takes care of all previous versions of OpenShift. The newer labels from 4.7 and beyond will be automatically handled in the backend process. In the future we will publish new documentation in case you don't want your operator to show up in a specific OpenShift version.

Here we showcase how to build an operator controller image targeting other (non-Intel/AMD) CPU architectures for hosting in an external registry. It also covers assembling the various architecture images into a single manifest list and tag. This section assumes that you've already made the required changes to the operator's Dockerfile (labels and licenses) when certifying for Intel/AMD.

Building with Legacy Operator SDK Releases (pre-1.0)

For older releases of the Operator SDK, where image building is handled by the operator-sdk binary, the process is fairly simple. There are no specific requirements to be made to the Dockerfile source for the operator controller. All modifications are made as flags or variables when calling operator-sdk build.

Specifically, you must set the GOARCH variable and --arch flag to match the CPU architecture that you're targeting. According to the CPU architecture table, the correct architecture (arch) for IBM Z would be s390x.

For cross compilation and container builds to work, you must install the qemu-user-static package, which is not available in RHEL (Fedora was used here):

$ sudo dnf install -y qemu-user-static

$ GOARCH=s390x operator-sdk build ubi-noop-go:v0.0.1-s390x --image-build-args="--arch=s390x"

Similarly, to build an operator image for IBM Power:

$ GOARCH=ppc64le operator-sdk build ubi-noop-go:v0.0.1-ppc64le --image-build-args="--arch=ppc64le"

OPTIONAL: If you do not yet have an Intel/AMD operator image (a requirement for multi-arch operator certification) and want to follow along, then build the amd64 image. Regardless, you must set the arch flag to amd64 or podman will default to using the most recent image/architecture (ppc64le) for future commands:

$ operator-sdk build ubi-noop-go:v0.0.1-amd64 --image-build-args="--arch=amd64"

With all of the images built, the next step is to create a manifest list and push it to an external registry.

Build and Push the Manifest List

In this section, we use podman exclusively for creating and pushing the manifest list. The syntax for docker is similar, with the exception of using a single argument when pushing the manifest to a registry.

Create a new manifest list, which will reference each of the underlying architecture-specific images:

$ podman manifest create ubi-noop-go:v0.0.1

Add each of the architecture specific images to the manifest list, in no particular order. To make sure that podman pulls the images from local container storage, we use the containers-storage:localhost/ prefix. Let's start with the Intel/AMD image:

$ podman manifest add ubi-noop-go:v0.0.1 containers-storage:localhost/ubi-noop-go:v0.0.1-amd64

Next, add the Power image to the manifest list:

$ podman manifest add ubi-noop-go:v0.0.1 containers-storage:localhost/ubi-noop-go:v0.0.1-ppc64le

Add the Z image to complete the manifest list:

$ podman manifest add ubi-noop-go:v0.0.1 containers-storage:localhost/ubi-noop-go:v0.0.1-s390x

The last step is to push the new manifest list to an external registry for hosting. Quay.io is suggested here, but you can use any registry which supports manifest lists, such as Docker Hub:

$ podman manifest push ubi-noop-go:v0.0.1 quay.io/exampleco/ubi-noop-go:v0.0.1

Supported Architectures

The full list of supported hardware platforms and associated architecture names is in the following table:

Core Requirements

There are a number of technical and non-technical requirements that must be met before being able to certify or publish a non-Intel/AMD operator. The list of requirements are as follows:

• A Certified Intel/AMD architecture operator and any applicable operands (container workloads deployed/managed by the operator controller)

• At least three (3) projects created and published in Red Hat Partner Connect:

  • An Operator / Bundle project

    • Project type: Operator Bundle

    • Distribution method: Red Hat

  • An Operator Controller image project

    • Project type: Container

    • Distribution method: External

  • One or more Operand images (as applicable)

    • Project type: Container

    • Distribution method: External

• All container images related to the operator must be hosted in an external registry

• Your operator and operands must both support the CPU architectures that you intend to run on. In other words, you can't build an operator for another platform (such as IBM Z) if your operand only runs on AMD64 / Intel x86. The product (operands) must also support the new platform.

• Each architecture image must be contained within a manifest list, with one manifest list for the operator and one for each of the operands. For example, a minimal multi-arch image layout might look like below (image and platform names are hypothetical):

  • An operator manifest list named/tagged as operator-image:v1.0 would be a manifest referring to these images:

    • operator-image:v1.0-amd64

    • operator-image:v1.0-othercpu

  • A single operand manifest list named/tagged as example-app:v1.0, and would refer to two other images matching the same architecture:

    • example-app:v1.0-amd64

    • example-app:v1.0-othercpu

Below is an illustration to help explain the relationship between the operator components and the various registries (RHCC, external, and scan). Please keep in mind that there could be more than one operand project/image whereas the diagram below only shows a single operand:

Current Limitations

There are a few limitations as well, which dictate the requirements:

• Using the RHCC (Red Hat Container Catalog) registry and in turn the Red Hat distribution method will not work due to lacking support for manifest lists

• There is no automated certification pipeline or test infrastructure in place for non-Intel/AMD architectures at this time

• Non-Intel/AMD operators cannot be certified or published on their own, and must be published along with an Intel/AMD operator

To actually publish your operator so that non-Intel/AMD architectures can consume it, the bundle image must be updated with the externally-hosted manifest lists for all images. We'll assume you're updating an existing bundle, so you'll want to regenerate a new bundle directory using your kustomize templates (applicable to post-1.0 SDK), or simply copy your latest bundle directory and work from that.

Using the latter method as an example, let's say we have a bundle directory named 0.0.1/ and we want to bump this to 0.0.2/. A simple recursive copy will get us started:

In the above case, we would work from the newly created bundle/0.0.2 directory, and make the necessary changes to the ClusterServiceVersion (CSV) yaml file within the manifests/ subdirectory of the bundle.

As part of the version bump, we need to rename the CSV, since the version number is part of the file name:

Now we can proceed with updating the CSV. The following field paths will need to be updated:

• metadata.annotations.alm-examples[] (as applicable, if image pull specs are used in the CR's)

• metadata.annotations.containerImage

• metadata.name

• spec.install.spec.deployments[].spec.template.spec.containers[].image

• spec.install.spec.deployments[].spec.template.spec.containers[].env[] (update RELATED_IMAGE environment variables for Red Hat Marketplace)

• spec.replaces

• spec.version

These are the same fields that should be updated for a typical release/upgrade cycle, say when an image tag/digest changes and you must update those references and bump the operator (CSV) version accordingly. Mainly, the point is here to ensure that the image manifest-lists are utilized instead of the single architecture Intel/AMD64 images used previously.

Lastly, there is one final edit that must be made to enable publishing your operator to other hardware platforms. You must add a label under metadata.labels for architecture supported by the operator.

For our previous example operator, ubi-noop-go, we would claim support for amd64, ppc64le, and s390x architectures: