Operator-Scorecard-Tests Failed

Issue

A partner has submitted the metadata but failing with operator-scorecard test. The pipeline currently runs version v0.15.1 of the operator-scorecard to test the metadata. Make sure you are testing using that version. In order to certify you must pass the first two Basic Tests: Spec Block Exists, and Status Block Exists. Passing the third basic test (Writing into CRs has an effect) requires adding the scorecard-proxy container to the operator deployment, which is not desired in a production operator and is therefore not required for certification.

Environment

• Operator Config section for Operator Project in Red Hat Partner Connect web site

Resolution

Method 1: Run the scorecard against the metadata

First you will need to deploy the operator via OLM using a custom operator source as described here. Assuming the operator deploys correctly you can then run the scorecard tests against the operator.

To run the scorecard locally against the operator metadata you are testing, you will need to set up a .osdk-scorecard.yaml file.The contents of this file are fairly simple to set up. Below is an example. You will need to substitute your actual file locations and paths in order for the scorecard to function properly with your Operator.

.osdk-scorecard.yaml

  plugins:
  - basic:
      olm-deployed: true
      csv-path: /home/operators/Wordpress/bundle/wordpress-operator.v0.0.1.clusterserviceversion.yaml
      crds-dir: ./bundle/
      cr-manifest: ./crs/cr.yaml

Setting up the .osdk-scorecard.yaml using the example above will ensure the scorecard tests are being run in the same way as how they are being run in the pipeline.

You will need create the cr-manifest: ./crs/cr.yaml in order for the scorecard to work. In the base of your project create the directory and the file.

mkdir crs/

vi crs/cr.yaml

To get the contents for cr.yaml open up the CSV and select everything within the [ ] of the metadata: annotations: alm-examples:

The alm-examples above was truncated to show the beginning and end of what needs to be selected

Paste the contents of the metadata: annotations: alm-examples: within the cr.yaml file you created and then change directories back to where the .osdk-scorecard.yaml file exists.

To run the scorecard simply run:

operator-sdk scorecard

Both the Spec Block Exists and Status Block Exists tests are looking at the CRs that are associated with the operator. If either one of these tests fails you will need to rebuild the operator image using the SDK, simply adjusting the metadata is not enough. Both will be automatically generated during the creation of the scaffolding (if using Helm or Ansible). For issues with adding the status using Golang see this documentation

Method 2: The local scorecard results are different from the pipeline results

If the workaround in Method 1 did not work as a solution or you've encountered other issues, a partner needs to open a support ticket and provide the following information:

• The Project URL

• Bundle Directory Archive

• Include test results for local run and pipeline run