Argo CD Operator E2E Test Guide¶
E2E tests are written using KUTTL.
Requirements¶
This test suite assumes that an Argo CD Operator is installed on the cluster or running locally using ARGOCD_CLUSTER_CONFIG_NAMESPACES=argocd-e2e-cluster-config make install run
.
The system executing the tests must have following tools installed:
kuttl
kubectl plugin (>= v0.11.1)oc
andkubectl
clientjq
for parsing JSON datacurl
There should be a kubeconfig
pointing to your cluster, user should have full admin privileges (i.e. kubeadm
).
Note
E2E tests utilize GNU Grep under the hood. Please make sure that you have the GNU compatible grep
installed.
If you are on OSX you can install GNU compatible grep using the below command. The package is installed as ggrep
by default. Please set this(ggrep) as an alias to grep
.
brew install grep
Use the below commands to install GNU compatible grep
on OSX.
Also, note that the e2e tests for Redis HA mode require a cluster with at least three worker nodes. A local three-worker node cluster can be created using k3d
Running the tests¶
In any case, you should have set up your kubeconfig
in such a way that your
default context points to the cluster you want to test. You can use the
kubectl login ...
command to set this up for you.
Run e2e tests¶
make e2e
Run Operator locally and execute e2e tests¶
make all
Running manual with kuttl¶
kubectl kuttl test ./tests/k8s --config ./tests/kuttl-tests.yaml
Running single tests¶
Sometimes (e.g. when initially writing a test or troubleshooting an existing
one), you may want to run single test cases isolated. To do so, you can pass
the name of the test using --test
to kuttl
, i.e.
kubectl kuttl test ./tests/k8s --config ./tests/kuttl-tests.yaml --test 1-004_validate_namespace_scoped_install
The name of the test is the name of the directory containing its steps and assertions.
If you are troubleshooting, you may want to prevent kuttl
from deleting the
test's namespace afterwards. In order to do so, just pass the additional flag
--skip-delete
to above command.
Writing new tests¶
Name of the test¶
Each test comes in its own directory, containing all its test steps. The name of the test is defined by the name of this directory.
The name of the test should be short, but expressive. The format for naming a
test is currently <test ID>_<short description>
.
The <test ID>
is the serial number of the test as defined in the Test Plan
document. The <short description>
is exactly that, a short description of
what happens in the test.
Name of the test steps¶
Each test step is a unique YAML file within the test's directory. The name of the step is defined by its file name.
The test steps must be named XX-<name>.yaml
. This is a kuttl
convention
and cannot be overriden. XX
is a number (prefixed with 0
, so step 1
must
be 01
), and <name>
is a free form value for the test step.
There are two reserved words you cannot use for <name>
:
assert
contains positive assertions (i.e. resources that must exist) anderrors
contains negative assertions (i.e. resources that must not exist)
Refer to the kuttl documentation for more information.
Documentation¶
Documentation is important, even for tests. You can should provide inline
documentation in your YAML files (using comments) and a README.md
in your
test case's directory. The README.md
should provide some context for the
test case, e.g. what it tries to assert for under which circumstances. This
will help others in troubleshooting failing tests.
Recipes¶
kuttl
unfortunately neither encourages or supports re-use of your test steps
and assertions yet.
Generally, you should try to use assert
and errors
declaration whenever
possible and viable. For some cases, you may need to use custom scripts to
get the results you are looking for.
Scripts general¶
Scripts can be executed in a kuttl.dev/TestStep
resources from a usual test
step declaration.
Your script probably will retrieve some information, and asserts it state. If the assertion fails, the script should exit with a code > 0, and also print some information why it failed, e.g.
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |
# Get some piece of information...
if test "$result" != "expected"; then
echo "Expectation failed, should 'expected', is '$result'"
exit 1
fi
Also, you may want to use set -e
and set -o pipefail
at the top of your
script to catch unexpected errors as test case failures, e.g.
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |
set -e
set -o pipefail
# rest of your script
Getting values of a resource's environment variables¶
YAML declarations used in assert
or errors
files unfortunately don't handle
arrays very well yet. You will always have to specify the complete expectation,
i.e. the complete array.
If you are just interested in a certain variable, and don't care about the rest,
you can use a script similar to the following using jq
. E.g. to get the value
of a variable named FOO
for the argocd-server
deployment in the test's
namespace:
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |
val=$(kubectl get -n $NAMESPACE deployments argocd-server -o json \
| jq -r '.spec.templates.spec.containers[0].env[]|select(.name=="FOO").value')
if test "$val" != "bar"; then
echo "Expectation failed for for env FOO in argocd-server: should 'bar', is '$val'"
exit 1
fi