Attention
Validation
Build-Time Kernel Configuration Check
After the kernel configuration has been produced during the build, it is checked to validate the presence of necessary kernel configuration to comply with specific EWAOL functionalities.
A list of required kernel configs is used as a reference, and compared against
the list of available configs in the kernel build. All reference configs need to
be present either as module (=m
) or built-in (=y
). A BitBake warning
message is produced if the kernel is not configured as expected.
The following kernel configuration checks are performed:
Container engine support:
Check performed via:
meta-ewaol-distro/classes/containers_kernelcfg_check.bbclass
. By default Yocto Docker config is used as the reference.K3s orchestration support:
Check performed via:
meta-ewaol-distro/classes/k3s_kernelcfg_check.bbclass
. By default Yocto K3s config is used as the reference.Xen virtualization support (available for EWAOL virtualization distribution images):
Check performed via:
meta-ewaol-distro/classes/xen_kernelcfg_check.bbclass
. By default Yocto Xen config is used as the reference.
Run-Time Integration Tests
The meta-ewaol-tests
Yocto layer contains recipes and configuration for
including run-time integration tests into an EWAOL distribution, to be run
manually after booting the image.
The EWAOL run-time integration tests are a mechanism for validating EWAOL core functionalities. The integration test suites included on an EWAOL distribution image depend on its target architecture, as follows:
- Baremetal architecture:
K3s Orchestration Tests (local deployment of a K3s pod)
- Virtualization architecture:
Control VM:
K3s Orchestration Tests (remote deployment of K3s pods on the Guest VMs, from the Control VM)
Guest VM:
The tests are built as a Yocto Package Test (ptest), and implemented using the Bash Automated Test System (BATS).
Run-time integration tests are not included on an EWAOL distribution image by default, and must instead be included explicitly. See Run-Time Integration Tests within the Build System documentation for details on how to include the tests.
The test suites are executed using the test
user account, which has sudo
privileges. More information about user accounts can be found at
User Accounts.
Running the Tests
If the tests have been included on the EWAOL distribution image, they may be run via the ptest framework, using the following command after booting the image and logging in:
ptest-runner [test-suite-id]
If the test suite identifier ([test-suite-id]
) is omitted, all integration
tests will be run. For example, running ptest-runner
on the Control VM of
an EWAOL virtualization distribution image produces output such as the
following:
$ ptest-runner
START: ptest-runner
[...]
PASS:container-engine-integration-tests
[...]
PASS:k3s-integration-tests
[...]
PASS:user-accounts-integration-tests
[...]
PASS:virtualization-integration-tests
[...]
STOP: ptest-runner
Note
As different EWAOL architectures support different test suites.
ptest-runner -l
is a useful command to list the available test suites on
the image.
Alternatively, a single standalone test suite may be run via a runner script included in the test suite directory:
/usr/share/[test-suite-id]/run-[test-suite-id]
Upon completion of the test-suite, a result indicator will be output by the
script, as one of two options: PASS:[test-suite-id]
or
FAIL:[test-suite-id]
, as well as an appropriate exit status.
A test suite consists of one or more ‘top-level’ BATS tests, which may be
composed of multiple assertions, where each assertion is considered a named
sub-test. If a sub-test fails, its individual result will be included in the
output with a similar format. In addition, if a test failed then debugging
information will be provided in the output of type DEBUG
. The format of
these results are described in Test Logging.
Test Logging
Test suite execution outputs results and debugging information into a log file.
As the test suites are executed using the test
user account, this log file
will be owned by the test
user and located in the test
user’s home
directory by default, at:
/home/test/runtime-integration-tests-logs/[test-suite-id].log
Therefore, reading this file as another user will require sudo
access. The
location of the log file for each test suite is customizable, as described in
the detailed documentation for each test suite below. The log file is replaced
on each new execution of a test suite.
The log file will record the results of each top-level integration test, as well as a result for each individual sub-test up until a failing sub-test is encountered.
Each top-level result is formatted as:
TIMESTAMP RESULT:[top_level_test_name]
Each sub-test result is formatted as:
TIMESTAMP RESULT:[top_level_test_name]:[sub_test_name]
Where TIMESTAMP
is of the format %Y-%m-%d %H:%M:%S
(see
Python Datetime Format Codes), and RESULT
is either PASS
, FAIL
,
or SKIP
.
On a test failure, a debugging message of type DEBUG
will be written to
the log. The format of a debugging message is:
TIMESTAMP DEBUG:[top_level_test_name]:[return_code]:[stdout]:[stderr]
Additional informational messages may appear in the log file with INFO
or
DEBUG
message types, e.g. to log that an environment clean-up action
occurred.
Test Suites
The test suites are detailed below.
Container Engine Tests
The container engine test suite is identified as:
container-engine-integration-tests
for execution via ptest-runner
or as a standalone BATS suite, as described
in Running the Tests.
On an EWAOL virtualization distribution image, the container engine test suite is available for execution on both the Control VM and the Guest VM. In addition, as part of running the test suite on the Control VM, an extra test will be performed which logs into each Guest VM and runs the container engine test suite on it, thereby reporting any test failures of a Guest VM as part of the Control VM’s test suite execution.
The test suite is built and installed in the image according to the following
BitBake recipe:
meta-ewaol-tests/recipes-tests/runtime-integration-tests/container-engine-integration-tests.bb
.
Currently the test suite contains three top-level integration tests, which run consecutively in the following order.
run container
is composed of four sub-tests:docker run
commanddocker inspect
commanddocker remove
commanddocker inspect
commandcontainer network connectivity
is composed of a single sub-test:docker run
commandrun container engine integration tests on Guest VMs from the Control VM
is only executed on the Control VM. On a Guest VM this test is skipped.
The test is composed of the following sub-tests:systemctl status
xendomains status
ptest-runner container-engine-integration-tests
commandThe tests can be customized via environment variables passed to the execution,
each prefixed by CE_
to identify the variable as associated to the
container engine tests:
CE_TEST_IMAGE
: defines the container imagedocker.io/library/alpine
CE_TEST_LOG_DIR
: defines the location of the log file/home/test/runtime-integration-tests-logs/
CE_TEST_CLEAN_ENV
: enable test environment clean-up1
(enabled)CE_TEST_GUEST_VM_BASENAME
: defines the base Xen domain name (hostname) to
determine which Guest VMs to include as part of the test suite execution${EWAOL_GUEST_VM_HOSTNAME}
ewaol-guest-vm
Container Engine Environment Clean-Up
A clean environment is expected when running the container engine tests. For example, if the target image already exists within the container engine environment, then the functionality to pull the image over the network will not be validated. Or, if there are running containers from previous (failed) tests then they may interfere with subsequent test executions.
Therefore, if CE_TEST_CLEAN_ENV
is set to 1
(as is default), running
the test suite will perform an environment clean before and after the suite
execution.
The environment clean operation involves:
Determination and removal of all running containers of the image given by
CE_TEST_IMAGE
Removal of the image given by
CE_TEST_IMAGE
, if it existsClearing the passwords set when the test suite accessed each Guest VM, performed only when running the test suite on a virtualization distribution image with Security Hardening enabled.
If enabled then the environment clean operations will always be run, regardless of test-suite success or failure.
K3s Orchestration Tests
The K3s test suite is identified as:
k3s-integration-tests
for execution via ptest-runner
or as a standalone BATS suite, as described
in Running the Tests.
The test suite is built and installed in the image according to the following
BitBake recipe within
meta-ewaol-tests/recipes-tests/runtime-integration-tests/k3s-integration-tests.bb
.
Currently the test suite contains a single top-level integration test which validates the deployment and high-availability of a test workload based on the Nginx webserver. The test suite is dependent on the target EWAOL architecture, as follows.
For EWAOL baremetal distribution images, the K3s integration tests consider a single-node cluster, which runs a K3s server together with its built-in worker agent. The containerized test workload is therefore deployed to this node for scheduling and execution.
For EWAOL virtualization distribution images, the K3s integration tests consider a cluster comprised of multiple nodes: the Control VM running a K3s server, and each target Guest VM running a K3s agent which is connected to the server. The containerized test workload is configured to only be schedulable on the Guest VMs, meaning that the server on the Control VM orchestrates a test application with replicas that are deployed and executed on each of the target Guest VMs. In addition to the same initialization procedure that is carried out when running the tests on a baremetal distribution image, initialization for virtualization distribution images includes connecting a K3s agent running on each target Guest VM to the Control VM’s K3s server (if it is not already connected). To do this, before the tests run, the Systemd service that provides the K3s agent on each Guest VM is configured with a Systemd service unit override that provides the IP and authentication token of the Control VM’s K3s server, and this service is then started. The K3s integration test suite therefore expects that there are target Guest VMs available when running on a virtualization distribution image, and will not create any if they are not found.
In both cases, the test suite will not be run until the appropriate K3s services are in the ‘active’ state, and all ‘kube-system’ pods are either running, or have completed their workload.
K3s container orchestration
is composed of many sub-tests, grouped here
by test area:kubectl apply
kubectl wait
kubectl create service
kubectl get
wget
kubectl get
kubectl get
kubectl set
kubectl wait
and kubectl get
kubectl get
wget
systemctl stop
wget
systemctl start
kubectl get
The tests can be customized via environment variables passed to the execution,
each prefixed by K3S_
to identify the variable as associated to the
K3s orchestration tests:
K3S_TEST_LOG_DIR
: defines the location of the log file/home/test/runtime-integration-tests-logs/
K3S_TEST_CLEAN_ENV
: enable test environment clean-up1
(enabled)K3S_TEST_GUEST_VM_BASENAME
: defines the base Xen domain name (hostname)
to determine which Guest VMs to include in the test suite execution${EWAOL_GUEST_VM_HOSTNAME}
ewaol-guest-vm
K3s Environment Clean-Up
A clean environment is expected when running the K3s integration tests, to ensure that the system is ready to be validated. For example, the test suite expects that the Pods created from any previous execution of the integration tests have been deleted, in order to test that a new Deployment successfully initializes new Pods for orchestration.
Therefore, if K3S_TEST_CLEAN_ENV
is set to 1
(as is default), running
the test suite will perform an environment clean before and after the suite
execution.
The environment clean operation involves:
Deleting any previous K3s test Service
Deleting any previous K3s test Deployment, ensuring corresponding Pods are also deleted
For virtualization distribution images, additional clean-up operations are performed:
Deleting each Guest VM node from the K3s cluster
Stopping the K3s agent running on each Guest VM, and deleting any test Systemd service override on each Guest VM
Clearing the passwords set when the test suite accessed each Guest VM, performed only when running the test suite on a virtualization distribution image with Security Hardening enabled.
If enabled then the environment clean operations will always be run, regardless of test-suite success or failure.
User Accounts Tests
The User Accounts test suite is identified as:
user-accounts-integration-tests
for execution via ptest-runner
or as a standalone BATS suite, as described
in Running the Tests.
The test suite is built and installed in the image according to the following
BitBake recipe within
meta-ewaol-tests/recipes-tests/runtime-integration-tests/user-accounts-integration-tests.bb
.
The test suite validates that the user accounts described in User Accounts are correctly configured on the EWAOL distribution image. Therefore, the validation performed by the test suite is dependent on the target architecture, and on whether or not it has been configured with EWAOL Security Hardening, as follows.
For a baremetal image, the test suite validates that the expected user accounts are configured and appropriate access permissions are in place. For a virtualization image, the test suite is available on both the Control VM and the Guest VM(s), and includes the same validation as the baremetal test suite on the respective VM’s local user accounts. However, as part of running the test suite on the Control VM, an extra test will be performed which logs into the Guest VMs and runs the user accounts test suite on them, thereby reporting any test failures of a Guest VM as part of the Control VM’s test suite execution.
As the configuration of user accounts is modified for EWAOL distribution images that are built with EWAOL security hardening, additional security-related validation is included in the test suite for these images, both on EWAOL baremetal and virtualization distribution images. These additional tests validate that the appropriate password requirements and root-user access restrictions are correctly imposed, and that the mask configuration for permission control of newly created files and directories is applied correctly.
The test suite therefore contains three top-level integration tests, two of which are conditionally executed, as follows:
user accounts management tests
is composed of three sub-tests:stat
utilitysudo
command
accesssudo
command accessuser accounts management additional security tests
is only included for
images configured with EWAOL security hardening, and is composed of four
sub-tests:run user accounts integration tests on the Guest VM from the Control VM
is only included for EWAOL virtualization distribution images, and is only
executed on the Control VM. On a Guest VM this test is skipped. The test
is composed of the following sub-tests:systemctl status
xendomains status
ptest-runner user-accounts-integration-tests
commandThe tests can be customized via environment variables passed to the execution,
each prefixed by UA_
to identify the variable as associated to the user
accounts tests:
UA_TEST_LOG_DIR
: defines the location of the log file/home/test/runtime-integration-tests-logs/
UA_TEST_CLEAN_ENV
: enable test environment clean-up1
(enabled)UA_TEST_GUEST_VM_BASENAME
: defines the base Xen domain name (hostname) to
determine which Guest VMs to include as part of the test suite execution${EWAOL_GUEST_VM_HOSTNAME}
ewaol-guest-vm
User Accounts Environment Clean-Up
As the user accounts integration tests only modify the system for images built with EWAOL security hardening, clean-up operations are only performed when running the test suite on these images.
In addition, the clean-up operations will only occur if UA_TEST_CLEAN_ENV
is
set to 1
(as is default).
The environment clean-up operations for images built with EWAOL security hardening are:
Reset the password for the
test
user accountReset the password for the non-privileged EWAOL user account
Clearing the passwords set when the test suite accessed each Guest VM, performed only when running the test suite on a virtualization distribution image with Security Hardening enabled.
After the environment clean-up, the user accounts will return to their original state where the first log-in will prompt the user for a new account password.
If enabled then the environment clean operations will always be run, regardless of test-suite success or failure.
Xen Virtualization Tests
The Xen Virtualization test suite is identified as:
virtualization-integration-tests
for execution via ptest-runner
or as a standalone BATS suite, as described
in Running the Tests.
The test suite is built and installed in the image according to the following
BitBake recipe within
meta-ewaol-tests/recipes-tests/runtime-integration-tests/virtualization-integration-tests.bb
.
The test suite is only available for images that target the virtualization architecture.
Currently the test suite contains two top-level integration tests, which validate that the target Guest VMs are correctly running, and validate that they can be managed successfully from the Control VM. These tests are as follows:
validate Guest VM is running
is composed of two sub-tests:xendomains status
xl console
ping
utilityvalidate Guest VM management
is composed of five sub-tests:xendomains status
systemctl stop
xendomains status
systemctl start
xendomains status
The tests can be customized via environment variables passed to the execution,
each prefixed by VIRT_
to identify the variable as associated to the
virtualization integration tests:
VIRT_TEST_LOG_DIR
: defines the location of the log file/home/test/runtime-integration-tests-logs/
VIRT_TEST_CLEAN_ENV
: enable test environment clean-up1
(enabled)VIRT_TEST_GUEST_VM_BASENAME
: defines the base Xen domain name (hostname)
to determine which Guest VMs to validate${EWAOL_GUEST_VM_HOSTNAME}
ewaol-guest-vm
Prior to execution, the Xen Virtualization test suite expects the
xendomains.service
Systemd service on the Control VM to be running or in the
process of initializing.
Xen Virtualization Environment Clean-Up
The Xen Virtualization integration tests only modify the system environment when the test suite is executed on an image with Security Hardening enabled, as accessing a Guest VM on a security hardened image requires setting the user account password.
There is therefore only a single environment clean operation performed for this test suite:
Clearing the passwords set when the tests accessed each Guest VM, performed only when running the test suite with Security Hardening enabled.
Cleaning up the account password will only occur if VIRT_TEST_CLEAN_ENV
is
set to 1
(as is default), in which case the environment clean will run
before and after the suite execution.
If enabled then the environment clean operation will always be run, regardless of test-suite success or failure.