Examples
Let’s have a look at a couple of real-life examples!
Coverage
Test coverage information can be stored in a single file, for
example wget/requirements.fmf
:
/protocols:
priority: high
/ftp:
requirement: Download a file using the ftp protocol.
coverage: wget/protocols/ftp
/http:
requirement: Download a file using the http protocol.
coverage: wget/protocols/http
/https:
requirement: Download a file using the https protocol.
coverage: wget/protocols/https
/download:
priority: medium
/output-document-pipe:
requirement: Save content to pipe.
coverage: wget/download
/output-document-file:
requirement: Save content to a file.
coverage: wget/download
/upload:
priority: medium
/post-file:
requirement: Upload a file to the server
coverage: wget/protocols/http
/post-data:
requirement: Upload a string to the server
coverage: wget/protocols/http
Or split by functionality area into separate files as desired, for
example wget/download/requirements.fmf
:
priority: medium
/output-document-pipe:
requirement: Save content to pipe.
coverage: wget/download
/output-document-file:
requirement: Save content to a file.
coverage: wget/download
Or integrated with test case metadata, e.g.
wget/download/main.fmf
:
description: Check basic download options
tags: [Tier2, TierSecurity]
test: runtest.sh
time: 3 min
/requirements:
requirement: Various download options working correctly
priority: low
/get-file:
coverage: wget/download
/output-document:
coverage: wget/download
/continue:
/timestamping:
/tries:
/no-clobber:
coverage: wget/download
/progress:
/quota:
/server-response:
/bind-address:
/spider:
In the example above three requirements are already covered, the rest still await for test coverage (attributes value is null).
Strategist
Here’s an example implementation of test-strategist data for openscap using the Flexible Metadata Format:
/probes:
description: Probes
/offline:
description: Offline scanning
/online:
description: Online scanning
/scanning:
description: Reading and understanding source datastreams
/oval:
influencers:
- openscap/probes/offline
- openscap/probes/online
/ds:
influencers:
- openscap/scanning/oval
- openscap/scanning/cpe
/cpe:
influencers:
- openscap/scanning/oval
Setups
This example shows how to use Flexible Metadata Format to run tests with different storage setups including cleanup. This is simplified metadata, whole example including tools can be found at storage_setup:
/setups:
description: Tests to prepare and clean up devices for tests
setup: True
/setup_local:
test: setup_local.py
requires_cleanup: setups/cleanup_local
/cleanup_local:
test: cleanup_local.py
/setup_remote:
test: setup_remote.py
requires_cleanup: setups/cleanup_remote
/cleanup_remote:
test: cleanup_remote.py
/setup_vdo:
test: setup_vdo.py
requires_cleanup: setups/cleanup_vdo
/cleanup_vdo:
test: cleanup_vdo.py
/tests:
description: Testing 'vdo' command line tool
requires_setup: [setups/setup_vdo]
/create
description: Testing 'vdo create'
/ack_threads
/activate
/modify
description: Testing 'vdo modify'
requires_setup+: [setups/setup_remote]
/block_map_cache_size
You can find here not only how to use FMF for setup/cleanup and group tests based on that, but also installing requirements, passing values from metadata to tests themself and much more.
Format
Custom format output using --format
and value
.
List object name and selected attribute:
fmf examples/wget --format '{0}: {1}\n' \
--value 'name' --value 'data["tester"]'
Show missing attributes in red:
fmf examples/wget/ --format '{}: {}\n' --value 'name' \
--value 'utils.color(str(data.get("priority")),
"red" if data.get("priority") is None else "green")'
List all test scripts with full path:
fmf examples --key test --format "{}/{}/{}\n" \
--value "os.getcwd()" \
--value "data.get('path') or name" \
--value "data['test']"