Modules

Flexible Metadata Format

class fmf.Context(*args, **kwargs)

Represents https://fmf.readthedocs.io/en/latest/context.html

property case_sensitive: bool

evaluate(expression)

matches(rule)

Does the rule match the current Context?

We have three outcomes: Yes, No and CannotDecide

CannotDecide and True == True and CannotDecide == CannotDecide CannotDecide and False == False and CannotDecide == False CannotDecide or True == True or CannotDecide == True CannotDecide or False == False or CannotDecide == CannotDecide

Parameters:

rule (str | bool) – Single rule to decide

Return type:

bool

Raises:

CannotDecide – Impossible to decide the rule wrt current Context, e.g. dimension is missing
InvalidRule – Syntax error in the rule

operator_map = {'!=': <function Context._op_not_eq>, '<': <function Context._op_less>, '<=': <function Context._op_less_or_equal>, '==': <function Context._op_eq>, '>': <function Context._op_greater>, '>=': <function Context._op_greater_or_equal>, 'is defined': <function Context._op_defined>, 'is not defined': <function Context._op_not_defined>, '~!=': <function Context._op_minor_not_eq>, '~<': <function Context._op_minor_less>, '~<=': <function Context._op_minor_less_or_eq>, '~=': <function Context._op_minor_eq>, '~>': <function Context._op_minor_greater>, '~>=': <function Context._op_minor_greater_or_equal>}

static parse_rule(rule)

Parses rule into expressions

Expression is a tuple of dimension_name, operator_str, list of value objects. Parsed rule is nested list of expression from OR and AND operators. Items of the first dimension are in OR relation. Items in the second dimension are in AND relation.

expr_1 and expr_2 or expr_3 is returned as [[expr_1, expr_2], expr_3] expr_4 or expr_5 is returned as [[expr_4], [expr_5]] expr_6 and expr_7 is returned as [[expr_6, expr_7]]

Parameters:: rule (str | bool) – rule to parse
Returns:: nested list of expressions from the rule
Raises:: InvalidRule – Syntax error in the rule

static parse_value(value): Single place to convert to ContextValue

re_and_split = re.compile('\\band\\b')

re_boolean = re.compile('(true|false)')

re_expression_double = re.compile('(\\w+)\\s*(is defined|is not defined)')

re_expression_triple = re.compile('(\\w+)\\s*(~>=|>|>=|~=|!=|~!=|~>|~<|~<=|<|<=|==)\\s*([^=].*)')

re_or_split = re.compile('\\bor\\b')

static split_expression(expression)

Split expression to dimension name, operator and values

When operator doesn’t have right side, None is returned instead of the list of values.

Parameters:: expression (str) – expression to split
Raises:: InvalidRule – When expression cannot be split, e.g. syntax error
Returns:: tuple(dimension name, operator, list of values)
Return type:: tuple(str|None, str|bool, list|None)

static split_rule_to_groups(rule)

Split rule into nested lists, no real parsing

expr0 and expr1 or expr2 is split into [[expr0, expr1], [expr2]]

Parameters:: rule (str) – rule to split
Raises:: InvalidRule – Syntax error in the rule

class fmf.Tree(data, name=None, parent=None)

Metadata Tree

adjust(context, key='adjust', undecided='skip', case_sensitive=True, decision_callback=None)

Adjust tree data based on provided context and rules

The ‘context’ should be an instance of the fmf.context.Context class describing the environment context. By default, the key ‘adjust’ of each node is inspected for possible rules that should be applied. Provide ‘key’ to use a custom key instead.

Optional ‘undecided’ parameter can be used to specify what should happen when a rule condition cannot be decided because context dimension is not defined. By default, such rules are skipped. In order to raise the fmf.context.CannotDecide exception in such cases use undecided=’raise’.

Optional ‘case_sensitive’ parameter can be used to specify if the context dimension values should be case-sensitive when matching the rules. By default, values are case-sensitive.

Optional ‘decision_callback’ callback would be called for every adjust rule inspected, with three arguments: current fmf node, current adjust rule, and whether it was applied or not.

child(name, data, source=None): Create or update child with given data

climb(whole=False): Climb through the tree (iterate leaf/all nodes)

property commit

Commit hash if tree grows under a git repo, False otherwise

Return current commit hash if the metadata tree root is located under a git repository. For metadata initialized from a dict or local directory with no git repo ‘False’ is returned instead.

copy()

Create and return a deep copy of the node and its subtree

It is possible to call copy() on any node in the tree, not only on the tree root node. Note that in that case, parent node and the rest of the tree attached to it is not copied in order to save memory.

find(name): Find node with given name

get(name=None, default=None)

Get attribute value or return default

Whole data dictionary is returned when no attribute provided. Supports direct values retrieval from deep dictionaries as well. Dictionary path should be provided as list. The following two examples are equal:

tree.data[‘hardware’][‘memory’][‘size’] tree.get([‘hardware’, ‘memory’, ‘size’])

However the latter approach will also correctly handle providing default value when any of the dictionary keys does not exist.

grow(path)

Grow the metadata tree for the given directory path

Note: For each path, grow() should be run only once. Growing the tree from the same path multiple times with attribute adding using the “+” sign leads to adding the value more than once!

inherit(): Apply inheritance

static init(path): Create metadata tree root under given path

merge(parent=None): Merge parent data

static node(reference)

Return Tree node referenced by the fmf identifier

Keys supported in the reference:

url …. git repository url (optional) ref …. branch, tag or commit (default branch if not provided) path … metadata tree root (‘.’ by default) name … tree node name (‘/’ by default)

See the documentation for the full fmf id specification: https://fmf.readthedocs.io/en/latest/concept.html#identifiers Raises ReferenceError if referenced node does not exist.

prune(whole=False, keys=None, names=None, filters=None, conditions=None, sources=None): Filter tree nodes based on given criteria

show(brief=False, formatting=None, values=None): Show metadata

update(data): Update metadata, handle virtual hierarchy

validate(schema, schema_store=None)

Validate node data with given JSON Schema and schema references.

schema_store is a dict of schema references and their content.

Return a named tuple utils.JsonSchemaValidationResult with the following two items:

result … boolean representing the validation result errors … A list of validation errors

Raises utils.JsonSchemaError if the supplied schema was invalid.

fmf.filter(filter, data, sensitive=True, regexp=False)

Return true if provided filter matches given dictionary of values

Filter supports disjunctive normal form with ‘|’ used for OR, ‘&’ for AND and ‘-’ for negation. Individual values are prefixed with ‘value:’, leading/trailing white-space is stripped. For example:

tag: Tier1 | tag: Tier2 | tag: Tier3
category: Sanity, Security & tag: -destructive

Note that multiple comma-separated values can be used as a syntactic sugar to shorten the filter notation:

tag: A, B, C ---> tag: A | tag: B | tag: C

Values should be provided as a dictionary of lists each describing the values against which the filter is to be matched. For example:

data = {tag: ["Tier1", "TIPpass"], category: ["Sanity"]}

Other types of dictionary values are converted into a string. A FilterError exception is raised when a dimension parsed from the filter is not found in the data dictionary. Set option ‘sensitive’ to False to enable case-insensitive matching. If ‘regexp’ option is True, regular expressions can be used in the filter values as well.

base

Base Metadata Classes

class fmf.base.AdjustCallback(*args, **kwargs)

A callback for per-rule notifications made by Tree.adjust()

Function be be called for every rule inspected by adjust(). It will be given three arguments: fmf tree being inspected, current adjust rule, and whether the rule was skipped (None), applied (True) or not applied (False).

class fmf.base.Tree(data, name=None, parent=None)

Metadata Tree

adjust(context, key='adjust', undecided='skip', case_sensitive=True, decision_callback=None)

Adjust tree data based on provided context and rules

The ‘context’ should be an instance of the fmf.context.Context class describing the environment context. By default, the key ‘adjust’ of each node is inspected for possible rules that should be applied. Provide ‘key’ to use a custom key instead.

Optional ‘undecided’ parameter can be used to specify what should happen when a rule condition cannot be decided because context dimension is not defined. By default, such rules are skipped. In order to raise the fmf.context.CannotDecide exception in such cases use undecided=’raise’.

Optional ‘case_sensitive’ parameter can be used to specify if the context dimension values should be case-sensitive when matching the rules. By default, values are case-sensitive.

Optional ‘decision_callback’ callback would be called for every adjust rule inspected, with three arguments: current fmf node, current adjust rule, and whether it was applied or not.

child(name, data, source=None): Create or update child with given data

climb(whole=False): Climb through the tree (iterate leaf/all nodes)

property commit

Commit hash if tree grows under a git repo, False otherwise

Return current commit hash if the metadata tree root is located under a git repository. For metadata initialized from a dict or local directory with no git repo ‘False’ is returned instead.

copy()

Create and return a deep copy of the node and its subtree

It is possible to call copy() on any node in the tree, not only on the tree root node. Note that in that case, parent node and the rest of the tree attached to it is not copied in order to save memory.

find(name): Find node with given name

get(name=None, default=None)

Get attribute value or return default

Whole data dictionary is returned when no attribute provided. Supports direct values retrieval from deep dictionaries as well. Dictionary path should be provided as list. The following two examples are equal:

tree.data[‘hardware’][‘memory’][‘size’] tree.get([‘hardware’, ‘memory’, ‘size’])

However the latter approach will also correctly handle providing default value when any of the dictionary keys does not exist.

grow(path)

Grow the metadata tree for the given directory path

Note: For each path, grow() should be run only once. Growing the tree from the same path multiple times with attribute adding using the “+” sign leads to adding the value more than once!

inherit(): Apply inheritance

static init(path): Create metadata tree root under given path

merge(parent=None): Merge parent data

static node(reference)

Return Tree node referenced by the fmf identifier

Keys supported in the reference:

url …. git repository url (optional) ref …. branch, tag or commit (default branch if not provided) path … metadata tree root (‘.’ by default) name … tree node name (‘/’ by default)

See the documentation for the full fmf id specification: https://fmf.readthedocs.io/en/latest/concept.html#identifiers Raises ReferenceError if referenced node does not exist.

prune(whole=False, keys=None, names=None, filters=None, conditions=None, sources=None): Filter tree nodes based on given criteria

show(brief=False, formatting=None, values=None): Show metadata

update(data): Update metadata, handle virtual hierarchy

validate(schema, schema_store=None)

Validate node data with given JSON Schema and schema references.

schema_store is a dict of schema references and their content.

Return a named tuple utils.JsonSchemaValidationResult with the following two items:

result … boolean representing the validation result errors … A list of validation errors

Raises utils.JsonSchemaError if the supplied schema was invalid.

utils

Logging, config, constants & utilities

class fmf.utils.Coloring(*args, **kwargs)

Coloring configuration

MODES = ['COLOR_OFF', 'COLOR_ON', 'COLOR_AUTO']

enabled(): True if coloring is currently enabled

get(): Get the current color mode

set(mode=None)

Set the coloring mode

If enabled, some objects (like case run Status) are printed in color to easily spot failures, errors and so on. By default the feature is enabled when script is attached to a terminal. Possible values are:

COLOR=0 ... COLOR_OFF .... coloring disabled
COLOR=1 ... COLOR_ON ..... coloring enabled
COLOR=2 ... COLOR_AUTO ... if terminal attached (default)

Environment variable COLOR can be used to set up the coloring to the desired mode without modifying code.

exception fmf.utils.FetchError: Fatal error in helper command while fetching

exception fmf.utils.FileError: File reading error

exception fmf.utils.FilterError: Missing data when filtering

exception fmf.utils.FormatError: Metadata format error

exception fmf.utils.GeneralError: General error

exception fmf.utils.JsonSchemaError: Invalid JSON Schema

class fmf.utils.JsonSchemaValidationResult(result: bool, errors: List[Any])

Represents JSON Schema validation result

errors: List[Any]: Alias for field number 1

result: bool: Alias for field number 0

class fmf.utils.Logging(name='fmf')

Logging Configuration

COLORS = {4: 'magenta', 7: 'cyan', 10: 'green', 20: 'blue', 30: 'yellow', 40: 'red'}

class ColoredFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)

Custom color formatter for logging

format(record)

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

LEVELS = ['CRITICAL', 'DEBUG', 'ERROR', 'FATAL', 'INFO', 'NOTSET', 'WARN', 'WARNING']

MAPPING = {0: 30, 1: 20, 2: 10, 3: 7, 4: 4, 5: 1}

get(): Get the current log level

set(level=None)

Set the default log level

If the level is not specified environment variable DEBUG is used with the following meaning:

DEBUG=0 ... LOG_WARN (default)
DEBUG=1 ... LOG_INFO
DEBUG=2 ... LOG_DEBUG
DEBUG=3 ... LOG_CACHE
DEBUG=4 ... LOG_DATA
DEBUG=5 ... LOG_ALL (log all messages)

exception fmf.utils.MergeError: Unable to merge data between parent and child

exception fmf.utils.ReferenceError: Referenced tree node cannot be found

exception fmf.utils.RootError: Metadata tree root missing

fmf.utils.clean_cache_directory(): Delete used cache directory if it exists

fmf.utils.color(text, color=None, background=None, light=False, enabled='auto')

Return text in desired color if coloring enabled

Available colors: black red green yellow blue magenta cyan white. Alternatively color can be prefixed with “light”, e.g. lightgreen.

fmf.utils.default_branch(repository, remote='origin'): Detect default branch from given local git repository

fmf.utils.dict_to_yaml(data, width=None, sort=False): Convert dictionary into yaml

fmf.utils.evaluate(expression, data, _node=None)

Evaluate arbitrary Python expression against given data

Expects data dictionary which will be used to populate local namespace. Used to provide flexible conditions for filtering.

fmf.utils.fetch(url, ref=None, destination=None, env=None): Deprecated: Use fetch_repo() instead

fmf.utils.fetch_repo(url, ref=None, destination=None, env=None)

Fetch remote git repository and return local directory

Fetch git repository from provided url into a local cache directory, checkout requested ref and return path to the repo. If no ref is provided, the default branch from the origin is used. If destination directory is provided, it should not exist or needs to be empty. Use dictionary env to set environment variables for git calls.

Raises FetchError upon failure with the original exception included.

fmf.utils.fetch_tree(url, ref=None, path='.')

Get initialized Tree from a remote git repository

url …. git repository url (required) ref …. branch, tag or commit (default branch if None) path … metadata tree root (default to ‘.’)

See fmf.base.Tree.node() to canonical default values.

Remote repository is cached locally (see get_cache_directory()), local directory with cache is locked during reading.

Raises GeneralError when lock couldn’t be acquired.

fmf.utils.filter(filter, data, sensitive=True, regexp=False)

Return true if provided filter matches given dictionary of values

Filter supports disjunctive normal form with ‘|’ used for OR, ‘&’ for AND and ‘-’ for negation. Individual values are prefixed with ‘value:’, leading/trailing white-space is stripped. For example:

tag: Tier1 | tag: Tier2 | tag: Tier3
category: Sanity, Security & tag: -destructive

Note that multiple comma-separated values can be used as a syntactic sugar to shorten the filter notation:

tag: A, B, C ---> tag: A | tag: B | tag: C

Values should be provided as a dictionary of lists each describing the values against which the filter is to be matched. For example:

data = {tag: ["Tier1", "TIPpass"], category: ["Sanity"]}

Other types of dictionary values are converted into a string. A FilterError exception is raised when a dimension parsed from the filter is not found in the data dictionary. Set option ‘sensitive’ to False to enable case-insensitive matching. If ‘regexp’ option is True, regular expressions can be used in the filter values as well.

fmf.utils.get_cache_directory(create=True)

Return cache directory, created by this call if necessary

Cache directory is (first existing): - Value of FMF_CACHE_DIRECTORY environment variable - Value set by the last call of set_cache_directory() - $XDG_CACHE_HOME/fmf - ~/.cache/fmf

Raise GeneralError if it is not possible to create it.

fmf.utils.info(message, newline=True): Log provided info message to the standard error output

fmf.utils.invalidate_cache(): Force fetch next time cache is used regardless its age

fmf.utils.listed(items, singular=None, plural=None, max=None, quote='', join='and')

Convert an iterable into a nice, human readable list or description:

listed(range(1)) .................... 0
listed(range(2)) .................... 0 and 1
listed(range(3), join='or') ......... 0, 1 or 2
listed(range(3), quote='"') ......... "0", "1" and "2"
listed(range(4), max=3) ............. 0, 1, 2 and 1 more
listed(range(5), 'number', max=3) ... 0, 1, 2 and 2 more numbers
listed(range(6), 'category') ........ 6 categories
listed(7, "leaf", "leaves") ......... 7 leaves

If singular form is provided but max not set the description-only mode is activated as shown in the last two examples. Also, an int can be used in this case to get a simple inflection functionality.

fmf.utils.pluralize(singular=None): Naively pluralize words

fmf.utils.run(command, cwd=None, check_exit_code=True, env=None)

Run command and return a (stdout, stderr) tuple

:command as list (name, arg1, arg2…) :cwd path to directory where to run the command :check_exit_code raise CalledProcessError if exit code is non-zero :env dictionary of the environment variables for the command

fmf.utils.set_cache_directory(cache_directory): Set preferred cache directory

fmf.utils.set_cache_expiration(seconds): Seconds until cache expires

fmf.utils.split(values, separator=re.compile('[ ,]+'))

Convert space-or-comma-separated values into a single list

Common use case for this is merging content of options with multiple values allowed into a single list of strings thus allowing any of the formats below and converts them into [‘a’, ‘b’, ‘c’]:

--option a --option b --option c ... ['a', 'b', 'c']
--option a,b --option c ............ ['a,b', 'c']
--option 'a b c' ................... ['a b c']

Accepts both string and list. By default space and comma are used as value separators. Use any regular expression for custom separator.

cli

This is command line interface for the Flexible Metadata Format.

Available commands are:

fmf ls      List identifiers of available objects
fmf show    Show metadata of available objects
fmf init    Initialize a new metadata tree
fmf clean   Remove cache directory and its content

See online documentation for more details and examples:

http://fmf.readthedocs.io/

Check also help message of individual commands for the full list of available options.

class fmf.cli.Parser(arguments=None, path=None)

Command line options parser

clean(): Remove cache directory

command_clean(): Clean cache

command_init(): Initialize tree

command_ls(): List names

command_show(): Show metadata

options_formatting(): Formating options

options_select(): Select by name, filter

options_utils(): Utilities

show(brief=False): Show metadata for each path given

fmf.cli.main(arguments=None, path=None): Parse options, do what is requested