Modules
Flexible Metadata Format
- class fmf.Context(*args, **kwargs)
Represents https://fmf.readthedocs.io/en/latest/context.html
- property case_sensitive: bool
- evaluate(expression)
- matches(rule)
Does the rule match the current Context?
We have three outcomes: Yes, No and CannotDecide
CannotDecide and True == True and CannotDecide == CannotDecide CannotDecide and False == False and CannotDecide == False CannotDecide or True == True or CannotDecide == True CannotDecide or False == False or CannotDecide == CannotDecide
- Parameters:
rule (str | bool) – Single rule to decide
- Return type:
bool
- Raises:
CannotDecide – Impossible to decide the rule wrt current Context, e.g. dimension is missing
InvalidRule – Syntax error in the rule
- operator_map = {'!=': <function Context._op_not_eq>, '<': <function Context._op_less>, '<=': <function Context._op_less_or_equal>, '==': <function Context._op_eq>, '>': <function Context._op_greater>, '>=': <function Context._op_greater_or_equal>, 'is defined': <function Context._op_defined>, 'is not defined': <function Context._op_not_defined>, '~!=': <function Context._op_minor_not_eq>, '~<': <function Context._op_minor_less>, '~<=': <function Context._op_minor_less_or_eq>, '~=': <function Context._op_minor_eq>, '~>': <function Context._op_minor_greater>, '~>=': <function Context._op_minor_greater_or_equal>}
- static parse_rule(rule)
Parses rule into expressions
Expression is a tuple of dimension_name, operator_str, list of value objects. Parsed rule is nested list of expression from OR and AND operators. Items of the first dimension are in OR relation. Items in the second dimension are in AND relation.
expr_1 and expr_2 or expr_3 is returned as [[expr_1, expr_2], expr_3] expr_4 or expr_5 is returned as [[expr_4], [expr_5]] expr_6 and expr_7 is returned as [[expr_6, expr_7]]
- Parameters:
rule (str | bool) – rule to parse
- Returns:
nested list of expressions from the rule
- Raises:
InvalidRule – Syntax error in the rule
- static parse_value(value)
Single place to convert to ContextValue
- re_and_split = re.compile('\\band\\b')
- re_boolean = re.compile('(true|false)')
- re_expression_double = re.compile('(\\w+)\\s*(is defined|is not defined)')
- re_expression_triple = re.compile('(\\w+)\\s*(!=|>=|<=|~=|~>=|~>|~<|<|~!=|==|>|~<=)\\s*([^=].*)')
- re_or_split = re.compile('\\bor\\b')
- static split_expression(expression)
Split expression to dimension name, operator and values
When operator doesn’t have right side, None is returned instead of the list of values.
- Parameters:
expression (str) – expression to split
- Raises:
InvalidRule – When expression cannot be split, e.g. syntax error
- Returns:
tuple(dimension name, operator, list of values)
- Return type:
tuple(str|None, str|bool, list|None)
- static split_rule_to_groups(rule)
Split rule into nested lists, no real parsing
expr0 and expr1 or expr2 is split into [[expr0, expr1], [expr2]]
- Parameters:
rule (str) – rule to split
- Raises:
InvalidRule – Syntax error in the rule
- class fmf.Tree(data, name=None, parent=None)
Metadata Tree
- adjust(context, key='adjust', undecided='skip', case_sensitive=True, decision_callback=None)
Adjust tree data based on provided context and rules
The ‘context’ should be an instance of the fmf.context.Context class describing the environment context. By default, the key ‘adjust’ of each node is inspected for possible rules that should be applied. Provide ‘key’ to use a custom key instead.
Optional ‘undecided’ parameter can be used to specify what should happen when a rule condition cannot be decided because context dimension is not defined. By default, such rules are skipped. In order to raise the fmf.context.CannotDecide exception in such cases use undecided=’raise’.
Optional ‘case_sensitive’ parameter can be used to specify if the context dimension values should be case-sensitive when matching the rules. By default, values are case-sensitive.
Optional ‘decision_callback’ callback would be called for every adjust rule inspected, with three arguments: current fmf node, current adjust rule, and whether it was applied or not.
- child(name, data, source=None)
Create or update child with given data
- climb(whole=False)
Climb through the tree (iterate leaf/all nodes)
- property commit
Commit hash if tree grows under a git repo, False otherwise
Return current commit hash if the metadata tree root is located under a git repository. For metadata initialized from a dict or local directory with no git repo ‘False’ is returned instead.
- copy()
Create and return a deep copy of the node and its subtree
It is possible to call copy() on any node in the tree, not only on the tree root node. Note that in that case, parent node and the rest of the tree attached to it is not copied in order to save memory.
- find(name)
Find node with given name
- get(name=None, default=None)
Get attribute value or return default
Whole data dictionary is returned when no attribute provided. Supports direct values retrieval from deep dictionaries as well. Dictionary path should be provided as list. The following two examples are equal:
tree.data[‘hardware’][‘memory’][‘size’] tree.get([‘hardware’, ‘memory’, ‘size’])
However the latter approach will also correctly handle providing default value when any of the dictionary keys does not exist.
- grow(path)
Grow the metadata tree for the given directory path
Note: For each path, grow() should be run only once. Growing the tree from the same path multiple times with attribute adding using the “+” sign leads to adding the value more than once!
- inherit()
Apply inheritance
- static init(path)
Create metadata tree root under given path
- merge(parent=None)
Merge parent data
- static node(reference)
Return Tree node referenced by the fmf identifier
Keys supported in the reference:
url …. git repository url (optional) ref …. branch, tag or commit (default branch if not provided) path … metadata tree root (‘.’ by default) name … tree node name (‘/’ by default)
See the documentation for the full fmf id specification: https://fmf.readthedocs.io/en/latest/concept.html#identifiers Raises ReferenceError if referenced node does not exist.
- prune(whole=False, keys=None, names=None, filters=None, conditions=None, sources=None)
Filter tree nodes based on given criteria
- show(brief=False, formatting=None, values=None)
Show metadata
- update(data)
Update metadata, handle virtual hierarchy
- validate(schema, schema_store=None)
Validate node data with given JSON Schema and schema references.
schema_store is a dict of schema references and their content.
Return a named tuple utils.JsonSchemaValidationResult with the following two items:
result … boolean representing the validation result errors … A list of validation errors
Raises utils.JsonSchemaError if the supplied schema was invalid.
- fmf.filter(filter, data, sensitive=True, regexp=False)
Return true if provided filter matches given dictionary of values
Filter supports disjunctive normal form with ‘|’ used for OR, ‘&’ for AND and ‘-’ for negation. Individual values are prefixed with ‘value:’, leading/trailing white-space is stripped. For example:
tag: Tier1 | tag: Tier2 | tag: Tier3 category: Sanity, Security & tag: -destructive
Note that multiple comma-separated values can be used as a syntactic sugar to shorten the filter notation:
tag: A, B, C ---> tag: A | tag: B | tag: C
Values should be provided as a dictionary of lists each describing the values against which the filter is to be matched. For example:
data = {tag: ["Tier1", "TIPpass"], category: ["Sanity"]}
Other types of dictionary values are converted into a string. A FilterError exception is raised when a dimension parsed from the filter is not found in the data dictionary. Set option ‘sensitive’ to False to enable case-insensitive matching. If ‘regexp’ option is True, regular expressions can be used in the filter values as well.
base
Base Metadata Classes
- class fmf.base.AdjustCallback(*args, **kwargs)
A callback for per-rule notifications made by Tree.adjust()
Function be be called for every rule inspected by adjust(). It will be given three arguments: fmf tree being inspected, current adjust rule, and whether the rule was skipped (
None
), applied (True
) or not applied (False
).
- class fmf.base.Tree(data, name=None, parent=None)
Metadata Tree
- adjust(context, key='adjust', undecided='skip', case_sensitive=True, decision_callback=None)
Adjust tree data based on provided context and rules
The ‘context’ should be an instance of the fmf.context.Context class describing the environment context. By default, the key ‘adjust’ of each node is inspected for possible rules that should be applied. Provide ‘key’ to use a custom key instead.
Optional ‘undecided’ parameter can be used to specify what should happen when a rule condition cannot be decided because context dimension is not defined. By default, such rules are skipped. In order to raise the fmf.context.CannotDecide exception in such cases use undecided=’raise’.
Optional ‘case_sensitive’ parameter can be used to specify if the context dimension values should be case-sensitive when matching the rules. By default, values are case-sensitive.
Optional ‘decision_callback’ callback would be called for every adjust rule inspected, with three arguments: current fmf node, current adjust rule, and whether it was applied or not.
- child(name, data, source=None)
Create or update child with given data
- climb(whole=False)
Climb through the tree (iterate leaf/all nodes)
- property commit
Commit hash if tree grows under a git repo, False otherwise
Return current commit hash if the metadata tree root is located under a git repository. For metadata initialized from a dict or local directory with no git repo ‘False’ is returned instead.
- copy()
Create and return a deep copy of the node and its subtree
It is possible to call copy() on any node in the tree, not only on the tree root node. Note that in that case, parent node and the rest of the tree attached to it is not copied in order to save memory.
- find(name)
Find node with given name
- get(name=None, default=None)
Get attribute value or return default
Whole data dictionary is returned when no attribute provided. Supports direct values retrieval from deep dictionaries as well. Dictionary path should be provided as list. The following two examples are equal:
tree.data[‘hardware’][‘memory’][‘size’] tree.get([‘hardware’, ‘memory’, ‘size’])
However the latter approach will also correctly handle providing default value when any of the dictionary keys does not exist.
- grow(path)
Grow the metadata tree for the given directory path
Note: For each path, grow() should be run only once. Growing the tree from the same path multiple times with attribute adding using the “+” sign leads to adding the value more than once!
- inherit()
Apply inheritance
- static init(path)
Create metadata tree root under given path
- merge(parent=None)
Merge parent data
- static node(reference)
Return Tree node referenced by the fmf identifier
Keys supported in the reference:
url …. git repository url (optional) ref …. branch, tag or commit (default branch if not provided) path … metadata tree root (‘.’ by default) name … tree node name (‘/’ by default)
See the documentation for the full fmf id specification: https://fmf.readthedocs.io/en/latest/concept.html#identifiers Raises ReferenceError if referenced node does not exist.
- prune(whole=False, keys=None, names=None, filters=None, conditions=None, sources=None)
Filter tree nodes based on given criteria
- show(brief=False, formatting=None, values=None)
Show metadata
- update(data)
Update metadata, handle virtual hierarchy
- validate(schema, schema_store=None)
Validate node data with given JSON Schema and schema references.
schema_store is a dict of schema references and their content.
Return a named tuple utils.JsonSchemaValidationResult with the following two items:
result … boolean representing the validation result errors … A list of validation errors
Raises utils.JsonSchemaError if the supplied schema was invalid.
utils
Logging, config, constants & utilities
- class fmf.utils.Coloring(*args, **kwargs)
Coloring configuration
- MODES = ['COLOR_OFF', 'COLOR_ON', 'COLOR_AUTO']
- enabled()
True if coloring is currently enabled
- get()
Get the current color mode
- set(mode=None)
Set the coloring mode
If enabled, some objects (like case run Status) are printed in color to easily spot failures, errors and so on. By default the feature is enabled when script is attached to a terminal. Possible values are:
COLOR=0 ... COLOR_OFF .... coloring disabled COLOR=1 ... COLOR_ON ..... coloring enabled COLOR=2 ... COLOR_AUTO ... if terminal attached (default)
Environment variable COLOR can be used to set up the coloring to the desired mode without modifying code.
- exception fmf.utils.FetchError
Fatal error in helper command while fetching
- exception fmf.utils.FileError
File reading error
- exception fmf.utils.FilterError
Missing data when filtering
- exception fmf.utils.FormatError
Metadata format error
- exception fmf.utils.GeneralError
General error
- exception fmf.utils.JsonSchemaError
Invalid JSON Schema
- class fmf.utils.JsonSchemaValidationResult(result: bool, errors: List[Any])
Represents JSON Schema validation result
- errors: List[Any]
Alias for field number 1
- result: bool
Alias for field number 0
- class fmf.utils.Logging(name='fmf')
Logging Configuration
- COLORS = {4: 'magenta', 7: 'cyan', 10: 'green', 20: 'blue', 30: 'yellow', 40: 'red'}
- class ColoredFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)
Custom color formatter for logging
- format(record)
Format the specified record as text.
The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.
- LEVELS = ['CRITICAL', 'DEBUG', 'ERROR', 'FATAL', 'INFO', 'NOTSET', 'WARN', 'WARNING']
- MAPPING = {0: 30, 1: 20, 2: 10, 3: 7, 4: 4, 5: 1}
- get()
Get the current log level
- set(level=None)
Set the default log level
If the level is not specified environment variable DEBUG is used with the following meaning:
DEBUG=0 ... LOG_WARN (default) DEBUG=1 ... LOG_INFO DEBUG=2 ... LOG_DEBUG DEBUG=3 ... LOG_CACHE DEBUG=4 ... LOG_DATA DEBUG=5 ... LOG_ALL (log all messages)
- exception fmf.utils.MergeError
Unable to merge data between parent and child
- exception fmf.utils.ReferenceError
Referenced tree node cannot be found
- exception fmf.utils.RootError
Metadata tree root missing
- fmf.utils.clean_cache_directory()
Delete used cache directory if it exists
- fmf.utils.color(text, color=None, background=None, light=False, enabled='auto')
Return text in desired color if coloring enabled
Available colors: black red green yellow blue magenta cyan white. Alternatively color can be prefixed with “light”, e.g. lightgreen.
- fmf.utils.default_branch(repository, remote='origin')
Detect default branch from given local git repository
- fmf.utils.dict_to_yaml(data, width=None, sort=False)
Convert dictionary into yaml
- fmf.utils.evaluate(expression, data, _node=None)
Evaluate arbitrary Python expression against given data
Expects data dictionary which will be used to populate local namespace. Used to provide flexible conditions for filtering.
- fmf.utils.fetch(url, ref=None, destination=None, env=None)
Deprecated: Use
fetch_repo()
instead
- fmf.utils.fetch_repo(url, ref=None, destination=None, env=None)
Fetch remote git repository and return local directory
Fetch git repository from provided url into a local cache directory, checkout requested ref and return path to the repo. If no ref is provided, the default branch from the origin is used. If destination directory is provided, it should not exist or needs to be empty. Use dictionary env to set environment variables for git calls.
Raises FetchError upon failure with the original exception included.
- fmf.utils.fetch_tree(url, ref=None, path='.')
Get initialized Tree from a remote git repository
url …. git repository url (required) ref …. branch, tag or commit (default branch if None) path … metadata tree root (default to ‘.’)
See
fmf.base.Tree.node()
to canonical default values.Remote repository is cached locally (see
get_cache_directory()
), local directory with cache is locked during reading.Raises GeneralError when lock couldn’t be acquired.
- fmf.utils.filter(filter, data, sensitive=True, regexp=False)
Return true if provided filter matches given dictionary of values
Filter supports disjunctive normal form with ‘|’ used for OR, ‘&’ for AND and ‘-’ for negation. Individual values are prefixed with ‘value:’, leading/trailing white-space is stripped. For example:
tag: Tier1 | tag: Tier2 | tag: Tier3 category: Sanity, Security & tag: -destructive
Note that multiple comma-separated values can be used as a syntactic sugar to shorten the filter notation:
tag: A, B, C ---> tag: A | tag: B | tag: C
Values should be provided as a dictionary of lists each describing the values against which the filter is to be matched. For example:
data = {tag: ["Tier1", "TIPpass"], category: ["Sanity"]}
Other types of dictionary values are converted into a string. A FilterError exception is raised when a dimension parsed from the filter is not found in the data dictionary. Set option ‘sensitive’ to False to enable case-insensitive matching. If ‘regexp’ option is True, regular expressions can be used in the filter values as well.
- fmf.utils.get_cache_directory(create=True)
Return cache directory, created by this call if necessary
Cache directory is (first existing): - Value of FMF_CACHE_DIRECTORY environment variable - Value set by the last call of set_cache_directory() - $XDG_CACHE_HOME/fmf - ~/.cache/fmf
Raise GeneralError if it is not possible to create it.
- fmf.utils.info(message, newline=True)
Log provided info message to the standard error output
- fmf.utils.invalidate_cache()
Force fetch next time cache is used regardless its age
- fmf.utils.listed(items, singular=None, plural=None, max=None, quote='', join='and')
Convert an iterable into a nice, human readable list or description:
listed(range(1)) .................... 0 listed(range(2)) .................... 0 and 1 listed(range(3), join='or') ......... 0, 1 or 2 listed(range(3), quote='"') ......... "0", "1" and "2" listed(range(4), max=3) ............. 0, 1, 2 and 1 more listed(range(5), 'number', max=3) ... 0, 1, 2 and 2 more numbers listed(range(6), 'category') ........ 6 categories listed(7, "leaf", "leaves") ......... 7 leaves
If singular form is provided but max not set the description-only mode is activated as shown in the last two examples. Also, an int can be used in this case to get a simple inflection functionality.
- fmf.utils.pluralize(singular=None)
Naively pluralize words
- fmf.utils.run(command, cwd=None, check_exit_code=True, env=None)
Run command and return a (stdout, stderr) tuple
:command as list (name, arg1, arg2…) :cwd path to directory where to run the command :check_exit_code raise CalledProcessError if exit code is non-zero :env dictionary of the environment variables for the command
- fmf.utils.set_cache_directory(cache_directory)
Set preferred cache directory
- fmf.utils.set_cache_expiration(seconds)
Seconds until cache expires
- fmf.utils.split(values, separator=re.compile('[ ,]+'))
Convert space-or-comma-separated values into a single list
Common use case for this is merging content of options with multiple values allowed into a single list of strings thus allowing any of the formats below and converts them into [‘a’, ‘b’, ‘c’]:
--option a --option b --option c ... ['a', 'b', 'c'] --option a,b --option c ............ ['a,b', 'c'] --option 'a b c' ................... ['a b c']
Accepts both string and list. By default space and comma are used as value separators. Use any regular expression for custom separator.
cli
This is command line interface for the Flexible Metadata Format.
Available commands are:
fmf ls List identifiers of available objects
fmf show Show metadata of available objects
fmf init Initialize a new metadata tree
fmf clean Remove cache directory and its content
See online documentation for more details and examples:
Check also help message of individual commands for the full list of available options.
- class fmf.cli.Parser(arguments=None, path=None)
Command line options parser
- clean()
Remove cache directory
- command_clean()
Clean cache
- command_init()
Initialize tree
- command_ls()
List names
- command_show()
Show metadata
- options_formatting()
Formating options
- options_select()
Select by name, filter
- options_utils()
Utilities
- show(brief=False)
Show metadata for each path given
- fmf.cli.main(arguments=None, path=None)
Parse options, do what is requested