Pipes
A pipeline is a YAML list of steps, called pipes. Each step is the name of a pipe, optionally followed by options on the same line and arguments after the colon:
- load fail_on_error True:
- http://example.com/metadata.xml
- select
- publish: /tmp/metadata.xml
This page is generated from the docstrings of the pipes in pyff.builtins.
break
Break out of a pipeline.
This sets the ‘done’ request property to True which causes the pipeline to terminate at that point. The method name is ‘_break’ but the keyword is ‘break’ to avoid conflicting with python builtin methods.
Examples
- one
- two
- break
- unreachable
certreport
Generate a report of the certificates (optionally limited by expiration time or key size) found in the selection.
Examples
- certreport:
error_seconds: 0
warning_seconds: 864000
error_bits: 1024
warning_bits: 2048
For key size checking this will report keys with a size less than the size specified, defaulting to errors for keys smaller than 1024 bits and warnings for keys smaller than 2048 bits. It should be understood as the minimum key size for each report level, as such everything below will create report entries.
Remember that you need a ‘publish’ or ‘emit’ call after certreport in your plumbing to get useful output. PyFF ships with a couple of xslt transforms that are useful for turning metadata with certreport annotation into HTML.
check_xml_namespaces
Ensure that all namespaces are http, https or urn URIs. A namespace URI with any other scheme causes a MetadataException.
discojson
Return a discojuice-compatible json representation of the tree
- discojson
If the config.load_icons directive is set the icons will be returned from a (possibly persistent) local cache & converted to data: URIs
discojson_sp
Return a json representation of the trust information
- discojson_sp
The returned json doc will have the following structure.
The root is a dictionary, in which the keys are the entityID’s of the SP entities that have trust information in their metadata, and the values are a representation of that trust information.
For the XML structure of the trust information see the XML Schema in this repo at /src/pyff/schema/saml-metadata-trustinfo-v1.0.xsd.
For each SP with trust information, the representation of that information is as follows.
If there are MetadataSource elements, there will be a key ‘extra_md’ pointing to a dictionary of the metadata from those additional sources, with entityIDs as keys and entities (with the format provided by the discojson function above) as values.
Then there will be a key ‘profiles’ pointing to a dictionary in which the keys are the names of the trust profiles, and the values are json representations of those trust profiles.
Each trust profile will have the following keys.
If the trust profile includes a FallbackHandler element, there will be a key ‘fallback_handler’ pointing to a dict with 2 keys, ‘profile’ which by default is ‘href’, and handler which is a string, commonly a URL.
Then there will be an ‘entity’ key pointing to a list of representations of individual trusted/untrusted entities, each of them a dictionary, with 2 keys: ‘entity_id’ pointing to a string with the entityID, and ‘include’, pointing to a boolean.
Finally there will be a key ‘entities’ pointing to a list of representations of groups of trusted/untrusted entities, each of them a dictionary with 3 keys: a ‘match’ key pointing to the property of the entities by which they will be selected, by default ‘registrationAuthority’, a key ‘select’ with the value that will be used to select the ‘match’ property, and ‘include’, pointing to a boolean.
discojson_sp_attr
Return a json representation of the trust information
- discojson_sp_attr
SP Entities can carry trust information as a base64 encoded json blob as an entity attribute with name https://refeds.org/entity-selection-profile. The schema of this json is the same as the one produced above from XML with the pipe discojson_sp, and published at:
https://github.com/TheIdentitySelector/thiss-mdq/blob/master/trustinfo.schema.json
drop_xsi_type
Remove all xsi:type attributes from the tree.
dump
Print a representation of the entities set on stdout. Useful for testing.
emit
Returns a UTF-8 encoded representation of the working tree.
Renders the working tree as text and sets the digest of the tree as the ETag. If the tree has already been rendered as text by an earlier step the text is returned as utf-8 encoded unicode. The mimetype (ctype) will be set in the Content-Type HTTP response header.
Examples
- emit application/xml:
- break
end
Exit with optional error code and message.
Examples
- end
- unreachable
The optional code and message arguments set the exit code and a message printed on stdout:
- end:
code: 1
message: "no metadata to publish"
Warning This is very bad if used with pyffd - the server will stop running. If you just want to break out of the pipeline, use break instead.
filter
Refines the working document by applying a filter. The filter expression is a subset of the select semantics and syntax:
- filter:
- "!//md:EntityDescriptor[md:SPSSODescriptor]"
- "https://idp.example.com/shibboleth"
This would select all SPs and any entity with entityID “https://idp.example.com/shibboleth” from the current working document and return as the new working document. Filter also supports the “as <alias>” construction from select allowing new synthetic collections to be created from filtered documents.
finalize
Prepares the working document for publication/rendering.
Set Name, ID, cacheDuration and validUntil on the toplevel EntitiesDescriptor element of the working document. Unless explicitly provided the @Name is set from the request URI if the pipeline is executed in the pyFF server. The @ID is set to a string representing the current date/time and will be prefixed with the string provided, which defaults to ‘_’. The @cacheDuration element must be a valid xsd duration (eg PT5H for 5 hrs) and @validUntil can be either an absolute ISO 8601 time string or (more commonly) a relative time in the form
\+?([0-9]+d)?\s*([0-9]+h)?\s*([0-9]+m)?\s*([0-9]+s)?
For instance +45d 2m results in a time delta of 45 days and 2 minutes. The ‘+’ sign is optional.
If operating on a single EntityDescriptor then @Name is ignored (cf first).
Examples
- finalize:
cacheDuration: PT8H
validUntil: +10d
ID: pyff
first
If the working document is a single EntityDescriptor, strip the outer EntitiesDescriptor element and return it.
Sometimes (eg when running an MDX pipeline) it is usually expected that if a single EntityDescriptor is being returned then the outer EntitiesDescriptor is stripped. This method does exactly that.
fork
Make a copy of the working tree and process the arguments as a pipeline. This essentially resets the working tree and allows a new plumbing to run. Useful for producing multiple outputs from a single source.
Examples
- select # select all entities
- fork:
- certreport
- publish:
output: "/tmp/annotated.xml"
- fork:
- xslt:
stylesheet: tidy.xsl
- publish:
output: "/tmp/clean.xml"
The second fork in this example is strictly speaking not necessary since the main plumbing is still active but it may help to structure your plumbings this way.
Merging
Normally the result of the “inner” plumbing is discarded - unless published or emit:ed to a calling client in the case of the MDX server - but by adding ‘merge’ to the options with an optional ‘merge strategy’ the behaviour can be changed to merge the result of the inner pipeline back to the parent working document.
The default merge strategy is ‘replace_existing’ which replaces each EntityDescriptor found in the resulting document in the parent document (using the entityID as a pointer). Any python module path (‘a.mod.u.le:callable’) ending in a callable is accepted. If the path doesn’t contain a ‘:’ then it is assumed to reference one of the standard merge strategies in pyff.merge_strategies.
For instance the following block can be used to set an attribute on a single entity:
- fork merge:
- select: http://sp.example.com/shibboleth-sp
- setattr:
attribute: value
Note that unless you have a select statement before your fork merge you’ll be merging into an empty active document which with the default merge strategy of replace_existing will result in an empty active document. To avoid this do a select before your fork, thus:
- select
- fork merge:
- select: http://sp.example.com/shibboleth-sp
- setattr:
attribute: value
parsecopy
Due to a hard to find bug, fork which uses deepcopy can lose some namespaces. The parsecopy argument is a workaround. It uses a brute force serialisation and deserialisation to get around the bug.
- select # select all entities
- fork parsecopy:
- certreport
- publish:
output: "/tmp/annotated.xml"
- fork:
- xslt:
stylesheet: tidy.xsl
- publish:
output: "/tmp/clean.xml"
indent
Transform the working document using proper indentation. Requires lxml >= 4.5
Indent the working document.
Examples
- indent:
space: ' '
info
Dumps the working document on stdout. Useful for testing.
load
General-purpose resource fetcher.
Supports both remote and local resources. Fetching remote resources is done in parallel using threads.
Warning
When downloading remote files over HTTPS the TLS server certificate is not validated by default
(verify_tls is False). Metadata signatures are only checked if a verification is given per
resource, see below.
Note: Default behaviour is to ignore metadata files or entities in MD files that cannot be loaded
Options are put directly after “load”. E.g:
- load fail_on_error True filter_invalid False:
- http://example.com/some_remote_metadata.xml
- local_file.xml
- /opt/directory_containing_md_files/
Options
Defaults are marked with (*).
max_workers <5>: Number of parallel threads to use for loading MD filestimeout <120>: Socket timeout (seconds) when downloading filesvalidate <True*|False>: When true downloaded metadata files are validated (schema validation)fail_on_error <True|False*>: Control whether an error during download, parsing or (optional) validation of a MD file aborts processing of the pipeline. When true a failure aborts and causes pyff to exit with a non zero exit code. Otherwise errors are logged but ignored.filter_invalid <True*|False>: Controls validation behaviour. When true entities that fail validation are filtered, i.e. are not loaded. When false the entire metadata file is either loaded, or not.fail_on_errorcontrols whether failure to validate the entire MD file will abort processing of the pipeline.verify_tls <True|False*>: Controls the validation of the host’s TLS certificate on fetching the resources
Resources
Each resource in the list has the form
<url or path> [as <alias>] [verify <fingerprint|certificate file>] [via <clause>]* [cleanup <clause>]*
as: name the resource, e.g. to refer to it in later pipeline steps.verify: check the XML signature of the resource, using either a PEM certificate file or a SHA-1 fingerprint of the signing certificate. With a fingerprint, the certificate embedded in the signature is used and the fingerprint pins it.verifymay be repeated to support key rollovers; a signature by any one of the given certificates (or fingerprints) is accepted. Certificates and fingerprints cannot be mixed.via: name of awhenclause in the same pipeline that is applied to the loaded metadata, for example to remove entities withfork merge remove.cleanup: name of awhenclause used to pre-process the parsed metadata before it is validated.
log_entity
log the request id as it is processed (typically the entity_id)
map
loop over the entities in a selection
Examples
- map:
- ...statements...
Executes a set of statements in parallel (using a thread pool).
nodecountry
Sets eidas:NodeCountry
Transforms the working document by setting NodeCountry
Examples
- nodecountry:
country: XX
Normally this would be combined with the ‘merge’ feature of fork or in a cleanup pipeline to add attributes to the working document for later processing.
pick
Select a set of EntityDescriptor elements as a working document but don’t validate it.
Useful for testing. See select for more information about selecting the document.
pipe
Run the argument list as a pipeline.
Unlike fork, pipe does not copy the working document but instead operates on the current active document. The done request property is reset to False after the pipeline has been processed. This allows for a classical switch/case flow using the following construction:
- pipe:
- when a:
- one
- break
- when b:
- two
- break
In this case if ‘a’ is present in the request state, then ‘one’ will be executed and the ‘when b’ condition will not be tested at all. Note that at the topmost level the pipe is implicit and may be left out.
- pipe:
- one
- two
is equivalent to
- one
- two
print
Print whatever is in the active tree without transformation
Examples
- print:
output: "somewhere.foo"
Without output the tree is printed on stdout.
prune
Prune the active tree, removing all elements matching
Examples
- prune:
- .//{http://www.w3.org/2000/09/xmldsig#}Signature
This example would drop all Signature elements. Note the use of namespaces.
- prune:
- .//{http://www.w3.org/2000/09/xmldsig#}Signature[1]
This example would drop the first Signature element below each parent element.
pubinfo
Sets publication info extension on EntityDescription element
Transforms the working document by adding a PublicationInfo extension to its top-level element. An error is raised if one is already present.
Examples
- pubinfo:
publisher: http://publisher.example.org/
creation_instant (an ISO 8601 time string) is optional and defaults to the current time.
publish
Publish the working document in XML form.
Publish takes one argument: path to a file where the document tree will be written.
Examples
- publish: /tmp/idp.xml
The full set of options with their corresponding defaults:
- publish:
output: output
raw: false
pretty_print: false
urlencode_filenames: false
hash_link: false
update_store: true
ext: .xml
If output is an existing directory, publish will write the working tree to a filename in the directory based on the @entityID or @Name attribute. Unless ‘raw’ is set to true the working tree will be serialized to a string before writing, with minimal formatting if ‘pretty_print’ is true (see ‘indent’ action for more extensive control). If true, ‘hash_link’ will generate a symlink based on the hash id (sha1) for compatibility with MDQ. Unless false, ‘update_store’ will cause the the current store to be updated with the published artifact. Setting ‘ext’ allows control over the file extension.
read_xml
Make an XML file the working document.
Unlike load this does not put anything in the store and does not split the file into entities, and it does
not depend on select: the file is the working document, e.g. for signing a document that is not
SAML metadata. The file is parsed as it is and is not modified: xi:include elements are not expanded.
Takes a dict with a single key, file, the path to the XML file.
Examples
- read_xml:
file: /opt/metadata/mdsl.xml
- sign:
key: signer.key
cert: signer.crt
- emit application/xml
Possible extensions
Expand xi:include elements, relative to the file, with an
xincludeoption.Ensure the root element, e.g. with a
rootoption in{namespace}nameform, to fail early when the file is not the kind of document the rest of the pipeline expects.Make the working document work with
finalize(and other pipes that assume entity metadata), to support more file types than SAML metadata.Support
mdsl(eIDAS MetadataServiceList) as a file type. The file type would ensure the root element and tellfinalizewhich attributes the document has. For an MDSL that isIssueDateandNextUpdateinstead ofvalidUntilandcacheDuration, which the schema does not allow on this root element. The pipeline could then look like this:- read_xml: file: /opt/metadata/mdsl.xml type: mdsl - finalize: validUntil: PT5M # sets NextUpdate, and IssueDate to the current time - sign: key: signer.key cert: signer.crt - emit application/xml
reginfo
Sets registration info extension on EntityDescription element
Transforms the working document by setting the specified attribute on all of the EntityDescriptor elements of the active document.
Examples
- reginfo:
authority: http://registrar.example.org/
policy:
en: http://registrar.example.org/policy.html
policy is optional and maps a language code to the registration policy URL.
select
Select a set of EntityDescriptor elements as the working document.
Select picks and expands elements (with optional filtering) from the active repository you setup using calls
to load. See pyff.repo.MDRepository.lookup() for a description of the syntax for selectors.
Examples
- select
This would select all entities in the active repository.
- select: "/var/local-metadata"
This would select all entities found in the directory /var/local-metadata. You must have a call to load to
load entities from this directory before the select statement.
- select: "/var/local-metadata!//md:EntityDescriptor[md:IDPSSODescriptor]"
This would selects all IdPs from /var/local-metadata
- select: "!//md:EntityDescriptor[md:SPSSODescriptor]"
This would select all SPs
Select statements are not cumulative - a select followed by another select in the plumbing resets the working documents to the result of the second select.
Most statements except load depend on having a select somewhere in your plumbing and will
stop the plumbing if the current working document is empty. For instance, running
- select: "!//md:EntityDescriptor[md:SPSSODescriptor]"
would terminate the plumbing at select if there are no SPs in the local repository. This is useful in combination with fork for handling multiple cases in your plumbings.
Options are put directly after “select”. E.g:
- select as /foo-2.0 dedup True: "!//md:EntityDescriptor[md:IDPSSODescriptor]"
Options
Defaults are marked with (*).
as <name>: The ‘as’ keyword allows a select to be stored as an alias in the local repository. For instance- select as /foo-2.0: "!//md:EntityDescriptor[md:IDPSSODescriptor]"
would allow you to use /foo-2.0.json to refer to the JSON-version of all IdPs in the current repository. Note that you should not include an extension in your “as foo-bla-something” since that would make your alias invisible for anything except the corresponding mime type.
dedup <True*|False>: Whether to deduplicate the results by entityID.Note: When select is used after a load pipe with more than one source, if dedup is set to True and there are entity properties that may differ from one source to another, these will be squashed rather than merged.
setattr
Sets entity attributes on the working document
Transforms the working document by setting the specified attribute on all of the EntityDescriptor elements of the active document.
Examples
- setattr:
attr1: value1
attr2: value2
Normally this would be combined with the ‘merge’ feature of fork to add attributes to the working document for later processing.
sign
Sign the working document.
Sign expects a single dict with at least a ‘key’ key and optionally a ‘cert’ key. The ‘key’ argument references either a PKCS#11 uri or the filename containing a PEM-encoded non-password protected private RSA key. The ‘cert’ argument may be empty in which case the cert is looked up using the PKCS#11 token, or may point to a file containing a PEM-encoded X.509 certificate.
When several initialized tokens are visible, select one with token_label and optional token_serial,
or with slot_id. Token selection requires pybergshamra 0.8.1 or newer. The slot may alternatively be
supplied using the historical :slot suffix in the key URI. Slot and token selectors are mutually exclusive.
PKCS11 URIs
A pkcs11 URI has the form
pkcs11://<absolute path to SO/DLL>[:slot]/<object label>[?pin=<pin>]
The pin parameter can be used to point to an environment variable containing the pin: “env:<ENV variable>”. By default pin is “env:PYKCS11PIN” which tells sign to use the pin found in the PYKCS11PIN environment variable. Authentication and signing are serialized per token within the worker process.
Examples
- sign:
key: pkcs11:///usr/lib/libCryptoki2_64.so/sctest2
token_label: sc_ha
token_serial: "11429933786539" # optional when the label is unique
cert: /etc/credentials/sc-test-md-signer.crt
This signs with the key-object label sctest2 on the token labelled sc_ha. The PKCS#11 client library
still reads vendor connectivity and HA settings from its normal configuration, such as /etc/Chrystoki.conf.
Note that you may need to run pyff with env PYKCS11PIN=<pin> …. for this to work. Consult the documentation
of your PKCS#11 module to find out about any other configuration you may need.
- sign:
key: signer.key
cert: signer.crt
This example signs the document using the plain key and cert found in the signer.key and signer.crt files.
signcerts
Logs the fingerprints of the signing certs found in the current working tree.
Useful for testing.
Examples
- signcerts
sort
Sorts the working entities by the value returned by the given xpath. By default, entities are sorted by ‘entityID’ when the ‘order_by [xpath]’ option is omitted and otherwise as second criteria. Entities where no value exists for a given xpath are sorted last.
Options are put directly after “sort”. E.g:
- sort order_by ./md:IDPSSODescriptor/md:Extensions/mdui:UIInfo/mdui:DisplayName
Options
order_by <xpath>: xpath expression selecting the value used for sorting the entities.
stats
Display statistics about the current working document.
Examples
- stats
store
Save the working document as separate files
Split the working document into EntityDescriptor-parts and save in directory/sha1(@entityID).xml. Note that this does not erase files that may already be in the directory. If you want a “clean” directory, remove it before you call store.
summary
Display a summary of the repository
then
Call a named ‘when’ clause and return - akin to macro invocations for pyFF
validate
Validate the working document
Generate an exception unless the working tree validates. Validation is done automatically during publication and loading of metadata so this call is seldom needed.
when
Conditionally execute part of the pipeline.
The inner pipeline is executed if the at least one of the condition values is present for the specified key in the request state.
Examples
- when foo:
- something
- when bar bill:
- other
The condition operates on the state: if ‘foo’ is present in the state (with any value), then the something branch is followed. If ‘bar’ is present in the state with the value ‘bill’ then the other branch is followed.
xslt
Transform the working document using an XSLT file.
Apply an XSLT stylesheet to the working document. The xslt pipe takes a set of keyword arguments. The only required argument is ‘stylesheet’ which identifies the xslt resource. This is looked up either in the package or as a user-supplied file. The rest of the keyword arguments are made available as string parameters to the XSLT transform.
Examples
- xslt:
stylesheet: foo.xsl
x: foo
y: bar