.. include:: ../global.inc
.. |br| raw:: html
#######################
Ruffus Decorators
#######################
.. seealso::
:ref:`Indicator objects `
.. _decorators:
.. toctree::
:maxdepth: 1
:name: core_toc
:caption: Core
:hidden:
originate.rst
split.rst
transform.rst
merge.rst
.. toctree::
:maxdepth: 1
:name: advanced_toc
:caption: Advanced
:hidden:
subdivide.rst
transform_ex.rst
collate.rst
graphviz.rst
mkdir.rst
jobs_limit.rst
posttask.rst
active_if.rst
follows.rst
.. toctree::
:maxdepth: 1
:name: combinatorics_toc
:caption: Combinatorics
:hidden:
product.rst
permutations.rst
combinations.rst
combinations_with_replacement.rst
.. toctree::
:maxdepth: 1
:name: esoteric_toc
:caption: Esoteric
:hidden:
files_ex.rst
check_if_uptodate.rst
parallel.rst
.. toctree::
:maxdepth: 1
:name: deprecated_toc
:caption: Deprecated
:hidden:
files.rst
files_re.rst
=============================================
*Core*
=============================================
.. csv-table::
:header: "Decorator", "Examples"
:widths: 400, 600,1
"**@originate** (:ref:`Summary ` / :ref:`Manual `)
- Creates (originates) a set of starting file from scratch (*ex nihilo*!)
- Only called to create files which do not exist.
- Each item in ``output`` is created by a separate job.
", "
* :ref:`@originate ` ( ``output``, [``extras``,...] )
\
", ""
"**@split** (:ref:`Summary ` / :ref:`Manual `)
- Splits a single input into multiple output
- Globs in ``output`` can specify an indeterminate number of files.
", "
* :ref:`@split ` ( ``input``, ``output``, [``extras``,...] )
\
", ""
"**@transform** (:ref:`Summary ` / :ref:`Manual `)
- | Applies the task function to
| transform input data to output.
", "
* :ref:`@transform ` ( ``input``, :ref:`suffix `\ () , ``output``, [``extras``,...] )
\
* :ref:`@transform ` ( ``input``, :ref:`regex `\ () , ``output``, [``extras``,...] )
\
* :ref:`@transform ` ( ``input``, :ref:`formatter `\ () , ``output``, [``extras``,...] )
\
", ""
"**@merge** (:ref:`Summary ` / :ref:`Manual `)
- Merges multiple input files into a single output.
", "
* :ref:`@merge ` (``input``, ``output``, [``extras``,...] )
\
", ""
.. _decorators.combinatorics:
=============================================
*Combinatorics*
=============================================
.. csv-table::
:header: "Decorator", "Examples"
:widths: 400, 600,1
"**@product** (:ref:`Summary ` / :ref:`Manual `)
- Generates the **product** between sets of input, i.e. all vs all comparisons.
", "
* :ref:`@product ` ( ``input``, :ref:`formatter `\ () ,*[* ``input``, :ref:`formatter `\ (), ``output``, [``extras``,...] )
\
", ""
"**@permutations** (:ref:`Summary ` / :ref:`Manual `)
- Generates the **permutations**, between all elements of a set of **Input**
- Analogous to the python `itertools.permutations `__
- permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC
", "
* :ref:`@permutations ` ( ``input``, :ref:`formatter `\ (), ``tuple_size``, ``output``, [``extras``,...] )
\
", ""
"**@combinations** (:ref:`Summary ` / :ref:`Manual `)
- Generates the **permutations**, between all elements of a set of **Input**
- Analogous to the python `itertools.combinations `__
- combinations('ABCD', 3) --> ABC ABD ACD BCD
- Generates the **combinations**, between all the elements of a set of **Input**:
i.e. r-length tuples of *input* elements with no repeated elements (**A A**)
and where order of the tuples is irrelevant
(either **A B** or **B A**, not both).
", "
* :ref:`@combinations ` ( ``input``, :ref:`formatter `\ (), ``tuple_size``, ``output``, [``extras``,...] )
\
", ""
"**@combinations_with_replacement** (:ref:`Summary ` / :ref:`Manual `)
- Generates the **permutations**, between all the elements of a set of **Input**
- Analogous to the python `itertools.permutations `__
- combinations('ABCD', 3) --> ABC ABD ACD BCD
- Generates the **combinations_with_replacement**,
between all the elements of a set of **Input**:
i.e. r-length tuples of *input* elements with no repeated elements (**A A**)
and where order of the tuples is irrelevant
(either **A B** or **B A**, not both).
", "
* :ref:`@combinations_with_replacement ` ( ``input``, :ref:`formatter `\ (), ``tuple_size``, ``output``, [``extras``,...] )
\
", ""
=============================================
*Advanced*
=============================================
.. csv-table::
:header: "Decorator", "Examples"
:widths: 400, 600,1
"**@subdivide** (:ref:`Summary ` / :ref:`Manual `)
- Subdivides each *input* into multiple *Outputs*.
- The number of *output* can be determined at runtime using globs.
- **Many to Even More** operator.
- Do not use **split** as a synonym for **subdivide**.
", "
* :ref:`@subdivide ` ( ``input``, :ref:`regex `\ () , [ :ref:`inputs ` | :ref:`add_inputs `\ *(*\ ``input_pattern``\ *)*\ , ] ``output``, [``extras``,...] )
\
* :ref:`@subdivide ` ( ``input``, :ref:`formatter `\ *(*\ [] *)*\ , [ :ref:`inputs ` | :ref:`add_inputs `\ *(*\ ``input_pattern``\ *)*\ , ] ``output``, [``extras``,...] )
\
", ""
"**@transform** (:ref:`Summary ` / :ref:`Manual `)
- Generates both **input** & **output** from regular expressions
- Useful for adding additional file dependencies
", "
* :ref:`@transform ` ( ``input``, :ref:`regex `\ () , [ :ref:`inputs ` | :ref:`add_inputs `\ *(*\ ``input_pattern``\ *)*\ , ] ``output``, [``extras``,...] )
\
* :ref:`@transform ` ( ``input``, :ref:`formatter `\ () , [ :ref:`inputs ` | :ref:`add_inputs `\ *(*\ ``input_pattern``\ *)*\ , ] ``output``, [``extras``,...] )
\
", ""
"**@collate** (:ref:`Summary ` / :ref:`Manual `)
- Groups multiple input using regular expression matching.
- Multiple input generating identical output are collated together.
", "
* :ref:`@collate ` (``input``, :ref:`regex `\ () , ``output``, [``extras``,...] )
\
* :ref:`@collate ` (``input``, :ref:`regex `\ () , :ref:`inputs ` | :ref:`add_inputs `\ *(*\ ``input_pattern``\ *)*\ , ``output``, [``extras``,...] )
\
* :ref:`@collate ` (``input``, :ref:`formatter `\ () , ``output``, [``extras``,...] )
\
* :ref:`@collate ` (``input``, :ref:`formatter `\ () , :ref:`inputs ` | :ref:`add_inputs `\ *(*\ ``input_pattern``\ *)*\ , ``output``, [``extras``,...] )
\
", ""
"**@follows** (:ref:`Summary ` / :ref:`Manual `)
- Indicates task dependency
- optional :ref:`mkdir ` prerequisite (:ref:`see Manual `)
", "
* :ref:`@follows ` ( ``task1``, ``'task2'`` ))
\
* :ref:`@follows ` ( ``task1``, :ref:`mkdir `\ ( ``'my/directory/'`` ))
\
", ""
"**@posttask** (:ref:`Summary ` / :ref:`Manual `)
- Calls function after task completes
- Optional :ref:`touch_file ` indicator (:ref:`Manual `)
", "
* :ref:`@posttask ` ( ``completion_signal_func`` )
\
* :ref:`@posttask ` (:ref:`touch_file `\ ( ``'task1.done'`` ))
\
", ""
"**@active_if** (:ref:`Summary ` / :ref:`Manual `)
- Switches tasks on and off at run time
- Evaluated each time you call
* :ref:`pipeline_run(...) `,
* :ref:`pipeline_printout(...) ` or
* :ref:`pipeline_printout_graph(...) `
- Dormant tasks behave as if they are :
* up to date and
* have no output.
", "
* :ref:`@active_if ` ( ``on_or_off1``, ``[on_or_off2, ...]`` )
\
", ""
"**@jobs_limit** (:ref:`Summary ` / :ref:`Manual `)
- Limits the amount of multiprocessing for the specified task
- Ensures that fewer than N jobs are run in parallel for this task
- Overrides ``multiprocess`` parameter in :ref:`pipeline_run(...) `
", "
* :ref:`@jobs_limit ` ( ``NUMBER_OF_JOBS_RUNNING_CONCURRENTLY`` )
\
", ""
"**@mkdir** (:ref:`Summary ` / :ref:`Manual `)
- Generates paths for `os.makedirs `__
", "
* :ref:`@mkdir ` ( ``input``, :ref:`suffix `\ () , ``output`` )
\
* :ref:`@mkdir ` ( ``input``, :ref:`regex `\ () , ``output`` )
\
* :ref:`@mkdir ` ( ``input``, :ref:`formatter `\ () , ``output``)
\
", ""
"**@graphviz** (:ref:`Summary ` / :ref:`Manual `)
- Customise the task graphics in
flowcharts from :ref:`pipeline_printout_graph(...) `
", "
* :ref:`@graphviz ` ( ``graphviz_parameter = XXX``, ``[graphviz_parameter2 = YYY ...]``)
\
", ""
=============================================
*Esoteric!*
=============================================
.. csv-table::
:header: "Decorator", "Examples"
:widths: 400, 600,1
"**@files** (:ref:`Summary ` / :ref:`Manual `)
- I/O parameters
- skips up-to-date jobs
- Should use :ref:`@transform ` etc instead
", "
* :ref:`@files `\ ( ``parameter_list`` )
\
* :ref:`@files `\ ( ``parameter_generating_function`` )
\
* :ref:`@files ` ( ``input_file``, ``output_file``, ``other_params``, ... )
\
", ""
"**@parallel** (:ref:`Summary ` / :ref:`Manual `)
- By default, does not check if jobs are up to date
- Best used in conjuction with :ref:`@check_if_uptodate `
", "
* :ref:`@parallel ` ( ``parameter_list`` )
\
* :ref:`@parallel ` ( ``parameter_generating_function`` )
\
", ""
"**@check_if_uptodate** (:ref:`Summary ` / :ref:`Manual `)
- Custom function to determine if jobs need to be run
", "
* :ref:`@check_if_uptodate ` ( ``is_task_up_to_date_function`` )
\
", ""
".. tip::
The use of this is discouraged.
**@files_re** (:ref:`Summary `)
- I/O file names via regular
expressions
- start from lists of file names
or |glob|_ results
- skips up-to-date jobs
", "
* :ref:`@files_re ` ( ``input``, ``matching_regex``, [``input_pattern``,] ``output``, ``...`` )
", ""