Ruffus Decorators¶

Core¶

Decorator	Examples
@originate (Summary / Manual) Creates (originates) a set of starting file from scratch (ex nihilo!) Only called to create files which do not exist. Each item in `output` is created by a separate job.	@originate ( `output`, [`extras`,...] )
@split (Summary / Manual) Splits a single input into multiple output Globs in `output` can specify an indeterminate number of files.	@split ( `input`, `output`, [`extras`,...] )
@transform (Summary / Manual) Applies the task function to transform input data to output.	@transform ( `input`, suffix() , `output`, [`extras`,...] ) @transform ( `input`, regex() , `output`, [`extras`,...] ) @transform ( `input`, formatter() , `output`, [`extras`,...] )
@merge (Summary / Manual) Merges multiple input files into a single output.	@merge (`input`, `output`, [`extras`,...] )

Combinatorics¶

Decorator	Examples
@product (Summary / Manual) Generates the product between sets of input, i.e. all vs all comparisons.	@product ( `input`, formatter() ,[ `input`, formatter(), `output`, [`extras`,...] )
@permutations (Summary / Manual) Generates the permutations, between all elements of a set of Input Analogous to the python itertools.permutations permutations(‘ABCD’, 2) –> AB AC AD BA BC BD CA CB CD DA DB DC	@permutations ( `input`, formatter(), `tuple_size`, `output`, [`extras`,...] )
@combinations (Summary / Manual) Generates the permutations, between all elements of a set of Input Analogous to the python itertools.combinations combinations(‘ABCD’, 3) –> ABC ABD ACD BCD Generates the combinations, between all the elements of a set of Input: i.e. r-length tuples of input elements with no repeated elements (A A) and where order of the tuples is irrelevant (either A B or B A, not both).	@combinations ( `input`, formatter(), `tuple_size`, `output`, [`extras`,...] )
@combinations_with_replacement (Summary / Manual) Generates the permutations, between all the elements of a set of Input Analogous to the python itertools.permutations combinations(‘ABCD’, 3) –> ABC ABD ACD BCD Generates the combinations_with_replacement, between all the elements of a set of Input: i.e. r-length tuples of input elements with no repeated elements (A A) and where order of the tuples is irrelevant (either A B or B A, not both).	@combinations_with_replacement ( `input`, formatter(), `tuple_size`, `output`, [`extras`,...] )

Advanced¶

Decorator Examples

@subdivide (Summary / Manual)

Subdivides each input into multiple Outputs.

The number of output can be determined at runtime using globs.

Many to Even More operator.

Do not use split as a synonym for subdivide.

@subdivide ( input, regex() , [ inputs | add_inputs(input_pattern), ] output, [extras,...] )

@subdivide ( input, formatter([] ), [ inputs | add_inputs(input_pattern), ] output, [extras,...] )

@transform (Summary / Manual)

Generates both input & output from regular expressions

Useful for adding additional file dependencies

@transform ( input, regex() , [ inputs | add_inputs(input_pattern), ] output, [extras,...] )

@transform ( input, formatter() , [ inputs | add_inputs(input_pattern), ] output, [extras,...] )

@collate (Summary / Manual)

Groups multiple input using regular expression matching.

Multiple input generating identical output are collated together.

@collate (input, regex() , output, [extras,...] )

@collate (input, regex() , inputs | add_inputs(input_pattern), output, [extras,...] )

@collate (input, formatter() , output, [extras,...] )

@collate (input, formatter() , inputs | add_inputs(input_pattern), output, [extras,...] )

@follows (Summary / Manual)

Indicates task dependency

optional mkdir prerequisite (see Manual)

@follows ( task1, 'task2' ))

@follows ( task1, mkdir( 'my/directory/' ))

@posttask (Summary / Manual)

Calls function after task completes

Optional touch_file indicator (Manual)

@posttask ( completion_signal_func )

@posttask (touch_file( 'task1.done' ))

@active_if (Summary / Manual)

Switches tasks on and off at run time

Evaluated each time you call

pipeline_run(...),

pipeline_printout(...) or

pipeline_printout_graph(...)

Dormant tasks behave as if they are :

up to date and

have no output.

@active_if ( on_or_off1, [on_or_off2, ...] )

@jobs_limit (Summary / Manual)

Limits the amount of multiprocessing for the specified task

Ensures that fewer than N jobs are run in parallel for this task

Overrides multiprocess parameter in pipeline_run(...)

@jobs_limit ( NUMBER_OF_JOBS_RUNNING_CONCURRENTLY )

@mkdir (Summary / Manual)

Generates paths for os.makedirs

@mkdir ( input, suffix() , output )

@mkdir ( input, regex() , output )

@mkdir ( input, formatter() , output)

@graphviz (Summary / Manual)

Customise the task graphics in flowcharts from pipeline_printout_graph(...)

@graphviz ( graphviz_parameter = XXX, [graphviz_parameter2 = YYY ...])

Esoteric!¶

Decorator Examples

@files (Summary / Manual)

I/O parameters

skips up-to-date jobs

Should use @transform etc instead

@files( parameter_list )

@files( parameter_generating_function )

@files ( input_file, output_file, other_params, ... )

@parallel (Summary / Manual)

By default, does not check if jobs are up to date

Best used in conjuction with @check_if_uptodate

@parallel ( parameter_list )

@parallel ( parameter_generating_function )

@check_if_uptodate (Summary / Manual)

Custom function to determine if jobs need to be run

@check_if_uptodate ( is_task_up_to_date_function )

Tip

The use of this is discouraged.

@files_re (Summary)

I/O file names via regular expressions

start from lists of file names or glob results

skips up-to-date jobs

@files_re ( input, matching_regex, [input_pattern,] output, ... )

Decorator	Examples
@subdivide (Summary / Manual) Subdivides each input into multiple Outputs. The number of output can be determined at runtime using globs. Many to Even More operator. Do not use split as a synonym for subdivide.	@subdivide ( `input`, regex() , [ inputs \| add_inputs(`input_pattern`), ] `output`, [`extras`,...] ) @subdivide ( `input`, formatter([] ), [ inputs \| add_inputs(`input_pattern`), ] `output`, [`extras`,...] )
@transform (Summary / Manual) Generates both input & output from regular expressions Useful for adding additional file dependencies	@transform ( `input`, regex() , [ inputs \| add_inputs(`input_pattern`), ] `output`, [`extras`,...] ) @transform ( `input`, formatter() , [ inputs \| add_inputs(`input_pattern`), ] `output`, [`extras`,...] )
@collate (Summary / Manual) Groups multiple input using regular expression matching. Multiple input generating identical output are collated together.	@collate (`input`, regex() , `output`, [`extras`,...] ) @collate (`input`, regex() , inputs \| add_inputs(`input_pattern`), `output`, [`extras`,...] ) @collate (`input`, formatter() , `output`, [`extras`,...] ) @collate (`input`, formatter() , inputs \| add_inputs(`input_pattern`), `output`, [`extras`,...] )
@follows (Summary / Manual) Indicates task dependency optional mkdir prerequisite (see Manual)	@follows ( `task1`, `'task2'` )) @follows ( `task1`, mkdir( `'my/directory/'` ))
@posttask (Summary / Manual) Calls function after task completes Optional touch_file indicator (Manual)	@posttask ( `completion_signal_func` ) @posttask (touch_file( `'task1.done'` ))
@active_if (Summary / Manual) Switches tasks on and off at run time Evaluated each time you call pipeline_run(...), pipeline_printout(...) or pipeline_printout_graph(...) Dormant tasks behave as if they are : up to date and have no output.	@active_if ( `on_or_off1`, `[on_or_off2, ...]` )
@jobs_limit (Summary / Manual) Limits the amount of multiprocessing for the specified task Ensures that fewer than N jobs are run in parallel for this task Overrides `multiprocess` parameter in pipeline_run(...)	@jobs_limit ( `NUMBER_OF_JOBS_RUNNING_CONCURRENTLY` )
@mkdir (Summary / Manual) Generates paths for os.makedirs	@mkdir ( `input`, suffix() , `output` ) @mkdir ( `input`, regex() , `output` ) @mkdir ( `input`, formatter() , `output`)
@graphviz (Summary / Manual) Customise the task graphics in flowcharts from pipeline_printout_graph(...)	@graphviz ( `graphviz_parameter = XXX`, `[graphviz_parameter2 = YYY ...]`)

Decorator	Examples
@files (Summary / Manual) I/O parameters skips up-to-date jobs Should use @transform etc instead	@files( `parameter_list` ) @files( `parameter_generating_function` ) @files ( `input_file`, `output_file`, `other_params`, ... )
@parallel (Summary / Manual) By default, does not check if jobs are up to date Best used in conjuction with @check_if_uptodate	@parallel ( `parameter_list` ) @parallel ( `parameter_generating_function` )
@check_if_uptodate (Summary / Manual) Custom function to determine if jobs need to be run	@check_if_uptodate ( `is_task_up_to_date_function` )
Tip The use of this is discouraged. @files_re (Summary) I/O file names via regular expressions start from lists of file names or glob results skips up-to-date jobs	@files_re ( `input`, `matching_regex`, [`input_pattern`,] `output`, `...` )