florin.pipelines

Deferred execution pipelines with different computational models.

Classes

BalsamPipeline

Distributed computation using the Balsam job submission database.

MPIPipeline

Multicore/multi-node parallel computation with MPI.

MultiprocessingPipeline

Multiprocessing using the standard fork/join model.

MultithreadingPipeline

Multithreading using the Python multithreading library.

SerialPipeline

Single-core serial deferred computation.

WorkQueuePipeline

Distributed computing using Work Queue to manage tasks.

class florin.pipelines.BalsamPipeline(*operations)

Methods

__call__(self, data)	Call self as a function.
add(self, func)	Append a callable to this pipeline.
dump(self, path)	Save this pipeline to file.
dumps(self)	Serialize this pipeline as a string.
run(self, data)	Run data through the pipeline.

add(self, func)

Append a callable to this pipeline.

Parameters:func (callable) – New function to add.

dump(self, path)

Save this pipeline to file.

Parameters:path (str) – Path to the file to write this pipeline to.

dumps(self)

Serialize this pipeline as a string.

run(self, data)

Run data through the pipeline.

Parameters:data – Input to the first function in the pipeline. Returns:The result of applying the pipeline to data. Return type:result

class florin.pipelines.MPIPipeline(*operations)

MPI-based multiprocessing pipeline.

Parameters:operations (callables) – Sequence of operations to run in the pipeline.

Notes

MPI is configured by wrapping Python in an mpiexec or mpirun call at runtime.

Methods

__call__(self, data)	Call self as a function.
add(self, func)	Append a callable to this pipeline.
dump(self, path)	Save this pipeline to file.
dumps(self)	Serialize this pipeline as a string.
run(self, data)	Run data through the pipeline.

add(self, func)

Append a callable to this pipeline.

Parameters:func (callable) – New function to add.

dump(self, path)

Save this pipeline to file.

Parameters:path (str) – Path to the file to write this pipeline to.

dumps(self)

Serialize this pipeline as a string.

run(self, data)

Run data through the pipeline.

Parameters:data – Input to the first function in the pipeline. Returns:The result of applying the pipeline to data. Return type:result

class florin.pipelines.MultiprocessingPipeline(*operations, processes=None)

Pipeline for multi-core parallel processing on a single machine.

Parameters:

• operations (callables) – The operations/functions/callable classes to run in this pipeline.
• processes (int, optional) – The number of processes to use. Setting None will attempt to use as many as can be supported.

Methods

__call__(self, data)	Call self as a function.
add(self, func)	Append a callable to this pipeline.
dump(self, path)	Save this pipeline to file.
dumps(self)	Serialize this pipeline as a string.
run(self, data)	Run data through the pipeline.

add(self, func)

Append a callable to this pipeline.

Parameters:func (callable) – New function to add.

dump(self, path)

Save this pipeline to file.

Parameters:path (str) – Path to the file to write this pipeline to.

dumps(self)

Serialize this pipeline as a string.

run(self, data)

Run data through the pipeline.

Parameters:data – Input to the first function in the pipeline. Returns:The result of applying the pipeline to data. Return type:result

class florin.pipelines.MultithreadingPipeline(*operations, threads=None)

Pipeline for multithreaded parallel processing on a single machine.

Parameters:

• operations (callables) – Sequence of operations to run in the pipeline.
• threads (int, optional) – The number of threads to use. Setting None will attempt to use as many as can be supported.

Methods

__call__(self, data)	Call self as a function.
add(self, func)	Append a callable to this pipeline.
dump(self, path)	Save this pipeline to file.
dumps(self)	Serialize this pipeline as a string.
run(self, data)	Run data through the pipeline.

add(self, func)

Append a callable to this pipeline.

Parameters:func (callable) – New function to add.

dump(self, path)

Save this pipeline to file.

Parameters:path (str) – Path to the file to write this pipeline to.

dumps(self)

Serialize this pipeline as a string.

run(self, data)

Run data through the pipeline.

Parameters:data – Input to the first function in the pipeline. Returns:The result of applying the pipeline to data. Return type:result

class florin.pipelines.SerialPipeline(*operations)

Pipeline for single-core serial computation.

Parameters:operations (callables) – The operations/functions/callable classes to run in this pipeline.

Methods

__call__(self, data)	Call self as a function.
add(self, func)	Append a callable to this pipeline.
dump(self, path)	Save this pipeline to file.
dumps(self)	Serialize this pipeline as a string.
run(self, data)	Run data through the pipeline.

add(self, func)

Append a callable to this pipeline.

Parameters:func (callable) – New function to add.

dump(self, path)

Save this pipeline to file.

Parameters:path (str) – Path to the file to write this pipeline to.

dumps(self)

Serialize this pipeline as a string.

run(self, data)

Run data through the pipeline.

Parameters:data – Input to the first function in the pipeline. Returns:The result of applying the pipeline to data. Return type:result

class florin.pipelines.WorkQueuePipeline(*operations)

Methods

__call__(self, data)	Call self as a function.
add(self, func)	Append a callable to this pipeline.
dump(self, path)	Save this pipeline to file.
dumps(self)	Serialize this pipeline as a string.
run(self, data)	Run data through the pipeline.

add(self, func)

Append a callable to this pipeline.

Parameters:func (callable) – New function to add.

dump(self, path)

Save this pipeline to file.

Parameters:path (str) – Path to the file to write this pipeline to.

dumps(self)

Serialize this pipeline as a string.

run(self, data)

Run data through the pipeline.

Parameters:data – Input to the first function in the pipeline. Returns:The result of applying the pipeline to data. Return type:result