SCOOP (Scalable Concurrent Operations in Python) is a Python software module for distributing concurrent tasks on various environments, from heterogeneous grids of workstations to supercomputers.

It uses ØMQ and the Greenlet package as building blocks to encapsulate and distribute tasks (named a Future) between processes and/or systems. Its interface is inspired from the PEP-3148 proposal.

SCOOP is targeted towards scientific applications that require execution of many loosely coupled tasks using all available hardware resources. These resources need to be accessible through SSH.

History

SCOOP was initiated by Yannick Hold and Marc Parizeau at the Computer Vision and Systems Laboratory of Université Laval. It is an iterative step over the now deprecated DTM module of the DEAP framework for developing evolutionary algorithm. While DTM used MPI for its communications, SCOOP uses instead ØMQ.

Network topology

SCOOP uses the Broker Architecture^[1] to distribute its Futures. It is based on a central element, called the Broker, that dispatches work to its workers. The main difference between this pattern and a Master/slave topology reside in the Future origin. In the Broker architecture, the Futures emanate from a worker, which is located on the periphery of the topology, instead of the master in the Master/slave architecture. This allows higher reliability in regard to worker faults and generally better performances due to the generic function of the Broker. Since he doesn't need to serialize nor deserialize any Future to route them around the grid, his workload consists of networking or interprocess I/O and almost no CPU processing time. This lowers the bottleneck of the Broker topology.

The Broker architecture won't stress the networking fabric and elements as much as a totally distributed topology since there is only one connection required on every worker.

Example

An introductory parallel "Hello, world!" example is implemented this way:

from scoop import futures

def hello_world(value) -> str:
    return "Hello World from Future #{}".format(value)

if __name__ == "__main__":
    return_values = futures.map(hello_world, range(16))
    print("\n".join(return_values))

References

External links

• Official website
• Official documentation