Overview of parallel processing in Transducers.jl

Transducers.jl supports thread-based (reduce) and process-based (dreduce) parallelisms with the same composable API; i.e. transducers. Having a uniform API to cover different parallelisms as well as sequential processing foldl is useful. Using multiple cores or machines for your computation is as easy as replacing foldl with reduce or dreduce; you don't need to re-write your transducers or reducing functions.

See also:

Thread-based parallelism

Note

To use multiple threads, the julia process must be started with appropriate environment variable JULIA_NUM_THREADS. See Julia manual for more information about how to enable and verify multi-threading in Julia:

Transducers.jl supports thread-based parallelism for Julia ≥ 1.0. You can use it by replacing foldl with reduce. With Julia ≥ 1.3, Transducers.jl supports early termination to avoid unnecessary computation while guaranteeing the result to be deterministic; i.e., it does not depend on how computation tasks are scheduled.

Process-based parallelism

Transducers.jl supports process-based parallelism using Distributed.jl. You can use it by replacing foldl with dreduce. It can be used for horizontally scaling the computation. It is also useful for using external libraries that are not "thread-safe."

Note that early termination is not supported in dreduce yet.