Transducers.jl supports thread-based (
reduce) and process-based (
dreduce) parallelisms with the same composable API; i.e. transducers. Having a uniform API to cover different parallelisms as well as sequential processing
foldl is useful. Using multiple cores or machines for your computation is as easy as replacing
dreduce; you don't need to re-write your transducers or reducing functions.
To use multiple threads, the
julia process must be started with appropriate environment variable
JULIA_NUM_THREADS. See Julia manual for more information about how to enable and verify multi-threading in Julia:
Transducers.jl supports thread-based parallelism for Julia ≥ 1.0. You can use it by replacing
reduce. With Julia ≥ 1.3, Transducers.jl supports early termination to avoid unnecessary computation while guaranteeing the result to be deterministic; i.e., it does not depend on how computation tasks are scheduled.
Transducers.jl supports process-based parallelism using Distributed.jl. You can use it by replacing
dreduce. It can be used for horizontally scaling the computation. It is also useful for using external libraries that are not "thread-safe."