pool.map - multiple arguments
pool.map accepts only a list of single parameters as input. Multiple parameters can be passed to pool by a list of parameter-lists, or by setting some parameters constant using partial.
Example 1: List of lists
A list of multiple arguments can be passed to a function via pool.map
(function needs to accept a list as single argument)
Example: calculate the product of each data pair
import multiprocessing
import numpy as np
data_pairs = [ [3,5], [4,3], [7,3], [1,6] ]
Define what to do with each data pair ( p=[3,5] ), example: calculate product
def myfunc(p):
product_of_list = np.prod(p)
return product_of_list
if __name__ == '__main__':
pool = multiprocessing.Pool(processes=4)
result_list = pool.map(myfunc, data_pairs)
print(result_list)
[15, 12, 21, 6]
Parallel run of a function with multiple arguments
To use pool.map for functions with multiple arguments, partial can be used to set constant values to all arguments which are not changed during parallel processing, such that only the first argument remains for iterating. (The variable input needs to be always the first argument of a function, not second or later arguments).
Example: multiply all numbers in a list by 10
import multiprocessing
from functools import partial
data_list = [1, 2, 3, 4]
def prod_xy(x,y):
return x * y
def parallel_runs(data_list):
pool = multiprocessing.Pool(processes=4)
prod_x=partial(prod_xy, y=10) # prod_x has only one argument x (y is fixed to 10)
result_list = pool.map(prod_x, data_list)
print(result_list)
if __name__ == '__main__':
parallel_runs(data_list)
[10, 20, 30, 40]
Partial creates a new simplified version of a function with part of the arguments fixed to specific values.
https://docs.python.org/3/library/multiprocessing.html
https://docs.python.org/3/library/functools.html#functools.partial