python | multiprocessing并行计算例子和资料

文章目录 1. 进程池 Pool1.1 典型用法——并行计算 获取返回值1.2 apply vs apply_async vs map vs map_async 2. Process2.1 典型用法——多进程执行 获取返回值 3.遇到的坑

1. 进程池 Pool 1.1 典型用法——并行计算 获取返回值

import time 
from multiprocessing import Pool
def f_sum(arr):
 time.sleep(len(arr))
 print( sleep %d s %len(arr))
 res sum(arr)
 return(res) 
a [1,2,3,4]
b [4,5,6,7,8]
c [1,2]
result Pool(3).map_async(f_sum, (a,b,c)).get()
print(result) 

1.2 apply vs apply_async vs map vs map_async

参考1 multiprocessing.Pool: When to use apply, apply_async or map?
参考2 map与map_async的区别的例子
参考3 python包中函数的原始注释

2. Process 2.1 典型用法——多进程执行 获取返回值

获取返回值 需要借助Queue() (收集返回值)

import time 
from multiprocessing import Process,Queue
multiprocess_Q Queue()
def f_sum(arr):
 time.sleep(len(arr))
 print( sleep %d s %len(arr))
 res sum(arr)
 multiprocess_Q.put(res)
 return(res) 
a [1,2,3,4]
b [4,5,6,7,8]
c [1,2]
all_p [Process(target f_sum,args (arr,)) for arr in [a,b,c]]
for p in all_p:
 p.start()
for p in all_p:
 p.join()
all_res [multiprocess_Q.get() for i in range(3)]
print(all_res)

3.遇到的坑

当时我并行的target函数是一个class的一个方法 报错信息莫名其妙

Traceback (most recent call last):
 File /usr/lib64/python3.6/multiprocessing/process.py , line 258, in _bootstrap
 self.run()
 File /usr/lib64/python3.6/multiprocessing/process.py , line 93, in run
 self._target(*self._args, **self._kwargs)
 File /usr/lib64/python3.6/multiprocessing/pool.py , line 108, in worker
 task get()
 File /usr/lib64/python3.6/multiprocessing/queues.py , line 337, in get
 return _ForkingPickler.loads(res)
TypeError: __init__() missing 1 required positional argument: high 

报错原因是 如果class比较复杂 有时候Pool会有问题。这时候用Process会容错性高一些。