你不需要线程来并行运行子流程:
from subprocess import Popencommands = [ 'date; ls -l; sleep 1; date', 'date; sleep 5; date', 'date; df -h; sleep 3; date', 'date; hostname; sleep 2; date', 'date; uname -a; date',]# run in parallelprocesses = [Popen(cmd, shell=True) for cmd in commands]# do other things here..# wait for completionfor p in processes: p.wait()
为了限制并发命令的数量,可以使用
multiprocessing.dummy.Pool线程并提供与
multiprocessing.Pool使用进程相同的接口:
from functools import partialfrom multiprocessing.dummy import Poolfrom subprocess import callpool = Pool(2) # two concurrent commands at a timefor i, returnpre in enumerate(pool.imap(partial(call, shell=True), commands)): if returnpre != 0: print("%d command failed: %d" % (i, returnpre))该答案演示了限制并发子进程数的各种技术:它显示了
multiprocessing.Pool,concurrent.futures,线程+基于队列的解决方案。
你可以限制并发子进程的数量,而无需使用线程/进程池:
from subprocess import Popenfrom itertools import islicemax_workers = 2 # no more than 2 concurrent processesprocesses = (Popen(cmd, shell=True) for cmd in commands)running_processes = list(islice(processes, max_workers)) # start new processeswhile running_processes: for i, process in enumerate(running_processes): if process.poll() is not None: # the process has finished running_processes[i] = next(processes, None) # start new process if running_processes[i] is None: # no new processes del running_processes[i] break



