栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Python

python学习记录一:关于分布式进程执行报错以及解决方案

Python 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

python学习记录一:关于分布式进程执行报错以及解决方案

源代码内容如下:(来自廖雪峰官方网站-python教程-进程和线程-分布式进程)

# test_07.py
import random, queue
from multiprocessing.managers import BaseManager

# 发送任务的队列:
task_queue = queue.Queue()
# 接收结果的队列
result_queue = queue.Queue()

# 从BaseManager继承的QueueManager:
class QueueManager(BaseManager):
    pass

# 把两个Queue都注册到网络上,callable参数关联了Queue对象
QueueManager.register('get_task_queue', callable=lambda: task_queue)
QueueManager.register('get_result_queue', callable=lambda: result_queue)
# 绑定端口5000,设置验证码‘abc’
manager = QueueManager(address=('', 5000), authkey=b'abc')
# 启动Queue:
manager.start()
# 获得通过网络访问的Queue对象
task = manager.get_task_queue()
result = manager.get_result_queue()
# 放几个任务进去:
for i in range(10):
n = random.randint(0, 10000)
print('Put task %d...' % n)
task.put(n)
# 从result队列读取结果:
print('Try get result...')
for i in range(10):
r = result.get(timeout=10)
print('Result: %s' % r)
# 关闭
manager.shutdown()
print('master exit.')

报错(一):
执行报错“_pickle.PicklingError: Can’t pickle : attribute lookup on main failed”,具体报错内容如下:

"D:Program Filespython-3.7.9python.exe" D:/py/test-three/test_07.py
Traceback (most recent call last):
  File "D:/py/test-three/test_07.py", line 22, in 
    manager.start()
  File "D:Program Filespython-3.7.9libmultiprocessingmanagers.py", line 563, in start
    self._process.start()
  File "D:Program Filespython-3.7.9libmultiprocessingprocess.py", line 112, in start
    self._popen = self._Popen(self)
  File "D:Program Filespython-3.7.9libmultiprocessingcontext.py", line 322, in _Popen
    return Popen(process_obj)
  File "D:Program Filespython-3.7.9libmultiprocessingpopen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "D:Program Filespython-3.7.9libmultiprocessingreduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle  at 0x0000021EED298438>: attribute lookup  on __main__ failed

首先查看报错内容,寻找错误源头‘manager.start()’,无法启动Queue,根据“_pickle.PicklingError: Can’t pickle : attribute lookup on main failed”,lambda表达式无法序列化,参考文章【https://blog.csdn.net/tpc4289/article/details/79280659/】,自定义函数,修改部分代码如下:

# 自定义函数return_tast_queue
def return_task_queue():
    global task_queue
    return task_queue

# 自定义函数return_result_queue
def return_result_queue():
    global result_queue
    return result_queue

# 把两个Queue都注册到网络上,callable参数关联了Queue对象
QueueManager.register('get_task_queue', callable=return_task_queue)
QueueManager.register('get_result_queue', callable=return_result_queue)

执行修改后的代码,再次报错,报错内容变化,说明上一个问题解决,新增其他问题。
报错(二):
is not going to be frozen to produce an executable.‘’')……具体报错内容为:

"D:Program Filespython-3.7.9python.exe" D:/py/test-three/test_07.py
Traceback (most recent call last):
  File "", line 1, in 
  File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 114, in _main
    prepare(preparation_data)
  File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "D:Program Filespython-3.7.9librunpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "D:Program Filespython-3.7.9librunpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "D:Program Filespython-3.7.9librunpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:pytest-threetest_07.py", line 32, in 
    manager.start()
  File "D:Program Filespython-3.7.9libmultiprocessingmanagers.py", line 563, in start
    self._process.start()
  File "D:Program Filespython-3.7.9libmultiprocessingprocess.py", line 112, in start
    self._popen = self._Popen(self)
  File "D:Program Filespython-3.7.9libmultiprocessingcontext.py", line 322, in _Popen
    return Popen(process_obj)
  File "D:Program Filespython-3.7.9libmultiprocessingpopen_spawn_win32.py", line 46, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

这个报错内容很明显,需要添加“if name == ‘main’:”,具体原理可参考文章【https://blog.csdn.net/shenfuli/article/details/103969964】

报错(三):

"D:Program Filespython-3.7.9python.exe" D:/py/test-three/test_08.py
Traceback (most recent call last):
  File "D:/py/test-three/test_08.py", line 36, in 
    task = manager.get_task_queue()
  File "D:Program Filespython-3.7.9libmultiprocessingmanagers.py", line 724, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "D:Program Filespython-3.7.9libmultiprocessingmanagers.py", line 607, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "D:Program Filespython-3.7.9libmultiprocessingconnection.py", line 492, in Client
    c = SocketClient(address)
  File "D:Program Filespython-3.7.9libmultiprocessingconnection.py", line 620, in SocketClient
    s.connect(address)
OSError: [WinError 10049] 在其上下文中,该请求的地址无效。

进程已结束,退出代码1

此处就是照着源代码输入时,丢失ip,需要填入自己的IP,cmd输入‘ipconfig’查看本机ip,填入此段代码:manager = QueueManager(address=(‘此处填入本机IP’, 5000), authkey=b’abc’)

修改后的正确代码如下:

import random, queue
from multiprocessing.managers import BaseManager

# 发送任务的队列:
task_queue = queue.Queue()
# 接收结果的队列
result_queue = queue.Queue()


def return_task_queue():
    global task_queue
    return task_queue


def return_result_queue():
    global result_queue
    return result_queue


# 从BaseManager继承的QueueManager:
class QueueManager(BaseManager):
    pass


if __name__ == '__main__':
    # 把两个Queue都注册到网络上,callable参数关联了Queue对象
    QueueManager.register('get_task_queue', callable=return_task_queue)
    QueueManager.register('get_result_queue', callable=return_result_queue)
    # 绑定端口5000,设置验证码‘abc’
    manager = QueueManager(address=('本机ip', 5000), authkey=b'abc')
    # 启动Queue:
    manager.start()
    # 获得通过网络访问的Queue对象
    task = manager.get_task_queue()
    result = manager.get_result_queue()
    # 放几个任务进去:
    for i in range(10):
        n = random.randint(0, 10000)
        print('Put task %d...' % n)
        task.put(n)
    # 从result队列读取结果:
    print('Try get result...')
    for i in range(10):
        r = result.get(timeout=10)
        print('Result: %s' % r)
    # 关闭
    manager.shutdown()
    print('master exit.')

总结报错1和2是Windows系统特有,由于python在Windows运行,有些使用的方法与其他系统不一样,导致学习过程中提供代码无法正常运行,需要自行排查。

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/998272.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号