栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Python

【Mindspore】【GPU并行训练】用2块GPUs可以,四块或八块就失败了 V2.0--后续报错

Python 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

【Mindspore】【GPU并行训练】用2块GPUs可以,四块或八块就失败了 V2.0--后续报错

问题:

【功能模块】

显卡配置:4 * Nvidia P100 

【操作步骤&问题现象】

1、使用mpirun --allow-run-as-root -n 4 python resnet50_distributed_training_gpu.py 进行分布式训练

【resnet50_distributed_training_gpu.py主函数代码】

遇到报错:(同链接 https://bbs.huaweicloud.com/forum/thread-111788-1-1.html)

2、通过运行docker时设置 "docker run --shm-size=2gb" 来解决

3.遇到后续报错:RuntimeError: mindspore/ccsrc/backend/session/kernel_build_client.h:141 GetScriptFilePath] popen failed, errno: 12

【截图信息】

【日志信息】(可选,上传日志内容或者附件)

Traceback (most recent call last):

  File "resnet50_distributed_training_gpu.py", line 139, in

    model.train(epoch_size, dataset, callbacks=[loss_cb,ckpoint_cb], dataset_sink_mode=False)

  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 592, in train

    sink_size=sink_size)

  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 385, in _train

    self._train_process(epoch, train_dataset, list_callback, cb_params)

  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 513, in _train_process

    outputs = self._train_network(*next_element)

  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 322, in __call__

    out = self.compile_and_run(*inputs)

  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 578, in compile_and_run

    self.compile(*inputs)

  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 565, in compile

    _executor.compile(self, *inputs, phase=self.phase, auto_parallel_mode=self._auto_parallel_mode)

  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/common/api.py", line 505, in compile

    result = self._executor.compile(obj, args_list, phase, use_vm)

RuntimeError: mindspore/ccsrc/backend/session/kernel_build_client.h:141 GetScriptFilePath] popen failed, errno: 12

#

--------------------------------------------------------------------------

Primary job  terminated normally, but 1 process returned

a non-zero exit code. Per user-direction, the job has been aborted.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpirun detected that one or more processes exited with non-zero status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[18168,1],0]

  Exit code:    1

解决方案:

设置export GLOG_v=1后再跑一下,然后保存log文件。

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/1022875.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号