============================= test session starts ============================== platform linux -- Python 3.9.21, pytest-6.2.5, py-1.11.0, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/networks, configfile: ../../../../../../sault/virtual_test/virtualenv_002/sault/config/pytest.ini plugins: forked-1.6.0, hydra-core-1.3.2, xdist-1.32.0, anyio-4.9.0 collected 1 item test_mindcv_overfit.py [WARNING] ME(563066:281473241902784,Process-1):2025-07-15-09:59:37.840.940 [mindspore/context.py:1412] For 'context.set_context', the parameter 'device_id' will be deprecated and removed in a future version. Please use the api mindspore.set_device() instead. [WARNING] ME(563066:281473241902784,Process-1):2025-07-15-09:59:37.841.693 [mindspore/context.py:1412] For 'context.set_context', the parameter 'deterministic' will be deprecated and removed in a future version. Please use the api mindspore.set_deterministic() instead. [WARNING] CORE(563066,ffff9898eec0,python3):2025-07-15-09:59:37.846.234 [mindspore/core/utils/ms_context.cc:535] GetJitLevel] Set jit level to O2 for rank table startup method. [WARNING] ME(563069:281473241902784,Process-2):2025-07-15-09:59:37.854.262 [mindspore/context.py:1412] For 'context.set_context', the parameter 'device_id' will be deprecated and removed in a future version. Please use the api mindspore.set_device() instead. [WARNING] ME(563069:281473241902784,Process-2):2025-07-15-09:59:37.854.988 [mindspore/context.py:1412] For 'context.set_context', the parameter 'deterministic' will be deprecated and removed in a future version. Please use the api mindspore.set_deterministic() instead. [WARNING] CORE(563069,ffff9898eec0,python3):2025-07-15-09:59:37.858.960 [mindspore/core/utils/ms_context.cc:535] GetJitLevel] Set jit level to O2 for rank table startup method. [WARNING] ME(563075:281473241902784,Process-3):2025-07-15-09:59:37.867.814 [mindspore/context.py:1412] For 'context.set_context', the parameter 'device_id' will be deprecated and removed in a future version. Please use the api mindspore.set_device() instead. [WARNING] ME(563075:281473241902784,Process-3):2025-07-15-09:59:37.868.552 [mindspore/context.py:1412] For 'context.set_context', the parameter 'deterministic' will be deprecated and removed in a future version. Please use the api mindspore.set_deterministic() instead. [WARNING] CORE(563075,ffff9898eec0,python3):2025-07-15-09:59:37.872.770 [mindspore/core/utils/ms_context.cc:535] GetJitLevel] Set jit level to O2 for rank table startup method. [WARNING] ME(563080:281473241902784,Process-4):2025-07-15-09:59:37.882.067 [mindspore/context.py:1412] For 'context.set_context', the parameter 'device_id' will be deprecated and removed in a future version. Please use the api mindspore.set_device() instead. [WARNING] ME(563080:281473241902784,Process-4):2025-07-15-09:59:37.882.859 [mindspore/context.py:1412] For 'context.set_context', the parameter 'deterministic' will be deprecated and removed in a future version. Please use the api mindspore.set_deterministic() instead. [WARNING] CORE(563080,ffff9898eec0,python3):2025-07-15-09:59:37.887.061 [mindspore/core/utils/ms_context.cc:535] GetJitLevel] Set jit level to O2 for rank table startup method. [WARNING] ME(563085:281473241902784,Process-5):2025-07-15-09:59:37.896.288 [mindspore/context.py:1412] For 'context.set_context', the parameter 'device_id' will be deprecated and removed in a future version. Please use the api mindspore.set_device() instead. [WARNING] ME(563085:281473241902784,Process-5):2025-07-15-09:59:37.897.019 [mindspore/context.py:1412] For 'context.set_context', the parameter 'deterministic' will be deprecated and removed in a future version. Please use the api mindspore.set_deterministic() instead. [WARNING] CORE(563085,ffff9898eec0,python3):2025-07-15-09:59:37.901.139 [mindspore/core/utils/ms_context.cc:535] GetJitLevel] Set jit level to O2 for rank table startup method. [WARNING] ME(563090:281473241902784,Process-6):2025-07-15-09:59:37.910.445 [mindspore/context.py:1412] For 'context.set_context', the parameter 'device_id' will be deprecated and removed in a future version. Please use the api mindspore.set_device() instead. [WARNING] ME(563090:281473241902784,Process-6):2025-07-15-09:59:37.911.182 [mindspore/context.py:1412] For 'context.set_context', the parameter 'deterministic' will be deprecated and removed in a future version. Please use the api mindspore.set_deterministic() instead. [WARNING] CORE(563090,ffff9898eec0,python3):2025-07-15-09:59:37.915.174 [mindspore/core/utils/ms_context.cc:535] GetJitLevel] Set jit level to O2 for rank table startup method. [WARNING] ME(563095:281473241902784,Process-7):2025-07-15-09:59:37.924.281 [mindspore/context.py:1412] For 'context.set_context', the parameter 'device_id' will be deprecated and removed in a future version. Please use the api mindspore.set_device() instead. [WARNING] ME(563095:281473241902784,Process-7):2025-07-15-09:59:37.924.998 [mindspore/context.py:1412] For 'context.set_context', the parameter 'deterministic' will be deprecated and removed in a future version. Please use the api mindspore.set_deterministic() instead. [WARNING] CORE(563095,ffff9898eec0,python3):2025-07-15-09:59:37.929.041 [mindspore/core/utils/ms_context.cc:535] GetJitLevel] Set jit level to O2 for rank table startup method. [WARNING] ME(563100:281473241902784,Process-8):2025-07-15-09:59:37.938.186 [mindspore/context.py:1412] For 'context.set_context', the parameter 'device_id' will be deprecated and removed in a future version. Please use the api mindspore.set_device() instead. [WARNING] ME(563100:281473241902784,Process-8):2025-07-15-09:59:37.938.882 [mindspore/context.py:1412] For 'context.set_context', the parameter 'deterministic' will be deprecated and removed in a future version. Please use the api mindspore.set_deterministic() instead. [WARNING] CORE(563100,ffff9898eec0,python3):2025-07-15-09:59:37.942.705 [mindspore/core/utils/ms_context.cc:535] GetJitLevel] Set jit level to O2 for rank table startup method. [WARNING] DEVICE(563066,ffff9898eec0,python3):2025-07-15-09:59:49.463.162 [mindspore/ccsrc/plugin/res_manager/ascend/collective/ascend_collective_comm_lib.cc:164] Initialize] Launch Ascend distributed job in RankTable manner. This manner will be deprecated in later version of MindSpore. Please switch to 'msrun' or 'mpirun'. You can refer to this link about how to use these commands: https://www.mindspore.cn/docs/zh-CN/r2.5.0/model_train/parallel/startup_method.html. [2025-07-15 09:59:49] mindcv.train INFO - We recommend installing `termcolor` via `pip install termcolor` and setup logger by `set_logger(..., color=True)` [WARNING] DEVICE(563075,ffff9898eec0,python3):2025-07-15-09:59:55.637.139 [mindspore/ccsrc/plugin/res_manager/ascend/collective/ascend_collective_comm_lib.cc:164] Initialize] Launch Ascend distributed job in RankTable manner. This manner will be deprecated in later version of MindSpore. Please switch to 'msrun' or 'mpirun'. You can refer to this link about how to use these commands: https://www.mindspore.cn/docs/zh-CN/r2.5.0/model_train/parallel/startup_method.html. [WARNING] DEVICE(563080,ffff9898eec0,python3):2025-07-15-09:59:58.835.182 [mindspore/ccsrc/plugin/res_manager/ascend/collective/ascend_collective_comm_lib.cc:164] Initialize] Launch Ascend distributed job in RankTable manner. This manner will be deprecated in later version of MindSpore. Please switch to 'msrun' or 'mpirun'. You can refer to this link about how to use these commands: https://www.mindspore.cn/docs/zh-CN/r2.5.0/model_train/parallel/startup_method.html. [2025-07-15 09:59:59] mindcv.train INFO - Essential Experiment Configurations: MindSpore mode[GRAPH(0)/PYNATIVE(1)]: 0 Number of devices: 8 Number of training samples: 32 Number of classes: 1000 Number of batches: 40036 Batch size: 32 Model: resnet50 Model parameters: 25610152 Number of epochs: 120 Optimizer: momentum Learning rate: 0.0125 LR Scheduler: cosine_decay Momentum: 0.9 Weight decay: 0.0001 Auto mixed precision: O2 Loss scale: 1024(fixed) [2025-07-15 09:59:59] mindcv.train INFO - Start training [WARNING] DEVICE(563085,ffff9898eec0,python3):2025-07-15-10:00:02.092.930 [mindspore/ccsrc/plugin/res_manager/ascend/collective/ascend_collective_comm_lib.cc:164] Initialize] Launch Ascend distributed job in RankTable manner. This manner will be deprecated in later version of MindSpore. Please switch to 'msrun' or 'mpirun'. You can refer to this link about how to use these commands: https://www.mindspore.cn/docs/zh-CN/r2.5.0/model_train/parallel/startup_method.html. [WARNING] DEVICE(563090,ffff9898eec0,python3):2025-07-15-10:00:05.300.050 [mindspore/ccsrc/plugin/res_manager/ascend/collective/ascend_collective_comm_lib.cc:164] Initialize] Launch Ascend distributed job in RankTable manner. This manner will be deprecated in later version of MindSpore. Please switch to 'msrun' or 'mpirun'. You can refer to this link about how to use these commands: https://www.mindspore.cn/docs/zh-CN/r2.5.0/model_train/parallel/startup_method.html. [WARNING] DEVICE(563095,ffff9898eec0,python3):2025-07-15-10:00:08.524.491 [mindspore/ccsrc/plugin/res_manager/ascend/collective/ascend_collective_comm_lib.cc:164] Initialize] Launch Ascend distributed job in RankTable manner. This manner will be deprecated in later version of MindSpore. Please switch to 'msrun' or 'mpirun'. You can refer to this link about how to use these commands: https://www.mindspore.cn/docs/zh-CN/r2.5.0/model_train/parallel/startup_method.html. [WARNING] DEVICE(563100,ffff9898eec0,python3):2025-07-15-10:00:11.885.906 [mindspore/ccsrc/plugin/res_manager/ascend/collective/ascend_collective_comm_lib.cc:164] Initialize] Launch Ascend distributed job in RankTable manner. This manner will be deprecated in later version of MindSpore. Please switch to 'msrun' or 'mpirun'. You can refer to this link about how to use these commands: https://www.mindspore.cn/docs/zh-CN/r2.5.0/model_train/parallel/startup_method.html. ................................................Process Process-2: Traceback (most recent call last): File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/jenkins/mindspore/testcases/testcases/tests/st/networks/test_mindcv_overfit.py", line 246, in compute_process _, loss_end, _, _ = train( File "/home/jenkins/mindspore/testcases/testcases/tests/st/networks/test_mindcv_overfit.py", line 64, in train init() File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/site-packages/mindspore/communication/management.py", line 203, in init init_hccl() RuntimeError: Call aclrtSetDevice failed, ret[507033]. Got device count[8] and device id[1], please check if device id is valid. ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/res_manager/ascend/hal_manager/ascend_hal_manager.cc:67 InitDevice ...................................................................................................................................................................................................................................................................................................................................................................[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! Process ForkServerProcess-8:10: Traceback (most recent call last): File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/common.py", line 110, in daemon_process os.kill(mgr_pid, signal.SIGKILL) ProcessLookupError: [Errno 3] No such process Process ForkServerProcess-1:10: Traceback (most recent call last): File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/common.py", line 110, in daemon_process os.kill(mgr_pid, signal.SIGKILL) ProcessLookupError: [Errno 3] No such process Process ForkServerProcess-5:10: Traceback (most recent call last): File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/common.py", line 110, in daemon_process os.kill(mgr_pid, signal.SIGKILL) ProcessLookupError: [Errno 3] No such process Process ForkServerProcess-7:10: Traceback (most recent call last): File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/common.py", line 110, in daemon_process os.kill(mgr_pid, signal.SIGKILL) ProcessLookupError: [Errno 3] No such process Process ForkServerProcess-3:10: Traceback (most recent call last): File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/common.py", line 110, in daemon_process os.kill(mgr_pid, signal.SIGKILL) ProcessLookupError: [Errno 3] No such process Process ForkServerProcess-6:10: Traceback (most recent call last): File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/common.py", line 110, in daemon_process os.kill(mgr_pid, signal.SIGKILL) ProcessLookupError: [Errno 3] No such process Process ForkServerProcess-4:10: Traceback (most recent call last): File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/common.py", line 110, in daemon_process os.kill(mgr_pid, signal.SIGKILL) ProcessLookupError: [Errno 3] No such process /home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 9 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 9 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 9 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 9 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 9 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 9 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /home/jenkins/anaconda3/envs/ci39/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 9 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '