Skip to content

[MACE v1.0.0] rpcmem_init failed on GPU #709

New issue
New issue
Closed
#710
Closed
[MACE v1.0.0] rpcmem_init failed on GPU#709
#710
@gasgallo

Description

@gasgallo
gasgallo
opened on Jan 25, 2021

Before you open an issue, please make sure you have tried the following steps:

  1. Make sure your environment is the same with (https://mace.readthedocs.io/en/latest/installation/env_requirement.html).
  2. Have you ever read the document for your usage?
  3. Check if your issue appears in HOW-TO-DEBUG or FAQ.
  4. The form below must be filled.

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
  • NDK version(e.g., 15c): 19c
  • GCC version(if compiling for host, e.g., 5.4.0): 5.4.0
  • MACE version (Use the command: git describe --long --tags): 1.0.0
  • Python version(2.7): 3.6
  • Bazel version (e.g., 0.13.0): 0.16.0
  • CMake version: 3.16.0

Model deploy file (*.yml)

library_name: model
target_abis: [arm64-v8a, armeabi-v7a]
model_graph_format: code
model_data_format: code

models:
  model:
    platform: onnx
    model_file_path: "/models/cb487a80da4a4431a846be8a0d73e5b5.onnx"
    model_sha256_checksum: ba2fa1b0d4850a1831327c98c56c97acc5bce949fd2a344b2713d4ee1e35a742
    subgraphs:
      - input_tensors:
          - data
        input_shapes:
          - 1,3,192,192
        input_data_formats:
          - NCHW
        output_tensors:
          - mobilepose0_conv1_fwd
        output_shapes:
          - 1,128,6,6
    obfuscate: 0
    runtime: cpu+gpu
    winograd: 2

Describe the problem

Model fails to run on GPU with openCL with below error.

To Reproduce

Steps to reproduce the problem:

1. cd /path/to/mace
2. python3 tools/converter.py convert --config=/path/to/yml
3. python3 tools/converter.py run --config=/path/to/yml

Error information / logs

Please include the full log and/or traceback here.

WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading: 
Loading: 0 packages loaded
WARNING: The major revision of the Android NDK referenced by android_ndk_repository rule 'androidndk' is 19. The major revisions supported by Bazel are [10, 11, 12, 13, 14, 15, 16]. Bazel will attempt to treat the NDK as if it was r16. This may cause compilation and linkage problems. Please download a supported NDK version.
Analyzing: target //mace/tools:mace_run_static (2 packages loaded)
Analyzing: target //mace/tools:mace_run_static (11 packages loaded)
INFO: Analysed target //mace/tools:mace_run_static (33 packages loaded).
INFO: Found 1 target...
[0 / 13] [-----] BazelWorkspaceStatusAction stable-status.txt
[29 / 205] no action
[191 / 237] [-----] Creating source manifest for //mace/tools:mace_run_static
[207 / 718] Compiling mace/ops/reverse.cc; 0s local ... (6 actions running)
[212 / 718] Compiling mace/ops/group_norm.cc; 2s local ... (6 actions running)
[242 / 718] Compiling mace/ops/ref/deconv_2d.cc; 1s local ... (3 actions, 2 running)
[573 / 718] Compiling mace/ops/opencl/buffer/softmax.cc; 1s local ... (5 actions running)
[581 / 718] Compiling mace/ops/opencl/image/resize_bicubic.cc; 2s local ... (5 actions running)
[591 / 718] Compiling mace/ops/opencl/image/extract_image_patches.cc; 2s local ... (3 actions running)
[598 / 718] Compiling mace/ops/opencl/image/eltwise.cc; 2s local ... (6 actions running)
[610 / 718] Compiling mace/ops/opencl/buffer/conv_2d_general.cc; 1s local ... (6 actions running)
[629 / 718] Compiling mace/ops/opencl/image/depthwise_deconv2d.cc; 1s local ... (5 actions running)
[650 / 718] Compiling mace/ops/arm/base/gemm.cc; 1s local ... (5 actions running)
[709 / 721] Compiling mace/libmace/mace.cc; 1s local ... (6 actions, 5 running)
[729 / 740] Compiling mace/ops/addn.cc; 2s local ... (5 actions running)
[754 / 762] Compiling mace/ops/eltwise.cc; 4s local ... (5 actions running)
[780 / 781] [-----] Linking mace/tools/mace_run_static
Target //mace/tools:mace_run_static up-to-date:
  bazel-bin/mace/tools/mace_run_static
INFO: Elapsed time: 76.118s, Critical Path: 6.67s
INFO: 191 processes: 191 local.
INFO: Build completed successfully, 194 total actions
INFO: Build completed successfully, 194 total actions
CMD> bazel version
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Build label: 0.16.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jul 31 17:01:24 2018 (1533056484)
Build timestamp: 1533056484
Build timestamp as int: 1533056484
CMD> bazel build //mace/proto:mace_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:mace_py (12 packages loaded)
INFO: Analysed target //mace/proto:mace_py (17 packages loaded).
INFO: Found 1 target...
[2 / 9] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:mace_py up-to-date:
bazel-genfiles/mace/proto/mace_pb2.py
INFO: Elapsed time: 1.681s, Critical Path: 0.03s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
CMD> cp -f bazel-genfiles/mace/proto/mace_pb2.py tools/python/py_proto
CMD> bazel build //mace/proto:micro_mem_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:micro_mem_py (12 packages loaded)
INFO: Analysed target //mace/proto:micro_mem_py (17 packages loaded).
INFO: Found 1 target...
[0 / 4] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:micro_mem_py up-to-date:
bazel-genfiles/mace/proto/micro_mem_pb2.py
INFO: Elapsed time: 1.648s, Critical Path: 0.05s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
CMD> cp -f bazel-genfiles/mace/proto/micro_mem_pb2.py tools/python/py_proto
CMD> bazel build //third_party/caffe:caffe_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //third_party/caffe:caffe_py (12 packages loaded)
INFO: Analysed target //third_party/caffe:caffe_py (17 packages loaded).
INFO: Found 1 target...
[0 / 5] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //third_party/caffe:caffe_py up-to-date:
bazel-genfiles/third_party/caffe/caffe_pb2.py
INFO: Elapsed time: 1.709s, Critical Path: 0.05s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
CMD> cp -f bazel-genfiles/third_party/caffe/caffe_pb2.py tools/python/py_proto
tools/converter.py:319: Downloading model, please wait ...
tools/converter.py:289: 
Downloaded successfully.
* Build //mace/tools:mace_run_static with ABI arm64-v8a
('build', '//mace/tools:mace_run_static', '--config', 'android', '--cpu=arm64-v8a', '--define', 'neon=true', '--define', 'opencl=true', '--define', 'quantize=false', '--define', 'bfloat16=false', '--define', 'fp16=false', '--define', 'rpcmem=true', '--define', 'hexagon=false', '--define', 'hta=false', '--define', 'apu=false', '--config', 'optimization', '--config', 'symbol_hidden', '--per_file_copt=mace/tools/mace_run.cc@-DMODEL_GRAPH_FORMAT_CODE')
('build', '//mace/tools:mace_run_static', '--config', 'android', '--cpu=arm64-v8a', '--define', 'neon=true', '--define', 'opencl=true', '--define', 'quantize=false', '--define', 'bfloat16=false', '--define', 'fp16=false', '--define', 'rpcmem=true', '--define', 'hexagon=false', '--define', 'hta=false', '--define', 'apu=false', '--config', 'optimization', '--config', 'symbol_hidden', '--per_file_copt=mace/tools/mace_run.cc@-DMODEL_GRAPH_FORMAT_CODE')
Build done!
******************************************************
          Run model model on TPS980P          
******************************************************
Generate input file:  build/model/_tmp/model/52ccb254544535bc260dea8cc3fcac30/TPS980P_rk3399/arm64-v8a/model_input_data
Generate input file done.
* Run 'model' with round=100, restart_round=1, tuning=False, out_of_range_check=False, num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/model/_tmp/model/52ccb254544535bc260dea8cc3fcac30/TPS980P_rk3399/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push third_party/nnlib/arm64-v8a/libhexagon_controller.so to /data/local/tmp/mace_run
Push build/model/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1610601805.2930682 to /data/local/tmp/mace_run/cmd_file-model-1610601805.2930682
I mace/tools/mace_run.cc:544] model name: model
I mace/tools/mace_run.cc:545] mace version: v1.0.0-0-g0945e49
I mace/tools/mace_run.cc:546] input node: data
I mace/tools/mace_run.cc:547] input shape: 1,3,192,192
I mace/tools/mace_run.cc:548] output node: mobilepose0_conv1_fwd
I mace/tools/mace_run.cc:549] output shape: 1,128,6,6
I mace/tools/mace_run.cc:550] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:551] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:552] input dir:
I mace/tools/mace_run.cc:553] output dir:
I mace/tools/mace_run.cc:554] model_data_file:
I mace/tools/mace_run.cc:555] model_file:
I mace/tools/mace_run.cc:556] apu_cache_policy: 0
I mace/tools/mace_run.cc:557] apu_binary_file:
I mace/tools/mace_run.cc:558] apu_storage_file:
I mace/tools/mace_run.cc:559] device: CPU
I mace/tools/mace_run.cc:560] round: 100
I mace/tools/mace_run.cc:561] restart_round: 1
I mace/tools/mace_run.cc:562] gpu_perf_hint: 3
I mace/tools/mace_run.cc:563] gpu_priority_hint: 3
I mace/tools/mace_run.cc:564] num_threads: -1
I mace/tools/mace_run.cc:565] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:568] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:573] opencl_queue_window_size: 0
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
I mace/libmace/mace.cc:628] Initializing MaceEngine
I mace/libmace/mace.cc:790] Destroying MaceEngine
I mace/tools/mace_run.cc:616] restart round 0
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
I mace/libmace/mace.cc:628] Initializing MaceEngine
I mace/tools/mace_run.cc:286] Create Mace Engine latency: 49.954 ms
I mace/tools/mace_run.cc:293] Total init latency: 50.098 ms
I mace/tools/mace_run.cc:387] Warm up run
I mace/tools/mace_run.cc:423] 1st warm up run latency: 106.728 ms
I mace/tools/mace_run.cc:431] Run model
I mace/tools/mace_run.cc:493] Average latency: 65.5742 ms
I mace/tools/mace_run.cc:508] Write output file /data/local/tmp/mace_run/model_out_mobilepose0_conv1_fwd with size 18432 done.
========================================================
     capability(CPU)        init      warmup     run_avg
========================================================
time          52.470      50.098     106.728      65.574
I mace/libmace/mace.cc:790] Destroying MaceEngine
Running finished!
Dana service is not available.
* Run 'model' with round=100, restart_round=1, tuning=False, out_of_range_check=False, num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/landmark/_tmp/model/52ccb254544535bc260dea8cc3fcac30/TPS980P_rk3399/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push third_party/nnlib/arm64-v8a/libhexagon_controller.so to /data/local/tmp/mace_run
Push build/landmark/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1610601817.087663 to /data/local/tmp/mace_run/cmd_file-model-1610601817.087663
I mace/tools/mace_run.cc:544] model name: model
I mace/tools/mace_run.cc:545] mace version: v1.0.0-0-g0945e49
I mace/tools/mace_run.cc:546] input node: data
I mace/tools/mace_run.cc:547] input shape: 1,3,192,192
I mace/tools/mace_run.cc:548] output node: mobilepose0_conv1_fwd
I mace/tools/mace_run.cc:549] output shape: 1,128,6,6
I mace/tools/mace_run.cc:550] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:551] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:552] input dir:
I mace/tools/mace_run.cc:553] output dir:
I mace/tools/mace_run.cc:554] model_data_file:
I mace/tools/mace_run.cc:555] model_file:
I mace/tools/mace_run.cc:556] apu_cache_policy: 0
I mace/tools/mace_run.cc:557] apu_binary_file:
I mace/tools/mace_run.cc:558] apu_storage_file:
I mace/tools/mace_run.cc:559] device: GPU
I mace/tools/mace_run.cc:560] round: 100
I mace/tools/mace_run.cc:561] restart_round: 1
I mace/tools/mace_run.cc:562] gpu_perf_hint: 3
I mace/tools/mace_run.cc:563] gpu_priority_hint: 3
I mace/tools/mace_run.cc:564] num_threads: -1
I mace/tools/mace_run.cc:565] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:568] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:573] opencl_queue_window_size: 0
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
I mace/libmace/mace.cc:628] Initializing MaceEngine
I mace/libmace/mace.cc:790] Destroying MaceEngine
I mace/tools/mace_run.cc:616] restart round 0
W ./mace/utils/tuner.h:201] Failed to read tuned param file: /data/local/tmp/mace_run/model_tuned_opencl_parameter.device.rk3399.bin
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
W mace/core/kv_storage.cc:109] Failed to read kv store file: /data/local/tmp/mace_run/interior//mace_cl_compiled_program.bin
W mace/core/runtime/opencl/opencl_runtime.cc:442] Load OpenCL cached compiled kernel file failed. Please make sure the storage directory exist and you have Write&Read permission
F mace/core/rpcmem.cc:23] Check failed: rm.flag == 0 rpcmem_init failed!
F mace/core/rpcmem.cc:23] backtrace:
F mace/core/rpcmem.cc:23]  pc 0x58cdeee2c4
F mace/core/rpcmem.cc:23]  pc 0x58cdef03a0
F mace/core/rpcmem.cc:23]  pc 0x58cdef0354
F mace/core/rpcmem.cc:23]  pc 0x58cdef0550
F mace/core/rpcmem.cc:23]  pc 0x58cdef05ac
F mace/core/rpcmem.cc:23]  pc 0x58cdee1148
F mace/core/rpcmem.cc:23]  pc 0x58cded0164
F mace/core/rpcmem.cc:23]  pc 0x58cdecff30
F mace/core/rpcmem.cc:23]  pc 0x58cde0ab68
F mace/core/rpcmem.cc:23]  pc 0x58cde0edec
F mace/core/rpcmem.cc:23]  pc 0x58cde0edb8
F mace/core/rpcmem.cc:23]  pc 0x58cddd10e0
F mace/core/rpcmem.cc:23]  pc 0x58cddd1a04
F mace/core/rpcmem.cc:23]  pc 0x58cddd422c
F mace/core/rpcmem.cc:23]  pc 0x58cddd43c4
F mace/core/rpcmem.cc:23]  pc 0x7d26d04590 __libc_init
F mace/core/rpcmem.cc:23]  pc 0x58cddd0f40
Aborted

Additional context

Add any other context about the problem here, e.g., what you have modified about the code.

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions