Skip to content

Regression

Build / HIP-5.6 / projectroot.benchmarks.distributed_tree_driver.ArborX_Benchmark_DistributedTree (from CTest)

Failing for the past 1 build (Since Failed #52 )
Took 13 sec.

Stacktrace

ArborX version: 1.6 (dev)
ArborX hash   : 978e59a3
Kokkos version: 4.2.0
[e621f6f40f56:18772] *** Process received signal ***
[e621f6f40f56:18772] Signal: Segmentation fault (11)
[e621f6f40f56:18772] Signal code: Address not mapped (1)
[e621f6f40f56:18772] Failing at address: 0x18
[e621f6f40f56:18772] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f04c81ae420]
[e621f6f40f56:18772] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7f04c6a0635d]
[e621f6f40f56:18772] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7f04c6a0a47d]
[e621f6f40f56:18772] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7f04c6a15926]
[e621f6f40f56:18772] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7f04c694dc90]
[e621f6f40f56:18772] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7f04c6958b5e]
[e621f6f40f56:18772] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe4e1eb]
[e621f6f40f56:18772] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23bcc]
[e621f6f40f56:18772] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23af8]
[e621f6f40f56:18772] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb303fc]
[e621f6f40f56:18772] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f04c6075083]
[e621f6f40f56:18772] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb2fffe]
[e621f6f40f56:18772] *** End of error message ***
[e621f6f40f56:18761] *** Process received signal ***
[e621f6f40f56:18761] Signal: Segmentation fault (11)
[e621f6f40f56:18761] Signal code: Address not mapped (1)
[e621f6f40f56:18761] Failing at address: 0x18
[e621f6f40f56:18761] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7feb59b82420]
[e621f6f40f56:18761] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7feb583da35d]
[e621f6f40f56:18761] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7feb583de47d]
[e621f6f40f56:18761] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7feb583e9926]
[e621f6f40f56:18761] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7feb58321c90]
[e621f6f40f56:18761] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7feb5832cb5e]
[e621f6f40f56:18761] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe4e1eb]
[e621f6f40f56:18761] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23bcc]
[e621f6f40f56:18761] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23af8]
[e621f6f40f56:18761] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb303fc]
[e621f6f40f56:18761] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7feb57a49083]
[e621f6f40f56:18761] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb2fffe]
[e621f6f40f56:18761] *** End of error message ***
[e621f6f40f56:18762] *** Process received signal ***
[e621f6f40f56:18762] Signal: Segmentation fault (11)
[e621f6f40f56:18762] Signal code: Address not mapped (1)
[e621f6f40f56:18762] Failing at address: 0x18
[e621f6f40f56:18762] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fe1c8022420]
[e621f6f40f56:18762] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7fe1c687a35d]
[e621f6f40f56:18762] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7fe1c687e47d]
[e621f6f40f56:18762] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7fe1c6889926]
[e621f6f40f56:18762] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7fe1c67c1c90]
[e621f6f40f56:18762] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7fe1c67ccb5e]
[e621f6f40f56:18762] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe4e1eb]
[e621f6f40f56:18762] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23bcc]
[e621f6f40f56:18762] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23af8]
[e621f6f40f56:18762] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb303fc]
[e621f6f40f56:18762] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fe1c5ee9083]
[e621f6f40f56:18762] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb2fffe]
[e621f6f40f56:18762] *** End of error message ***
[e621f6f40f56:18767] *** Process received signal ***
[e621f6f40f56:18767] Signal: Segmentation fault (11)
[e621f6f40f56:18767] Signal code: Address not mapped (1)
[e621f6f40f56:18767] Failing at address: 0x18
[e621f6f40f56:18767] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fb3dd310420]
[e621f6f40f56:18767] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7fb3dbb6835d]
[e621f6f40f56:18767] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7fb3dbb6c47d]
[e621f6f40f56:18767] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7fb3dbb77926]
[e621f6f40f56:18767] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7fb3dbaafc90]
[e621f6f40f56:18767] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7fb3dbabab5e]
[e621f6f40f56:18767] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe4e1eb]
[e621f6f40f56:18767] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23bcc]
[e621f6f40f56:18767] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23af8]
[e621f6f40f56:18767] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb303fc]
[e621f6f40f56:18767] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fb3db1d7083]
[e621f6f40f56:18767] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb2fffe]
[e621f6f40f56:18767] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec noticed that process rank 3 with PID 0 on node e621f6f40f56 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Standard Output

ArborX version: 1.6 (dev)
ArborX hash   : 978e59a3
Kokkos version: 4.2.0
[e621f6f40f56:18772] *** Process received signal ***
[e621f6f40f56:18772] Signal: Segmentation fault (11)
[e621f6f40f56:18772] Signal code: Address not mapped (1)
[e621f6f40f56:18772] Failing at address: 0x18
[e621f6f40f56:18772] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f04c81ae420]
[e621f6f40f56:18772] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7f04c6a0635d]
[e621f6f40f56:18772] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7f04c6a0a47d]
[e621f6f40f56:18772] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7f04c6a15926]
[e621f6f40f56:18772] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7f04c694dc90]
[e621f6f40f56:18772] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7f04c6958b5e]
[e621f6f40f56:18772] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe4e1eb]
[e621f6f40f56:18772] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23bcc]
[e621f6f40f56:18772] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23af8]
[e621f6f40f56:18772] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb303fc]
[e621f6f40f56:18772] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f04c6075083]
[e621f6f40f56:18772] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb2fffe]
[e621f6f40f56:18772] *** End of error message ***
[e621f6f40f56:18761] *** Process received signal ***
[e621f6f40f56:18761] Signal: Segmentation fault (11)
[e621f6f40f56:18761] Signal code: Address not mapped (1)
[e621f6f40f56:18761] Failing at address: 0x18
[e621f6f40f56:18761] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7feb59b82420]
[e621f6f40f56:18761] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7feb583da35d]
[e621f6f40f56:18761] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7feb583de47d]
[e621f6f40f56:18761] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7feb583e9926]
[e621f6f40f56:18761] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7feb58321c90]
[e621f6f40f56:18761] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7feb5832cb5e]
[e621f6f40f56:18761] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe4e1eb]
[e621f6f40f56:18761] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23bcc]
[e621f6f40f56:18761] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23af8]
[e621f6f40f56:18761] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb303fc]
[e621f6f40f56:18761] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7feb57a49083]
[e621f6f40f56:18761] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb2fffe]
[e621f6f40f56:18761] *** End of error message ***
[e621f6f40f56:18762] *** Process received signal ***
[e621f6f40f56:18762] Signal: Segmentation fault (11)
[e621f6f40f56:18762] Signal code: Address not mapped (1)
[e621f6f40f56:18762] Failing at address: 0x18
[e621f6f40f56:18762] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fe1c8022420]
[e621f6f40f56:18762] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7fe1c687a35d]
[e621f6f40f56:18762] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7fe1c687e47d]
[e621f6f40f56:18762] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7fe1c6889926]
[e621f6f40f56:18762] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7fe1c67c1c90]
[e621f6f40f56:18762] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7fe1c67ccb5e]
[e621f6f40f56:18762] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe4e1eb]
[e621f6f40f56:18762] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23bcc]
[e621f6f40f56:18762] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23af8]
[e621f6f40f56:18762] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb303fc]
[e621f6f40f56:18762] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fe1c5ee9083]
[e621f6f40f56:18762] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb2fffe]
[e621f6f40f56:18762] *** End of error message ***
[e621f6f40f56:18767] *** Process received signal ***
[e621f6f40f56:18767] Signal: Segmentation fault (11)
[e621f6f40f56:18767] Signal code: Address not mapped (1)
[e621f6f40f56:18767] Failing at address: 0x18
[e621f6f40f56:18767] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fb3dd310420]
[e621f6f40f56:18767] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7fb3dbb6835d]
[e621f6f40f56:18767] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7fb3dbb6c47d]
[e621f6f40f56:18767] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7fb3dbb77926]
[e621f6f40f56:18767] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7fb3dbaafc90]
[e621f6f40f56:18767] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7fb3dbabab5e]
[e621f6f40f56:18767] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe4e1eb]
[e621f6f40f56:18767] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23bcc]
[e621f6f40f56:18767] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xe23af8]
[e621f6f40f56:18767] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb303fc]
[e621f6f40f56:18767] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fb3db1d7083]
[e621f6f40f56:18767] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xb2fffe]
[e621f6f40f56:18767] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec noticed that process rank 3 with PID 0 on node e621f6f40f56 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------