Regression
Build / HIP-5.6 / projectroot.benchmarks.distributed_tree_driver.ArborX_Benchmark_DistributedTree (from CTest)
Failing for the past 1 build
(Since Failed
)
Stacktrace
ArborX version: 1.7 (dev) ArborX hash : a403dad8 Kokkos version: 4.2.0 [6a99dc0fa1df:28104] *** Process received signal *** [6a99dc0fa1df:28104] Signal: Segmentation fault (11) [6a99dc0fa1df:28104] Signal code: Address not mapped (1) [6a99dc0fa1df:28104] Failing at address: 0x18 [6a99dc0fa1df:28104] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f1abb47d420] [6a99dc0fa1df:28104] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7f1ab9cd535d] [6a99dc0fa1df:28104] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7f1ab9cd947d] [6a99dc0fa1df:28104] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7f1ab9ce4926] [6a99dc0fa1df:28104] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7f1ab9c1cc90] [6a99dc0fa1df:28104] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7f1ab9c27b5e] [6a99dc0fa1df:28104] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0x1006b4b] [6a99dc0fa1df:28104] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc52c] [6a99dc0fa1df:28104] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc458] [6a99dc0fa1df:28104] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3fec] [6a99dc0fa1df:28104] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f1ab9344083] [6a99dc0fa1df:28104] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3bee] [6a99dc0fa1df:28104] *** End of error message *** [6a99dc0fa1df:28115] *** Process received signal *** [6a99dc0fa1df:28115] Signal: Segmentation fault (11) [6a99dc0fa1df:28115] Signal code: Address not mapped (1) [6a99dc0fa1df:28115] Failing at address: 0x18 [6a99dc0fa1df:28115] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f67b8bc9420] [6a99dc0fa1df:28115] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7f67b742135d] [6a99dc0fa1df:28115] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7f67b742547d] [6a99dc0fa1df:28115] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7f67b7430926] [6a99dc0fa1df:28115] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7f67b7368c90] [6a99dc0fa1df:28115] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7f67b7373b5e] [6a99dc0fa1df:28115] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0x1006b4b] [6a99dc0fa1df:28115] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc52c] [6a99dc0fa1df:28115] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc458] [6a99dc0fa1df:28115] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3fec] [6a99dc0fa1df:28115] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f67b6a90083] [6a99dc0fa1df:28115] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3bee] [6a99dc0fa1df:28115] *** End of error message *** [6a99dc0fa1df:28110] *** Process received signal *** [6a99dc0fa1df:28110] Signal: Segmentation fault (11) [6a99dc0fa1df:28110] Signal code: Address not mapped (1) [6a99dc0fa1df:28110] Failing at address: 0x18 [6a99dc0fa1df:28110] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fd5d5fb4420] [6a99dc0fa1df:28110] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7fd5d480c35d] [6a99dc0fa1df:28110] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7fd5d481047d] [6a99dc0fa1df:28110] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7fd5d481b926] [6a99dc0fa1df:28110] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7fd5d4753c90] [6a99dc0fa1df:28110] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7fd5d475eb5e] [6a99dc0fa1df:28110] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0x1006b4b] [6a99dc0fa1df:28110] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc52c] [6a99dc0fa1df:28110] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc458] [6a99dc0fa1df:28110] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3fec] [6a99dc0fa1df:28110] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fd5d3e7b083] [6a99dc0fa1df:28110] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3bee] [6a99dc0fa1df:28110] *** End of error message *** [6a99dc0fa1df:28105] *** Process received signal *** [6a99dc0fa1df:28105] Signal: Segmentation fault (11) [6a99dc0fa1df:28105] Signal code: Address not mapped (1) [6a99dc0fa1df:28105] Failing at address: 0x18 [6a99dc0fa1df:28105] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fbb9dcf5420] [6a99dc0fa1df:28105] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7fbb9c54d35d] [6a99dc0fa1df:28105] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7fbb9c55147d] [6a99dc0fa1df:28105] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7fbb9c55c926] [6a99dc0fa1df:28105] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7fbb9c494c90] [6a99dc0fa1df:28105] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7fbb9c49fb5e] [6a99dc0fa1df:28105] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0x1006b4b] [6a99dc0fa1df:28105] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc52c] [6a99dc0fa1df:28105] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc458] [6a99dc0fa1df:28105] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3fec] [6a99dc0fa1df:28105] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fbb9bbbc083] [6a99dc0fa1df:28105] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3bee] [6a99dc0fa1df:28105] *** End of error message *** -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpiexec noticed that process rank 3 with PID 0 on node 6a99dc0fa1df exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
Standard Output
ArborX version: 1.7 (dev) ArborX hash : a403dad8 Kokkos version: 4.2.0 [6a99dc0fa1df:28104] *** Process received signal *** [6a99dc0fa1df:28104] Signal: Segmentation fault (11) [6a99dc0fa1df:28104] Signal code: Address not mapped (1) [6a99dc0fa1df:28104] Failing at address: 0x18 [6a99dc0fa1df:28104] [6a99dc0fa1df:28104 [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f1abb47d420] [6a99dc0fa1df:28104] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7f1ab9cd535d] [6a99dc0fa1df:28104] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7f1ab9cd947d] [6a99dc0fa1df:28104] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7f1ab9ce4926] [6a99dc0fa1df:28104] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7f1ab9c1cc90] [6a99dc0fa1df:28104] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7f1ab9c27b5e] [6a99dc0fa1df:28104] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0x1006b4b] [6a99dc0fa1df:28104] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc52c] [6a99dc0fa1df:28104] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc458] [6a99dc0fa1df:28104] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3fec] [6a99dc0fa1df:28104] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f1ab9344083] [6a99dc0fa1df:28104] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3bee] [6a99dc0fa1df:28104] *** End of error message *** [6a99dc0fa1df:28115] *** Process received signal *** [6a99dc0fa1df:28115] Signal: Segmentation fault (11) [6a99dc0fa1df:28115] Signal code: Address not mapped (1) [6a99dc0fa1df:28115] Failing at address: 0x18 [6a99dc0fa1df:28115] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f67b8bc9420] [6a99dc0fa1df:28115] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7f67b742135d] [6a99dc0fa1df:28115] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7f67b742547d] [6a99dc0fa1df:28115] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7f67b7430926] [6a99dc0fa1df:28115] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7f67b7368c90] [6a99dc0fa1df:28115] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7f67b7373b5e] [6a99dc0fa1df:28115] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0x1006b4b] [6a99dc0fa1df:28115] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc52c] [6a99dc0fa1df:28115] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc458] [6a99dc0fa1df:28115] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3fec] [6a99dc0fa1df:28115] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f67b6a90083] [6a99dc0fa1df:28115] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3bee] [6a99dc0fa1df:28115] *** End of error message *** [6a99dc0fa1df:28110] *** Process received signal *** [6a99dc0fa1df:28110] Signal: Segmentation fault (11) [6a99dc0fa1df:28110] Signal code: Address not mapped (1) [6a99dc0fa1df:28110] Failing at address: 0x18 [6a99dc0fa1df:28110] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fd5d5fb4420] [6a99dc0fa1df:28110] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7fd5d480c35d] [6a99dc0fa1df:28110] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7fd5d481047d] [6a99dc0fa1df:28110] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7fd5d481b926] [6a99dc0fa1df:28110] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7fd5d4753c90] [6a99dc0fa1df:28110] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7fd5d475eb5e] [6a99dc0fa1df:28110] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0x1006b4b] [6a99dc0fa1df:28110] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc52c] [6a99dc0fa1df:28110] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc458] [6a99dc0fa1df:28110] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3fec] [6a99dc0fa1df:28110] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fd5d3e7b083] [6a99dc0fa1df:28110] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3bee] [6a99dc0fa1df:28110] *** End of error message *** [6a99dc0fa1df:28105] *** Process received signal *** [6a99dc0fa1df:28105] Signal: Segmentation fault (11) [6a99dc0fa1df:28105] Signal code: Address not mapped (1) [6a99dc0fa1df:28105] Failing at address: 0x18 [6a99dc0fa1df:28105] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fbb9dcf5420] [6a99dc0fa1df:28105] [ 1] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33235d)[0x7fbb9c54d35d] [6a99dc0fa1df:28105] [ 2] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x33647d)[0x7fbb9c55147d] [6a99dc0fa1df:28105] [ 3] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x341926)[0x7fbb9c55c926] [6a99dc0fa1df:28105] [ 4] /opt/rocm-5.6.0/lib/libamdhip64.so.5(+0x279c90)[0x7fbb9c494c90] [6a99dc0fa1df:28105] [ 5] /opt/rocm-5.6.0/lib/libamdhip64.so.5(hipStreamCreate+0x14e)[0x7fbb9c49fb5e] [6a99dc0fa1df:28105] [ 6] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0x1006b4b] [6a99dc0fa1df:28105] [ 7] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc52c] [6a99dc0fa1df:28105] [ 8] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xfdc458] [6a99dc0fa1df:28105] [ 9] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3fec] [6a99dc0fa1df:28105] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fbb9bbbc083] [6a99dc0fa1df:28105] [11] /var/jenkins/workspace/ArborX_master/build/benchmarks/distributed_tree_driver/ArborX_Benchmark_DistributedTree.exe[0xcb3bee] [6a99dc0fa1df:28105] *** End of error message *** -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpiexec noticed that process rank 3 with PID 0 on node 6a99dc0fa1df exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------