Changes

Introduce constructor for multi-GPU support.

Fix typo

Co-authored-by: Dong Hun Lee <59181952+ldh4@users.noreply.github.com>

Add test

Fix typo.

Co-authored-by: Dong Hun Lee <59181952+ldh4@users.noreply.github.com>

Explicitly check for valid device id

Set the device id in cuda_kernel_arch

Check for default device

Check that device associated with stream matches requested device

Remove extra constructor

Remove logical memory spaces

Address reviewer comments

m_cudaDev isn't static anymore

Set the device id explicitly for CUDA API calls in impl_initialize

Add warp sync for Cuda parallel reduce

compute-sanitizer --tool=racecheck discovered a potential racecondition for Cuda parallel reductions (using range policy) where data was being updated on a single thread inside a warp, but the warp was not being synchronized before being read.

update comment to include final() mention

This PR fixes the too-much-OpenACC-warning issue, mentioned in PR #6639.
This PR also re-enables the OpenACC CI test.

add missing header fix #6644

Fixup test math functions ulp should double -> int

Drop DualView converting copy assignment operator

It is unnecessary, we already have a converting constructor and a
regular copy assignment operator.

Don't use rocm-docker for clang-format

Diable HIP CI

Remove deprecation warning for AllocationMechanism for gcc <11.0

OpenMPTarget: clang extensions for dynamic shared memory. (#6380)

* OpenMPTarget: clang extensions for dynamic shared memory.

* OpenMPTarget: Rebase and use macros for pragma.

* OpenMPTarget: Fix bug. Add Macros file.

* OpenMPTarget: Fix resize_scratch.

* OpenMPTarget: restore map clause.

* OpenMPTarget: Fix comment.

* OpenMPTarget: change L0 values in resize_scratch.

* OpenMPTarget: Fix comment and spaces.

---------

Co-authored-by: Rahulkumar Gayatri <rgayatri@lbl.gov>

Fix builtin_unreachable use for MSVC/CUDA

Also split math functions test differently to avoid need for bigobj

Fix missing include on msvc/cuda

Avoid lambdas in constexpr branch for msvc/cuda

Sidestep lacking CTAD support msvc/cuda

Fix formatting

Move header for Damien because he is right

Unit test for issue 3371 (negative vector length should not yield a negative max_team_size) (#6076)

* #3371: Added TeamPolicy construction test

* Check recommended team size

* Move size_max and size_recommended tests to TeamPolicy constructor test

* Define tag type outside of test body

* Add missing overload

* tests: remove spurious check

* Remove redundant qualifier

Co-authored-by: Damien L-G <dalg24+github@gmail.com>

* Fix formatting

---------

Co-authored-by: Cezary Skrzyński <cezary.skrzynski@ng-analytics.com>
Co-authored-by: Cezary Skrzyński <czars1988@gmail.com>
Co-authored-by: Damien L-G <dalg24+github@gmail.com>

Add CMakeLists.txt for stream benchmark

Do not negate the dependent true traits helper

Drop guards to accommodate external code defining KOKKOS_ASSERT

Use omp_get_max_active_levels() when supported

Add missing gfx940

Add Impl::always_false type-dendent false trait

Per review prefer always_false<Arg>::value to is_void_v<Arg>

Improve "no copy mechanism" exception message

Add a unit test for new deep_copy exception msg

Add missing include sstream

src->source, dst->destination

and reformat a bit

Workaround for ROCm 6.0 failing to compile with AVX2 SIMD support

SYCL: Force inlining of Kokkos::printf (#6650)

* SYCL: Force inlining of Kokkos::printf

* Always force inline Kokkos::printf

Improve handling of printf in OMPT on Intel GPUs

OpenMP: Use `omp_get_nested` for older gcc versions (#6685)

* OpenMP: fix for issue 6670.

* OpenMP: Update gcc version when using max_active_level.

* OpenMP: Edit execute_in_serial.

* Apply suggestions from code review

Co-authored-by: Daniel Arndt <arndtd@ornl.gov>

* clang-format

---------

Co-authored-by: Rahulkumar Gayatri <rgayatri@lbl.gov>
Co-authored-by: Damien L-G <dalg24+github@gmail.com>
Co-authored-by: Daniel Arndt <arndtd@ornl.gov>

Disable more Bessel tests for SYCL on INtel GPUs

fill_random without exceution space instance should fence

Drop unnecessary guarding for a tool library being loaded in ProfilingSection

Drop unnecessary header include in Kokkos_Profiling_ProfileSection.hpp

#5333: CUDA: Use scratch space appropriate to small reduction elements in Team reductions (#5334)

* #5333: Add maybe failing test case?

* Revise test to try to target the right code

* NOMERGE #5333: Partial implementation of fix

* #5333: Update tests

* #5333: Fix comparision types in reducers test

* #5333: Re-enable tests

* #5333: Add test for Scalar with size of 1

* Fix formatting

* Remove unrelated changes

* Extract TeamPolicy tests into a separate method

* Disable new tests (OpenACC, OpenMPTarget, Serial, SYCL, point_t)

* Use smaller number of teams for bhalf_t

* Use pointer_type for m_scratch_space parameter

* Skip failing tests (SYCL)

* Fix tests for OpenMPTarget

* Fix num_teams_done variable in SYCL TeamPolicy reduction

* Restore guards for array_reduce<float, 7>

* Also make array_reduce<float, 7> work for SYCL

* tests: change point_t default constructor

---------

Co-authored-by: Arkadiusz Szczepkowicz <arek.szczepkowicz@ng-analytics.com>
Co-authored-by: Cezary Skrzyński <cezary.skrzynski@ng-analytics.com>
Co-authored-by: Daniel Arndt <arndtd@ornl.gov>

Cuda: Allocate using the correct device (#6392)

* Cuda: Allocate using the correct device

* Avoid warning about uninitialized variable

* exec_space_provided -> stream_sync_only

* Fix up ASYNC support

* Only check for errors in synchronization if allocation was successful

* Rename arguments and make constructor private

---------

Co-authored-by: Daniel Arndt <arndtd@ornl.com>

Let `Profiling::ProfilingSection(std::string)` constructor be explicit and nodiscard (#6690)

* Add nodiscard attribute to Profiling::ProfilingSection

* Make Profiling::ProfilingSection constructor explicit

Cosmetic changes to ProfilingSection

GitHub CI: Test with AddressSanitizer (#6676)

* GitHub CI: Test with AddressSanitizer

* Disable view_allocation_error when running with AddressSanitizer

* Try  __attribute__((no_sanitize(address)))

* Revert "Try  __attribute__((no_sanitize(address)))"

This reverts commit 335f7f26ae96034ceae221e5a80bd93b4769ed1d.

Kokkos::Array deduction guide (#6373)

* Added a deduction guide for Kokkos::Array

* Added unit test for Kokkos::Array deduction guide

* Decorated is_equal with KOKKOS_FUNCTION

Add CI for MSVC+Cuda (#6661)

SYCL: Address deprecations after oneAPI 2023.2.0 (#6577)

* Address deprecations in oneAPI 2023.2.0

* Define KOKKOS_IMPL_SYCL_GET_MULTI_PTR

Fixup cast tolerance to double before printing

Try linking against CUDA libararies even with KOKKOS_ENABLE_COMPILE_AS_CMAKE_LANGUAGE

test_array_ctad: disable test for intel versions < 2021

Address issue #6702

Enable `{transform_}exclusive_scan` in place (#6667)

* Enable exclusive_scan in place

* add test for inplace operation

* improve tests

* formatting

* fix test for transform_*

* simplify args

* revise tests

---------

Co-authored-by: Francesco Rizzi <fnrizzi@sandia.gov>

Add `ATOMICS_BYPASS` configuration option to disable atomics (#6692)

* Add NAME_TBD_UNSAFE_ATOMICS configuration option to disable atomics

* Rename configuration option and macro NAME_TBD_UNSAFE_ATOMICS -> ATOMICS_BYPASS

* Improve option description and configuration time error message

Co-Authored-By: Christian Trott <crtrott@sandia.gov>
Co-Authored-By: Daniel Arndt <arndtd@ornl.gov>

* Mention Kokkos_ENABLE_ATOMICS_BYPASS option in error msssage

* Simplify sanity check for disabling atomics in <Kokkos_Macros.hpp>

Co-Authored-By: Daniel Arndt <arndtd@ornl.gov>

---------

Co-authored-by: Christian Trott <crtrott@sandia.gov>
Co-authored-by: Daniel Arndt <arndtd@ornl.gov>

Check matching static extents in View constructor (#5190)

* Also check static extents

* Use separate bools for the two checks and improve error messages

* Improve error messages

* Refactor test

* Add another test

* Extend existing tests

* Use make_index_sequence

* Remove LIVE and DIE

* Try removing guards for OpenMPTarget

* Use different style in view_construction_with_wrong_static_extents

* Remove template template parameter

* Also check unmanaged views

* Fix DynRankView

* Add FIXME

* Pass label by const char *

* fix terminating define

* Deduce rank and dynamic_rank from View

* Only call runtime_check_rank when KOKKOS_ENABLE_DEBUG_BOUNDS_CHECK is defined

* Guard with if constexpr

---------

Co-authored-by: Francesco Rizzi <fnrizzi@sandia.gov>

Remove Kokkos::[b]half_t volatile overloads (#6579)

* Deprecate Kokkos::[b]half_t volatile overloads

* Remove volatile [b]half_t altogether

add tests

Provide `kokkos_swap` as part of Core and deprecate `Experimental::swap` in Algorithms (#6697)

* Move Kokkos_Swap.hpp header to core

* Include only what we use (in <Kokkos_Swap.hpp>)

* Rename header guards in <Kokkos_Swap.hpp> (because I can)

* Drop useless comment

* Fix noexcept specification of Kokkos::swap

* Constexprify Kokkos::swap

* Fixup type requirements in Kokkos::swap

* Drop the inline (because I can)

* fixup! Move Kokkos_Swap.hpp header to core

* Promote swap to namespace Kokkos::

* Move and update Kokkos::swap unit test to core

* Add support for swapping arrays

* Fixup size_t

* Provide Kokkos::swap overload for a few classes

* Rename Kokkos::swap -> Kokkos::kokkos_swap

* For backward compatibility provide deprecated Experimental::swap

Provide new public headers `<Kokkos_Clamp.hpp>` and `<Kokkos_MinMax.hpp>` (#6687)

* Promote min, max, clamp to public

* Drop unnecessary <Kokkos_MinMaxClamp.hpo> header includes

* Split Kokkos_MinMaxClamp.hpp into Kokkos_MinMax.hpp and Kokkos_Clamp.hpp

---------

Co-authored-by: Damien L-G <dalg24@gmail.com>

Fix TeamThreadMDRange parallel_reduce (#6511)

* Fix TeamThreadRangeMD parallel_reduce

* Use vector_reduce for Cuda, HIP, and SYCL

* Initialize reduction variable

* Test with maximal vector length

* TeamHandle::execution_space->typename TeamHandle::execution_space

* Fix OpenMPTarget

* Restore tests

* reducer is unused

* Workarounds for OpenMPTarget

* Reduce

* Fix OpenMPTarget tests

* Restore tests

* Strengthen test case for OpenMPTarget

* Add some static_asserts

add tests for in-place `inclusive_scan` (#6682)

* add test for inclusive scan in place

* refine tests

* remove comment

* revert transform since it is moved to separate pr

Drop pointless Kokkos::Impl::CudaExec forward declaration

There is no defintion for it...

Don't use the compiler launcher script if the compile language is CUDA. (#6704)

* Don't use the compiler launcher script if the compile language is CUDA.

* Updated descriptive comment.

Deprecate `{Cuda,HIP}::detect_device_count()` and `Cuda::[detect_]device_arch()` (#6710)

* CUDA/HIP: Inline getting device count

get_device_count() -> int cannot be generic. Get rid of the extra
indirection because it brings no benefit.

* Get rid of CudaInternalDevices and cleanup Cuda::print_configuration()

* Get rid of cuda_get_device_{count,properties}_wrapper()

* Missed a few CudaInternalDevices and get rid of Cuda::detect_device_arch()

* Get rid of Cuda::device_arch()

* Fixup

Co-authored-by: Bruno Turcksin <bruno.turcksin@gmail.com>

* Don’t mess with Voodoo

* Be more conservative and deprecate before removing

* Clang-format for suggestion made on GH

* Remove stray const qualifier

* Forgot that device_arch() was static

* Who Let the Bugs Out??

* The same thing we do every night, Pinky - try to take over the world!

* this should fix it

---------

Co-authored-by: Bruno Turcksin <bruno.turcksin@gmail.com>

Get rid of CudaInternal::cuda_get_error_{name,string}_wrapper

No need to jump through so many hoops to print the error message

HIP: Forgot to delete matching brace closing the namespace

Make initialize and finalize of the Cuda/HIP singleton less special (#6714)

Make initialization of the Cuda/HIP singleton less special

Kokkos_HIP.cpp: include Kokkos_Core.hpp to resolve errors

Resolves errors of form:
/home/jenkins/caraway-new/workspace/KokkosKernels_PullRequest_VEGA90A_ROCM560/kokkos/core/src/HIP/Kokkos_HIP.cpp:96:15: error: no member named 'hip_global_unique_token_locks' in namespace 'Kokkos::Impl'
  (void)Impl::hip_global_unique_token_locks(true);

Note: include of Kokkos_HIP_UniqueToken.hpp was insufficient as this triggered new errors:
/home/ndellin/kokkos/core/src/HIP/Kokkos_HIP_UniqueToken.hpp:40:29: error: implicit instantiation of undefined template 'Kokkos::View<unsigned int *, Kokkos::HIPSpace>'
  View<uint32_t*, HIPSpace> m_locks;

Add bound checks in RangePolicy and MDRangePolicy (#6617)

* Added a bounds check in MDRangePolicy that checks that all lower bounds are less than its upper bound

* Modified the wording on the abort

* Converted the error msg from a stringstream to a string

* Modified abort msg

* Fixed the unit test output based on backend's default iterate direction

* Update core/unit_test/TestMDRangePolicyConstructors.hpp

Formatting.

Co-authored-by: Damien L-G <dalg24+github@gmail.com>

* Updated RangePolicy to have the same precondition as MDRangePolicy

---------

Co-authored-by: Damien L-G <dalg24+github@gmail.com>

Temporary fix to reenable HIP CI

Let the smart pointer manage the CUDA/HIP stream (#6721)

* Let the smart pointer manage the CUDA/HIP stream

* Fixup do not null the stream in {Cuda/HIP}Internal::finalize()

* Fixup hip not tested either

* Not breaking HIP backend for 3rd time in a row

Co-authored-by: Bruno Turcksin <bruno.turcksin@gmail.com>

---------

Co-authored-by: Bruno Turcksin <bruno.turcksin@gmail.com>

Fix Docker env variables

Ensure view_allocation_error does not silently ignore that no exception was thrown

Add RawMemoryAllocationFailure::AllocationMechanism::OpenACCMalloc enumerator

[OpenACC] throw if acc_malloc returned nullptr

Fixup using declaration

Disable openacc.view_allocation_error test

Guard `[MD]RangePolicy` precondition check for deprecated code 4 (#6726)

* Guard [MD]RangePolicy precondition check for deprecated code 4

* No good reason to use a raw string literal for the warning msg

* Drop pointless inline specifier

* Restore original behavior when deprecated code 4 is enabled

* first blush

* Fixup obviously wont display both message when aborting

* Trust me, it's not badly written. It's just way above your head.

* Add assertions for the old behavior

* Fiddling with string comparison

* Attempt to resolve death tests error msg matching issues

* Fix that stupid regex

Add C++26 standard to CMake Setup

Add support for C++26 in generated makefiles

Add KOKKOS_ENABLE_CXX26 to the configuration metadata

Reenable HIP testing

Disabling failing HIP test in the CI

Use team_size_max to fix "Team size too large" error in reducer test (#6725)

* Fix test to not use team size too large

For some configurations, one of the reducer test was setting a team size too large.

* Remove team_size calculation from test not using it

* Remove unused typedef

* Use int in for-loop to avoid warning

Co-authored-by: Damien L-G <dalg24+github@gmail.com>

* Fix teamsize request for OPENMPTARGET

* Cleanup test and match TestTeam.hpp

Use 32 as a minimum team size like in TestTeam.hpp. Better match layout
in the 3 tests here.

---------

Co-authored-by: Damien L-G <dalg24+github@gmail.com>

Don't touch my records! (refactor Cuda/HIP/SYCL/Threads to not directly mess with `SharedAllocationRecord`) (#6732)

* Do not use SharedAllocationRecord directly

* Purge Cuda/HIP graph implementation from SharedAllocationRecord

* MemorySpace::{free -> allocate} and FIXMEs for size argument

* Properly cast allocated pointer from void* to size_type*

* Fixup Cuda/HIP graph header includes and pointer casting

* Fix the FIXMEs

Untangle SharedAllocationRecord spaghetti code

Trilinos: Don't let Kokkos set CMAKE_CXX_FLAGS

Fix TestThreadVectorMDRangeParallelReduce (#6734)

* Fix TestThreadVectorMDRangeParallelReduce

* Eliminate teamSum

Cuda multi-GPU support: Allow execution space instance constructor to run (#6706)

* Cuda multi-GPU support: Allow execution space instance constructor to run

* Skip a test

* Use cuda_stream/device also for UVM and HostPinned

* Clean up

* Revert test changes

add more warp sync for cuda reductions

Drop support for deprecated command-line arguments and environment variables (#6744)

* Drop support for deprecated cmdline args and env vars

* Deprecate num_devices and skip_device in InitializationSettings

* Fixup rm now unused get_flag utility lambda

Avoid unnecessary zero-memset of the scratch flags in SYCL (#6739)

* SYCL: Homogenize scratch_flags with CUDA and HIP

* Add comments for CUDA and HIP

* Fix typo [ci skip]

---------

Co-authored-by: Daniel Arndt <arndtd@ornl.gov>

std(remove-if): fixing tmp view alloc + avoid evaluating twice the predicate during final pass