03. GPU Version: Moving Arrays and Loops to the GPU

You rewrite the program of the 02. MPI version into heat1d_gpu, which can be built for either the GPU or the CPU. GPU memory, MISO_LAMBDA, and capturing this are explained in Chapter 9, and the points to watch in this assignment in Chapter 10. Copy the source of the MPI version to 03_gpu.cpp and rewrite it from there.

What you build

Switching the place of computation

At the top of the source, write using Backend = backend::CUDA; if __CUDACC__ is defined, and using Backend = backend::Host; otherwise.
Change the arrays used in the computation to Array1D<Real, Backend>, and the first argument of for_each and reduce to Backend{}.
Write every lambda with MISO_LAMBDA, and use util::sin and the like for math functions (std::sin cannot be used on the GPU).

Ghost-cell exchange

Create a struct Halo that bundles the exchange. Its members are as follows.

Member	Role
`int n`	The number of physical cells on this rank
`int left, right`	The left and right neighboring ranks (`MPI_PROC_NULL` if there is none)
`MPI_Comm comm`	The communicator used for communication
`Array1D<Real, Backend> send, recv`	Send and receive arrays on the computing side (2 elements; `[0]` is the left side and `[1]` is the right side)
`Array1D<Real> h_send, h_recv`	Send and receive arrays on the host (2 elements)

Allocate the arrays in the constructor’s member initializer list (Chapter 4), and do not reallocate them at every step.

The member function void apply(Array1D<Real, Backend> &u) processes in the following order.

With a for_each (Range1D{0, 1}), pack u[1] into send[0] and u[n] into send[1].
Copy to the host with h_send.copy_from(send);.
Perform the same two MPI_Sendrecv calls as in the MPI version, using h_send and h_recv. The first call sends h_send[0] to the left neighbor and receives into h_recv[1] from the right neighbor. The second call sends h_send[1] to the right neighbor and receives into h_recv[0] from the left neighbor. You can pass the addresses as h_send.data() and h_send.data() + 1.
Copy to the computing side with recv.copy_from(h_recv);.
With a for_each (Range1D{0, 1}), set u[0] = recv[0] if there is a left neighbor and u[0] = -u[1] if not, and do the same for the right side.

Do not use member variables (such as n, left, and right) inside a lambda. Before the lambda, copy them into local variables, as in const int n_ = n; and const bool has_left = (left != MPI_PROC_NULL);, and use those inside the lambda.

Error and output

Compute the error with reduce(Backend{}, ...). The return value is a host value, so you can pass it directly to MPI_Allreduce.
Before writing the output, create Array1D<Real> u_host(n + 2);, copy to the host with u_host.copy_from(u);, and then call MPI_Gather.
At the start of the first printed line, add [gpu] for the GPU build and [cpu] for the CPU build. Also distinguish the file names, as in result_N64_P2_gpu.txt / result_N64_P2_cpu.txt.

Build setup

For the CPU, as with the MPI version, just create heat1d_gpu from 03_gpu.cpp and link MISO::MISO.

For the GPU, add to CMakeLists.txt so that the following two things happen when you give -DMISO_USE_CUDA=ON to cmake.

Before bringing in MISO with add_subdirectory, enable CUDA (enable_language(CUDA)). Before that, decide the generation of the target GPU, CMAKE_CUDA_ARCHITECTURES (native if not specified).
Make 03_gpu.cpp be compiled as a CUDA source (set_source_files_properties(03_gpu.cpp PROPERTIES LANGUAGE CUDA)).

The part you add looks, for example, like the following.

option(MISO_USE_CUDA "Enable CUDA support" OFF)
if(MISO_USE_CUDA)
    if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
        set(CMAKE_CUDA_ARCHITECTURES native CACHE STRING "CUDA architectures")
    endif()
    enable_language(CUDA)
    set(CMAKE_CUDA_STANDARD 17)
    set(CMAKE_CUDA_STANDARD_REQUIRED ON)
endif()

MISO_USE_CUDA has the same name as the setting in MISO’s miso/CMakeLists.txt, so CUDA is also enabled on the MISO side, and options such as --expt-extended-lambda are added. Create the GPU build in a separate folder from the CPU build.

cmake -B build-gpu -S . -DMISO_DIR=/path/to/MISO -DMISO_USE_CUDA=ON
cmake --build build-gpu -j

What to check

CPU build (possible even without a GPU)

With cmp, check that the results of mpiexec -np P ./build/heat1d_gpu 64 ($P = 1, 2, 4$) match result_N64_P<P>.txt of the MPI version bit for bit. If they match, the rewrite to Backend and the exchange through the send and receive arrays are written correctly.

GPU build

Error: Run $N = 32, 64, 128, 256, 512$ on 1 rank, and check that max error (exact) agrees with the table of the serial version to the 4th or 5th digit, and that max error (discrete) is at most $10^{-11}$.
Number of ranks: Run with $P = 1, 2, 4$, and check that the files from the GPU builds match one another bit for bit. Even with a single GPU, multiple ranks can share the same GPU (miso::Env selects the GPU to use from the rank number within the node).
Comparison with the CPU: Compute the maximum difference over all cells from the CPU result for the same $N$ and $P$.
```
paste result_N64_P1_cpu.txt result_N64_P1_gpu.txt \
  | awk '{d = $2 - $4; if (d < 0) d = -d; if (d > m) m = d} END {print m}'
```
Treat the results as matching if the maximum difference is about $10^{-12}$ or less. The last digits can differ because of FMA and differences in the implementation of math functions, so we do not require bit-for-bit agreement (it may still match). The printed max error (discrete) can also differ from the CPU in the last digits (for example 1.249e-14 and 1.247e-14).

If an illegal memory access was encountered is printed at run time, first check whether a lambda contains a member variable or a view of a host array (Array1D<Real>). For other errors, see the reference “GPU Troubleshooting”.

Criteria for moving on

Once the CPU build matches the MPI version and, if you can use a GPU, you have checked 2 to 4 above with the GPU build, the heat conduction Practice is complete. Move on to Chapter 11 and read MISO’s time evolution.