GpuKernelLaunch¶
Launches a GPU kernel.
Declaration¶
Syntax¶
GpuKernelLaunch(symbol kernel, expr thread_count, expr block_size,
call_arg* args, symbol? component_fit, gpu_scratch_buffer* scratch,
expr? stop_status, stmt* stop_replay, expr* overwritten,
expr* overwritten_if, gpu_launch_dimension* chunk_dimensions,
int chunk_fixed_iterations, symbol? first_chunk_parameter)
gpu_scratch_buffer = (int argument_index, expr size, expr? read_back)
gpu_launch_dimension = (symbol start_parameter, symbol end_parameter)
Arguments¶
Argument |
Description |
|---|---|
|
the Function to run. Its signature has |
|
the number of threads the launch numbers: one per iteration of the loop, or, when the nest is launched in tiles, one per place in whole tiles; a count below one runs none. |
|
the number of threads per block. |
|
the arguments passed to every thread. |
|
the Block the host runs before the launch to give the allocatable array components of elements of an array argument the loop writes the storage the kernel writes into, or |
|
the arguments that are buffers only the device holds (below). |
|
the host array passed as the status buffer of a kernel that can |
|
the host statements that run the |
|
array arguments the kernel may assign whole before it reads them (below). |
|
for each of |
|
the dimensions of the iteration space a launch of more iterations than the chunk limit splits into chunks, each run as a launch of the kernel, in the order the kernel numbers its iterations (below); empty when it always runs as one. |
|
the product of the constant extents of the dimensions every chunk covers whole: 1 when there are none, and 0 when |
|
the kernel parameter a launch with |
Return values¶
None.
Description¶
A launch is not a call: it starts thread_count threads, rounded up to whole
blocks of block_size, and returns without waiting for them. The threads are
numbered from zero, block after block; the kernel sends every thread numbered
thread_count or higher back before its body, so the host sizes what the
launch holds per thread – a workspace slice – by thread_count, not by the
threads it starts.
On a device that asks for tiles (CUDA), a nest of two or more dimensions
without a reduction can be launched in tiles: a block covers a tile of 32
iterations of the dimension its stores are contiguous in by the rest of the
block along a second dimension, the one the stores step through next in
memory (or, if they do not agree on one, the next one in loop order). The
block index numbers the tiles, and thread_count counts whole tiles. Where
whole tiles would pad the nest by more than a quarter, as a short dimension
does, or would start more threads than the chunk limit (below), the host
tells the kernel to number the iterations flat instead, and thread_count
counts iterations. Padding to whole tiles can take
thread_count past the kind of the loop indices, so it is an integer(8).
The kernel checks the zero-based tile coordinates against the extents before
adding the loop lower bounds. A padded thread returns before constructing a
source index that could overflow even when every valid index is representable.
GpuSync is what waits; the gpu_launch_sync
pass places it where the host next needs what a kernel wrote (see
GPU offloading).
The execution configuration is part of the node rather than of the kernel, because the same kernel is normally launched with different sizes.
The kernel is an ordinary Function; what makes
it launchable is that its signature has exec_space = Kernel, and a kernel
has no result. See exec_space.
The gpu_offload pass replaces a parallel loop assigned to the device with its
launch, together with the copies the host makes before and after it; nothing
of the original loop is kept. gpu_kernel_finalize records the kernel’s
Function.gpu layout after shared lowering. device_launch_expand and both
device emitters consume that layout. Workspace sizes are evaluated once on the host and passed as
explicit scalar arguments, so host allocation and device indexing use the
same values.
Before creating a launch, GPU offload checks the allocation of allocatable array results in callees that remain out of line. Nested or multiple allocation sites must agree with a buffer shape established by an unconditional fixed allocation. Otherwise the loop is rejected while the original loop is still available, which is an error: it is a lowering LFortran does not have yet. Reallocation and conditional assignments remain eligible when they preserve every extent, not just the total element count.
A loop reaches GPU offload only after the unsupported-construct check in the
parallel_dispatch pass has assigned it to the device (see
GPU offloading), so a loop the offload cannot
lower is a compile-time error, whatever its reason, whether or not
--gpu-allow-cpu-fallback is given, and also with --show-gpu-kernel-source.
A scratch entry makes the argument argument_index a buffer of size
elements of the kernel argument’s element type that only the device holds,
such as an arena of the device heap: the host allocates nothing for it and
copies nothing in or out. The actual argument is an allocatable array of the
launching procedure, not allocated before the launch. When read_back is
present, the host allocates the actual with read_back elements after the
launch, at most size, and copies the leading elements of the buffer into
it; otherwise the actual stays unallocated.
A kernel that can stop records the statement a thread reached in the
status buffer stop_status. The statements of stop_replay read it and run
that statement on the host. They run after the first GpuSync that follows
the launch, which the gpu_launch_sync pass places, copies of them after
each GpuSync the launch may still be pending at; that pass also clears the
first element of stop_status once, on entry to the procedure.
When overwritten_if[i], a scalar logical the host evaluates at the launch,
is true, the kernel assigns every element of the array overwritten[i], an
actual argument of the launch, before it reads any of them. What the host
holds there then does not reach the kernel, so device_launch_expand hands
the buffer to the runtime as LFORTRAN_GPU_WRITE_DISCARD, and a runtime
with memory of its own only makes room for it instead of copying it in.
gpu_offload finds such arrays in the loop as the source wrote it: an
assignment a(i, j, ...) = value subscripted by the loop variables, one per
loop, that each iteration reaches before anything else of it names a, at
the top of the iteration or inside IFs that compare loop variables with
values the loop cannot change. The condition is that those loops, narrowed
by the IFs, run over exactly the bounds of a.
This proof does not imply that the buffer is unaliased. If any other buffer
argument of the launch overlaps it, the runtime treats the discard as an
ordinary write so that the other argument can still read its previous
contents. The kernel layout’s gpu_kernel_argument.noalias is a separate
shared-ASR proof; it is not inferred from overwritten.
A launch with chunk_dimensions runs as several launches of its kernel when
it has more iterations than the chunk limit (huge(0), which the hidden,
test-only option --gpu-chunk-limit lowers): one per chunk, a box of the
iteration space of at most that many iterations. The iterations of a do concurrent may run in any order, so the chunks run one after another, and
the kernel numbers the iterations of each chunk from the bounds of its box,
from zero. A chunk has at most huge(0) iterations, so a kernel whose loop
indices are all integer(4) numbers them in integer(4), while the launch
counts the iterations of the whole nest in integer(8); an integer(8)
index keeps integer(8) numbering.
A chunk covers every dimension not in chunk_dimensions whole; their
extents are constants, and their product is chunk_fixed_iterations, at
most the limit. A bound whose value is known at compile time (a literal, a
named constant or a constant expression) stays a constant in the kernel
unless its dimension is in chunk_dimensions. Each dimension in
chunk_dimensions names the kernel parameters that hold its bounds,
start_parameter and end_parameter, which each chunk sets to the bounds
of its box, and the launch reads the bounds of the whole dimension from its
arguments for them. They are listed fastest-varying first, and a chunk
covers as many values of each as fit beside the dimensions covered whole
and the ones before it in the list, and at least one. gpu_offload splits
every dimension whose extent is not a constant, and those with constant
extents beyond what fits the limit. It gives a launch no
chunk_dimensions when its nest cannot have more iterations than the
limit, as the kind of each loop index and its constant bounds bound its
extent: a one-dimensional do concurrent (i = 1:n) with an integer(4)
index has at most huge(0) iterations. device_launch_expand sets the
buffers, scratch buffers and workspaces once, sized for the largest chunk,
and then launches each chunk with its own scalar arguments and grid. A
launch that may run in tiles runs flat when its whole tiles would start
more threads than the chunk limit, with or without chunk_dimensions, so no
launch starts more threads than the limit. Where every extent of the nest is
a constant, that is decided
at compile time: a nest whose whole tiles come to more threads than
the limit gets a kernel that numbers its iterations flat, so a nest of at
most that many iterations runs as one launch with no chunk_dimensions,
and its constant bounds stay constants in the kernel. A buffer handed over as
LFORTRAN_GPU_WRITE_DISCARD is handed over again as written after each
chunk, so the chunks after the first keep what it wrote. A launch whose
kernel runs several threads per iteration (the layout’s lanes) starts
that many per iteration of each chunk. CUDA and Metal launches have
chunk_dimensions. The host counts the iterations of a launch in
integer(8), so a Metal kernel, which numbers them in 32 bits, runs a nest
of more than huge(0) iterations in chunks.
A launch with a reduction gives each thread an accumulator of its own, an
element of a buffer argument, which the kernel sets to the identity of the
operator before the body combines into it; the host folds the accumulators
into the scalar after the launch. The threads of each chunk are numbered
from zero, so a launch that may run in chunks has one accumulator per
iteration of its largest chunk, the first one, when it has more iterations
than the chunk limit. Its kernel takes first_chunk_parameter, which the
launch sets to 1 when it runs as one or for its first chunk, and to 0 for
the later chunks: the threads of the first chunk start their
accumulators, and those of the later chunks keep combining into what the
thread of the same number left there. A thread past the end of a partial
chunk returns before it touches its accumulator, and the host folds only
the ones the first chunk started, once, after the last chunk. A launch
without chunk_dimensions, or whose kernel has no accumulators, has no
first_chunk_parameter.
Private array storage is placed after shared lowering by gpu_kernel_storage,
before address-space specialization. Fixed-size arrays that fit the device’s
stack budget have Thread memory space. Larger arrays have Global memory
space and the kernel layout records their per-thread workspace slices. This
placement never turns an iteration-private value into a shared value.
A workspace may record allocation_if, its captured allocation_parameter,
and allocation_on_true. Shared kernel planning accepts only stable, total
scalar predicates that are host-evaluable at every launch and dominate every
data use with the recorded polarity. The host captures the predicate once at
the launch and the device uses the same value. When inactive, launch expansion
requests zero scratch bytes without suppressing unrelated kernel work. Array
shape and bounds are unchanged; inactive or empty slices do not form pointers
into absent storage. Thread-varying tests, optional or aliased scalars,
descriptor reads, calls, and other non-total expressions conservatively retain
normal allocation unless another safe enclosing guard proves it unnecessary.
Examples¶
(GpuKernelLaunch
:kernel (SymbolRef 1 "zero")
:thread_count (IntegerConstant
:n 1
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
:block_size (IntegerConstant
:n 256
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
:args [
(call_arg
:value (Var
:v (SymbolRef 2 "a")
)
)
]
:component_fit nil
:scratch []
:stop_status nil
:stop_replay []
:overwritten []
:overwritten_if []
:chunk_dimensions []
:chunk_fixed_iterations 0
:first_chunk_parameter nil
)
It comes from this complete ASR text document:
(TranslationUnit
:symtab (SymbolTable
:id 0
:symbols {
"m" (Module
:symtab (SymbolTable
:id 1
:symbols {
"launch" (Function
:symtab (SymbolTable
:id 2
:symbols {
"a" (Variable
:parent_symtab 2
:name "a"
:dependencies []
:intent :InOut
:symbolic_value nil
:value nil
:storage :Default
:type (Array
:type (Real
:kind 4
)
:dims [
(dimension
:start (IntegerConstant
:n 1
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
:length (IntegerConstant
:n 256
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
)
]
:physical_type :FixedSizeArray
:memory_space :Global
)
:type_declaration nil
:abi :Source
:access :Public
:presence :Required
:value_attr false
:target_attr false
:contiguous_attr false
:bindc_name nil
:is_volatile false
:is_asynchronous false
:is_protected false
:pass_attr :NotMethod
:self_argument nil
:codims []
:common_block nil
)
}
)
:name "launch"
:function_signature (FunctionType
:arg_types [
(Array
:type (Real
:kind 4
)
:dims [
(dimension
:start (IntegerConstant
:n 1
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
:length (IntegerConstant
:n 256
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
)
]
:physical_type :FixedSizeArray
:memory_space :Global
)
]
:return_var_type nil
:abi :Source
:deftype :Implementation
:bindc_name nil
:elemental false
:pure false
:module false
:inline false
:static false
:restrictions []
:is_restriction false
:exec_space :Host
)
:dependencies []
:args [
(Var
:v (SymbolRef 2 "a")
)
]
:body [
(GpuKernelLaunch
:kernel (SymbolRef 1 "zero")
:thread_count (IntegerConstant
:n 1
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
:block_size (IntegerConstant
:n 256
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
:args [
(call_arg
:value (Var
:v (SymbolRef 2 "a")
)
)
]
:component_fit nil
:scratch []
:stop_status nil
:stop_replay []
:overwritten []
:overwritten_if []
:chunk_dimensions []
:chunk_fixed_iterations 0
:first_chunk_parameter nil
)
(GpuWait)
(GpuSync)
]
:return_var nil
:access :Public
:deterministic false
:side_effect_free false
:module_file nil
)
"zero" (Function
:symtab (SymbolTable
:id 3
:symbols {
"a" (Variable
:parent_symtab 3
:name "a"
:dependencies []
:intent :InOut
:symbolic_value nil
:value nil
:storage :Default
:type (Array
:type (Real
:kind 4
)
:dims [
(dimension
:start (IntegerConstant
:n 1
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
:length (IntegerConstant
:n 256
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
)
]
:physical_type :FixedSizeArray
:memory_space :Global
)
:type_declaration nil
:abi :Source
:access :Public
:presence :Required
:value_attr false
:target_attr false
:contiguous_attr false
:bindc_name nil
:is_volatile false
:is_asynchronous false
:is_protected false
:pass_attr :NotMethod
:self_argument nil
:codims []
:common_block nil
)
}
)
:name "zero"
:function_signature (FunctionType
:arg_types [
(Array
:type (Real
:kind 4
)
:dims [
(dimension
:start (IntegerConstant
:n 1
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
:length (IntegerConstant
:n 256
:type (Integer
:kind 4
)
:intboz_type :Decimal
)
)
]
:physical_type :FixedSizeArray
:memory_space :Global
)
]
:return_var_type nil
:abi :Source
:deftype :Implementation
:bindc_name nil
:elemental false
:pure false
:module false
:inline false
:static false
:restrictions []
:is_restriction false
:exec_space :Kernel
)
:dependencies []
:args [
(Var
:v (SymbolRef 3 "a")
)
]
:body []
:return_var nil
:access :Public
:deterministic false
:side_effect_free false
:module_file nil
)
}
)
:name "m"
:parent_module nil
:dependencies []
:loaded_from_mod false
:intrinsic false
:has_submodules false
:host_only_object false
:global_init nil
)
"main" (Program
:symtab (SymbolTable
:id 4
:symbols {}
)
:name "main"
:dependencies []
:body []
:global_init nil
)
}
)
:items []
:global_init nil
)