Skip to content

Grid Unit

Kamayan is built on parthenon's grid, which can support uniform, static and adaptively refined meshes as a runtime option. The domain is initialized as a uniform grid of nx1xnx2xnx3 cells. These can then be partitioned onto an initial grid of MeshBlocks, which can be further refined based on various conditions. See Parameters for more details on the runtime parameters that control this behavior.

Fields

Kamayan strongly prefers to reference field data stored on the MeshBlocks through the type based interface. Fields are associated with a struct, and instantiations of that type can be used to reference a desired element of that field if it is declared with some tensor shape. Kamayan exports all possible fields in kamayan/fields.hpp, but which fields have been registered to the simulation will depend on the configuration of the various units. Here fields are declared with the VARIABLE macro, which takes in the type name in all capital letters, which also serves as the string label associated with the field, and optionally a list of ints that will become the tensor shape of the field when registered with the kamayan::AddField[s] interfaces.

kamayan/fields.hpp:cons
// conserved variables
VARIABLE(DENS);
VARIABLE(MOMENTUM, 3);  // will register with shape=std::vector<int>{3}
VARIABLE(ENER);
VARIABLE(MAG);

Note

KamayanUnits will register fields through the Initialize callback interface that returns a StateDescriptor.

Packs

Field data can be accessed through the SparsePacks provided by parthenon that will pack variables by type into a single data structure that can be used to index into cell level data on MeshBlocks and MeshData partitions. Kamayan provides an interface to directly get these packs, either directly through listing explicitly the desired variables or by passing a TypeList of the variable structs you wish to access.

problems/isentropic_vortex.cpp:pack
physics/hydro/hydro_add_flux_tasks.cpp:pack
// explicitly specify the fields we want
    auto pack = grid::GetPack<DENS, VELOCITY, PRES>(mb);
// packing with a decl'ed `TypeList` from the HydroTraits
    auto pack_recon = grid::GetPack(reconstruct_vars(), md);
    auto pack_flux = grid::GetPack(conserved_vars(), md, {PDOpt::WithFluxes});

Note

Additional pack options can be specified. In this example the PDOpt::WithFluxes option means that the associated fluxes if any for the packed variables will also be accessible with the pack_flux.flux interface.

Packs can then be indexed by block, variable and (k,j,i) index

problems/isentropic_vortex.cpp:index
          pack(0, DENS(), k, j, i) = state(DENS());
          pack(0, PRES(), k, j, i) = state(PRES());
          pack(0, VELOCITY(0), k, j, i) = state(VELOCITY(0));
          pack(0, VELOCITY(1), k, j, i) = state(VELOCITY(1));

Subpacks

Often we don't need full access to all block, and cell indices of the pack. Rather it is advantageous only worry about a relative offset along the dimensions of interest for a particular function. Some examples are the conversion of primitive to conservative variables, which only cares about indexing into different fields at the same cell, or the reconstruction of Riemann states, which need only to index into the same field at neighboring zones along a one dimensional stencil.

Kamayan provides the SubPack abstractions to wrap packs and Kokkos::Views to do just that.

physics/hydro/primconsflux.cpp:make-idx
          auto U = SubPack(pack, b, k, j, i);
          Prim2Cons<hydro_traits>(U, U);
physics/hydro/primconsflux.hpp:use-idx
  U(DENS()) = V(DENS());
  Real emag = 0.;
  Real ekin = 0.;
  for (int dir = 0; dir < 3; dir++) {
    U(MOMENTUM(dir)) = V(DENS()) * V(VELOCITY(dir));
    ekin += V(VELOCITY(dir)) * V(VELOCITY(dir));
    if constexpr (hydro_traits::MHD != Mhd::off) {
      U(MAGC(dir)) = V(MAGC(dir));
      emag += V(MAGC(dir)) * V(MAGC(dir));
    }
  }

physics/hydro/hydro_add_flux_tasks.cpp:make-stncl
                auto stencil = SubPack<Axis::IAXIS>(pack_recon, b, var, k, j, i);
                Reconstruct<reconstruction_traits>(stencil, vM(var, i), vP(var, i));
physics/hydro/reconstruction.hpp:use-stncl
  const Real dvL = stencil(0) - stencil(-1);
  const Real dvR = stencil(1) - stencil(0);

Scratch Variables

Kamayan provides the ability to represent fields on the mesh that are temporary in nature using collections called ScratchVariableLists. These provide types that can be registered to a package that will alias common Metadata::derived fields across different regions where they are used. In this way temporary block storage is achieved without needing to allocate more memory than is minimally required to satisfy a single ScratchVariableList.

ScratchVarableLists are constructed from specializations of the ScratchVariable type that is used to describe the variable's TopologicalType and vector/tensor shape, and finally are used to index into the ScratchVariableList in order to get a type that can be used exactly as any other field through the type-based packing.

grid/grid_refinement.hpp:scratch
// first-order derivative at centers used in Loehner estimator
using FirstDer_t = ScratchVariable<"firstder", TopologicalType::Cell, 3>;

using RefinementScratch = ScratchVariableList<FirstDer_t>;
using FirstDer = RefinementScratch::type<FirstDer_t>;
grid/grid.cpp:addscratch
  if (nref_vars > 0) AddScratch<RefinementScratch>(pkg.get());

In the above example a single scratch variable to represent the first-order derivatives in each 3D direction is registered as a cell centered variable as a 3 component vector. When it is registered it will define three fields, scratch_cell_n, that can be reused by other scratch variables, possibly of different shape. The alias FirstDer is pulled out of the ScratchVariableList can then be used with the pack overloads just like any other field.

grid/grid_refinement.cpp:FirstDer
        pack_der(b, FirstDer(0), k, j, i) =
            0.5 * (pack(b, var, k, j, i + 1) - pack(b, var, k, j, i - 1)) /
            coords.Dxc<1>();

Note

If cmake is configured with -Dkamayan_DEBUG_SCRATCH then each scratch variable will be independently registered using the name string template parameter to the ScratchVariable. In the above example a shape {3} field scratch_firstder will be registered.

Parameters

Paramter Type Default Allowed Description
<kamayan/refinement0>
derefine_tol Real 2.00000e-01 Error threshold for derefinement
field String NO FIELD WAS SET Field to refine on.
filter Real 1.00000e-02 Noise filtering strength used in Loehner estimator.
max_level Integer 1 max refinement level for this field.
method String loehner [ loehner, derivative_order_1, derivative_order_2, ] Method to use for refinement
refine_tol Real 8.00000e-01 Error threshold for refinement
<parthenon/mesh>
ix1_bc String outflow [ periodic, outflow, reflect, user, ] Inner boundary condition along x1.
ix2_bc String outflow [ periodic, outflow, reflect, user, ] Inner boundary condition along x2.
ix3_bc String outflow [ periodic, outflow, reflect, user, ] Inner boundary condition along x3.
nghost Integer 3 Number of ghost zones to use on each block.
numlevel Integer 1 Number of refinement levels.
nx1 Integer 32 Number of cells across the domain at level 0.
nx2 Integer 32 Number of cells across the domain at level 0. Set to 1 for 1D.
nx3 Integer 32 Number of cells across the domain at level 0. Set to 1 for 2D.
ox1_bc String outflow [ periodic, outflow, reflect, user, ] Outer boundary condition along x1.
ox2_bc String outflow [ periodic, outflow, reflect, user, ] Outer boundary condition along x2.
ox3_bc String outflow [ periodic, outflow, reflect, user, ] Outer boundary condition along x3.
refinement String adaptive [ adaptive, static, none, ] Mesh refinement strategy.
x1max Real 1.00000e+00 Maximum x1 value of domain.
x1min Real 0.00000e+00 Minimum x1 value of domain.
x2max Real 1.00000e+00 Maximum x2 value of domain.
x2min Real 0.00000e+00 Minimum x2 value of domain.
x3max Real 1.00000e+00 Maximum x3 value of domain.
x3min Real 0.00000e+00 Minimum x3 value of domain.
<parthenon/meshblock>
nx1 Integer 16 Size of meshblocks in x1.
nx2 Integer 16 Size of meshblocks in x2.
nx3 Integer 16 Size of meshblocks in x3.