Driver Unit

The KamayanDriver implements the parthenon MultiStageDriver, and so works with any of the provided multistage integrators. Kamayan allows any KamayanUnit to hook into the main evolution loop through the provided callback interfaces. The primary evolution hooks can be classified as

Flux-based multi-stage operators
Non-flux multi-stage operators
Split operators

For multi-stage integrators we require our primary independent variables to be replicated across buffers corresponding to the various stages. Parthenon buffers will only duplicate fields marked Metadata::Independent across these various buffers, while all others will point to the same data.

physics/hydro/hydro.cpp:hydro_add_fields

    // conserved variables are Independent in each multi-stage buffer
    AddFields(typename hydro_vars::WithFlux(), pkg,
              {CENTER_FLAGS(Metadata::Independent, Metadata::WithFluxes)});
    // primitive variables reference same data on each multi-stage buffer
    AddFields(typename hydro_vars::NonFlux(), pkg, {CENTER_FLAGS()});

RK-Stages

The multi-stage integrator is based of a method-of-lines approach where the temporal discretization at each point in space is determined from the state of the system at the current time, resulting in an ODE system of equations for each point on the grid.

$\dfrac{du^n_{i,j,k}}{dt} = \mathcal{L}_{i,j,k}(u^n_{l,m,n},...).$

The KamayanDriver will accumulate $\mathcal{L}$ over all the units that have been registered to it's UnitCollection. The driver will call into each unit's AddTasksOneStep to accumulate the right hand side into a provided MeshData container dudt, and the driver handles apply the changes to the solution.

Flux-based Tasks

For the case that terms in $\mathcal{L}$ can be expressed as a difference of fluxes then it can be advantageous to have the driver accumulate the fluxes over all the units that provide a callback for AddFluxTasks.

$\begin{align*} \mathcal{L}_{i,j,k} = \frac{1}{\mathcal{V}_{i,j,k}} & \left ( A_{i+1/2,j,k}f^{x_1}_{i+1/2,j,k} - A_{i-1/2,j,k}f^{x_1}_{i-1/2,j,k} \right . \\ & \left . + A_{i,j+1/2,k}f^{x_2}_{i,j+1/2,k} - A_{i,j-1/2,k}f^{x_2}_{i,j-1/2,k} \right . \\ & \left . + A_{i,j,k+1/2}f^{x_3}_{i,j,k+1/2} - A_{i,j,k-1/2}f^{x_3}_{i,j,k-1/2} \right ) \end{align*}$

The driver will then take care of calling into the parthenon routines for flux correction at block and fine-coarse boundaries.

Tasks

The KamayanDriver uses parthenon's tasking infrastructure to build out the execution for the components making up an evolution cycle. It is helpful to add named labels when registering tasks into the TaskLists, as that is how the driver can generate the above task graph.

physics/hydro/hydro_add_flux_tasks.cpp:add_task

    get_fluxes = tl.AddTask(
        prev, "hydro::CalculateFluxes",
        [](MeshData *md, Config *cfg) {
          return Dispatcher<CalculateFluxesNested>(PARTHENON_AUTO_LABEL, cfg).execute(md);
        },
        md, cfg.get());

Unit Callbacks

Tasks are registered in the driver by using callbacks that are registered to the units. When callbacks are registered they can optionally specify string lists of other units whose callbacks should be called before or after the callback being registered.

physics/hydro/hydro.cpp:register

  hydro->SetupParams.Register(SetupParams);
  hydro->InitializeData.Register(InitializeData);
  // need to convert to primitives before calling equation of state
  hydro->PreparePrimitive.Register(PreparePrimitive, /*after=*/{}, /*before=*/{"eos"});
  hydro->PostMeshInitialization.Register(PostMeshInitialization, {"eos"});
  hydro->AddFluxTasks.Register(AddFluxTasks);

When the tasks are added into the driver's task list a directed acyclic graph (DAG) is generated for the callback so that they can be executed in order. If there is a cycle in the graph then kamayan will throw a runtime error.

Parameters

Paramter	Type	Default	Allowed	Description
<parthenon/time>
dt_ceil	Real	1.79769e+308		The maximum allowed timestep.
dt_factor	Real	2.00000e+00		The maximum allowed relative increase of the timestep over the previous value.
dt_floor	Real	2.22507e-308		The minimum allowed timestep.
dt_force	Real	-1.79769e+308		Force the timestep to this value, ignoring all other conditions.
dt_init	Real	1.79769e+308		The maximum allowed timestep during the first cycle.
dt_init_force	Boolean	true		If set to true, force the first cycle's timestep to the value given by dt_init.
dt_max	Real	1.79769e+308		If the timestep falls above dt_max for dt_max_cycle_limit cycles, Parthenon fatals.
dt_max_cycle_limit	Integer	1		The maximum number of cycles the timestep an be above dt_max.
dt_min	Real	-1.79769e+308		If the timestep falls below dt_min for dt_min_cycle_limit cycles, Parthenon fatals.
dt_min_cycle_limit	Integer	10		The maximum number of cycles the timestep can be below dt_min.
dt_user	Real	1.79769e+308		Set a global timestep limit.
integrator	String	rk2	[ rk1, rk2, rk3, ]	Which multi-stage Runge-Kutta method to use
ncrecv_bdry_buf_timeout_sec	Real	-1.00000e+00		Timeout in seconds for the ReceiveBoundaryBuffers tasks. Disabed (negative) by default. Typically no need in production runs. Useful for debugging MPI calls.
ncycle_out	Integer	1		Number of cycles between short diagnostic output to standard out containing, e.g., current time, dt, zone-update/wsec. Default: 1 (i.e, every cycle).
ncycle_out_mesh	Integer	0		Number of cycles between printing the mesh structure to standard out. Use a negative number to also print every time the mesh was modified. Default: 0 (i.e, off).
nlim	Integer	-1		Stop criterion on total number of steps taken. Ignored if < 0.
perf_cycle_offset	Integer	0		Skip the first N cycles when calculating the final performance (e.g., zone-cycles/wall_second). Allows to hide the initialization overhead in Parthenon.
tlim	Real	1.79769e+308		Stop criterion on simulation time.