inciter::Discretization class

Discretization Charm++ chare array holding common functinoality to all discretization schemes.

Public static functions

static void registerReducers()
Configure Charm++ reduction types.

Constructors, destructors, conversion operators

Discretization(CkMigrateMessage* m) explicit
Migrate constructor.

Public functions

auto Discretization(std::size_t meshid, const CProxy_Transporter& transporter, const tk::CProxy_MeshWriter& meshwriter, const tk::UnsMesh::CoordMap& coordmap, const tk::UnsMesh::Chunk& el, const std::map<int, std::unordered_set<std::size_t>>& nodeCommMap, int nc) -> Discretization_SDAG_CODE explicit
Constructor.
void resizePostAMR(const tk::UnsMesh::Chunk& chunk, const tk::UnsMesh::Coords& coord, const std::unordered_map<int, std::unordered_set<std::size_t>>& nodeCommMap, const std::set<std::size_t>& removedNodes)
Resize mesh data structures after mesh refinement.
void startvol()
Get ready for (re-)computing/communicating nodal volumes.
void vol()
Sum mesh volumes to nodes, start communicating them on chare-boundaries.
void setRefiner(const CProxy_Refiner& ref)
Set Refiner Charm++ proxy.
void comvol(int c, const std::vector<std::size_t>& gid, const std::vector<tk::real>& nodevol)
Collect nodal volumes across chare boundaries.
void totalvol()
Sum mesh volumes and contribute own mesh volume to total volume.
void stat(tk::real mesh_volume)
Compute mesh cell statistics.
auto deastart() -> bool
Decide whether to start the deactivation procedure in this time step.
auto deactivate() const -> bool
Decide whether to run deactivation procedure in this time step.
void deactivated(int n)
Receive deactivation report.
auto lb() const -> bool
Decide whether to do load balancing this time step.
void boxvol()
Compute total box IC volume.

Charm++ pack/unpack serializer member functions

std::size_t m_meshid
Mesh ID.
int m_nchare
Total number of Discretization chares.
uint64_t m_it
Iteration count.
uint64_t m_itr
uint64_t m_itf
std::size_t m_initial
Flag that is nonzero during setup and zero during time stepping.
tk::real m_t
Physical time.
tk::real m_lastDumpTime
Physics time at last field output.
tk::real m_physFieldFloor
Recent floor of physics time divided by field output interval time.
tk::real m_physHistFloor
Recent floor of physics time divided by history output interval time.
tk::real m_physIntegFloor
Recent floor of physics time divided by integral output interval time.
std::vector<tk::real> m_rangeFieldFloor
Recent floors of physics time divided by field output time for ranges.
std::vector<tk::real> m_rangeHistFloor
Recent floors of physics time divided by history output time for ranges.
std::vector<tk::real> m_rangeIntegFloor
Recent floors of physics time divided by integral output time for ranges.
tk::real m_dt
Physical time step size.
tk::real m_dtn
Physical time step size at the previous time step.
std::size_t m_nvol
Number of chares from which we received nodal volume contributions on chare boundaries.
std::vector<std::unordered_set<std::size_t>> m_boxnodes
List of nodes at which box user ICs are set for each IC box.
CProxy_Transporter m_transporter
Transporter proxy.
tk::CProxy_MeshWriter m_meshwriter
Mesh writer proxy.
CProxy_Refiner m_refiner
Mesh refiner proxy.
tk::UnsMesh::Chunk m_el
Elements of the mesh chunk we operate on.
std::vector<std::size_t>& m_inpoel
Alias to element connectivity.
std::vector<std::size_t>& m_gid
Alias to global node IDs of owned elements.
std::unordered_map<std::size_t, std::size_t>& m_lid
Alias to local node ids associated to the global ones of owned elements.
tk::UnsMesh::Coords m_coord
Mesh point coordinates.
std::unordered_map<int, std::unordered_set<std::size_t>> m_nodeCommMap
Global mesh node IDs bordering the mesh chunk held by fellow Discretization chares associated to their chare IDs.
tk::real m_meshvol
Total mesh volume.
std::vector<tk::real> m_v
std::vector<tk::real> m_vol
std::unordered_map<std::size_t, tk::real> m_volc
std::unordered_map<int, std::unordered_map<std::size_t, tk::real>> m_cvolc
tk::real m_boxvol
Volume of user-defined box IC.
tk::Timer m_timer
Timer measuring a time step.
int m_refined
1 if mesh was refined in a time step, 0 if it was not
Clock::time_point m_prevstatus
Time point storing clock state at status()
int m_nrestart
Number of times restarted.
std::vector<HistData> m_histdata
Data at history point locations.
tk::real m_res
Current residual (during convergence to steady state)
tk::real m_res0
Residual at previous ETA calcuation (during convergence to steady state)
tk::real m_res1
Residual at next ETA calcuation (during convergence to steady state)
int m_dea
Numberf of deactived chares.
int m_deastarted
Flag: 1 if deactivation procedure has started.
std::size_t m_pit
Numberf of recent pressure solve iterations taken.
void pup(PUP::er& p) override
Pack/Unpack serialize member function.
void operator|(PUP::er& p, Discretization& i)
Pack/Unpack serialize operator|.

Accessors

auto Coord() const -> const tk::UnsMesh::Coords&
auto Coord() -> tk::UnsMesh::Coords&
Coordinates accessor as reference.
auto Gid() const -> const std::vector<std::size_t>&
Global ids accessors as const-ref.
auto Lid() const -> const std::unordered_map<std::size_t, std::size_t>&
Local ids accessors as const-ref.
auto Inpoel() const -> const std::vector<std::size_t>&
Tetrahedron element connectivity (with local ids) accessors as const-ref.
auto Chunk() const -> const tk::UnsMesh::Chunk&
Mesh chunk accessor as const-ref.
auto meshvol() const -> tk::real
Total mesh volume accessor.
auto V() const -> const std::vector<tk::real>&
Nodal mesh volume accessors const-ref.
auto Vol() const -> const std::vector<tk::real>&
Nodal mesh volumes at current time step accessors as const-ref.
auto Cvolc() const -> const std::unordered_map<int, std::unordered_map<std::size_t, tk::real>>&
Access chare-volume contributions per chare per chare-boundary node.
auto nchare() const -> int
Access number of chares.
void Initial(std::size_t i)
auto Initial() const -> bool
auto Hist() const -> const std::vector<HistData>&
History points data accessor as const-ref.
auto Boxvol() -> tk::real&
Box volume accessor.
auto MeshId() const -> std::size_t
Mesh ID accessor.
auto Dt() const -> tk::real
Time step size accessor.
auto Dtn() const -> tk::real
Time step size at previous time step accessor.
auto T() const -> tk::real
Physical time accessor.
auto It() const -> uint64_t
Iteration count accessor.
auto Itr() -> uint64_t&
Non-const-ref refinement iteration count accessor.
auto Itf() -> uint64_t&
Non-const-ref field-output iteration count accessor.
auto Nrestart() -> int&
Non-const-ref number of restarts accessor.
auto Timer() const -> const tk::Timer&
Timer accessor as const-ref.
auto Timer() -> tk::Timer&
Timer accessor as non-const-ref.
auto refined() const -> int
Accessor to flag indicating if the mesh was refined as a value.
auto refined() -> int&
Accessor to flag indicating if the mesh was refined as non-const-ref.
auto Tr() const -> const CProxy_Transporter&
Transporter proxy accessor as const-ref.
auto Tr() -> CProxy_Transporter&
Transporter proxy accessor as non-const-ref.
auto Ref() const -> Refiner*
Access bound Refiner class pointer.
auto NodeCommMap() const -> const std::unordered_map<int, std::unordered_set<std::size_t>>&
Node communication map accessor as const-ref.
auto BoxNodes() const -> const std::vector<std::unordered_set<std::size_t>>&
IC boxnodes accessor.
void setdt(tk::real newdt)
Set time step size.
void next()
Prepare for next step.
void status()
Otput one-liner status report.
auto histfilename(const std::string& id, std::streamsize precision) -> std::string
Construct history output filename.
void histheader(std::vector<std::string>&& names)
Output headers for time history files (one for each point)
void history(std::vector<std::vector<tk::real>>&& data)
Output time history for a time step.
void write(const std::vector<std::size_t>& inpoel, const tk::UnsMesh::Coords& coord, const std::map<int, std::vector<std::size_t>>& bface, const std::map<int, std::vector<std::size_t>>& bnode, const std::vector<std::size_t>& triinpoel, const std::vector<std::string>& elemfieldnames, const std::vector<std::string>& nodefieldnames, const std::vector<std::string>& elemsurfnames, const std::vector<std::string>& nodesurfnames, const std::vector<std::vector<tk::real>>& elemfields, const std::vector<std::vector<tk::real>>& nodefields, const std::vector<std::vector<tk::real>>& elemsurfs, const std::vector<std::vector<tk::real>>& nodesurfs, CkCallback c)
Output mesh and fields data (solution dump) to file(s)
void grindZero()
Zero grind-timer.
auto restarted(int nrestart) -> bool
Detect if just returned from a checkpoint and if so, zero timers.
void remap(const std::unordered_map<std::size_t, std::size_t>& map)
Remap mesh data due to new local ids.
auto fielditer() const -> bool
Decide if field output iteration count interval is hit.
auto fieldtime() const -> bool
Decide if field output physics time interval is hit.
auto fieldrange() const -> bool
Decide if physics time falls into a field output time range.
auto histiter() const -> bool
Decide if history output iteration count interval is hit.
auto histtime() const -> bool
Decide if history output physics time interval is hit.
auto histrange() const -> bool
Decide if physics time falls into a history output time range.
auto integiter() const -> bool
Decide if integral output iteration count interval is hit.
auto integtime() const -> bool
Decide if integral output physics time interval is hit.
auto integrange() const -> bool
Decide if physics time falls into a integral output time range.
auto finished() const -> bool
Decide if this is the last time step.
void residual(tk::real r)
Update residual (during convergence to steady state)
void pit(std::size_t it)
Update number of pressure linear solve iterations taken.

Function documentation

static void inciter::Discretization::registerReducers()

Configure Charm++ reduction types.

Since this is a [initnode] routine, see the .ci file, the Charm++ runtime system executes the routine exactly once on every logical node early on in the Charm++ init sequence. Must be static as it is called without an object. See also: Section "Initializations at Program Startup" at in the Charm++ manual http://charm.cs.illinois.edu/manuals/html/charm++/manual.html.

Discretization_SDAG_CODE inciter::Discretization::Discretization(std::size_t meshid, const CProxy_Transporter& transporter, const tk::CProxy_MeshWriter& meshwriter, const tk::UnsMesh::CoordMap& coordmap, const tk::UnsMesh::Chunk& el, const std::map<int, std::unordered_set<std::size_t>>& nodeCommMap, int nc) explicit

Constructor.

Parameters
meshid in Mesh ID
transporter in Host (Transporter) proxy
meshwriter in Mesh writer proxy
coordmap in Coordinates of mesh nodes and their global IDs
el in Elements of the mesh chunk we operate on
nodeCommMap in Node lists associated to chare IDs bordering the mesh chunk we operate on
nc in Total number of Discretization chares

void inciter::Discretization::resizePostAMR(const tk::UnsMesh::Chunk& chunk, const tk::UnsMesh::Coords& coord, const std::unordered_map<int, std::unordered_set<std::size_t>>& nodeCommMap, const std::set<std::size_t>& removedNodes)

Resize mesh data structures after mesh refinement.

Parameters
chunk in New mesh chunk (connectivity and global<->local id maps)
coord in New mesh node coordinates
nodeCommMap in New node communication map
removedNodes in Newly removed mesh node local ids

void inciter::Discretization::setRefiner(const CProxy_Refiner& ref)

Set Refiner Charm++ proxy.

Parameters
ref in Incoming refiner proxy to store

void inciter::Discretization::comvol(int c, const std::vector<std::size_t>& gid, const std::vector<tk::real>& nodevol)

Collect nodal volumes across chare boundaries.

Parameters
in Sender chare id
gid in Global mesh node IDs at which we receive volume contributions
nodevol in Partial sums of nodal volume contributions to chare-boundary nodes

void inciter::Discretization::stat(tk::real mesh_volume)

Compute mesh cell statistics.

Parameters
mesh_volume in Total mesh volume

bool inciter::Discretization::deastart()

Decide whether to start the deactivation procedure in this time step.

Returns True to start the deactivation prcedure in this time step

bool inciter::Discretization::deactivate() const

Decide whether to run deactivation procedure in this time step.

Returns True to run deactivation prcedure in this time step

void inciter::Discretization::deactivated(int n)

Receive deactivation report.

Parameters
in Sum of deactivated chares

bool inciter::Discretization::lb() const

Decide whether to do load balancing this time step.

Returns True to do load balancing in this time step

void inciter::Discretization::pup(PUP::er& p) override

Pack/Unpack serialize member function.

Parameters
in/out Charm++'s PUP::er serializer object reference

void inciter::Discretization::operator|(PUP::er& p, Discretization& i)

Pack/Unpack serialize operator|.

Parameters
in/out Charm++'s PUP::er serializer object reference
in/out Discretization object reference

const tk::UnsMesh::Coords& inciter::Discretization::Coord() const

Coordinates accessor as const-ref

void inciter::Discretization::Initial(std::size_t i)

Parameters
in Value to put in 'initial'

Set 'initial' flag

bool inciter::Discretization::Initial() const

Returns True during setup, false durign time stepping

Query 'initial' flag

void inciter::Discretization::setdt(tk::real newdt)

Set time step size.

Parameters
newdt in Size of the new time step

std::string inciter::Discretization::histfilename(const std::string& id, std::streamsize precision)

Construct history output filename.

Parameters
id in History point id
precision in Floating point precision to use for output
Returns History file name

void inciter::Discretization::histheader(std::vector<std::string>&& names)

Output headers for time history files (one for each point)

Parameters
names in History output variable names

void inciter::Discretization::history(std::vector<std::vector<tk::real>>&& data)

Output time history for a time step.

Parameters
data in Time history data for all variables and equations integrated

void inciter::Discretization::write(const std::vector<std::size_t>& inpoel, const tk::UnsMesh::Coords& coord, const std::map<int, std::vector<std::size_t>>& bface, const std::map<int, std::vector<std::size_t>>& bnode, const std::vector<std::size_t>& triinpoel, const std::vector<std::string>& elemfieldnames, const std::vector<std::string>& nodefieldnames, const std::vector<std::string>& elemsurfnames, const std::vector<std::string>& nodesurfnames, const std::vector<std::vector<tk::real>>& elemfields, const std::vector<std::vector<tk::real>>& nodefields, const std::vector<std::vector<tk::real>>& elemsurfs, const std::vector<std::vector<tk::real>>& nodesurfs, CkCallback c)

Output mesh and fields data (solution dump) to file(s)

Parameters
inpoel in Mesh connectivity for the mesh chunk to be written
coord in Node coordinates of the mesh chunk to be written
bface in Map of boundary-face lists mapped to corresponding side set ids for this mesh chunk
bnode in Map of boundary-node lists mapped to corresponding side set ids for this mesh chunk
triinpoel in Interconnectivity of points and boundary-face in this mesh chunk
elemfieldnames in Names of element fields to be output to file
nodefieldnames in Names of node fields to be output to file
elemsurfnames in Names of elem surface fields to be output to file
nodesurfnames in Names of node surface fields to be output to file
elemfields in Field data in mesh elements to output to file
nodefields in Field data in mesh nodes to output to file
elemsurfs in Surface field data in mesh elements to output to file
nodesurfs in Surface field data in mesh nodes to output to file
in Function to continue with after the write

Since m_meshwriter is a Charm++ chare group, it never migrates and an instance is guaranteed on every PE. We index the first PE on every logical compute node. In Charm++'s non-SMP mode, a node is the same as a PE, so the index is the same as CkMyPe(). In SMP mode the index is the first PE on every logical node. In non-SMP mode this yields one or more output files per PE with zero or non-zero virtualization, respectively. If there are multiple chares on a PE, the writes are serialized per PE, since only a single entry method call can be executed at any given time. In SMP mode, still the same number of files are output (one per chare), but the output is serialized through the first PE of each compute node. In SMP mode, channeling multiple files via a single PE on each node is required by NetCDF and HDF5, as well as ExodusII, since none of these libraries are thread-safe.

bool inciter::Discretization::restarted(int nrestart)

Detect if just returned from a checkpoint and if so, zero timers.

Parameters
nrestart in Number of times restarted
Returns True if restart detected

void inciter::Discretization::remap(const std::unordered_map<std::size_t, std::size_t>& map)

Remap mesh data due to new local ids.

Parameters
map in Mapping of old->new local ids

bool inciter::Discretization::fielditer() const

Decide if field output iteration count interval is hit.

Returns True if field output iteration count interval is hit

bool inciter::Discretization::fieldtime() const

Decide if field output physics time interval is hit.

Returns True if field output physics time interval is hit

bool inciter::Discretization::fieldrange() const

Decide if physics time falls into a field output time range.

Returns True if physics time falls into a field output time range

bool inciter::Discretization::histiter() const

Decide if history output iteration count interval is hit.

Returns True if history output iteration count interval is hit

bool inciter::Discretization::histtime() const

Decide if history output physics time interval is hit.

Returns True if history output physics time interval is hit

bool inciter::Discretization::histrange() const

Decide if physics time falls into a history output time range.

Returns True if physics time falls into a history output time range

bool inciter::Discretization::integiter() const

Decide if integral output iteration count interval is hit.

Returns True if integral output iteration count interval is hit

bool inciter::Discretization::integtime() const

Decide if integral output physics time interval is hit.

Returns True if integral output physics time interval is hit

bool inciter::Discretization::integrange() const

Decide if physics time falls into a integral output time range.

Returns True if physics time falls into a integral output time range

bool inciter::Discretization::finished() const

Decide if this is the last time step.

Returns True if this is the last time step

void inciter::Discretization::residual(tk::real r)

Update residual (during convergence to steady state)

Parameters
in Current residual

void inciter::Discretization::pit(std::size_t it)

Update number of pressure linear solve iterations taken.

Parameters
it in Number of pressure linear solve iterations taken

Variable documentation

uint64_t inciter::Discretization::m_itr

Iteration count with mesh refinement

Used as the restart sequence number {RS} in saving output in an ExodusII sequence

uint64_t inciter::Discretization::m_itf

Field output iteration count without mesh refinement

Counts the number of field outputs to file during two time steps with mesh efinement

tk::UnsMesh::Chunk inciter::Discretization::m_el

Elements of the mesh chunk we operate on.

Initialized by the constructor. The first vector is the element connectivity (local IDs), the second vector is the global node IDs of owned elements, while the third one is a map of global->local node IDs.

std::vector<tk::real> inciter::Discretization::m_v

Nodal mesh volumes

This is the volume of the mesh associated to nodes of owned elements (sum of surrounding cell volumes / 4) without contributions from other chares on chare-boundaries

std::vector<tk::real> inciter::Discretization::m_vol

Volume of nodes

This is the volume of the mesh associated to nodes of owned elements (sum of surrounding cell volumes / 4) with contributions from other chares on chare-boundaries

std::unordered_map<std::size_t, tk::real> inciter::Discretization::m_volc

Receive buffer for volume of nodes (with global node id as key)

This is a communication buffer used to compute the volume of the mesh associated to nodes of owned elements (sum of surrounding cell volumes / 4) with contributions from other chares on chare-boundaries.

std::unordered_map<int, std::unordered_map<std::size_t, tk::real>> inciter::Discretization::m_cvolc

Chare-volume contributions per chare per chare-boundary node

Outer key: chare id, value: local node id and node volume