inciter::Sorter class
#include <src/Inciter/Sorter.hpp>

Mesh sorter for global distributed mesh node reordering.

Public static functions

static void registerReducers(): Configure Charm++ reduction types.

Constructors, destructors, conversion operators

Sorter(std::size_t meshid, const CProxy_Transporter& transporter, const tk::CProxy_MeshWriter& meshwriter, const tk::SorterCallback& cbs, const CProxy_Discretization& discretization, const CProxy_RieCG& riecg, const CProxy_LaxCG& laxcg, const CProxy_ZalCG& zalcg, const CProxy_KozCG& kozcg, const CProxy_ChoCG& chocg, const CProxy_LohCG& lohcg, const tk::CProxy_ConjugateGradients& cgpre, const tk::CProxy_ConjugateGradients& cgmom, CkCallback reorderRefiner, const std::vector<std::size_t>& ginpoel, const tk::UnsMesh::CoordMap& coordmap, const tk::UnsMesh::Chunk& el, const std::map<int, std::vector<std::size_t>>& bface, const std::vector<std::size_t>& triinpoel, const std::map<int, std::vector<std::size_t>>& bnode, int nchare) explicit: Constructor.
Sorter(CkMigrateMessage* m) explicit: Migrate constructor.

Public functions

void setup(std::size_t npoin): Setup chare mesh boundary node communication map.
void query(int fromch, const std::unordered_set<std::size_t>& bnd): Incoming query for a list mesh nodes for which this chare compiles communication maps.
void recvquery(): Report receipt of boundary node lists.
void response(): Respond to boundary node list queries.
void bnd(int fromch, const std::map<int, std::unordered_set<std::size_t>>& nodeCommMap): Receive boundary node communication map for our mesh chunk.
void recvbnd(): Receive receipt of boundary node communication map.
void start(): Start reordering (if user enabled it)
void offset(int c, std::size_t u): Receive number of uniquely assigned global mesh node IDs from chares with lower indices.
void request(int c, const std::unordered_set<std::size_t>& nd): Request new global node IDs for old node IDs.
void lower(std::size_t low): Receive lower bound of node IDs our PE operates on after reordering.
void neworder(const std::unordered_map<std::size_t, std::tuple<std::size_t, tk::UnsMesh::Coord>>& nodes): Receive new (reordered) global node IDs and coordinates.
void createWorkers(): Create worker chare array elements on this PE.
void mesh(std::vector<std::size_t>& ginpoel, tk::UnsMesh::CoordMap& coordmap, std::vector<std::size_t>& triinpoel, std::map<int, std::vector<std::size_t>>& bnode): Update mesh data we hold for whoever calls this function.

Charm++ pack/unpack serializer member functions

std::size_t m_meshid: Mesh ID.
CProxy_Transporter m_host: Host proxy.
tk::CProxy_MeshWriter m_meshwriter: MeshWriter proxy.
tk::SorterCallback m_cbs: Charm++ callbacks associated to compile-time tags for sorter.
CProxy_Discretization m_discretization: Discretization base proxy.
CProxy_RieCG m_riecg: Discretization scheme proxy.
CProxy_LaxCG m_laxcg: Discretization scheme proxy.
CProxy_ZalCG m_zalcg: Discretization scheme proxy.
CProxy_KozCG m_kozcg: Discretization scheme proxy.
CProxy_ChoCG m_chocg: Discretization scheme proxy.
CProxy_LohCG m_lohcg: Discretization scheme proxy.
tk::CProxy_ConjugateGradients m_cgpre: Conjugate Gradients Charm++ proxy for pressure solve.
tk::CProxy_ConjugateGradients m_cgmom: Conjugate Gradients Charm++ proxy for momentum solve.
CkCallback m_reorderRefiner: Callback to use to send reordered mesh to Refiner.
std::vector<std::size_t> m_ginpoel: Tetrtahedron element connectivity of our chunk of the mesh (global ids)
tk::UnsMesh::CoordMap m_coordmap: Coordinates associated to global node IDs of our mesh chunk.
tk::UnsMesh::Chunk m_el: Elements of the mesh chunk we operate on.
std::size_t m_nbnd: Counter for number of chares contributing to chare boundary nodes.
std::map<int, std::vector<std::size_t>> m_bface: List of boundary faces associated to side-set IDs.
std::vector<std::size_t> m_triinpoel: Boundary face-node connectivity.
std::map<int, std::vector<std::size_t>> m_bnode: List of boundary nodes associated to side-set IDs.
int m_nchare: Total number of sorter chares.
std::set<std::size_t> m_nodeset: Unique global node IDs chares on our PE will contribute to.
int m_noffset: Counter for the number of chares from which this chare has received node reordering offsets from.
std::unordered_map<std::size_t, std::vector<int>> m_nodech: Node->chare map used to build boundary node communication maps.
std::unordered_map<int, std::unordered_set<std::size_t>> m_chnode: Chare->node map used to build boundary node communication maps.
std::map<int, std::unordered_set<std::size_t>> m_nodeCommMap: Node lists associated to chare IDs.
std::unordered_map<int, std::unordered_set<std::size_t>> m_reordcomm: Communication map used for distributed mesh node reordering.
std::size_t m_start: Starting global mesh node ID for node reordering on this chare during mesh node reordering.
std::unordered_map<std::size_t, std::size_t> m_newnodes: Map associating new node IDs (value) to old node IDs (key)
tk::UnsMesh::CoordMap m_newcoordmap: Coordinates associated to global (new) node IDs during reordering.
std::vector<std::pair<int, std::unordered_set<std::size_t>>> m_reqnodes: Queue of requested node IDs from chares.
std::size_t m_lower: Lower bound of node IDs this chare contributes to in a linear system.
std::size_t m_upper: Upper bound of node IDs this chare contributes to in a linear system.
void pup(PUP::er& p) override: Pack/Unpack serialize member function.
void operator|(PUP::er& p, Sorter& s): Pack/Unpack serialize operator|.

Function documentation

inciter::Sorter::Sorter(std::size_t meshid, const CProxy_Transporter& transporter, const tk::CProxy_MeshWriter& meshwriter, const tk::SorterCallback& cbs, const CProxy_Discretization& discretization, const CProxy_RieCG& riecg, const CProxy_LaxCG& laxcg, const CProxy_ZalCG& zalcg, const CProxy_KozCG& kozcg, const CProxy_ChoCG& chocg, const CProxy_LohCG& lohcg, const tk::CProxy_ConjugateGradients& cgpre, const tk::CProxy_ConjugateGradients& cgmom, CkCallback reorderRefiner, const std::vector<std::size_t>& ginpoel, const tk::UnsMesh::CoordMap& coordmap, const tk::UnsMesh::Chunk& el, const std::map<int, std::vector<std::size_t>>& bface, const std::vector<std::size_t>& triinpoel, const std::map<int, std::vector<std::size_t>>& bnode, int nchare) explicit

Constructor.

Parameters
meshid in	Mesh ID
transporter in	Transporter (host) Charm++ proxy
meshwriter in	Mesh writer Charm++ proxy
cbs in	Charm++ callbacks for Sorter
discretization in	Discretization Charm++ proxy
riecg in	RieCG Charm++ proxy
laxcg in	RieCG Charm++ proxy
zalcg in	ZalCG Charm++ proxy
kozcg in	KozCG Charm++ proxy
chocg in	ChoCG Charm++ proxy
lohcg in	LohCG Charm++ proxy
cgpre in	ConjugateGradients Charm++ proxy for pressure solve
cgmom in	ConjugateGradients Charm++ proxy for momentum solve
reorderRefiner in	Callback to use to send reordered mesh to Refiner
ginpoel in	Mesh connectivity (this chare) using global node IDs
coordmap in	Mesh node coordinates (this chare) for global node IDs
el in	Elements of the mesh chunk we operate on
bface in	Face lists mapped to side set ids
triinpoel in	Interconnectivity of points and boundary-faces
bnode in	Node ids mapped to side set ids
nchare in	Total number of Charm++ worker chares

void inciter::Sorter::setup(std::size_t npoin)

Setup chare mesh boundary node communication map.

Parameters
npoin in	Total number of mesh points in mesh. Note that the number of mesh points does not have to be exactly the total number of points in the mesh. It can be a larger number, but not less. This is only used here to assign nodes to workers that will assign ids to mesh nodes during node reordering.

void inciter::Sorter::query(int fromch, const std::unordered_set<std::size_t>& bnd)

Incoming query for a list mesh nodes for which this chare compiles communication maps.

Parameters
fromch in	Sender chare ID
bnd in	Chare-boundary data from another chare

void inciter::Sorter::bnd(int fromch, const std::map<int, std::unordered_set<std::size_t>>& nodeCommMap)

Receive boundary node communication map for our mesh chunk.

Parameters
fromch in	Sender chare ID
nodeCommMap in	Communication map assembled by chare fromch

void inciter::Sorter::offset(int c, std::size_t u)

Receive number of uniquely assigned global mesh node IDs from chares with lower indices.

Parameters
c in	Chare ID
u in	Number of mesh node IDs chare c will assign IDs to

This function computes the offset each chare will need to start assigning its new node IDs from. The offset for a chare is the offset for the previous chare plus the number of node IDs the previous chare (uniquely) assigns new IDs for minus the number of node IDs the previous chare receives from others (lower chares). This is computed here in a parallel/distributed fashion by each chare sending its number of node IDs (that it uniquely assigns) to all chares. Note that each chare would only need to send this information to chares with higher IDs, but instead this function is called in a broadcast fashion, because that is more efficient than individual calls to only chares with higher IDs. Therefore when computing the offsets, we only count the lower chares. When this is done, we have the precise asymmetric communication map as well as the start offset on all chares and so we can start the distributed global mesh node ID reordering.

void inciter::Sorter::request(int c, const std::unordered_set<std::size_t>& nd)

Request new global node IDs for old node IDs.

Parameters
c in	Chare request coming from and to which we send new IDs to
nd in	Set of old node IDs whose new IDs are requested

void inciter::Sorter::neworder(const std::unordered_map<std::size_t, std::tuple<std::size_t, tk::UnsMesh::Coord>>& nodes)

Receive new (reordered) global node IDs and coordinates.

Parameters
nodes in	Map associating new to old node IDs

void inciter::Sorter::mesh(std::vector<std::size_t>& ginpoel, tk::UnsMesh::CoordMap& coordmap, std::vector<std::size_t>& triinpoel, std::map<int, std::vector<std::size_t>>& bnode)

Update mesh data we hold for whoever calls this function.

Parameters
ginpoel in/out	Mesh connectivity using global IDs
coordmap in/out	Map of mesh node coordinates
triinpoel in/out	Boundary face-node connectivity
bnode in	Node lists of side sets

void inciter::Sorter::pup(PUP::er& p) override

Pack/Unpack serialize member function.

Parameters
p in/out	Charm++'s PUP::er serializer object reference

void inciter::Sorter::operator|(PUP::er& p, Sorter& s)

Pack/Unpack serialize operator|.

Parameters
p in/out	Charm++'s PUP::er serializer object reference
s in/out	Sorter object reference

Variable documentation

std::unordered_map<int, std::unordered_set<std::size_t>> inciter::Sorter::m_reordcomm

Communication map used for distributed mesh node reordering.

This map associates the list of global mesh point indices to fellow chare IDs from which this chare receives new node IDs during reordering. Only data that will be received from chares with a lower index are stored, thus this is an asymmetric communication map.

inciter::Sorter class #include <src/Inciter/Sorter.hpp>