inciter::Sorter class

Mesh sorter for global distributed mesh node reordering.

Public static functions

static void registerReducers()
Configure Charm++ reduction types.

Constructors, destructors, conversion operators

Sorter(std::size_t meshid, const CProxy_Transporter& transporter, const tk::CProxy_MeshWriter& meshwriter, const tk::SorterCallback& cbs, const CProxy_Discretization& discretization, const CProxy_RieCG& riecg, const CProxy_LaxCG& laxcg, const CProxy_ZalCG& zalcg, const CProxy_KozCG& kozcg, const CProxy_ChoCG& chocg, const tk::CProxy_ConjugateGradients& cgpre, CkCallback reorderRefiner, const std::vector<std::size_t>& ginpoel, const tk::UnsMesh::CoordMap& coordmap, const tk::UnsMesh::Chunk& el, const std::map<int, std::vector<std::size_t>>& bface, const std::vector<std::size_t>& triinpoel, const std::map<int, std::vector<std::size_t>>& bnode, int nchare) explicit
Constructor.
Sorter(CkMigrateMessage* m) explicit
Migrate constructor.

Public functions

void setup(std::size_t npoin)
Setup chare mesh boundary node communication map.
void query(int fromch, const std::unordered_set<std::size_t>& bnd)
Incoming query for a list mesh nodes for which this chare compiles communication maps.
void recvquery()
Report receipt of boundary node lists.
void response()
Respond to boundary node list queries.
void bnd(int fromch, const std::map<int, std::unordered_set<std::size_t>>& nodeCommMap)
Receive boundary node communication map for our mesh chunk.
void recvbnd()
Receive receipt of boundary node communication map.
void start()
Start reordering (if user enabled it)
void offset(int c, std::size_t u)
Receive number of uniquely assigned global mesh node IDs from chares with lower indices.
void request(int c, const std::unordered_set<std::size_t>& nd)
Request new global node IDs for old node IDs.
void lower(std::size_t low)
Receive lower bound of node IDs our PE operates on after reordering.
void neworder(const std::unordered_map<std::size_t, std::tuple<std::size_t, tk::UnsMesh::Coord>>& nodes)
Receive new (reordered) global node IDs and coordinates.
void createWorkers()
Create worker chare array elements on this PE.
void mesh(std::vector<std::size_t>& ginpoel, tk::UnsMesh::CoordMap& coordmap, std::vector<std::size_t>& triinpoel, std::map<int, std::vector<std::size_t>>& bnode)
Update mesh data we hold for whoever calls this function.

Charm++ pack/unpack serializer member functions

std::size_t m_meshid
Mesh ID.
CProxy_Transporter m_host
Host proxy.
tk::CProxy_MeshWriter m_meshwriter
MeshWriter proxy.
tk::SorterCallback m_cbs
Charm++ callbacks associated to compile-time tags for sorter.
CProxy_Discretization m_discretization
Discretization base proxy.
CProxy_RieCG m_riecg
Discretization scheme proxy.
CProxy_LaxCG m_laxcg
Discretization scheme proxy.
CProxy_ZalCG m_zalcg
Discretization scheme proxy.
CProxy_KozCG m_kozcg
Discretization scheme proxy.
CProxy_ChoCG m_chocg
Discretization scheme proxy.
tk::CProxy_ConjugateGradients m_cgpre
Conjugate Gradients Charm++ proxy for pressure solve.
CkCallback m_reorderRefiner
Callback to use to send reordered mesh to Refiner.
std::vector<std::size_t> m_ginpoel
Tetrtahedron element connectivity of our chunk of the mesh (global ids)
tk::UnsMesh::CoordMap m_coordmap
Coordinates associated to global node IDs of our mesh chunk.
tk::UnsMesh::Chunk m_el
Elements of the mesh chunk we operate on.
std::size_t m_nbnd
Counter for number of chares contributing to chare boundary nodes.
std::map<int, std::vector<std::size_t>> m_bface
List of boundary faces associated to side-set IDs.
std::vector<std::size_t> m_triinpoel
Boundary face-node connectivity.
std::map<int, std::vector<std::size_t>> m_bnode
List of boundary nodes associated to side-set IDs.
int m_nchare
Total number of sorter chares.
std::set<std::size_t> m_nodeset
Unique global node IDs chares on our PE will contribute to.
int m_noffset
Counter for the number of chares from which this chare has received node reordering offsets from.
std::unordered_map<std::size_t, std::vector<int>> m_nodech
Node->chare map used to build boundary node communication maps.
std::unordered_map<int, std::unordered_set<std::size_t>> m_chnode
Chare->node map used to build boundary node communication maps.
std::map<int, std::unordered_set<std::size_t>> m_nodeCommMap
Node lists associated to chare IDs.
std::unordered_map<int, std::unordered_set<std::size_t>> m_reordcomm
Communication map used for distributed mesh node reordering.
std::size_t m_start
Starting global mesh node ID for node reordering on this chare during mesh node reordering.
std::unordered_map<std::size_t, std::size_t> m_newnodes
Map associating new node IDs (value) to old node IDs (key)
tk::UnsMesh::CoordMap m_newcoordmap
Coordinates associated to global (new) node IDs during reordering.
std::vector<std::pair<int, std::unordered_set<std::size_t>>> m_reqnodes
Queue of requested node IDs from chares.
std::size_t m_lower
Lower bound of node IDs this chare contributes to in a linear system.
std::size_t m_upper
Upper bound of node IDs this chare contributes to in a linear system.
void pup(PUP::er& p) override
Pack/Unpack serialize member function.
void operator|(PUP::er& p, Sorter& s)
Pack/Unpack serialize operator|.

Function documentation

inciter::Sorter::Sorter(std::size_t meshid, const CProxy_Transporter& transporter, const tk::CProxy_MeshWriter& meshwriter, const tk::SorterCallback& cbs, const CProxy_Discretization& discretization, const CProxy_RieCG& riecg, const CProxy_LaxCG& laxcg, const CProxy_ZalCG& zalcg, const CProxy_KozCG& kozcg, const CProxy_ChoCG& chocg, const tk::CProxy_ConjugateGradients& cgpre, CkCallback reorderRefiner, const std::vector<std::size_t>& ginpoel, const tk::UnsMesh::CoordMap& coordmap, const tk::UnsMesh::Chunk& el, const std::map<int, std::vector<std::size_t>>& bface, const std::vector<std::size_t>& triinpoel, const std::map<int, std::vector<std::size_t>>& bnode, int nchare) explicit

Constructor.

Parameters
meshid in Mesh ID
transporter in Transporter (host) Charm++ proxy
meshwriter in Mesh writer Charm++ proxy
cbs in Charm++ callbacks for Sorter
discretization in Discretization Charm++ proxy
riecg in RieCG Charm++ proxy
laxcg in RieCG Charm++ proxy
zalcg in ZalCG Charm++ proxy
kozcg in KozCG Charm++ proxy
chocg in KozCG Charm++ proxy
cgpre in ConjugateGradients Charm++ proxy for pressure solve
reorderRefiner in Callback to use to send reordered mesh to Refiner
ginpoel in Mesh connectivity (this chare) using global node IDs
coordmap in Mesh node coordinates (this chare) for global node IDs
el in Elements of the mesh chunk we operate on
bface in Face lists mapped to side set ids
triinpoel in Interconnectivity of points and boundary-faces
bnode in Node ids mapped to side set ids
nchare in Total number of Charm++ worker chares

void inciter::Sorter::setup(std::size_t npoin)

Setup chare mesh boundary node communication map.

Parameters
npoin in Total number of mesh points in mesh. Note that the number of mesh points does not have to be exactly the total number of points in the mesh. It can be a larger number, but not less. This is only used here to assign nodes to workers that will assign ids to mesh nodes during node reordering.

void inciter::Sorter::query(int fromch, const std::unordered_set<std::size_t>& bnd)

Incoming query for a list mesh nodes for which this chare compiles communication maps.

Parameters
fromch in Sender chare ID
bnd in Chare-boundary data from another chare

void inciter::Sorter::bnd(int fromch, const std::map<int, std::unordered_set<std::size_t>>& nodeCommMap)

Receive boundary node communication map for our mesh chunk.

Parameters
fromch in Sender chare ID
nodeCommMap in Communication map assembled by chare fromch

void inciter::Sorter::offset(int c, std::size_t u)

Receive number of uniquely assigned global mesh node IDs from chares with lower indices.

Parameters
in Chare ID
in Number of mesh node IDs chare c will assign IDs to

This function computes the offset each chare will need to start assigning its new node IDs from. The offset for a chare is the offset for the previous chare plus the number of node IDs the previous chare (uniquely) assigns new IDs for minus the number of node IDs the previous chare receives from others (lower chares). This is computed here in a parallel/distributed fashion by each chare sending its number of node IDs (that it uniquely assigns) to all chares. Note that each chare would only need to send this information to chares with higher IDs, but instead this function is called in a broadcast fashion, because that is more efficient than individual calls to only chares with higher IDs. Therefore when computing the offsets, we only count the lower chares. When this is done, we have the precise asymmetric communication map as well as the start offset on all chares and so we can start the distributed global mesh node ID reordering.

void inciter::Sorter::request(int c, const std::unordered_set<std::size_t>& nd)

Request new global node IDs for old node IDs.

Parameters
in Chare request coming from and to which we send new IDs to
nd in Set of old node IDs whose new IDs are requested

void inciter::Sorter::neworder(const std::unordered_map<std::size_t, std::tuple<std::size_t, tk::UnsMesh::Coord>>& nodes)

Receive new (reordered) global node IDs and coordinates.

Parameters
nodes in Map associating new to old node IDs

void inciter::Sorter::mesh(std::vector<std::size_t>& ginpoel, tk::UnsMesh::CoordMap& coordmap, std::vector<std::size_t>& triinpoel, std::map<int, std::vector<std::size_t>>& bnode)

Update mesh data we hold for whoever calls this function.

Parameters
ginpoel in/out Mesh connectivity using global IDs
coordmap in/out Map of mesh node coordinates
triinpoel in/out Boundary face-node connectivity
bnode in Node lists of side sets

void inciter::Sorter::pup(PUP::er& p) override

Pack/Unpack serialize member function.

Parameters
in/out Charm++'s PUP::er serializer object reference

void inciter::Sorter::operator|(PUP::er& p, Sorter& s)

Pack/Unpack serialize operator|.

Parameters
in/out Charm++'s PUP::er serializer object reference
in/out Sorter object reference

Variable documentation

std::unordered_map<int, std::unordered_set<std::size_t>> inciter::Sorter::m_reordcomm

Communication map used for distributed mesh node reordering.

This map associates the list of global mesh point indices to fellow chare IDs from which this chare receives new node IDs during reordering. Only data that will be received from chares with a lower index are stored, thus this is an asymmetric communication map.