Restruct global linear solvers to use a greater amount of local coefficient format
This Merge Request helps reduce the set up communication of the global Linear system solutions by keeping the data formal in the local coefficient space rather than adopting the global/universal ordering so early in the call stack. The updates keeps the local coefficient structure right down to the GlobalLinSysSolve classes and could possibly be pushed even further, however now we can make this decision at the GlobalinSysSolve factory pattern level.
As an example of the changes the original implementation has communication for
- Inner product in global space within HelmSolve
- Weak forcing is the imposed by mapping back to the local space and then back again.
- Impose dirichlet conditions invoked a call to UniversalAssembleBnd.
- LocalToGlobal space of inout array to solver
By keeping everything in local space for longer it is possible to just have communication for
- Assemble RHS (after imposing weak forcing)
- LocalToGlobal of inout array after imnposing Diriichlet conditions,
Note that the ImposeDirichlet condition does now operate on the local coefficient space but there is still the potential for communication when a vertex (or edge in 3D) only just touches a Dirichlet boundary within a partition. To handle this there is now a new gs handle which calls a new methods amax (absolute max). This therefore has required a modification to gslib and this will need to potentially be pushed to the gitlab libary of gs?
To do this we have replaced m_bndCondCoeffsToGlobalCoeffs with a m_bndCondCoeffToLocalCoeffs in AssemblyMap. I have also renamed AssemblyMap::m_bndCondTraceToGlobalTraceMap to AssemblyMap::m_bndCondIDToGlobalID which seemed more descriptive.
In making this update I have removed the option to have eGlobal format for the coefficients and have also removed the Global Optimisation parameters since they no longer make any sense if we keep everything in local coefficient space.
Note: In calls to GlobalSolve such as ContField::FwdTrans the outarray contains the initial condition for the solve must be given and so some care must be made this is initialised before calling the methods. Previously this was by default zeroed. Also it has been necessary to do a LocalToGlobal and a GlobalToLocal call on the initial conditions in Continuous expansions to ensure the condition is C0 continuous (otherwise tests have problems)