Implementation of additional scatter/gather operations in panzer

[kincaidkc]

Good afternoon all,

I am implementing an equation set in Panzer which includes a term calculated from the lumped L2 projection of a field, an operation which requires me to integrate fields from points to nodes, and sum contributions from all cells sharing a single node. I am aware that there is a L2Projection class within Panzer, but would prefer to handle things myself as it is not clear to me how the existing class interfaces with the rest of the code. I have a first implementation completed which follows this series of steps:

1. Calculate the value of the field on which we are doing the lumped projection at the integration points, let's call it _phi(cell, point).
2. Use the Panzer_Integrator_BasisTimesScalar evaluator to integrate the point field to the nodes, giving us _phi_integrated(cell, basis).
3. Use Panzer_ScatterResidual_Tpetra to scatter the nodal information into a single vector, in the process performing the summation of each cell's contribution to the true integrated nodal value.
4. Use Panzer_GatherSolution_Tpetra to pull this information back from the vector into a local form
5. In a final evaluator, use the basis weights to move the integrated and summed nodal values back to the integration points to calculate the field use in the governing equations

I have this pipeline set up and 90% working in my code. However, there are some abnormalities in the field at physical and workset boundaries which lead me to believe there is an issue with how the ghosting is being handled in the scatter and gather operations. Right now, I am making no specific effort to ensure this is done correctly, as I was (perhaps naively) hoping Panzer would handle this in the back end for me. The scatter and gather evaluators are pushing to and pulling from the same TpetraLinearObjContainer, which I created with the following steps:

1. Build the container using the TpetraLinearObjFactory::buildGhostedLinearObjContainer method
2. Get a Thyra::VectorSpaceBase using the TpetraLinearObjFactory::getThyraDomainSpace
3. Create a vector in this space using the Thyra::createMember function
4. Set both X and F in the Tpetra LOC to point to this vector using the set_x_th and set_f_th methods
5. Call the initialize() function of the Tpetra LOC to set all fields to zero

This procedure is done in the buildAndRegisterGatherAndOrientationEvaluators function in my equation set. I have made some modifications to the panzer back end to allow me to insert the new data containers into the panzer::AssemblyEngine_InArgs so that they are included in the main global evaluation data container, but haven't modified any other part of the code.

The data seems to flow through this pipeline correctly, except that the values at physical and workset boundaries varies slightly from the expected value. It does not seem to be off by any consistent factor (i.e. a factor of 2, 4, etc). I can eliminate the abnormal values on workset boundaries just by increasing the workset size so that all DOFs fit on one set (not a long-term solution), but cannot find a way to debug the behavior at physical boundaries. I would greatly appreciate it if anyone could offer some insight as to what I might be doing wrong. I have tried to follow the implementation for scattering and gathering DOFs as closely as possible, but some parts of the code are a little opaque to me.

Thanks,
Kellis

@rppawlo @cgcgcg

[rppawlo]

@kincaidkc - I suspect there's an issue with the mpi communication to sum the ghosted contributions. Let's set up a meeting this week to talk about this.