Skip to content

ThomasFermi: Fix Domain Decomposition #2007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
psychocoderHPC merged 1 commit into from
May 5, 2017
Merged

ThomasFermi: Fix Domain Decomposition #2007

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion src/picongpu/include/particles/ionization/byCollision/ThomasFermi/ThomasFermi_Impl.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,11 @@ namespace ionization

using FrameType = typename SrcSpecies::FrameType;

/* specify field to particle interpolation scheme */
/** specify field to particle interpolation scheme
*
* @todo this needs to be done independently/twice if ion species (rho) and electron
* species (ene) are of different shape
Copy link
Member

@n01r n01r May 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In principle one could add another todo here
If one were to use two different atomic species they would currently each do their own density and energy calculations. Yet they would have no knowledge of each other. But of course IPD effects do not care about the species and so it remains a technical detail to be solved on how to include that. 😃

Copy link
Member Author

@ax3l ax3l May 4, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, that's a todo I have in my collisional methods, too xD
but let's keep it that way for the bugfix, feel free to add a .. note:: or .. warning:: to the user docs you write right now

Copy link
Member

@n01r n01r May 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure sure, that's a follow-up feature and I have a hunch that it requires some careful thinking. 😆

*/
using Field2ParticleInterpolation = typename PMacc::traits::Resolve<
typename GetFlagType<FrameType,interpolation<> >::type
>::type;
Expand Down Expand Up @@ -166,13 +170,20 @@ namespace ionization
/* kernel call for weighted ion density calculation */
density->template computeValue< CORE + BORDER, DensitySolver >(*srcSpecies, currentStep);
dc.releaseData( SrcSpecies::FrameType::getName() );
EventTask densityEvent = density->asyncCommunication( __getTransactionEvent() );
densityEvent += density->asyncCommunicationGather( densityEvent );
Copy link
Member

@n01r n01r May 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for me again: the first of the two lines above gets the event of the FieldTmp operation and the second line says that in addition to this event the results from the previous event in the BORDER region of the neigboring GPU have ton be communicated to the GUARD of this GPU. Only afterwards the density values for particles whose shape extends outside the BORDER region will be correct, right?

Copy link
Member Author

@ax3l ax3l May 4, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, this code had two bugs with regards to domain decomposition:

  • the BORDER was not filled correct (with additions from neighbor-GPU particles that have shape contributions) and
  • since you access the density and energy density fields with respect to shape you also access the guard which was not created at all.

The first event, asyncCommunication builds a valid BORDER, the second event, asyncCommunicationGather builds a valid GUARD. Both APIs are also doxygen documented now since #2010


/* load species without copying the particle data to the host */
auto destSpecies = dc.get< DestSpecies >( DestSpecies::FrameType::getName(), true );

/* kernel call for weighted electron energy density calculation */
eneKinDens->template computeValue< CORE + BORDER, EnergyDensitySolver >(*destSpecies, currentStep);
dc.releaseData( DestSpecies::FrameType::getName() );
EventTask eneKinEvent = eneKinDens->asyncCommunication( __getTransactionEvent() );
eneKinEvent += eneKinDens->asyncCommunicationGather( eneKinEvent );

/* contributions from neighboring GPUs to our border area */
__setTransactionEvent( densityEvent + eneKinEvent );

/* initialize device-side density- and energy density field databox pointers */
rhoBox = density->getDeviceDataBox();
Expand Down