False density initialization with several GPUs using FreeFormula?

[ssstuvz]

I have 3 GPUs in y (laser propagation direction)

namespace gasFreeFormula
    {
        namespace SI
        {
            struct GasProfile
            {
                /** Normalized Gas Profile [0.0:1.0]
                 *
                 * This formula should use SI quantities only!
                 * The normalized profile will be multiplied by GAS_DENSITY.
                 *
                 * @param pos vector with offset to the global (left top front) cell
                 * @return float_X between 0.0 and 1.0
                 */
                DINLINE float_64 operator()( floatD_64 pos )
                {
                    const float_64 x = pos.x() * 100.0; // m -> cm
                    const float_64 cell_width_cm = ::picongpu::SI::CELL_WIDTH_SI * 100.0;
                    const float_64 Length_X_cm = 384 * cell_width_cm;
                    const float_64 Center = Length_X_cm / 2.0;
                    const float_64 W0_cm = 5e-6 * 100.0;
                    const float_64 r = x-Center;
                    const float_64 Channel_Depth = 1.0/3.1415/2.818e-13/W0_cm/W0_cm;
                    const float_64 Density_cm = ::picongpu::SI::GAS_DENSITY_SI * 1e-6;


                        float_64 s = 0.1 + 0.9*sin(x/1e-4);
                    //float_64 s = 1.0 + Channel_Depth/Density_cm*r*r/W0_cm/W0_cm;

                    /* all parts of the function MUST be > 0 */
                    s *= float_64( s >= 0.0 );

                    return s;
                }
            };
        }
    }

[ax3l]

on the very first few, make sure you do not mix doubles and int's in c++.

or to be more precise, run g++ calc.cpp && ./a.out on this code:

#include <iostream>
#include <cmath>

int main()
{
    std::cout << 4/3 * M_PI << std::endl;
}

affected lines:

const float_64 Length_X_cm = 384. * cell_width_cm;
const float_64 W0_cm = 5.e-6 * 100.0;
const float_64 Density_cm = ::picongpu::SI::GAS_DENSITY_SI * 1.e-6;

That will probably not fix it already, so pls post your .cfg settings and your number or particles per cell, too. Also, pls some output lines that you got (e.g. enhanced by enabling PIC_VERBOSE=9 during configure or with ccmake).

[ssstuvz]

Right now, it is only this line, Lengths and W0 and Density are not used:

float_64 s = 0.1 + 0.9*sin(x/1e-4);

[ax3l]

uh, an other line. please use:

float_64 s = 0.1 + 0.9*math::sin(x/1.e-4);

update: I was too slow :)

[ssstuvz]

Forgot to mention, it works on a single GPU, so I don't think that's the issue.

Here is config file from tbg folder:

TBG_wallTime="1:00:00"

TBG_gpu_x=1
TBG_gpu_y=3
TBG_gpu_z=1

TBG_gridSize="-g 384 2304 1"
TBG_steps="-s 10000 -m"


#################################
## Section: Optional Variables ##
#################################

# png image output (electron density)
TBG_pngYX="--png_e.period 100 --png_e.axis yx --png_e.slicePoint 0.5 --png_e.folder pngElectronsYX"

# binary density output for slides
TBG_binaryDensity="--binDensity_e.period 100 --binDensity_e.axis yx --binDensity_e.slicePoint 0.5 --binDensity_e.folder densityElectrons"

TBG_hdf5="--hdf5.period 100"

TBG_plugins="!TBG_pngYX                    \
             !TBG_hdf5                     \
             !TBG_binaryDensity            \
             --elec_cnt.period 100"     


#################################
## Section: Program Parameters ##
#################################

TBG_devices="-d !TBG_gpu_x !TBG_gpu_y !TBG_gpu_z"

TBG_programParams="!TBG_devices      \
                   !TBG_gridSize     \
                   !TBG_steps        \
                   !TBG_plugins  | tee output"

# TOTAL number of GPUs
TBG_tasks="$(( TBG_gpu_x * TBG_gpu_y * TBG_gpu_z ))"

"$TBG_cfgPath"/submitAction.sh

[ssstuvz]

And output (should I put long outputs here or somewhere else?):
p.s. This is run on jetsons

 ========================   JOB MAP   ========================

 Data for node: jetson0 Num procs: 1
    Process OMPI jobid: [57778,1] Process rank: 0

 Data for node: jetson1 Num procs: 1
    Process OMPI jobid: [57778,1] Process rank: 1

 Data for node: jetson2 Num procs: 1
    Process OMPI jobid: [57778,1] Process rank: 2

 =============================================================
[1,1]<stdout>:PIConGPUVerbose DOMAINS(2) | rank {0,1}; localsize {384,768}; localoffset {0,768};
[1,2]<stdout>:PIConGPUVerbose DOMAINS(2) | rank {0,2}; localsize {384,768}; localoffset {0,1536};
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | rank {0,0}; localsize {384,768}; localoffset {0,0};
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | Sliding Window is ON
[1,1]<stdout>:PIConGPUVerbose MEMORY(8) | size for all exchange = 36 MiB
[1,2]<stdout>:PIConGPUVerbose MEMORY(8) | size for all exchange = 36 MiB
[1,0]<stdout>:PIConGPUVerbose MEMORY(8) | size for all exchange = 36 MiB
[1,1]<stdout>:PIConGPUVerbose MEMORY(8) | Shared RAM between GPU and host detected - using only half of the 'device' memory.
[1,1]<stdout>:PIConGPUVerbose MEMORY(8) | create 36 MiB for species e
[1,1]<stdout>:mem for particles=36 MiB = 20926 Frames = 1339264 Particles
[1,1]<stdout>:PIConGPUVerbose MEMORY(8) | free mem after all mem is allocated 421 MiB
[1,0]<stdout>:PIConGPUVerbose MEMORY(8) | Shared RAM between GPU and host detected - using only half of the 'device' memory.
[1,0]<stdout>:PIConGPUVerbose MEMORY(8) | create 92 MiB for species e
[1,0]<stdout>:mem for particles=92 MiB = 53527 Frames = 3425728 Particles
[1,0]<stdout>:PIConGPUVerbose MEMORY(8) | free mem after all mem is allocated 454 MiB
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | max weighting 18.8269
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | Courant c*dt <= 1.79196 ? 1
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | omega_pe * dt <= 0.1 ? 0.00451317
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | y-cells per wavelength: 22.5734
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | macro particles per gpu: 2949120
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | typical macro particle weighting: 18.8269
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | UNIT_SPEED 2.99792e+08
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | UNIT_TIME 8e-17
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | UNIT_LENGTH 2.39834e-08
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | UNIT_MASS 1.71501e-29
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | UNIT_CHARGE 3.0164e-18
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | UNIT_EFIELD 2.13064e+13
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | UNIT_BFIELD 71070.4
[1,0]<stdout>:PIConGPUVerbose PHYSICS(1) | UNIT_ENERGY 1.54138e-12
[1,2]<stdout>:PIConGPUVerbose MEMORY(8) | Shared RAM between GPU and host detected - using only half of the 'device' memory.
[1,2]<stdout>:PIConGPUVerbose MEMORY(8) | create 410 MiB for species e
[1,2]<stdout>:mem for particles=410 MiB = 238714 Frames = 15277696 Particles
[1,2]<stdout>:PIConGPUVerbose MEMORY(8) | free mem after all mem is allocated 451 MiB
[1,0]<stdout>:PIConGPUVerbose MEMORY(8) | free mem after all particles are initialized 441 MiB
[1,1]<stdout>:PIConGPUVerbose MEMORY(8) | free mem after all particles are initialized 421 MiB
[1,0]<stdout>:initialization time: 36sec 273msec = 36 sec
[1,2]<stdout>:PIConGPUVerbose MEMORY(8) | free mem after all particles are initialized 438 MiB
[1,1]<stdout>:Ringbuffer: memory overflow
[1,0]<stdout>:PIConGPUVerbose CRITICAL(4) | maximum number of  particles on a GPU : 1289472
[1,0]<stdout>:
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | Master create image
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 0byte=0elements | size {768,384}  | offset {0,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 1179648byte=294912elements | size {768,384}  | offset {768,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 2359296byte=589824elements | size {768,384}  | offset {1536,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | Master create image
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 0byte=0elements | size {768,384}  | offset {0,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 3538944byte=294912elements | size {768,384}  | offset {768,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 7077888byte=589824elements | size {768,384}  | offset {1536,0}
[1,0]<stdout>:  0 % =        0 | time elapsed:            37sec  91msec | avg time per step:   0msec
[1,0]<stdout>:PIConGPUVerbose CRITICAL(4) | maximum number of  particles on a GPU : 1289472
[1,0]<stdout>:
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | Master create image
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 0byte=0elements | size {768,384}  | offset {0,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 1179648byte=294912elements | size {768,384}  | offset {768,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 2359296byte=589824elements | size {768,384}  | offset {1536,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | Master create image
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 0byte=0elements | size {768,384}  | offset {0,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 3538944byte=294912elements | size {768,384}  | offset {768,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 7077888byte=589824elements | size {768,384}  | offset {1536,0}
[1,0]<stdout>:PIConGPUVerbose CRITICAL(4) | maximum number of  particles on a GPU : 1289472
[1,0]<stdout>:
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | Master create image
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 0byte=0elements | size {768,384}  | offset {0,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 1179648byte=294912elements | size {768,384}  | offset {768,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 2359296byte=589824elements | size {768,384}  | offset {1536,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | Master create image
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 0byte=0elements | size {768,384}  | offset {0,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 3538944byte=294912elements | size {768,384}  | offset {768,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 7077888byte=589824elements | size {768,384}  | offset {1536,0}
[1,0]<stdout>:PIConGPUVerbose CRITICAL(4) | maximum number of  particles on a GPU : 1289472
[1,0]<stdout>:
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | Master create image
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 0byte=0elements | size {768,384}  | offset {0,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 1179648byte=294912elements | size {768,384}  | offset {768,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 2359296byte=589824elements | size {768,384}  | offset {1536,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | Master create image
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 0byte=0elements | size {768,384}  | offset {0,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 3538944byte=294912elements | size {768,384}  | offset {768,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 7077888byte=589824elements | size {768,384}  | offset {1536,0}
[1,0]<stdout>:PIConGPUVerbose CRITICAL(4) | maximum number of  particles on a GPU : 1289472
[1,0]<stdout>:
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | Master create image
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 0byte=0elements | size {768,384}  | offset {0,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 1179648byte=294912elements | size {768,384}  | offset {768,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 2359296byte=589824elements | size {768,384}  | offset {1536,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | Master create image
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 0byte=0elements | size {768,384}  | offset {0,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 3538944byte=294912elements | size {768,384}  | offset {768,0}
[1,0]<stdout>:PIConGPUVerbose DOMAINS(2) | part image with offset 7077888byte=589824elements | size {768,384}  | offset {1536,0}

[ax3l]

thanks. the problem is, that you do not have enough memory for your particles:

mem for particles=36 MiB = 20926 Frames = 1339264 Particles

... and this number (1339264) is an estimate if the memory is nice and unperturbed.

While your number of cells is below 300'000, which seems to be a healty number for jetsons 2GB RAM it seems to be still an issue that on your ranks 0 and 1 the available memory is quite tight.

Please check is something or someone is running on these nodes. PIConGPU is simply not able to get all the available memory from the node since it is already allocated (outside of the simulation, or the fields == # of cells takes up too much).

[ssstuvz]

Just to confirm: reducing the size of simulation helped.

[ax3l]

all right, thanks for the feedback!