Opt Executable Segfaults with Adaptive Mesh and User-Defined Material Model

[vikramroybarc]

Check these boxes if you have followed the posting rules.

• Q&A General is the most appropriate section for my question
• I have consulted the posting Guidelines on the Discussions front page
• I have searched the Discussions forum and my question has not been asked before
• I have searched the MOOSE website and the documentation does not answer my question
• I have formatted my post following the posting guidelines (screenshots as a last resort, triple back quotes around pasted text)

Question

Hi MOOSE Team,

I am using a user-defined material model in my MOOSE application.

With initial mesh adaptivity, my input file contains:

[Adaptivity]
  marker = marker
  initial_marker = marker
  initial_steps = 1
  stop_time = 0
  max_h_level = 2
  [Markers]
    [marker]
      type = BoxMarker
      bottom_left = '0 0 0'
      top_right = '1 1 0'
      outside = DO_NOTHING
      inside = REFINE
    []
  []
[]

When I enable mesh adaptivity, my opt executable encounters an error during the initial residual calculation and prints:

*** ERROR ***
The following occurred in the Problem 'MOOSE Problem' of type FEProblem.

The following parallel-communicated exception was detected during INITIAL evaluation:
J2 plasticity Newton Raphson did not converge
Because this did not occur during residual evaluation, there is no way to handle this, so the solution is aborting.

Stack frames: 16
0: libMesh::print_trace(std::ostream&)
1: moose::internal::mooseErrorRaw(...)
2: MooseBase::callMooseError(...)
3: MooseBase::callMooseError(...)
4: /home/vikramr/projects/moose/framework/libmoose-opt.so.0(+0xccddf8)
5: FEProblemBase::checkExceptionAndStopSolve(bool)
6: DisplacedProblem::updateMesh(bool)
7: FEProblemBase::initialSetup()
8: TransientBase::init()
9: Transient::init()
10: MooseApp::executeExecutioner()
11: MooseApp::run()
12: main
13: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)
14: __libc_start_main
15: /home/vikramr/projects/rhocp-vr/rhocp-opt(+0x538e)
Abort(1) on node 0: application called MPI_Abort(MPI_COMM_WORLD, 1)
Segmentation fault (core dumped)

However, the dbg executable runs successfully and solves the problem.

Additionally, when I use uniform refinement via the -r option, both the opt and dbg executables work perfectly.

This problem appears only with this specific user-defined material model; all other material models run without any errors.

I do use _state_var[_qp] in my material model, but I only update these variables when _t_step > 0. Therefore, I don’t think the segmentation fault is coming from state variable initialization.

I’m unable to understand why mesh adaptivity causes the opt version to crash during initial evaluation, while the dbg build runs fine.

If anyone has insights or suggestions, please help.

Thank you!

[GiudGiud]

Hello

Therefore, I don’t think the segmentation fault is coming from state variable initialization.

What do you mean by segmentation fault here? The error message looks like a regular error message, not a crash by segmentation fault.

I’m unable to understand why mesh adaptivity causes the opt version to crash during initial evaluation, while the dbg build runs fine.

It is peculiar indeed, as the code is essentially the same in the two versions, with additional checks in debug and less optimization.
If you use a debugger and use a breakpoint on this exception, we could take a look at the code.
Otherwise the typical instructions for failed solves should still apply:
https://mooseframework.inl.gov/moose/application_usage/failed_solves.html

[GiudGiud]

yes you can run the debugger in opt.
However, I would recommend recompiling in devel so the backtrace includes line numbers

[lindsayad]

If it passes cleanly in dbg mode then this sounds like a memory error for which I would use a tool like valgrind

[GiudGiud]

Yeah see the other thread (#31948), it runs cleanly for me in devel, and I just checked in debug, and completed a full run in an hour without any issue either.

Valgrind is the way forward imo as well.

[vikramroybarc]

Hi @GiudGiud,

Thank you for testing this example.

I’m currently travelling until 30 November. Once I’m back in the office, I will retest these examples after updating the Moose environment and repository.

Regarding the memory error: does it indicate insufficient system memory? The machine I’m using to run these examples has an Intel Xeon 4114 processor, 128 GB RAM, and Ubuntu 22 LTS.

[GiudGiud]

Regarding the memory error: does it indicate insufficient system memory?

No we would see a different error for that.
Ok thanks for letting us know