Performance issue: MOOSE+NEML2 computation time increases significantly with DOF compared to COMSOL #32133
Replies: 3 comments 16 replies
-
|
Thank you for the report and more so for providing all of the input! We can take an attempt at profiling this. @hugary1995 do you want to look into this or should I do the profile? |
Beta Was this translation helpful? Give feedback.
-
|
I will look into this. |
Beta Was this translation helpful? Give feedback.
-
are the profiles from before or after the change? Is the issue with nonlinear convergence due to a problem in the Jacobian the reason for poor scaling?
Do the boundary condition depend both on this data and on the nonlinear variables? Are you interpolating a value from the CSV data using the boundary value of a variable? |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your suggestions. I would like to provide more details based on my latest tests. Correction on input files: In the moose_input.i I uploaded, line 52 should be corrected to input = "neml2_input.i". I apologize for the error. I have two functions that read from CSV files: temperature_load and strain_load, both using PiecewiseLinear. They are used in the T_function and xdisp blocks, respectively. These are used to specify the domain temperature (for calculating thermal expansion and the inelastic constitutive model, both of which are nonlinear and implemented via MOOSE and NEML2 interpolation) and the displacement boundary condition on one face. My simulation time steps (in [Executioner]/[TimeStepper]) exactly match the time points in the temperature and strain CSVs. The analyzejacobian.py (with the mesh set to 1x1x1) reports Jacobian errors. To investigate the performance, I conducted the following tests: ADPiecewiseLinear: I replaced PiecewiseLinear with ADPiecewiseLinear, but the output from the Jacobian check remained the same. Hardcoding Temperature: When I changed the function in the T_function block to a constant 298.15K, the Jacobian error became very small (around 0.1%). When I changed it to 273.15K (the initial temperature in the CSV), the Jacobian check showed "no errors detected". Performance with a "Perfect" Jacobian: Since I am only running the first few steps to study calculation efficiency, and the temperature in these steps is almost 273.15K, I proceeded by hardcoding T_function to 273.15K. At this point, the Jacobian check showed no errors, but the execution time remained virtually unchanged (for the 40x8x8 mesh instance). Removing Thermal Expansion: I then completely removed the thermal expansion calculation block [./thermal_expansion_strain1], the eigenstrain_names = thermal_expansion line, and the [Functions]/[temperature_load] block. At this point, the temperature was constant at 273.15K and only served as a reference for the temperature-dependent material properties. I found that the execution time still did not change. Removing CSV Displacement: Finally, I removed the [Functions]/[strain_load] block and replaced the boundary condition with an analytic function: [BCs]/[xdisp]/function = 0.025*t (since the first few lines of my straincsv.csv can be represented by this expression). The calculation time still showed no improvement. Conclusion: In summary, I believe the issue of long calculation time is not related to the piecewise linear functions, CSV reading, thermal expansion calculations, or the Jacobian matrix. It appears the bottleneck lies elsewhere. I would appreciate any further insights into what might be causing this performance issue. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for investigating this.
For the first issue, |
Beta Was this translation helpful? Give feedback.
-
|
What solver/preconditioner are you using in COMSOL? It appears you are using LU in MOOSE which does not scale linearly with degrees of freedom |
Beta Was this translation helpful? Give feedback.
-
|
It might be because my custom model zjzmodelpow.cxx/.h didn't define the second derivative. I'll add it later. |
Beta Was this translation helpful? Give feedback.
-
|
What do you mean by directly? I ran the solid mechanics executable with your |
Beta Was this translation helpful? Give feedback.
-
|
I thought you hit the error during some kind of debugging, with some modification on my input file. |
Beta Was this translation helpful? Give feedback.
-
|
I have revised zjzmodelpow.cxx, and updated it in https://github.com/zhangjunzheng136/moose-neml2. I tested it and it should be able to calculate the 2nd derivative. Maybe you can try recompiling neml2-custom-lib-template with the new zjzmodelpow.cxx. The .tar file I packed is 27GB, and it's heavier than I thought. It would be pretty hard to find specificly on which package the executable is relying, and remove the others. Maybe there're some difference between our enviornments? Not sure if the following information could help. My env: NEML2: |
Beta Was this translation helpful? Give feedback.
-
|
Works now. Here is the call graph shown by gperftools. The kernel cost comes from
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello developers,
I am using MOOSE + NEML2 to improve the simulation efficiency of inelastic constitutive models. However, I have observed that as the degrees of freedom (DOF) increase, the computation time in MOOSE increases much more rapidly compared to my previous experience with COMSOL.
All my MOOSE+NEML2 input files:
https://github.com/zhangjunzheng136/moose-neml2/tree/main
I believe there's something need to be done in my MOOSE+NEML2 input files, but failed. The input files gave correct result, so there's only problem of effeciency, not correctness.
It would be nice of you for reading and analysing the problem.
time (seconds)
Context and Comparison
My previous implementation in COMSOL was quite cumbersome. Therefore, I expected the NEML2 implementation (which is with many constitutive models) to be significantly more efficient. However, this performance advantage disappears when the mesh density increases.
Problem Description
Geometry: A rectangular block.
Physics: Inelastic constitutive model implemented in NEML2. Thermal expansion calculated in MOOSE.
Boundary Conditions: Uniaxial tension. Displacements on three faces are constrained to prevent rigid body motion; a displacement BC is applied to the loading face.
Mesh: Same mesh used in both MOOSE and COMSOL (both 2nd-order Lagrangian elements).
Hardware: CPU, 4 cores.
Issue with Input Data and Jacobian
I initially performed a Jacobian check and found large errors. I investigated and found this was caused by using linear interpolation on CSV data for the boundary conditions (causing non-smooth derivatives).
When I replaced the BC function with a smooth function (e.g., time$t$ ), the Jacobian check passed (no error), and the simulation results matched literature values. This confirms the physics implementation is correct.
The Dilemma: I plan to use extensive time-series data from experiments as boundary conditions, so reading from CSV files is unavoidable. I am concerned that using Spline interpolation might be inefficient or unstable for very large datasets.
MOOSE logs:
COMSOL log:
Beta Was this translation helpful? Give feedback.
All reactions