Skip to content

Add developer documentation for managers in NEST#3465

Open
jessica-mitchell wants to merge 36 commits intonest:masterfrom
jessica-mitchell:devdocs-html
Open

Add developer documentation for managers in NEST#3465
jessica-mitchell wants to merge 36 commits intonest:masterfrom
jessica-mitchell:devdocs-html

Conversation

@jessica-mitchell
Copy link
Contributor

@jessica-mitchell jessica-mitchell commented Apr 10, 2025

This PR creates pages dedicated to explaining how managers work in NEST.
The goal is to provide newer developers with an overview.

Currently a general overview is given in developer_space/managers/index.html
And each manager has its own page with a brief description and generated docs from doxygen output (extracting the C++ comments) using the extension breathe. Additional details are provided for MUSICManager and SPManager, after discussion with @alpinangu on what would be useful to include. If the review of these pages is positive than we can include such details in each manager page.

The analysis of code and text descriptions was aided by AI.

@jessica-mitchell jessica-mitchell added T: Enhancement New functionality, model or documentation S: Normal Handle this with default priority I: No breaking change Previously written code will work as before, no one should note anything changing (aside the fix) labels Apr 10, 2025
@github-project-automation github-project-automation bot moved this to In progress in Documentation Apr 10, 2025
Copy link

@alpinangu alpinangu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Jessica, I think the changes you made are really helpful.

I really like the Relationships with Other Managers section. I think it’s super useful for someone who doesn’t have a clear big-picture understanding. This section provides that well from the manager’s point of view. I also like that there are links to the listed managers.

I also really like the detailed operation sequence. The explanations are clear and well-ordered, and I appreciate how it includes the code too—so you can look at the function without needing to switch tabs. Maybe it could also mention which parts of the source code call these functions, if it’s not too hard to trace.

I also really liked how MUSICManager has a section for MPI, since both it and MUSIC rely heavily on it.

The Overview section and the description at the top also provide a nice explanation. It’s helpful for people who just want a quick reminder instead of going through the detailed breakdowns.

It might be helpful to include a super brief description of what the class members are for. Personally, when I start working on a class, I like to understand the members before diving into the functions.

Lastly, I found that some of the Doxygen descriptions could be more detailed.

@github-actions
Copy link

Pull request automatically marked stale!

@github-actions github-actions bot added the stale Automatic marker for inactivity, please have another look here label Jun 14, 2025
@github-actions github-actions bot removed the stale Automatic marker for inactivity, please have another look here label Jul 9, 2025
@github-actions
Copy link

github-actions bot commented Sep 8, 2025

Pull request automatically marked stale!

@github-actions github-actions bot added the stale Automatic marker for inactivity, please have another look here label Sep 8, 2025
@jessica-mitchell
Copy link
Contributor Author

block because of another PR #3558 thats uses GHpages

@jessica-mitchell jessica-mitchell added the P: Blocked Work on this can not continue, see comments for the particular reason label Sep 15, 2025
@github-actions github-actions bot removed the stale Automatic marker for inactivity, please have another look here label Sep 17, 2025
@jessica-mitchell jessica-mitchell moved this from In progress to Blocked in Documentation Nov 8, 2025
@github-actions
Copy link

Pull request automatically marked stale!

@github-actions github-actions bot added the stale Automatic marker for inactivity, please have another look here label Nov 17, 2025
Copy link
Contributor

@terhorstd terhorstd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conceptually I think this is a good way to bridge between the user and developer docs.

  • The descriptions of the managers read well, but particularly the last sentence of each sometimes seems to stretch out a bit too far and borders on hallucination of the AI. → We definitely need experts to confirm that's not nonsense.
  • The choice of two long examples is a bit unfortunate, as the MUSIC and SP managers are quite special cases and it may be more difficult to find experts to review them.
    • I think this PR should only add the intro praragraphs, then each extended manager description should be a separate PR to split review processes to enough people.
    • The long descriptions seem very extensive and have the feel of AI generated bloat. Maybe choosing a subset only and putting e.g. structural information on separate pages could reduce the wall-of-text feeling. (See also next point)
  • The "Managers in NEST" page is an important anchor page for understanding the internal workings and aligns well with our concept of having a "Concepts in NEST"-kind-of-page also for deeper diving users. The content seems a bit arbitrarily stretched throughout the sections though, without providing deeper links or relations. The items in "main control flow" section repeat the list of managers, so both could be combined. Also the "Advanced architectural features" could be listed together with the managers they are implemented in.

Comment on lines +8 to +9
status queries, and simulation resets. Additionally, it tracks a fingerprint to detect changes and maintain consistency
across multiple runs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
status queries, and simulation resets. Additionally, it tracks a fingerprint to detect changes and maintain consistency
across multiple runs.
status queries, and simulation resets.

Comment on lines +10 to +13
The class also provides functionality for calibrating models and retrieving their status.
The code is designed to work in a multi-threaded environment, ensuring that
models are correctly managed across different threads. The ModelManager plays a crucial
role in the simulation framework by providing a centralized way to handle various models and their configurations.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The class also provides functionality for calibrating models and retrieving their status.
The code is designed to work in a multi-threaded environment, ensuring that
models are correctly managed across different threads. The ModelManager plays a crucial
role in the simulation framework by providing a centralized way to handle various models and their configurations.
The code is designed to work in a multi-threaded environment, ensuring that
models are correctly managed across different threads.

Comment on lines +9 to +10
handling file I/O for spike data or parameter logging). It ensures devices communicate with the correct backend while
deferring logging responsibilities to the LoggingManager.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
handling file I/O for spike data or parameter logging). It ensures devices communicate with the correct backend while
deferring logging responsibilities to the LoggingManager.
handling file I/O for spike data or parameter logging). It ensures devices communicate with the correct backend.

Comment on lines +7 to +8
efficiency. It tracks node IDs, handles special node types like MUSIC interfaces, and detects network changes to update
states, while storing exceptions from parallel operations for error resolution.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It tracks node IDs, handles special node types like MUSIC interfaces, and detects network changes to update states, while storing exceptions from parallel operations for error resolution.

Is that true?

@github-project-automation github-project-automation bot moved this from Blocked to Review in Documentation Jan 23, 2026
Comment on lines +12 to +82
The KernelManager acts as the core, coordinating the other managers that specialize in different aspects of the
simulation.

:ref:`KernelManager <kernel_manager>` initializes all managers in dependency order:

- :ref:`nest::LoggingManager <logging_manager>`
- :ref:`nest::MPIManager <mpi_manager>`
- :ref:`nest::VPManager <vp_manager>`
- :ref:`nest::ModuleManager <module_manager>`
- :ref:`nest::RandomManager <random_manager>`
- :ref:`nest::SimulationManager <simulation_manager>`
- :ref:`nest::ModelRangeManager <modelrange_manager>`
- :ref:`nest::ConnectionManager <connection_manager>`
- :ref:`nest::SPManager <sp_manager>`
- :ref:`nest::EventDeliveryManager<event_delivery_manager>`
- :ref:`nest::IOManager <io_manager>`
- :ref:`nest::ModelManager <model_manager>`
- :ref:`nest::MUSICManager <music_manager>`
- :ref:`nest::NodeManager <node_manager>`



Main control flow
-----------------

1. **Initialization**:
- The Kernel Manager initializes the simulation environment, allocating resources and setting up initial configurations.
- The Simulation Manager sets up the simulation based on defined parameters.

2. **Node and Connection Setup**:
- The Node Manager creates and configures nodes.
- The Connection Manager establishes connections between nodes.

3. **Model Application**:
- The Model Manager loads and applies models to nodes and connections.
- The SP Manager applies synaptic plasticity rules as needed.

4. **Simulation Execution**:
- The Simulation Manager oversees the execution phase, with the Event Delivery Manager ensuring events are processed correctly.
- The MPI Manager handles parallel processing tasks if the simulation is distributed.

5. **Data Handling**:
- The IO Manager reads configuration files and writes simulation results.
- The Logging Manager captures diagnostic information for debugging.

6. **Cleanup**:
- The Simulation Manager ensures resources are cleaned up after the simulation completes.


Advanced Architectural Features
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- **Policy-based template pattern for manager customization**:
- The policy-based template pattern is used to customize specific manager behaviors, allowing developers to extend functionality without modifying core logic.

- **Thread-local storage for performance-critical components**:
- Thread-local storage is employed for performance-critical components to minimize contention and improve parallel execution efficiency.

- **Double dispatch pattern for event handling**:
- The double dispatch pattern is used for event handling, enabling dynamic and type-safe event processing across different node and connection types.

- **MPI-agnostic interfaces through manager abstraction**:
- MPI-agnostic interfaces are provided through manager abstraction, allowing the simulation to run seamlessly with or without MPI, enhancing flexibility and portability.

- **Multiple synchronization strategies**:

- **MPI_Barrier** is used for global synchronization to ensure all processes reach a certain point before proceeding.
- **OpenMP locks** are employed for shared memory parallelism to manage access to shared resources.
- **Atomic operations** are used for spike counter updates to ensure thread-safe increments without locks.
- **RAII patterns** are used for resource management to ensure proper acquisition and release of resources.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To illustrate the last point in my review:

Suggested change
The KernelManager acts as the core, coordinating the other managers that specialize in different aspects of the
simulation.
:ref:`KernelManager <kernel_manager>` initializes all managers in dependency order:
- :ref:`nest::LoggingManager <logging_manager>`
- :ref:`nest::MPIManager <mpi_manager>`
- :ref:`nest::VPManager <vp_manager>`
- :ref:`nest::ModuleManager <module_manager>`
- :ref:`nest::RandomManager <random_manager>`
- :ref:`nest::SimulationManager <simulation_manager>`
- :ref:`nest::ModelRangeManager <modelrange_manager>`
- :ref:`nest::ConnectionManager <connection_manager>`
- :ref:`nest::SPManager <sp_manager>`
- :ref:`nest::EventDeliveryManager<event_delivery_manager>`
- :ref:`nest::IOManager <io_manager>`
- :ref:`nest::ModelManager <model_manager>`
- :ref:`nest::MUSICManager <music_manager>`
- :ref:`nest::NodeManager <node_manager>`
Main control flow
-----------------
1. **Initialization**:
- The Kernel Manager initializes the simulation environment, allocating resources and setting up initial configurations.
- The Simulation Manager sets up the simulation based on defined parameters.
2. **Node and Connection Setup**:
- The Node Manager creates and configures nodes.
- The Connection Manager establishes connections between nodes.
3. **Model Application**:
- The Model Manager loads and applies models to nodes and connections.
- The SP Manager applies synaptic plasticity rules as needed.
4. **Simulation Execution**:
- The Simulation Manager oversees the execution phase, with the Event Delivery Manager ensuring events are processed correctly.
- The MPI Manager handles parallel processing tasks if the simulation is distributed.
5. **Data Handling**:
- The IO Manager reads configuration files and writes simulation results.
- The Logging Manager captures diagnostic information for debugging.
6. **Cleanup**:
- The Simulation Manager ensures resources are cleaned up after the simulation completes.
Advanced Architectural Features
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- **Policy-based template pattern for manager customization**:
- The policy-based template pattern is used to customize specific manager behaviors, allowing developers to extend functionality without modifying core logic.
- **Thread-local storage for performance-critical components**:
- Thread-local storage is employed for performance-critical components to minimize contention and improve parallel execution efficiency.
- **Double dispatch pattern for event handling**:
- The double dispatch pattern is used for event handling, enabling dynamic and type-safe event processing across different node and connection types.
- **MPI-agnostic interfaces through manager abstraction**:
- MPI-agnostic interfaces are provided through manager abstraction, allowing the simulation to run seamlessly with or without MPI, enhancing flexibility and portability.
- **Multiple synchronization strategies**:
- **MPI_Barrier** is used for global synchronization to ensure all processes reach a certain point before proceeding.
- **OpenMP locks** are employed for shared memory parallelism to manage access to shared resources.
- **Atomic operations** are used for spike counter updates to ensure thread-safe increments without locks.
- **RAII patterns** are used for resource management to ensure proper acquisition and release of resources.
The KernelManager acts as the core, coordinating the other managers that specialize in different aspects of the simulation.
:ref:`KernelManager <kernel_manager>` initializes all managers in dependency order, initializes the simulation environment, allocating resources and sets up initial configurations:
- :ref:`nest::LoggingManager <logging_manager>` – Central mechanisms handling diagnostic information for debugging.
- :ref:`nest::MPIManager <mpi_manager>` – Handles parallel processing tasks if the simulation is distributed.
MPI-agnostic interfaces are provided through manager abstraction, allowing the simulation to run seamlessly with or without MPI, enhancing flexibility and portability.
- :ref:`nest::VPManager <vp_manager>`
- :ref:`nest::ModuleManager <module_manager>`
The **policy-based template** pattern is used to customize specific manager behaviors, allowing developers to extend functionality without modifying core logic.
- :ref:`nest::RandomManager <random_manager>`
- :ref:`nest::SimulationManager <simulation_manager>` – Sets up the simulation based on defined parameters, oversees the execution phase, and ensures resources are cleaned up after the simulation completes.
- Multiple synchronization strategies are used to maximize parallel efficiency
- **MPI_Barrier** is used for global synchronization to ensure all processes reach a certain point before proceeding.
- **OpenMP locks** are employed for shared memory parallelism to manage access to shared resources.
- **Atomic operations** are used for spike counter updates to ensure thread-safe increments without locks.
- :ref:`nest::ModelRangeManager <modelrange_manager>`
- :ref:`nest::ConnectionManager <connection_manager>` – Establishes connections between nodes.
Uses **Thread-local storage for performance-critical components** for performance-critical components to minimize contention and improve parallel execution efficiency.
- :ref:`nest::SPManager <sp_manager>` – Applies synaptic plasticity rules as needed.
- :ref:`nest::EventDeliveryManager<event_delivery_manager>` – Ensures events are processed correctly.
The **double dispatch pattern** is used for event handling, enabling dynamic and type-safe event processing across different node and connection types.
- :ref:`nest::IOManager <io_manager>` – Reads configuration files (?) and writes simulation results.
- :ref:`nest::ModelManager <model_manager>` – Loads and applies models to nodes and connections.
- :ref:`nest::MUSICManager <music_manager>`
- :ref:`nest::NodeManager <node_manager>` – Creates and configures nodes such as neurons and devices.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@terhorstd this PR is blocked due to conversation we've had about changing how we do developer documentation over the last few months - we wanted to get the C++ docs into gh-pages and then look at adding additional docs

@terhorstd
Copy link
Contributor

For the short-descriptions of the managers, I'd like to ask @suku248 to quickly check there is no nonsense. Do you have time?

@terhorstd terhorstd requested a review from suku248 January 23, 2026 12:33
@jessica-mitchell jessica-mitchell moved this from Review to Blocked in Documentation Feb 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

I: No breaking change Previously written code will work as before, no one should note anything changing (aside the fix) P: Blocked Work on this can not continue, see comments for the particular reason S: Normal Handle this with default priority stale Automatic marker for inactivity, please have another look here T: Enhancement New functionality, model or documentation

Projects

Status: Blocked

Development

Successfully merging this pull request may close these issues.

3 participants