Anytopo by ziyuezzy · Pull Request #2598 · sstsimulator/sst-elements

ziyuezzy · 2025-11-04T12:40:44Z

Hi Devs,

This is a pull request, previously mentioned in issue #2584

There are documentations of this work in
"sst-elements-src/src/sst/elements/merlin/interfaces/endpointNIC/documentations/README.md", and
"sst-elements-src/src/sst/elements/merlin/topology/anytopo_ultility/README.md"

In a few words, this branch extends the Merlin module in SST-element, such that any network topology can be built with an input networkx graph. Another key feature is to support source routing (the endpoint NIC determines the routed paths). And a few popular HPC network topologies (dragonfly, slimfly, polarfly, jellyfish) are already defined in
"sst-elements-src/src/sst/elements/merlin/topology/anytopo_ultility/" dir, together with some tests in "sst-elements-src/src/sst/elements/merlin/topology/anytopo_ultility/tests".

If you would like to accept this pull requests, maybe we can move these tests to the main merlin test dir?

It worth mention that the reorderedlinkcontrol has been modified to fit in the new 'ExtendedRequest' framework, this has been included in tests as well, see EndpointNIC(use_reorderLinkControl=True ... in the tests.

I believe that this pull request will have the following contribution to Merlin:

supporting more network topologies (any topology from input graph)
supporting source routing, with is increasingly interesting in HPC network traffic engineering
Adds a framework of EndpointNIC that allows different NIC/smartNIC functionalities to be plugged in the packet-processing pipeline. For now source routing is implemented through this framework.

Please let me know if there is any feedback/comments/questions for the code.

Best regards,
Z.

sst-autotester · 2025-11-04T12:47:43Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

feldergast · 2025-11-04T15:11:08Z

In order to merge the PR, the tests will need to be part of the main merlin testing. Go ahead and move them there and we can review the code and release the PR for testing.

ziyuezzy · 2025-11-05T12:01:24Z

Before I add new tests, I want to make sure the current tests all pass.
I don't really understand why the 'macos-14 / macos-15' are failing here?

When I run 'sst-test-elements' locally (ubuntu 22, python3.8), all tests pass (see screenshot below).

For example the test "test_EmberSweep_*" all passed locally, but failed in the github action...
Does anyone have an idea what is wrong there?

feldergast · 2025-11-05T15:04:32Z

There are some known issues with the GitHub macOS runners and how they interact with SST. You can safely ignore the results of the GitHub macOS builders for now. They are experimental and the official testing is done on other machines.

feldergast · 2025-11-05T15:41:44Z

Looks like that those issues were actually fixed a week or so ago. I dug a little deeper and it looks like you introduced a dependance on a third-party library that isn't installed on those systems (NetworkX). You'll need to do a check for that and not load it if it's not there (i.e. anytopo won't be available if NetworkX isn't there. The polarfly and polarstar topos have a similar issue with a different library. Give me a bit to look through the code and I can send some pointers on the best way to do this.

feldergast · 2025-11-05T15:53:23Z

+from collections import defaultdict
+from typing import Callable, Optional
+
+try:


This looks like the code segment that is failing when networks isn't installed. It looks like this code is executing when the module is loaded, so the ImportError below ends up failing in the module loading. So, this causes every simulation that imports sst.merlin to fail, not just those that use topoany. Probably need to set a variable if the modules are missing and fail when trying to use the topology instead of at module load time.

feldergast · 2025-11-05T16:24:28Z

Looks like polarstar and polarfly depend on the same third-party Python libraries as anytopo, so we already have builders set up with those libraries, so anytopo should test on those and be skipped on all the ones without those libraries.

sst-autotester · 2025-11-05T16:47:44Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

ziyuezzy · 2025-11-05T16:53:05Z

Looks like polarstar and polarfly depend on the same third-party Python libraries as anytopo, so we already have builders set up with those libraries, so anytopo should test on those and be skipped on all the ones without those libraries.

Ok that's good news. Thanks for checking.
Will the github action automatically skip or continue with the tests, depending on whether or not networkx is installed? Or Should I add some indications in the test files?

Another question is that some of my tests require extra python libraries, for example I imported the library "galois" for generating the 'Slimfly; topology. This import is not built with SST, but only is required when running the test.
Is there any solution there?

feldergast · 2025-11-05T17:15:08Z

You'll need to add an @unittest.skipIf similar to what the polar* topology tests do so that they don't try to execute if the libraries aren't found. Note that the testsuite_Default_merlin.py file also tries to load the required libraries so that they can check if they are in sys.modules.

sst-autotester · 2025-11-06T08:47:42Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

ziyuezzy · 2025-11-06T10:03:38Z

I have added new tests for Anytopo, and all tests passed. So I think this PR is ready for review.

feldergast · 2025-11-25T15:33:39Z

Sorry, with the SC conference last week, Thanksgiving this week and vacation next week, it will be a bit before I can dive deep into this PR. I took a quick glance through it and didn't see anything that immediately stood out as an issue. I'll revisit this the week of December 8th and do a complete review.

sst-autotester · 2026-01-05T11:47:45Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

feldergast · 2026-01-05T17:36:09Z

Sorry for the delay, things got a bit hectic during December. I will be looking at this PR this week, though it may take me a bit to get through since I'll have to remind myself of the details in the merlin code.

feldergast · 2026-01-05T19:05:08Z

+class ExtendedRequest : public SST::Interfaces::SimpleNetwork::Request {
+private:
+    // Map from plugin name/key to metadata
+    std::map<std::string, std::shared_ptr<PluginMetadata>> metadata;


Using shard_ptr in classes that end up in events is not supported since std::shared_ptr is not serializable for events used in synchronization. This is because pointers are not tracked during event serialization, so nothing will end up being shared outside of the rank it was created on. I know this will cause some memory increase, but, in general, elements in the merlin library are assumed to be able to run across multiple ranks. I'm going to continue to review the code, but I think this will need to be changed before we can merge it.

feldergast · 2026-01-05T19:38:12Z

+{
+public:
+    // Static shared data accessible by both SourceRoutingPlugin and topology classes
+    static std::map<int, int> endpoint_to_router_map;  // Shared endpoint-to-router mapping


Generally, static variables shouldn't be used in SubComponents. They should use the SharedObjects provided by the core. Since both of the classes that need access are derived from SubComponent, they will have access to those APIs.

Ok, a minor question:
these SharedObjects support array of common data types (int, bool, ...) right?
Can we also share arbitrary objects like mutex of iostream?

The SharedObjects pretty much support any Object that is serializable, which means that a mutex would not be supported. If you need to have a shared Mutex, then that will still need to be a static variable. I/O generally doesn't fall under the same restrictions as variables used during the simulation.

ziyuezzy · 2026-01-06T16:31:09Z

Hi, no worries, we are not in any hurry.
And thanks for giving feedbacks on the code, I will try to fix the points you raised.

Recently I have been thinking about the design of this branch.
Of course importing network topology from graph, and making the routers do source-routing, is probably necessary to be implemented in Merlin.
But this whole concept of 'EndpointNIC with plugins', feels suitable to be an individual sst-element, similar to the 'rdmaNic' element?
Implementing these in Merlin, may benefit its extensibility. For example, I recently implemented another small plugin (not in this branch) that dumps traffic traces from Merlin, it was really clean and efficient. I wonder whether it is even feasible to implement circuit-switching within this framework....
But on the other hand, with a lot of plugins, this framework might also make Merlin complicated/confusing to read for people just started to use this simulator... I don't know if that is a concern or not.

sst-autotester · 2026-01-07T10:17:48Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

sst-autotester · 2026-01-16T11:17:49Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

sst-autotester · 2026-01-16T18:17:45Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

feldergast · 2026-01-20T18:33:33Z

+    void serialize_order(SST::Core::Serialization::serializer &ser) override {
+        Request::serialize_order(ser);
+
+        // Manually serialize the metadata map since std::variant isn't directly supported


std::variant is supported by the serializer now, but it was developed primarily for checkpointing and may not work properly with pointer tracking turned off. I'd have to look through that code to know if pointer tracking matters. It may be worth trying to see if you can just serialize the vector directly.

Co-authored-by: Copilot <copilot@github.com>

…ing.h/cc" -> "sourceRouting.h/cc"

sst-autotester · 2026-05-26T07:47:22Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

ziyuezzy · 2026-05-26T12:58:36Z

I have updated the branch, but the CI action is not triggered.. I don't know why.

ziyuezzy · 2026-05-26T14:09:16Z

I have built and ran tests locally, all passed

sst-autotester · 2026-05-26T15:47:35Z

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; this inspection will remain valid until a new commit to source branch is performed.

sst-autotester · 2026-05-26T15:50:30Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements

Build Num: 2776
Status: STARTED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist

Build Num: 1552
Status: STARTED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2

Build Num: 2717
Status: STARTED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2

Build Num: 2719
Status: STARTED

Build Information

Test Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements

Build Num: 1253
Status: STARTED

Using Repos:

Repo: ELEMENTS (ziyuezzy/sst-elements)

Repo: SQE (sstsimulator/sst-sqe)

Repo: CORE (sstsimulator/sst-core)

Repo: MACRO (sstsimulator/sst-macro)

Pull Request Author: ziyuezzy

sst-autotester · 2026-05-26T16:33:45Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 8 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements

Result: PASSED
Build #: 2776
URL: Jenkins server at https://sst-jenkins.sandia.gov/view/SST/job/SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements/2776/consoleFull

Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist

Result: FAILED
Build #: 1552
URL: Jenkins server at https://sst-jenkins.sandia.gov/view/SST/job/SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist/1552/consoleFull
Job: - Status: FAILURE

Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2

Result: PASSED
Build #: 2717
URL: Jenkins server at https://sst-jenkins.sandia.gov/view/SST/job/SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2/2717/consoleFull

Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2

Result: PASSED
Build #: 2719
URL: Jenkins server at https://sst-jenkins.sandia.gov/view/SST/job/SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2/2719/consoleFull

Job: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements

Result: PASSED
Build #: 1253
URL: Jenkins server at https://sst-jenkins.sandia.gov/view/SST/job/SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements/1253/consoleFull

feldergast · 2026-05-26T16:48:51Z

Looks like at least one header file (ExtendedRequest.h) is missing from Makefile.am so the make dist test is failing. Please make sure any files that were added are in Makefile.am. If you have any questions about where specific files belong in the file, let me know.

sst-autotester · 2026-05-26T19:13:30Z

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; this inspection will remain valid until a new commit to source branch is performed.

sst-autotester · 2026-05-26T19:16:57Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements

Build Num: 2779
Status: STARTED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist

Build Num: 1554
Status: STARTED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2

Build Num: 2720
Status: STARTED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2

Build Num: 2722
Status: STARTED

Build Information

Test Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements

Build Num: 1256
Status: STARTED

Using Repos:

Repo: ELEMENTS (ziyuezzy/sst-elements)

Repo: SQE (sstsimulator/sst-sqe)

Repo: CORE (sstsimulator/sst-core)

Repo: MACRO (sstsimulator/sst-macro)

Pull Request Author: ziyuezzy

sst-autotester · 2026-05-26T20:00:23Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements

Build Num: 2779
Status: PASSED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist

Build Num: 1554
Status: PASSED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2

Build Num: 2720
Status: PASSED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2

Build Num: 2722
Status: PASSED

Build Information

Test Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements

Build Num: 1256
Status: PASSED

sst-autotester · 2026-05-26T20:00:41Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

sst-autotester · 2026-05-26T20:00:48Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester · 2026-05-27T07:17:25Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

feldergast reviewed Nov 5, 2025

View reviewed changes

ziyuezzy force-pushed the anytopo branch 2 times, most recently from 560ee03 to 18399d4 Compare November 6, 2025 08:22

ziyuezzy requested a review from feldergast November 12, 2025 08:35

ziyuezzy force-pushed the anytopo branch from e79f6d3 to 018fe89 Compare January 5, 2026 11:06

feldergast reviewed Jan 5, 2026

View reviewed changes

Comment thread src/sst/elements/merlin/interfaces/endpointNIC/NICPlugin.h

feldergast reviewed Jan 5, 2026

View reviewed changes

ziyuezzy force-pushed the anytopo branch from 3b28033 to 7bf9a31 Compare January 7, 2026 12:19

ziyuezzy requested a review from feldergast January 14, 2026 10:16

ziyuezzy force-pushed the anytopo branch from 064192c to 3e4fe83 Compare January 16, 2026 17:54

feldergast reviewed Jan 20, 2026

View reviewed changes

ziyuezzy and others added 3 commits May 26, 2026 09:22

adding SST_ELI_DOCUMENT_PARAMS

688c2a3

Co-authored-by: Copilot <copilot@github.com>

Renaming class endpointNIC -> EndpointNIC; renaming files "SourceRout…

70f63e7

…ing.h/cc" -> "sourceRouting.h/cc"

cleaning up; adding new lines at the end of files

868f61d

ziyuezzy force-pushed the anytopo branch 2 times, most recently from 8efb1e9 to cb8dbc2 Compare May 26, 2026 07:29

reducing dynamic casts

1415148

ziyuezzy force-pushed the anytopo branch from cb8dbc2 to 1415148 Compare May 26, 2026 11:59

ziyuezzy requested a review from feldergast May 26, 2026 12:58

berquist added the AT: PRE-TEST INSPECTED label May 26, 2026

sst-autotester removed the AT: PRE-TEST INSPECTED label May 26, 2026

feldergast added AT: WIP Mark PR as a Work in Progress (No Autotesting Performed) AT: PRE-TEST INSPECTED and removed AT: WIP Mark PR as a Work in Progress (No Autotesting Performed) labels May 26, 2026

sst-autotester removed the AT: PRE-TEST INSPECTED label May 26, 2026

cleaning up

976b77d

ziyuezzy force-pushed the anytopo branch 2 times, most recently from cb8dbc2 to 976b77d Compare May 27, 2026 07:07

Uh oh!

Conversation

ziyuezzy commented Nov 4, 2025

Uh oh!

sst-autotester commented Nov 4, 2025

Uh oh!

feldergast commented Nov 4, 2025

Uh oh!

ziyuezzy commented Nov 5, 2025

Uh oh!

feldergast commented Nov 5, 2025

Uh oh!

feldergast commented Nov 5, 2025

Uh oh!

feldergast Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

feldergast commented Nov 5, 2025

Uh oh!

sst-autotester commented Nov 5, 2025

Uh oh!

ziyuezzy commented Nov 5, 2025

Uh oh!

feldergast commented Nov 5, 2025

Uh oh!

sst-autotester commented Nov 6, 2025

Uh oh!

ziyuezzy commented Nov 6, 2025

Uh oh!

feldergast commented Nov 25, 2025

Uh oh!

sst-autotester commented Jan 5, 2026

Uh oh!

feldergast commented Jan 5, 2026

Uh oh!

feldergast Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

feldergast Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

ziyuezzy Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

feldergast Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

ziyuezzy commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sst-autotester commented Jan 7, 2026

Uh oh!

sst-autotester commented Jan 16, 2026

Uh oh!

sst-autotester commented Jan 16, 2026

Uh oh!

feldergast Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

sst-autotester commented May 26, 2026

Uh oh!

ziyuezzy commented May 26, 2026

Uh oh!

ziyuezzy commented May 26, 2026

Uh oh!

sst-autotester commented May 26, 2026

Uh oh!

sst-autotester commented May 26, 2026

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2

Build Information

Test Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements

Using Repos:

ziyuezzy commented Jan 6, 2026 •

edited

Loading