diff --git a/doc/bmc/BMC-SONiC-support.md b/doc/bmc/BMC-SONiC-support.md new file mode 100644 index 00000000000..07890796ca8 --- /dev/null +++ b/doc/bmc/BMC-SONiC-support.md @@ -0,0 +1,137 @@ +# BMC Support in SONiC + +## Table of Content +## Table of Content +- [BMC Support in SONiC](#bmc-support-in-sonic) + - [Table of Content](#table-of-content) + - [1. Revision](#1-revision) + - [2. Scope](#2-scope) + - [3. Definitions/Abbreviations](#3-definitionsabbreviations) + - [4. Overview](#4-overview) + - [5. High-Level Requirements](#5-high-level-requirements) + - [5.1. Functional Requirements](#51-functional-requirements) + - [5.1.1 Baseboard Management Controller (BMC)](#511-baseboard-management-controller-bmc) + - [5.1.2 BMC Hardware](#512-bmc-hardware) + - [5.1.3 Operating System (OS)](#513-operating-system-os) + - [5.1.4 Out-of-Band Management (OOB Management)](#514-out-of-band-management-oob-management) + - [5.1.5 Serial Console](#515-serial-console) + - [5.1.6 OBMC Console](#516-obmc-console) + - [5.1.7 OBMC Web](#517-obmc-web) + - [5.2 BMC Components Deployment](#52-bmc-components-deployment) + - [6. Boot Loader Components](#6-boot-loader-components) + - [7. eMMC Writes](#7-emmc-writes) + - [8. Restrictions/Limitations](#8-restrictionslimitations) + + +### 1. Revision + +| Rev | Date | Author | Change Description | +| :---- | :---- | :---- | :---- | +| 0.1 | 2025-07-16 | ctikku-nexthop | Initial Draft | +| 0.2 | 2025-08-28 | ctikku-nexthop | Revised Draft | +| 0.3 | 2025-09-25 | chander-nexthop | Revised Draft | +| 0.4 | 2025-11-11 | chinmoy-nexthop | Revised Draft | +| 0.5 | 2026-02-02 | chander-nexthop | Revised Draft | + +### 2. Scope +This document describes the addition of BMC support in SONiC and the execution of SONiC on a BMC controller for out-of-band management of network devices. This design supports Aspeed AST2720 platform, and can be extended to other BMC chipsets. + +### 3. Definitions/Abbreviations + +| Term | Definition | +| :---- | :---- | +| BMC | Baseboard Management Controller | +| NOS | Network Operating System | +| PSU | Power Supply Unit | +| CLI | Command Line Interface | +| SEL | System Event Log | + +### 4. Overview + +The purpose of this HLD is to enable a SONiC image with BMC functionality to run on a BMC controller within a network switch, providing out-of-band monitoring and management of the device. This enhancement improves reliability, manageability, and automation by enabling interaction with the switch through the BMC controller. +Initial capabilities include power control of the main CPU and redirection of its serial console over the network. + +Future enhancements may include advanced hardware health monitoring, telemetry data collection, proactive fault management, and BMC-driven operations for provisioning and diagnostics. + +### 5. High-Level Requirements + +#### 5.1. Functional Requirements + +##### 5.1.1 Baseboard Management Controller (BMC) +- The BMC will be a dedicated hardware component, independent of the main switch CPU. +- It will operate on its own OS, separate from the main switch CPU’s OS. +- It will include a dedicated management network port for out-of-band management. +- It will have an independent serial port, separate from the main switch system. +- The BMC uses eMMC storage; therefore, write operations must be minimized. +- The BMC will provide staged boot control, manage main system power on/off, and handle leak detection and management. +- The BMC will support a network device over USB to connect to the x86 CPU for a secure private network between the x86 and the BMC. + +##### 5.1.2 BMC Hardware +- The BMC controller will include its own CPU, memory, storage, and network port. +- SOC: Aspeed 2720 (arm64), Memory: 4GB, Storage: 32GB eMMC. +- The main switch CPU can operate independently of the BMC controller. +- The BMC controller can also operate independently of the main switch CPU. + +##### 5.1.3 Operating System (OS) +- The operating system running on the BMC will be referred to as **BMC-SONiC-OS**. +- **BMC-SONiC-OS** will be built from the SONiC-OS [sonic-buildimage]( https://github.com/sonic-net/sonic-buildimage.git) codebase. +- All BMC-related enhancements will be committed to the same repository. +- The image will use SONiC build tools and workflows but with BMC-specific build flags. +- The SONiC-OS image build process and functionality will remain unchanged. +- **BMC-SONiC-OS** will include BMC-specific services packaged as Docker containers. +- Switching-related components and associated containers will be excluded from the BMC image. +- A multi-port ethernet switch may connect the BMC and main CPU. The switch will be managed by the BMC, for example, to configure VLANs for traffic separation. +- Support will be added to enable a USB-based network interface for connectivity between the x86 and the BMC. + +##### 5.1.4 Out-of-Band Management (OOB Management) +- **BMC-SONiC-OS** will provide a dedicated management network port and IP address. +- It will support standard network access protocols (e.g., SSH). +- It will allow management of the main switch CPU (e.g., reboot or power cycle) from the BMC controller. +- Rebooting or power cycling the main switch CPU will not impact the BMC. +- Rebooting or power cycling the BMC will not affect the main switch CPU. + +##### 5.1.5 Serial Console +- The BMC will have its own UART-based serial port, separate from the main switch. +- The front-panel serial port can be dynamically switched between the BMC and main CPU. +- Predefined hotkeys will be used to toggle between the two UART interfaces. + +##### 5.1.6 OBMC Console +- **BMC-SONiC-OS** will use [obmc-console](https://github.com/openbmc/obmc-console) for console management. +- A clone of this project will be created and added as a submodule in the sonic-buildimage repository. +- The switch CPU console will be accessible by invoking the obmc-console-client inside the obmc-console docker. +- The switch CPU console will also be accessible over the network via ssh to a specific port on the BMC. +- It will capture and log the main switch CPU’s serial console output for later review. +- Log rotation, compression and export to an external server will be supported + +##### 5.1.7 OBMC Web +- **BMC-SONiC-OS** will use [OpenBMC Web (bmcweb)](https://github.com/openbmc/bmcweb) as the RESTful (Redfish) API server. +- A clone of this project will be created and added as a submodule in the sonic-buildimage repository. +- A new module SONiC-Dbus-Bridge will be created to populate necessary state from REDIS and other data sources into DBus. This way bmcweb can continue to read dbus paths. +- **bmcweb** will provide a web-based interface to the switch CPU console. +- The **staged boot process** will be managed through the Redfish API. +- **Redfish API** for Power On/Off control of the main system. +- **Redfish API** for leak detection and management. + - The leak detection mechanism within the switch. + +#### 5.2 BMC Components Deployment +- **BMC-SONiC-OS** will deploy BMC functionalities as Docker containers. +- **Kubernetes** will be used for container orchestration. + +### 6. Boot Loader Components +- **BMC-SONiC-OS** will use **U-Boot** as the boot loader. +- The image will be installed on an **eMMC partition**. +- The **OpenBMC image** will continue to serve as the **golden firmware** in the SPI boot flash +- **Initial Load Process**: + - Boot the SONiC kernel with a minimal initramfs from U-Boot to launch a shell, and use the included script to download the image via TFTP and flash it to the eMMC. + - If SPI flash is running OpenBMC, boot into that, download a pre-formatted disk image and flash that to eMMC +- **Subsequent Boots**: + - Once the image is installed, the system can boot directly from the eMMC, eliminating the need to download or reinstall the image over TFTP. + - The image will support the sonic-installer utility for image management, including adding a new image, removing an existing image, and selecting the default image for boot. + - There will be only one partition in the eMMC and multiple images can be installed in that partition, conforming to the standard SONiC behaviour (Ex: /host/image-1, /host/image-2, etc.). +- The boot process may be refined in future revisions to improve efficiency and usability. + +### 7. eMMC Writes + - eMMC writes are minimized to protect the device from wear. + +### 8. Restrictions/Limitations +- This functionality is under development. The final implementation details and affected areas may differ from this design. diff --git a/doc/bmc/images/multi_dtb_design.png b/doc/bmc/images/multi_dtb_design.png new file mode 100644 index 00000000000..1699daf001c Binary files /dev/null and b/doc/bmc/images/multi_dtb_design.png differ diff --git a/doc/bmc/multi_dtb_design.md b/doc/bmc/multi_dtb_design.md new file mode 100644 index 00000000000..b8f3a670808 --- /dev/null +++ b/doc/bmc/multi_dtb_design.md @@ -0,0 +1,254 @@ +# Multi-DTB Design for SONiC BMC Support + +## Table of Contents + +### 1. Revision + +| Rev | Date | Author | Change Description | +| --- | ---- | ------ | ------------------ | +| 0.1 | 2026-02-25 | chander-nexthop | Initial draft | + +### 2. Scope + +This document describes the scope of multi-DTB (Device Tree Binary) support for SONiC BMC platform. It enables a single SONiC image to support multiple vendor BMC implementations based on AspeedTech AST27xx SoC, eliminating the need for per-vendor images. + +### 3. Definitions/Abbreviations + +| Term | Definition | +| ---- | ---------- | +| BMC | Baseboard Management Controller | +| DTB | Device Tree Binary | +| DTS | Device Tree Source | +| FIT | Flattened Image Tree | +| SoC | System on Chip | + +### 4. Overview + +Many vendors that make switches running SONiC are looking to add a Baseboard Management Controller (BMC) based on AspeedTech AST27xx SoC to their system designs. These BMC SoCs will run SONiC. It is desirable to have a single SONiC image for this purpose instead of having per-vendor images. This document proposes a design to achieve the same. + +#### Background Information + +Support for the necessary kernel drivers for the AST27xx SOC has already been added to the SONiC Linux Kernel Repository via https://github.com/sonic-net/sonic-linux-kernel/pull/522. + +When compiled with the necessary kernel options, the SONiC arm64 kernel will have all the requisite device drivers needed for this SoC. + +Support for building a SONiC image, `sonic-aspeed-arm64.bin` with `PLATFORM=aspeed` and `PLATFORM_ARCH=arm64` is being done via PR https://github.com/sonic-net/sonic-buildimage/pull/24898. + +In this, we will add the common makefiles, exclude docker containers and debian packages not applicable to the BMC platform, add new docker containers for console management and Redfish APIs, add aspeed specific systemd services like bring up usb network device etc. It will also have the infrastructure outlined here for all vendors to use this same image for their SoC's. + +### 5. Requirements + +- SONiC image should boot with the DTB compiled from the DTS file a vendor has created. +- Vendor specific configuration changes (for example, which console tty of the SoC connects to their control plane's console) must be supported. +- Vendors should be able to add their own software like kernel modules, utility scripts etc to the image and that software must take effect when the image is booted on their card. +- U-Boot should use the correct DTB to load during SONiC boot. +- SONiC should properly set up `/host/machine.conf`, so PLATFORM is inferred correctly. +- Vendors should be able to package their own software like kernel modules, utility scripts, configuration files etc to the image. +- Changes to bring up SONiC on a new SOC should be minimal. + +The proposed design outlined below attempts to address these requirements with a caveat. It is expected that vendors will submit their DTS files as patches, using the standard `src/sonic-linux-kernel` patch workflow, so that those DTS files will be compiled into device tree binaries (DTB's) as part of the kernel build. This is to avoid any extraneous dependency for the sonic-linux-kernel build. + +### 6. Architecture Design + +#### 6.1. Design Overview + +The design hinges on the fact that a single FIT image can contain multiple DTB's and the boot loader, U-Boot, in this case, can pick a specific DTB to load. U-Boot can use any mechanism available to arrive at the DTB to load (ex. Builtin during compilation or reading some EEPROM region or some device registers). It is beyond the scope of this document to precisely specify the mechanism U-Boot would use. + +However it is assumed that U-Boot will store the DTB file name to load into an environment variable called `bootconf`. This way, SONiC installer can use that variable for in place substitution when setting up the U-Boot environment during install of a SONiC image. + +#### 6.2. Boot Flow + +For example, SONiC installer, when setting up an image to boot, would use bootconf without specifying what it is. U-Boot is expected to define it to the right DTB: + +```bash +fw_printenv sonic_image_1 +sonic_image_1=run sonic_bootargs; run sonic_boot_load; bootm $loadaddr#conf-$bootconf +``` + +This way when during boot the appropriate DTB too is loaded. The kernel exposes the information from that DTB in `/proc/device-tree`. + +#### 6.3. Platform Detection and Configuration + +After SONiC boots up, the file `/host/machine.conf` is used to arrive at the PLATFORM and SONiC would set up `/usr/share/sonic/platform` to point into the vendor+platform directory under `/device`. So, it becomes imperative that `/host/machine.conf` will have to be set up properly for the SONiC's workflow. + +To do this, this design proposes to use a systemd service that will run once upon first boot. This systemd service will look into `/proc/device-tree/compatible` and will map the string therein to the platform and set up the `/host/machine.conf`. + +The design assumes a single DTB per vendor card. Revisions, if any, need to have a different DTB name and U-Boot must be able to differentiate between the revisions to set up bootconf. However this design doesn't preclude the possibility of multiple DTB names pointing to the same PLATFORM name. + +#### 6.4. Vendor-Specific Software Support + +To support vendor specific software like kernel modules, scripts etc, we would use the existing lazy install feature of SONIC that lets platform specific packages to be packaged into the image but installed at run time during first boot. Therefore there would be a per vendor `sonic-platform-module-` directory under `platform/aspeed`, where a vendor can define debian packages per card they create. All these .debs would be packaged into the image but installed at runtime using the existing lazy install mechanism supported by the rc.local service. + +#### 6.5. Directory Layout + +``` +platform/aspeed/ +├── one-image.mk +├── onie-image-arm64.conf +├── platform-modules-ast-evb.mk # Makefile for EvalBrd .deb +├── platform-modules-nexthop.mk # Makefile for NextHop .deb +├── platform_arm64.conf +├── rules.mk +├── scripts +│ ├── sonic-machine-conf-init.sh +│ ├── sonic-platform-init.sh +│ ├── sonic-uboot-env-init.sh +│ └── sonic-usb-network-init.sh +├── sonic-platform-modules-ast-evb # Vendor specific Dir +│ ├── ast2700 # Vendor card sub-dir +│ │ ├── build +│ │ │ ├── bdist.linux-aarch64 +│ │ │ └── lib +│ │ │ └── sonic_platform +│ │ │ ├── __init__.py +│ │ │ ├── chassis.py +│ │ │ └── platform.py +│ │ ├── obmc-console +│ │ │ └── server.tty.conf +│ │ ├── scripts +│ │ ├── setup.py +│ │ ├── sonic_platform +│ │ │ ├── __init__.py +│ │ │ ├── chassis.py +│ │ │ ├── fan.py +│ │ │ ├── fan_drawer.py +│ │ │ ├── platform.py +│ │ │ ├── thermal.py +│ │ │ └── watchdog.py +│ │ └── systemd +│ ├── debian +│ │ ├── changelog +│ │ ├── control +│ │ ├── rules +│ │ └── sonic-platform-ast-evb-ast2700.postinst +│ └── setup.py +├── sonic-platform-modules-nexthop # Vendor Nexthop +│ ├── b27 # Nexthop's BMC card +│ │ ├── build +│ │ │ ├── bdist.linux-aarch64 +│ │ │ └── lib +│ │ │ └── sonic_platform +│ │ │ ├── __init__.py +│ │ │ ├── chassis.py +│ │ │ └── platform.py +│ │ ├── obmc-console +│ │ │ └── server.tty.conf +│ │ ├── scripts +│ │ │ └── switch_cpu_utils.sh +│ │ ├── setup.py +│ │ └── sonic_platform +│ │ ├── __init__.py +│ │ ├── chassis.py +│ │ ├── fan.py +│ │ ├── fan_drawer.py +│ │ ├── platform.py +│ │ ├── thermal.py +│ │ └── watchdog.py +│ ├── debian +│ │ ├── changelog +│ │ ├── control +│ │ ├── rules +│ │ └── sonic-platform-aspeed-nexthop-b27.postinst +│ └── setup.py +├── sonic_fit.its +└── systemd + ├── sonic-machine-conf-init.service + ├── sonic-platform-init.service + ├── sonic-uboot-env-init.service + └── sonic-usb-network-init.service + +device/aspeed/arm64-aspeed_ast2700_evb-r0/ +├── asic.conf +├── default_sku +├── installer.conf +├── platform.json +├── platform_asic +├── platform_components.json +├── platform_env.conf +├── pmon_daemon_control.json +└── system_health_monitoring_config.json + +device/nexthop/arm64-nexthop_b27-r0/ +├── asic.conf +├── default_sku +├── installer.conf +├── platform.json +├── platform_asic +├── platform_components.json +├── platform_env.conf +├── pmon_daemon_control.json +└── system_health_monitoring_config.json +``` + +#### 6.7. Build & Boot Flow Diagram + +![Build & Boot Flow Diagram](images/multi_dtb_design.png) + +### 7. Vendor Work Flow For New SoCs + +#### 7.1. Create DTS File +- Create `.dts` file for their card and submit to `src/sonic-linux-kernel` as a patch. This approach is preferred over say having the DTS file in a vendor specific directory under `platform/aspeed/vendor/` to avoid a dependency on such files to build the kernel. + +#### 7.2. Update U-Boot +- Make changes to U-Boot to detect the card type, arrive at DTB file to use and set it in the U-Boot var `bootconf` + +#### 7.3. Create Device Directory +- Create a directory under `device/` ex. `device/aspeed/arm64-aspeed_ast2700_evb-r0/` and put `platform_env.conf`, `default_sku` etc as files inside it. `device/aspeed/arm64-aspeed_ast2700_evb-r0/` defines the same for the evaluation board and can be used as reference. + +#### 7.4. Create Platform Modules Directory +Add a directory under `platform/aspeed/sonic-platform-modules-/` (ex. `platform/aspeed/sonic-platform-modules-ast-evb`) and, for each card type they have, add a subdirectory with the card name and under that the following directories and files: + +##### 7.4.1. sonic_platform/ +- `{__init__.py, platform.py, chassis.py etc}` +- These need inherit from their respective classes defined by `sonic_platform_base` + +##### 7.4.2. scripts/ +- Any scripts vendors want to install go here + +##### 7.4.3. obmc-console/server.tty.conf +- Specify the BMC tty that maps to the switch cpu's console tty. + +##### 7.4.4. setup.py +- Defines setup with vendor specific attributes +- Can use `platform/aspeed/sonic-platform-modules-ast-evb/ast2700/` as reference + +##### 7.4.5. Create (first time) OR Update platform-modules-ast-evb.mk + +#### 7.5. Edits to Common Files + +##### 7.5.1. Edit sonic_fit.its + +Add details about their DTB in `platform/aspeed/sonic_fit.its`. Use `ast2700-evb` in this file as reference. +- Under `images` section +- Under `configurations` section + +##### 7.5.2. Edit sonic-machine-conf-init.sh + +Edit `platform/aspeed/sonic-machine-conf-init.sh` and add a condition to the 'if block' at line 37, something like below. Please note that the PLATFORM string MUST match the directory name created for the vendor under `device/aspeed`: + +```bash +if echo "$COMPATIBLE" | grep -q "vendorA-evb"; then + PLATFORM="arm64-aspeed_ast2700_vendorA" + MACHINE="aspeed_ast2700" + log "Detected Aspeed AST2700 EVB platform" +fi +``` + +#### 7.6. Create or Update Makefile + +Create or Update Makefile `platform/aspeed/platform-modules-.mk`: + +```makefile +# NextHop Platform modules +# +# NOTE: When adding more platforms (e.g., b28), use add_extra_package: +# ASPEED_NEXTHOP_B28_PLATFORM_MODULE = sonic-platform-aspeed-nexthop-b28_1.0_arm64.deb +# $(ASPEED_NEXTHOP_B28_PLATFORM_MODULE)_PLATFORM = arm64-nexthop_b28-r0 +# $(eval $(call add_extra_package,$(ASPEED_NEXTHOP_B27_PLATFORM_MODULE),$(ASPEED_NEXTHOP_B28_PLATFORM_MODULE))) + +ASPEED_NEXTHOP_B27_PLATFORM_MODULE = sonic-platform-aspeed-nexthop-b27_1.0_arm64.deb +$(ASPEED_NEXTHOP_B27_PLATFORM_MODULE)_SRC_PATH = $(PLATFORM_PATH)/sonic-platform-modules-nexthop +$(ASPEED_NEXTHOP_B27_PLATFORM_MODULE)_PLATFORM = arm64-nexthop_b27-r0 +SONIC_DPKG_DEBS += $(ASPEED_NEXTHOP_B27_PLATFORM_MODULE) +``` +