Skip to content

Adds FSIO R/W sync script for stormond implementation#18927

Closed
ashwnsri wants to merge 7 commits intosonic-net:masterfrom
ashwnsri:syncserv-syncfiletopmon
Closed

Adds FSIO R/W sync script for stormond implementation#18927
ashwnsri wants to merge 7 commits intosonic-net:masterfrom
ashwnsri:syncserv-syncfiletopmon

Conversation

@ashwnsri
Copy link
Contributor

@ashwnsri ashwnsri commented May 9, 2024

Why I did it

This is part of a larger feature implementation: Storage Monitoring Daemon. It adds a script to sync the FSIO reads and writes from the STATE_DB to the disk, which is called by the various reboot scripts during planned reboot.

Work item tracking
  • Microsoft ADO (number only): 17468992

How I did it

  1. Added the sync script to image_config folder

  2. Made changes in sonic_debian_extension.j2 that copies the above file to the appropriate location in the image.

  3. Made changes that instruct the build_template to copy 'fsio-rw-sync' into the appropriate location in the pmon container, so that the script can be called by the daemon from within pmon.

How to verify it

  1. Flash image with these changes onto a DUT

  2. Write sample values of the appropriate fields to STATE_DB

  3. Erase the contents of the fsio-rw-stats.json file and replace with empty JSON

  4. Reboot the device with this sister sonic-utilities change: Modified reboot scripts to sync FSIO reads/writes to disk before OS-level reboot sonic-utilities#3171

  5. On reboot, verify that the values from Step 2 are in the JSON file.

if the sonic-utilities PR is not yet merged, you can verify the presence of the sync script in /usr/local/bin and call it manually, after which you can verify that the empty JSON file has been populated with the contents of the STATE_DB values we previously posted to the database.

root@str3-s6100-acs-6:~# redis-cli -n 6 HSET "STORAGE_INFO|sda" fs_io_reads 1000 ; redis-cli -n 6 HSET "STORAGE_INFO|sda" fs_io_writes 2000 ; redis-cli -n 6 HSET "STORAGE_INFO|sda" latest_fs_io_reads 10 ; redis-cli -n 6 HSET "STORAGE_INFO|sda" latest_fs_io_writes 20
(integer) 1
(integer) 1
(integer) 1
(integer) 1
root@str3-s6100-acs-6:~# echo "{}" > /host/pmon/stormond/fsio-rw-stats.json
root@str3-s6100-acs-6:~#
root@str3-s6100-acs-6:~#
root@str3-s6100-acs-6:~#
root@str3-s6100-acs-6:~# sudo reboot
requested COLD shutdown
/host: 9.3 GiB (9988685824 bytes) trimmed on /dev/sda4
Thu 09 May 2024 06:50:47 PM UTC Issuing OS-level reboot ...
root@str3-s6100-acs-6:~# client_loop: send disconnect: Broken pipe
assrinivasan@assrinivasan-dev-vm-1:/data/sonic$ ash str3-s6100-acs-6
10.3.146.155
Warning: Permanently added '10.3.146.155' (RSA) to the list of known hosts.
Linux str3-s6100-acs-6 5.10.0-23-2-amd64 #1 SMP Debian 5.10.179-3 (2023-07-27) x86_64
You are on
  ____   ___  _   _ _  ____
 / ___| / _ \| \ | (_)/ ___|
 \___ \| | | |  \| | | |
  ___) | |_| | |\  | | |___
 |____/ \___/|_| \_|_|\____|

-- Software for Open Networking in the Cloud --

.
.
.
admin@str3-s6100-acs-6:~$ cat /host/pmon/stormond/fsio-rw-stats.json
{"sda": {"fs_io_reads": "1000", "fs_io_writes": "2000", "latest_fs_io_reads": "10", "latest_fs_io_writes": "20"}}admin@str3-s6100-acs-6:~$

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106
  • 202111
  • 202205
  • 202211
  • 202305

Tested branch (Please provide the tested image version)

  • 20230531.22

Description for the changelog

Added logic to sync stormond STATE_DB values to disk in planned reboot scenarios

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Ashwin Srinivasan added 7 commits May 9, 2024 04:57
A service 'fsio-sync' that calls a shell script 'fsio-sync.sh' that
parses the latest fsio reads/writes from /proc/diskstats and saves it
to disk using 'fsio-rw-sync' script, as well as updates the StateDB
upon successful parse and sync.

Changes that copy these files to the appropriate locations in the image.

Changes that instruct the build_template to copy 'fsio-rw-sync' into
the appr. location in the pmon container
… boot

Changed fsio-sync.sh so that sleep interval can by dynamically changed per loop
@ashwnsri ashwnsri closed this May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant