SYSTEM STOP SWARM MODE command for graceful shutdown swarm node merge attempt v2

[ianton-ru]

Changelog category (leave one):

• Experimental Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

SYSTEM STOP SWARM MODE command for graceful shutdown swarm node

Documentation entry for user-facing changes

Solved #759
New command SYSTEM STOP SWARM MODE.
Scenario:
We want to scale down swarm cluster. On node which we want to shutdown we call SYSTEM STOP SWARM MODE, after that node stops to accept new distributed commands. It can still processed objects which is started processed before SYSTEM STOP SWARM MODE. When all that objects successfully processed, we can kill that node without any errors or lost data in responses on initiator.

After SYSTEM STOP SWARM MODE on swarm node:

• unregister node from autodiscovery clusters if exists
• stop getting new tasks for objectStorageCluster-family functions (s3Cluster/icebergCluster/etc.)

On initiator node:

• for all distributed requests with setting skip_unavailable_shards=true unexpected closing of socket is legal if no data packets were accepted before. This allow to shutdown non-autodiscovery node too.

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

[github-actions]

Workflow [PR], commit [e74130f]

[Enmk]

test_database_hms/test.py seems to be failing at random