[resolv-config] Improve container resolv.conf update mechanism

[oleksandrivantsiv]

Why I did it

Fix a possible race condition during the config reload caused by the concurrent restart of the Docker containers and the resolv-config service. The update of the DNS configuration inside the container is triggered by the container's post-start action.

Work item tracking

• Microsoft ADO (number only):

How I did it

• Add support for single container updates with container name argument
• Implement parallel updates for bulk operations
• Add comprehensive logging with syslog integration
• Add container state handling (start/wait for stopped containers)
• Add proper error handling and status reporting
• Remove temporary file usage during resolv.conf updates
• Add networking service check for bulk updates only

Integration changes:

• Add automatic DNS update call in docker_image_ctl post-start action

How to verify it

Run sonic-mgmt DNS tests

Which release branch to backport (provide reason below if selected)

• 201811
• 201911
• 202006
• 202012
• 202106
• 202111
• 202205
• 202211
• 202305

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[qiluo-msft]

How does "waits up to 5 seconds for containers to start" help solving a possible race condition? Is it just reduce the possibility but not completely resolve it?

Can we add a testcase to prevent future regression?

[ganglyu]

Is it possible to add a hook to run the update_container script after Docker starts?
Can we use a timer to run the update_container script?

[oleksandrivantsiv]

How does "waits up to 5 seconds for containers to start" help solving a possible race condition? Is it just reduce the possibility but not completely resolve it?

Can we add a testcase to prevent future regression?

There is a small time window when the race condition can occur. It always happens only during the config reload when the DNS configuration changes.

1. Dockerd starts the container with the current content of the "resolv.conf". It takes some time for the container to start and move into the "running" state.
2. Meanwhile, concurrently, hostcfgd receives an event with the DNS configuration. It runs resolv-config.service, which writes the DNS configuration into the "/etc/resolv.conf" file and triggers the update of the containers.
3. When the "update-containers" script starts, some Docker containers might not be in the "running" state yet, so the update for these containers can be missed.

The time window for this scenario is very small, which is why the issue has such a low reproducibility rate. In this scenario, waiting 5 seconds ensures that the Docker container has enough time to move into the running state.

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[oleksandrivantsiv]

@qiluo-msft, @ganglyu I updated the implementation to remove 5seconds wait period. Please review

[qiluo-msft]

Did you ever try this easier solution https://unix.stackexchange.com/a/348406/161486?

[qiluo-msft]

networking service is active

If "networking service is active" is a dependency to fix the issue, we need to worry about some container start during "networking service is not active", but later on the service started.

[oleksandrivantsiv]

I'm not sure I fully understand this comment

[oleksandrivantsiv]

The solution described in https://unix.stackexchange.com/a/348406/161486 is redundant. This is a default behavior Docker has. When the container starts dockerd copies "/etc/resolv.conf" file from the host OS. The issue with this behavior is described in my previous comment:

There is a small time window when the race condition can occur. It always happens only during the config reload when the DNS configuration changes.

1. Dockerd starts the container with the current content of the "resolv.conf" (copies it from the host). It takes some time for the container to start and move into the "running" state.
2. Meanwhile, concurrently, hostcfgd receives an event with the DNS configuration. It runs resolv-config.service, which writes the DNS configuration into the "/etc/resolv.conf" file and triggers the update of the containers.
3. When the "update-containers" script starts, some Docker containers might not be in the "running" state yet, so the update for these containers can be missed.

[mssonicbld]

Cherry-pick PR to 202411: #22462