swarm/src/lib: Continue polling network when behaviour is blocked (#2304)

With #2248 a connection task
`await`s sending an event to the behaviour before polling for new events
from the behaviour [1].

When `Swarm::poll` is unable to deliver an event to a connection task it
returns `Poll::Pending` even though (a) polling `Swarm::network` might be
able to make progress (`network_not_ready` being `false`) and (b) it
does not register a waker to be woken up [2].

In combination this can lead to a deadlock where a connection task waits
to send an event to the behaviour and `Swarm::poll` returns
`Poll::Pending` failing to send an event to the connection task, not
registering a waiker in order to be polled again.

With this commit `Swarm::poll` will only return `Poll::Pending`, when
failing to deliver an event to a connection task, if the network is
unable to make progress (i.e. `network_not_ready` being `true`).

In the long-run `Swarm::poll` should likely be redesigned, prioritizing
the behaviour over the network, given the former is the control plane
and the latter potentially yields new work from the outside.

[1]: https://github.com/libp2p/rust-libp2p/blob/ca1b7cf043b4264c69b19fe75de488330a7a1f2f/core/src/connection/pool/task.rs#L224-L232
[2]: https://github.com/libp2p/rust-libp2p/blob/ca1b7cf043b4264c69b19fe75de488330a7a1f2f/swarm/src/lib.rs#L756-L783

Author: mxinden

swarm/src/lib.rs

@@ -765,7 +765,11 @@ where
                             if let Some(mut conn) = peer.connection(conn_id) {
                                 if let Some(event) = notify_one(&mut conn, event, cx) {
                                     this.pending_event = Some((peer_id, handler, event));
-                                    return Poll::Pending;
+                                    if network_not_ready {
+                                        return Poll::Pending;
+                                    } else {
+                                        continue;
+                                    }
                                 }
                             }
                         }
@@ -775,7 +779,11 @@ where
                             {
                                 let handler = PendingNotifyHandler::Any(ids);
                                 this.pending_event = Some((peer_id, handler, event));
-                                return Poll::Pending;
+                                if network_not_ready {
+                                    return Poll::Pending;
+                                } else {
+                                    continue;
+                                }
                             }
                         }
                     }
@@ -843,7 +851,11 @@ where
                                     if let Some(event) = notify_one(&mut conn, event, cx) {
                                         let handler = PendingNotifyHandler::One(connection);
                                         this.pending_event = Some((peer_id, handler, event));
-                                        return Poll::Pending;
+                                        if network_not_ready {
+                                            return Poll::Pending;
+                                        } else {
+                                            continue;
+                                        }
                                     }
                                 }
                             }
@@ -854,7 +866,11 @@ where
                                 {
                                     let handler = PendingNotifyHandler::Any(ids);
                                     this.pending_event = Some((peer_id, handler, event));
-                                    return Poll::Pending;
+                                    if network_not_ready {
+                                        return Poll::Pending;
+                                    } else {
+                                        continue;
+                                    }
                                 }
                             }
                         }