Spread connections across multiple gateways

[jchambers]

According to the APNs docs:

You may establish multiple connections to the same gateway or to multiple gateway instances. If you need to send a large number of push notifications, spread them out over connections to several different gateways. This improves performance compared to using a single connection: it lets you send the push notifications faster, and it lets APNs deliver them faster.

No further guidance is given as to how that might happen, though. We had previously assumed that connection-spreading would happen automatically, but that does not appear to be the case in testing. It looks like there are a number of IPs associated with the gateway's DNS entry:

$ dig gateway.push.apple.com

; <<>> DiG 9.8.3-P1 <<>> gateway.push.apple.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64507
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;gateway.push.apple.com.        IN  A

;; ANSWER SECTION:
gateway.push.apple.com. 181 IN  CNAME   gateway.push-apple.com.akadns.net.
gateway.push-apple.com.akadns.net. 3 IN A   17.172.232.33
gateway.push-apple.com.akadns.net. 3 IN A   17.172.232.30
gateway.push-apple.com.akadns.net. 3 IN A   17.172.233.149
gateway.push-apple.com.akadns.net. 3 IN A   17.172.232.35
gateway.push-apple.com.akadns.net. 3 IN A   17.172.232.231
gateway.push-apple.com.akadns.net. 3 IN A   17.172.233.155
gateway.push-apple.com.akadns.net. 3 IN A   17.172.233.147
gateway.push-apple.com.akadns.net. 3 IN A   17.172.238.221

To spread connections across multiple gateways, it seems like we'll need to do our own DNS resolution and connect directly by IP.

[eager]

We had previously assumed that connection-spreading would happen automatically, but that does not appear to be the case in testing.

It looks like the DNS is round-robined on Apple’s side, but the TTL is 60 seconds, so it would presumably only affect connections made more than a minute apart.

[marcomorain]

I wonder do Apple just intend for clients to to open more than one connection to APNS? If they have only 8 end-points, then each end-point must be able to handle a large number of push notifications per-second. I assume 2 connections to the same end-point is faster than one.

/CC @jmason

[jchambers]

If they have only 8 end-points, then each end-point must be able to handle a large number of push notifications per-second.

Probably true. A tech note from Apple suggests that we ought to be able to get ~9k notifications/second through a single connection, but I'm not sure how they arrived at that number.

I assume 2 connections to the same end-point is faster than one.

Maybe, but the main benefit there may just be that you have one connection still going if the other closes due to a rejected notification and needs to spend time reconnecting. That said, that's purely speculation on my part.

This bit from the docs makes me think that we'd want to spread across multiple different endpoints for maximum throughput:

You may establish multiple connections to the same gateway or to multiple gateway instances. If you need to send a large number of push notifications, spread them out over connections to several different gateways.

(emphasis added)

[jchambers]

Now that we live in a post-HTTP/2 world, I'm actually thinking this isn't a great idea. To recap, the main benefits in the past were:

1. Be more resilient in the face of frequently-closing connections.
2. Realize some performance gains by spreading requests across upstream computing resources.

Now that connections don't close every time a notification gets rejected, the connection resilience isn't something we need to worry about in quite the same way, and I think the benefit provided by multiple connections would be negligible in nearly all cases.

The second point—spreading requests across upstream resources—may still be A Thing, but it seems like we'd have to add an enormous amount of complexity (DNS resolution, pooled connections, etc.) to get there. It also seems like we'd only see gains in cases where we're saturating a CPU core (and could benefit from using more cores) for a single topic, but aren't close to saturating a network connection. I suspect most users who are sending zillions of notifications are generally sending them (a) on multiple topics or (b) from multiple servers.

My inclination right now is to close this unless somebody chimes in saying "this is totally my use case!" What do you all think?

[jmason]

I would be in favour of closing this, too. The primary use case was to deal with the rejected-notification-causes-connection-close lossage, which is no longer an issue (thankfully).

Users who still want greater performance, or more resilience, by using multiple parallel connections will probably have parallelism further up the stack -- ie. using multiple parallel ApnsClients, or multiple servers (as you suggested). (I know we do!)

[jchambers]

Closing for now, but will certainly be listening for comments.

[jasonjoo2010]

@jchambers

I am working on this currently.

We have over 3 million messages to push in one hour with 6 nodes concurrently. Maybe 3~4 sending tasks per day.
But i found something strange when trying to improve the performance of the system.

Every node of the six has a separate wan-ip communicating with gateways.
I pick up three nodes to dig the performance and we call them A, B and C.
Messages per 30 seconds:
A: 4000 ~ 16000 with avg. 7000, 6 connections with 17.188.155.155
B: 16000 ~ 32000 with avg. 24000, 16 connections with 17.188.151.161
C: 3000 ~ 13000 with avg. 4000, 7 connections with 17.188.x.x(forgotten, but surely different from the other two)

I try to restart the instance in poor performance and maybe i get different result, but hardly reach the performance of node B, and hardly select the same ip as node B did.

So i assume several possible reasons:

i. the loads of different gateway are different.
ii. the performances of different gateway are different.(similar to No. i)
iii. the limitations, especially to the sending nodes's ip, are different.

And i am doing various experiments to figure it out now.