Nest source encounters 409 and retry attempts hit 429 rate-limit

[trvrnrth]

I have a nest hello gen 1 wired doorbell source which, after some time experiences a 409 response:

error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door

It appears that there is no back-off on attempts to re-connect so these are then rate-limited:

error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door

The full error message grabbed from the Device Access Sample App at this time is:

{
  "error": {
    "code": 429,
    "message": "Rate limited for the GenerateWebRtcStream API for the user.",
    "status": "RESOURCE_EXHAUSTED"
  }
}

At this point a restart is not sufficient to re-connect. go2rtc must be stopped for some time to allow the rate limit to "clear" before being started again (in my case this is a case of manually stopping, waiting a few minutes then starting the HA Frigate add-on).

I am unclear what causes the initial 409 conflict response, but implementing a back-off on at least the 429 response would be good. I guess it might make sense to just back-off on any error which would lessen the chance of hitting the rate limit in the first place.

Here is a full extract from the logs at the time of last failure:

2024-10-12 15:16:29.799077795  16:16:29.798 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url=ffmpeg:front_door#audio=aac
2024-10-12 15:16:50.871397445  16:16:50.871 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:16:55.987067326  16:16:55.986 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:01.104080094  16:17:01.103 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:06.220382167  16:17:06.220 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:11.337320055  16:17:11.337 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:16.453962537  16:17:16.453 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:21.569329842  16:17:21.569 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:26.684904880  16:17:26.684 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:31.799685805  16:17:31.799 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:36.916763767  16:17:36.916 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:42.031527043  16:17:42.031 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:47.146558111  16:17:47.146 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:52.264210419  16:17:52.264 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:57.380472619  16:17:57.380 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:02.495376712  16:18:02.495 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:07.613469154  16:18:07.613 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:12.728694209  16:18:12.728 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:17.846215123  16:18:17.845 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:22.962979410  16:18:22.962 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:28.079459957  16:18:28.079 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:33.194993369  16:18:33.194 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:38.313387009  16:18:38.313 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:43.428449201  16:18:43.428 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:48.543098321  16:18:48.542 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:53.660881178  16:18:53.660 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:58.779714906  16:18:58.779 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:03.892740883  16:19:03.892 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:09.009398444  16:19:09.009 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:14.128929625  16:19:14.128 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:19.246458352  16:19:19.246 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:24.360992865  16:19:24.360 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:29.477766616  16:19:29.477 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:34.593178617  16:19:34.592 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:39.709458289  16:19:39.709 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:44.825299325  16:19:44.825 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:49.941154786  16:19:49.940 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:55.060783500  16:19:55.060 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:00.177158992  16:20:00.177 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:00.177373129  16:20:00.177 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37294: write: broken pipe"
2024-10-12 15:20:05.294597204  16:20:05.293 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:05.294615471  16:20:05.294 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37302: write: broken pipe"
2024-10-12 15:20:10.410614638  16:20:10.410 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:10.410963374  16:20:10.410 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37314: write: broken pipe"
2024-10-12 15:20:15.525828154  16:20:15.525 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:20.643628744  16:20:20.643 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:25.758949395  16:20:25.758 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:30.874456050  16:20:30.874 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:30.874686424  16:20:30.874 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:45290: write: broken pipe"
2024-10-12 15:20:35.989727907  16:20:35.989 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:35.990225268  16:20:35.989 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:45292: write: broken pipe"
2024-10-12 15:20:41.106395530  16:20:41.106 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:41.106723038  16:20:41.106 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:45308: write: broken pipe"
2024-10-12 15:20:46.222196430  16:20:46.221 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:46.222541964  16:20:46.222 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37156: write: broken pipe"
2024-10-12 15:20:51.336820870  16:20:51.336 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:51.336971870  16:20:51.336 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37166: write: broken pipe"
2024-10-12 15:20:56.452101686  16:20:56.451 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:56.452461684  16:20:56.452 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37168: write: broken pipe"

[trvrnrth]

I had a bit of time to do some local debugging of this and realised I'd not provided my stream config in the issue above:

streams:
  front_door:
    - nest:?client_id=REDACTED&client_secret=REDACTED&device_id=REDACTED&project_id=REDACTED&refresh_token=REDACTED
    - "ffmpeg:front_door#audio=aac

The testing I've done suggests that the fixed 5 second retry I'm seeing stems from the ffmpeg audio transcode I have in place so I am currently testing with a 60 second timeout as follows to see if this helps:

      - "ffmpeg:front_door#audio=aac#input=-fflags nobuffer -flags low_delay -timeout 60000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i {input}"

[softcoder]

Did this ever get resolved? I'm trying this inn the go2rtc addon in the frigate docker container and can't seem to get my nest cam working?

[trvrnrth]

Adjusting the timeouts (in both the ffmpeg transcode and also externally in frigate) mitigates the 429 rate limiting, however I am still seeing the stream getting stuck receiving 409 responses (now at 60 second intervals).

At this time I can replicate the responses in the Device Access Sample App as follows:

{
  "error": {
    "code": 409,
    "message": "Failed to retrieve answer SDP due to timeout, please try again.",
    "status": "ABORTED"
  }
}

Restarting appears to be sufficient to resolve whatever the underlying issue is, although on my last restart I did see a single 409 response before the stream started working again.

[hnws]

@AlexxIT may I know that if I want to add this exponential backoff ability, or maybe even restart the nest subscription, which file should I start looking into?

[AlexxIT]

Related to nest files:

• https://github.com/AlexxIT/go2rtc/tree/master/internal/nest
• https://github.com/AlexxIT/go2rtc/tree/master/pkg/nest

[tom42530]

Hi,
@AlexxIT any update about this issue?

Thanks !

[AlexxIT]

I don't use Nest. Can't fix it myself.

[astyrrian1]

I hit this quite often unfortunately and often have to remote in, turn off my container for 15 minutes, turn it back on to workaround rate limit issues. I end up with nest wrong status 429 - too many requests. I thought I would have addressed it with -timeout flag but 6000000 doesn't seem to do it.

[hnws]

Just want to share a bit progress on this.
I believe somehow I have the retry implemented. I will test for a few more days and submit PR.

[itsnotlupus]

Did this ever get resolved? I'm trying this inn the go2rtc addon in the frigate docker container and can't seem to get my nest cam working?

I bumped into this a couple days ago. The exact same go2rtc.yaml config, with the exact same nest: URL with the same magic tokens in it, fails to bring up any feed on frigate (and indeed fails to show any stream if you expose port 1984 in the frigate container), but works when ran from a standalone go2rtc binary, or a go2rtc home assistant add-on.
If you fiddle with frigate configuration to point at an external go2rtc service, it is possible to have your nest cams working within frigate.

Except I'm having an issue where the camera feed stops working after 5 minutes and there's nothing in the logs (trace level) to account for that (yes, I did grab a fresh refresh_token, didn't help), which is what sent me searching for clues 'round these parts.

[astyrrian1]

Thanks, I see this was merged a few days ago (#1669), so now we just wait for next go2rtc release, then for Frigate to pick that up - or run a custom version sooner (https://docs.frigate.video/configuration/advanced/#custom-go2rtc-version)

[astyrrian1]

I am testing this out now via nightly build and custom binary in frigate. So far working really well. Thank you!