Postgres: use `pg_try_advisory_lock` instead of `pg_advisory_lock` by AkuSilvenius · Pull Request #962 · golang-migrate/migrate

AkuSilvenius · 2023-07-26T16:27:22Z

Issue

Changes

Use pg_try_advisory_lock instead of pg_advisory_lock in Postgres Lock to fix the expected behavior of waiting for acquiring lock indefinitely
Add configurable exponential backoff for retrying when lock is not available.
Add config parameter x-lock-retry-max-interval to configure the max interval to be used for exponential backoff when retrying Lock

coveralls · 2024-01-23T10:36:47Z

coverage: 56.463% (+0.1%) from 56.335%
when pulling 21469a7 on AkuSilvenius:960_add_try_advisory_lock
into 0c456c4 on golang-migrate:master.

dhui

left a comment in case we decide to go this route

dhui · 2024-01-25T06:56:46Z

+				break
+			}
+
+			time.Sleep(100 * time.Millisecond)


Use the backoff library with an exponential backoff + jitter as the default. Note, this may cause other nodes/hosts to take longer to deploy due to the longer wait period so the backoff should be configurable.

@dhui thanks for the feedback, I've added a configurable exponential retry.

this may cause other nodes/hosts to take longer to deploy due to the longer wait period

I've added defaults and kept them quite small to address the above

AkuSilvenius · 2024-02-20T20:55:54Z

unfortunately the tests keep failing because device is running out of space

eelco · 2024-04-18T18:29:26Z

I assume the tests were temporarily broken, so this might work after a rebase? I’m interested in this fix landing 😊

AkuSilvenius · 2024-04-21T18:47:24Z

I assume the tests were temporarily broken, so this might work after a rebase? I’m interested in this fix landing 😊

@eelco thanks for noting, looks like #1072 fixed the tests and pipeline here in the PR is working and passing now - unfortunately this PR got stuck in sleep mode for quite a while due to the unstable pipeline, I hope we can merge this soon :)

simonjpartridge · 2024-07-03T12:48:58Z

I assume the tests were temporarily broken, so this might work after a rebase? I’m interested in this fix landing 😊

@eelco thanks for noting, looks like #1072 fixed the tests and pipeline here in the PR is working and passing now - unfortunately this PR got stuck in sleep mode for quite a while due to the unstable pipeline, I hope we can merge this soon :)

Is there anything we can do to help get this PR unblocked and merged?

AkuSilvenius · 2024-07-03T13:26:37Z

@simonjpartridge all tests are passing now, so just approvals are needed I guess :)

AkuSilvenius · 2024-07-06T17:04:48Z

anyone of the recent participants here in the PR willing to give it a review as the pipeline is stable now, and approve if it looks good so we get this one merged and the issue closed? @dhui @eelco @simonjpartridge

simonjpartridge

Looks good overall, one comment that should be actioned IMO around setting of the max retry interval

simonjpartridge · 2024-07-08T09:51:46Z

+		if maxRetryInterval > DefaultLockInitialRetryInterval {
+			lockConfig.MaxRetryInterval = maxRetryInterval
 		}


This if statement means a user can't set the max retry interval to less than the default. I can't see a good reason to limit it in this way so why not do this?

Suggested change

if maxRetryInterval > DefaultLockInitialRetryInterval {

lockConfig.MaxRetryInterval = maxRetryInterval

}

if maxRetryInterval >= 0 {

lockConfig.MaxRetryInterval = maxRetryInterval

}

thanks for the suggestion - the retry strategy here is to use exponential backoff, meaning if we end up retrying, the later retries should wait for longer than the previous ones. If we'd allow the user to set maxRetryInterval to less than InitialRetryInterval then the retry interval wouldn't increase which is against the idea of exponential backoff

however your comment raises a valid point that the user might not be aware of this initial retry interval and could try setting it to less than that without knowing it would be ignored - I've added a note about this to the readme be32ada

simonjpartridge · 2024-07-08T10:18:09Z

 	})
 }

+func testPostgresConcurrentMigrations(t *testing.T) {


Ran these tests against master and can confirm they fail 🎉. This change makes the tests pass

Co-authored-by: Simon Partridge <4527871+simonjpartridge@users.noreply.github.com>

AkuSilvenius · 2024-11-01T19:06:48Z

Looks like there has been some updates on the pipeline as the tests run without issues now. Maybe we could finally get this one merged? 😁 pinging reviewers @dhui @eelco @simonjpartridge

eelco · 2024-12-03T11:44:27Z

@AkuSilvenius I would approve but I don’t have write access in this repo, so it would not matter…

simonjpartridge

nice work! Unfortunately I don't think I have write access either. @dhui is the reviewer we need

amirsalarsafaei · 2026-09-17T13:22:59Z

Hi I believe this to be a serious issue in the library. Our prod got affected by it. Seems the like the pr is stale is there anything I can do to move it forward?
@Fontinalis

amirsalarsafaei · 2026-09-28T19:41:51Z

@AkuSilvenius any chance you could rebase this on master? It's pretty far behind now and has conflicts. If you don't have time for it I can open a new PR with your commits rebased (you'd stay the author) and link it back here.

@Fontinalis pinging again in case you missed my last mention. We hit #960 in production and this PR fixes it. dhui's backoff feedback was addressed and simonjpartridge already approved, so it should be pretty close. Happy to help with whatever is needed to get it merged.

pg_advisory_lock blocks until the lock is free, so concurrent migrations can hang forever (golang-migrate#960). Poll pg_try_advisory_lock instead and retry with an exponential backoff. The max retry interval can be set with x-lock-retry-max-interval. Rebased from golang-migrate#962. Fixes golang-migrate#960

AkuSilvenius added 7 commits July 26, 2023 18:40

Add failing test

362c98e

add simple trylock

cd50933

Add comment

2f8e47b

update test

5dadc1d

remove comment

021b8b8

retry pipeline

4ff8627

retry pipeline

a403d5d

This was referenced Sep 12, 2023

Postgres deadlocking when multiple processes run CREATE INDEX CONCURRENTLY #960

Open

Pipelines fail because running out of memory #977

Open

retry pipeline

8f10bfc

AkuSilvenius marked this pull request as ready for review January 23, 2024 10:39

dhui reviewed Jan 25, 2024

View reviewed changes

add configurable retry for lock

c989d69

update README.md

6f06f0a

AkuSilvenius added 3 commits April 21, 2024 21:01

Merge branch 'master' into 960_add_try_advisory_lock

e858f5e

update test name

1644f0c

actually run test

49bee23

simonjpartridge suggested changes Jul 8, 2024

View reviewed changes

AkuSilvenius and others added 5 commits July 8, 2024 16:01

Apply suggestions from code review

b8e39d0

Co-authored-by: Simon Partridge <4527871+simonjpartridge@users.noreply.github.com>

mention initial retry interval in readme

be32ada

add missing millisecond conversion

993b736

retry pipeline

8580c15

retry pipeline

a43c2f1

retry pipeline

21469a7

simonjpartridge approved these changes Dec 3, 2024

View reviewed changes

This was referenced Jan 14, 2026

use try advisory lock #1352

Closed

use try advisory lock DefinitiveCo/migrate#1

Merged

amirsalarsafaei mentioned this pull request Oct 3, 2026

fix(postgres,pgx): use pg_try_advisory_lock with exponential backoff in Lock #1451

Open

Conversation

AkuSilvenius commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Changes

Uh oh!

coveralls commented Jan 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhui left a comment

Choose a reason for hiding this comment

Uh oh!

dhui Jan 25, 2024

Choose a reason for hiding this comment

Uh oh!

AkuSilvenius Feb 20, 2024

Choose a reason for hiding this comment

Uh oh!

AkuSilvenius commented Feb 20, 2024

Uh oh!

eelco commented Apr 18, 2024

Uh oh!

AkuSilvenius commented Apr 21, 2024

Uh oh!

simonjpartridge commented Jul 3, 2024

Uh oh!

AkuSilvenius commented Jul 3, 2024

Uh oh!

AkuSilvenius commented Jul 6, 2024

Uh oh!

simonjpartridge left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simonjpartridge Jul 8, 2024

Choose a reason for hiding this comment

Uh oh!

AkuSilvenius Jul 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AkuSilvenius Jul 8, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

simonjpartridge Jul 8, 2024

Choose a reason for hiding this comment

Uh oh!

AkuSilvenius commented Nov 1, 2024

Uh oh!

eelco commented Dec 3, 2024

Uh oh!

simonjpartridge left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amirsalarsafaei commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amirsalarsafaei commented Sep 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AkuSilvenius commented Jul 26, 2023 •

edited

Loading

coveralls commented Jan 23, 2024 •

edited

Loading

simonjpartridge left a comment •

edited

Loading

AkuSilvenius Jul 8, 2024 •

edited

Loading

simonjpartridge left a comment •

edited

Loading

amirsalarsafaei commented Sep 17, 2026 •

edited

Loading