Fix rare deadlock when using read/write locks

[erikjohnston]

Introduced in #15782

[reivilibre]

I can't really follow what change this is making or what race it's trying to cover up, any chance you can elaborate? :)

[erikjohnston]

Whoops! Yes, basically: we have a _mode table that tracks (for a given lock name/key) that tracks whether its been locked and if so if its a read or a write lock and a token of a currently held lock. When we drop a lock we need to either delete that row, or update the token to another token (if there is another read lock).

If we try and drop two locks at the same time, we'll try and update the _mode table to point to the token of the other lock (that we're also dropping in another transaction), and that results in a deadlock.

The fix here is to make sure that we "serialise" changes to the lock with a given name/key. We do this by taking a row level lock on _mode table before we insert and delete (and we already do that for insert).

[reivilibre]

not dead obvious (but mostly just because I didn't have a clue how this locking stuff was put together) but makes sense now.