Skip to content

HDFS-17496. DataNode supports more fine-grained dataset lock based on blockid. #6764

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Hexiaoqiao merged 12 commits into from
Jan 2, 2025
Merged

HDFS-17496. DataNode supports more fine-grained dataset lock based on blockid. #6764

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
fcf8021
HDFS-17496. DataNode supports more fine-grained dataset lock based on…
hfutatzhanghb Apr 23, 2024
db36a2b
remove used statement.
hfutatzhanghb Apr 23, 2024
dc09713
fix bug
hfutatzhanghb Jun 28, 2024
89ca2cd
fix checkstyles.
hfutatzhanghb Nov 27, 2024
c723265
fix checkstyles.
hfutatzhanghb Nov 27, 2024
3d579ae
fix failed UTs
hfutatzhanghb Nov 28, 2024
94d6660
trigger yetus.
hfutatzhanghb Nov 29, 2024
4751846
fix checkstyle
hfutatzhanghb Dec 3, 2024
263a5ca
checkAndUpdate use subdir lock
hfutatzhanghb Dec 3, 2024
d4f9eb1
fix npe check.
hfutatzhanghb Dec 8, 2024
9eb153c
Add interface DataSetSubLockStrategy
hfutatzhanghb Dec 16, 2024
226cd5b
fix failed UT.
hfutatzhanghb Dec 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ...t/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/DataNodeLockManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ public interface DataNodeLockManager<T extends AutoCloseDataSetLock> {
*/
enum LockLevel {
BLOCK_POOl,
VOLUME
VOLUME,
DIR
}

/**
Expand Down
33 changes: 31 additions & 2 deletions .../hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataSetLockManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,8 @@ private String generateLockName(LockLevel level, String... resources) {
+ resources[0] + "volume lock :" + resources[1]);
}
return resources[0] + resources[1];
} else if (resources.length == 3 && level == LockLevel.DIR) {
return resources[0] + resources[1] + resources[2];
} else {
throw new IllegalArgumentException("lock level do not match resource");
}
Expand Down Expand Up @@ -153,7 +155,7 @@ public DataSetLockManager() {
public AutoCloseDataSetLock readLock(LockLevel level, String... resources) {
if (level == LockLevel.BLOCK_POOl) {
return getReadLock(level, resources[0]);
} else {
} else if (level == LockLevel.VOLUME){
AutoCloseDataSetLock bpLock = getReadLock(LockLevel.BLOCK_POOl, resources[0]);
AutoCloseDataSetLock volLock = getReadLock(level, resources);
volLock.setParentLock(bpLock);
Expand All @@ -162,14 +164,25 @@ public AutoCloseDataSetLock readLock(LockLevel level, String... resources) {
resources[0]);
}
return volLock;
} else {
AutoCloseDataSetLock bpLock = getReadLock(LockLevel.BLOCK_POOl, resources[0]);
AutoCloseDataSetLock volLock = getReadLock(LockLevel.VOLUME, resources[0], resources[1]);
volLock.setParentLock(bpLock);
AutoCloseDataSetLock dirLock = getReadLock(level, resources);
dirLock.setParentLock(volLock);
if (openLockTrace) {
LOG.debug("Sub lock " + resources[0] + resources[1] + resources[2] + " parent lock " +
resources[0] + resources[1]);
}
return dirLock;
}
}

@Override
public AutoCloseDataSetLock writeLock(LockLevel level, String... resources) {
if (level == LockLevel.BLOCK_POOl) {
return getWriteLock(level, resources[0]);
} else {
} else if (level == LockLevel.VOLUME) {
AutoCloseDataSetLock bpLock = getReadLock(LockLevel.BLOCK_POOl, resources[0]);
AutoCloseDataSetLock volLock = getWriteLock(level, resources);
volLock.setParentLock(bpLock);
Expand All @@ -178,6 +191,17 @@ public AutoCloseDataSetLock writeLock(LockLevel level, String... resources) {
resources[0]);
}
return volLock;
} else {
AutoCloseDataSetLock bpLock = getReadLock(LockLevel.BLOCK_POOl, resources[0]);
AutoCloseDataSetLock volLock = getReadLock(LockLevel.VOLUME, resources[0], resources[1]);
volLock.setParentLock(bpLock);
AutoCloseDataSetLock dirLock = getWriteLock(level, resources);
dirLock.setParentLock(volLock);
if (openLockTrace) {
LOG.debug("Sub lock " + resources[0] + resources[1] + resources[2] + " parent lock " +
resources[0] + resources[1]);
}
return dirLock;
}
}

Expand Down Expand Up @@ -224,8 +248,13 @@ public void addLock(LockLevel level, String... resources) {
String lockName = generateLockName(level, resources);
if (level == LockLevel.BLOCK_POOl) {
lockMap.addLock(lockName, new ReentrantReadWriteLock(isFair));
} else if (level == LockLevel.VOLUME) {
lockMap.addLock(resources[0], new ReentrantReadWriteLock(isFair));
lockMap.addLock(lockName, new ReentrantReadWriteLock(isFair));
} else {
lockMap.addLock(resources[0], new ReentrantReadWriteLock(isFair));
lockMap.addLock(generateLockName(LockLevel.VOLUME, resources[0], resources[1]),
new ReentrantReadWriteLock(isFair));
lockMap.addLock(lockName, new ReentrantReadWriteLock(isFair));
}
}
Expand Down
27 changes: 27 additions & 0 deletions ...roject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DatanodeUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.hdfs.protocol.Block;
Expand Down Expand Up @@ -127,6 +129,31 @@ public static File idToBlockDir(File root, long blockId) {
return new File(root, path);
}

/**
* Take an example. We hava a block with blockid mapping to:
* "/data1/hadoop/hdfs/datanode/current/BP-xxxx/current/finalized/subdir0/subdir0"
* We return "subdir0/subdir0"
* @param blockId blockId
* @return The two-level subdir name
*/
public static String idToBlockDirSuffixName(long blockId) {
Copy link
Contributor

@Hexiaoqiao Hexiaoqiao Dec 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's different between idToBlockDir and idToBlockDirSuffixName?

Copy link
Member Author

@hfutatzhanghb hfutatzhanghb Dec 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

idToBlockDirSuffixName method is used to generate SubDir lock name, just like BlockPool id or storageuuid.

int d1 = (int) ((blockId >> 16) & 0x1F);
int d2 = (int) ((blockId >> 8) & 0x1F);
return DataStorage.BLOCK_SUBDIR_PREFIX + d1 + SEP +
DataStorage.BLOCK_SUBDIR_PREFIX + d2;
}

public static List<String> getAllSubDirNameForDataSetLock() {
Copy link
Contributor

@Hexiaoqiao Hexiaoqiao Dec 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't get what this method want to do, and why there are double layer loop (include the invoke enter) and user 0x1F as end condition.

Copy link
Member Author

@hfutatzhanghb hfutatzhanghb Dec 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is used to generate all subdir lock name. We use 0x1F as end condition because the layout of DataNode.
Each blockpool dir under one volume has two layer subdir, like subdir1/subdir31.

Copy link
Member Author

@hfutatzhanghb hfutatzhanghb Dec 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When DataNode starts, we will initial all dataset subdir lock in advance. Every volume has its subdir dataset locks.
In current layout of datanode's storage directory， we have 32 multipy 32 subdirs in each volume.

Copy link
Contributor

@Hexiaoqiao Hexiaoqiao Dec 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is good idea to follow the current directory layout to construct lock based blockid. since:
a. We have to keep same logic between the directory layout and block/subdir lock, right? However it implements at different place without any constraint, which will involve some cost for the future improvement when change the directory layout.
b. I think we could decouple block/subdir lock from directory layout, define another rule for block/subdir lock with blockid only(such as id mod 1000, split one volume lock to 1000 locks is enough to improve the performance), which I think it make sense to block/subdir.
What do you think about? Thanks.

List<String> res = new ArrayList<>();
for (int d1 = 0; d1 <= 0x1F; d1++) {
for (int d2 = 0; d2 <= 0x1F; d2++) {
res.add(DataStorage.BLOCK_SUBDIR_PREFIX + d1 + SEP +
DataStorage.BLOCK_SUBDIR_PREFIX + d2);
}
}
return res;
}

/**
* @return the FileInputStream for the meta data of the given block.
* @throws FileNotFoundException
Expand Down
Loading
Loading