Skip to content

HDDS-10316. Speed up TestReconTasks #6223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 40 commits into from
Apr 2, 2024
Merged

HDDS-10316. Speed up TestReconTasks #6223

Changes from 11 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
53e2c11
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 9, 2024
399b76a
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 12, 2024
23df399
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 12, 2024
e37e875
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 13, 2024
d3f021e
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 13, 2024
9b22c3c
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 13, 2024
64e35f9
HDDS-10316 common code is extracted to check the flow
Feb 13, 2024
e42e398
HDDS-10316 common code is extracted to check the flow
Feb 14, 2024
3ebe2a1
HDDS-10316 common code is extracted to check the flow
Feb 14, 2024
833e008
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 14, 2024
27d7902
Merge branch 'apache:master' into raju-b-hdds-10316
raju-balpande Feb 15, 2024
c1fe332
HDDS-10346 make test independent of ordering.
Feb 26, 2024
c540681
Merge branch 'raju-b-hdds-10316' of https://github.com/raju-balpande/…
Feb 26, 2024
9665426
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 26, 2024
8cc60b4
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 26, 2024
68a7697
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 26, 2024
c8a858a
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 26, 2024
cb9d7c2
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 26, 2024
317dcfa
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 26, 2024
dc25ff1
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 26, 2024
59c8920
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 27, 2024
50033f7
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 27, 2024
46f97dd
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 27, 2024
3c5305e
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
cd8ba26
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
7725dfa
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
14b4cd9
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
3b1b51c
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
f54cdbd
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
2844cec
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
922f92d
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
1f4a91e
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
e75f857
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
eada75b
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
608631b
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 28, 2024
285c084
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 29, 2024
dc519e1
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 29, 2024
c566ab6
HDDS-10316 reducing the initiation, method ordering introduced.
Feb 29, 2024
1fd1b5b
Merge branch 'master' into raju-b-hdds-10316
raju-balpande Feb 29, 2024
8a8db39
Merge remote-tracking branch 'origin/master' into raju-b-hdds-10316
adoroszlai Apr 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 25 additions & 15 deletions ...op-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconTasks.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,12 @@
import org.apache.ozone.test.LambdaTestUtils;
import org.hadoop.ozone.recon.schema.ContainerSchemaDefinition;
import org.hadoop.ozone.recon.schema.tables.pojos.UnhealthyContainers;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.TestMethodOrder;
import org.junit.jupiter.api.TestInstance;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Timeout;
import org.slf4j.event.Level;
Expand All @@ -52,18 +56,21 @@
import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
import static org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainer.runTestOzoneContainerViaDataNode;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertNotEquals;

/**
* Integration Tests for Recon's tasks.
*/
@Timeout(300)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
@TestMethodOrder(OrderAnnotation.class)
public class TestReconTasks {
private MiniOzoneCluster cluster = null;
private OzoneConfiguration conf;

@BeforeEach
public void init() throws Exception {
@BeforeAll
void init() throws Exception {
conf = new OzoneConfiguration();
conf.set(HDDS_CONTAINER_REPORT_INTERVAL, "5s");
conf.set(HDDS_PIPELINE_REPORT_INTERVAL, "5s");
Expand All @@ -81,14 +88,15 @@ public void init() throws Exception {
Level.DEBUG);
}

@AfterEach
public void shutdown() {
@AfterAll
void shutdown() {
if (cluster != null) {
cluster.shutdown();
}
}

@Test
@Order(2)
public void testSyncSCMContainerInfo() throws Exception {
ReconStorageContainerManagerFacade reconScm =
(ReconStorageContainerManagerFacade)
Expand Down Expand Up @@ -121,6 +129,7 @@ public void testSyncSCMContainerInfo() throws Exception {
}

@Test
@Order(3)
public void testMissingContainerDownNode() throws Exception {
ReconStorageContainerManagerFacade reconScm =
(ReconStorageContainerManagerFacade)
Expand All @@ -141,7 +150,7 @@ public void testMissingContainerDownNode() throws Exception {
(ReconContainerManager) reconScm.getContainerManager();
ContainerInfo containerInfo =
scmContainerManager
.allocateContainer(RatisReplicationConfig.getInstance(ONE), "test");
.allocateContainer(RatisReplicationConfig.getInstance(ONE), "testMissingContainer");
Copy link
Contributor

@devmadhuu devmadhuu Mar 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are increasing datanodes, then better to keep replication factor also as THREE

long containerID = containerInfo.getContainerID();

try (RDBBatchOperation rdbBatchOperation = new RDBBatchOperation()) {
Expand All @@ -156,8 +165,8 @@ public void testMissingContainerDownNode() throws Exception {
runTestOzoneContainerViaDataNode(containerID, client);

// Make sure Recon got the container report with new container.
assertEquals(scmContainerManager.getContainers(),
reconContainerManager.getContainers());
assertArrayEquals(scmContainerManager.getContainers().toArray(),
reconContainerManager.getContainers().toArray());

// Bring down the Datanode that had the container replica.
cluster.shutdownHddsDatanode(pipeline.getFirstNode());
Expand All @@ -168,7 +177,7 @@ public void testMissingContainerDownNode() throws Exception {
.getUnhealthyContainers(
ContainerSchemaDefinition.UnHealthyContainerStates.MISSING,
0, 1000);
return (allMissingContainers.size() == 1);
return (allMissingContainers.size() >= 1);
});

// Restart the Datanode to make sure we remove the missing container.
Expand Down Expand Up @@ -202,6 +211,7 @@ public void testMissingContainerDownNode() throws Exception {
* @throws Exception
*/
@Test
@Order(1)
public void testEmptyMissingContainerDownNode() throws Exception {
ReconStorageContainerManagerFacade reconScm =
(ReconStorageContainerManagerFacade)
Expand All @@ -221,7 +231,7 @@ public void testEmptyMissingContainerDownNode() throws Exception {
(ReconContainerManager) reconScm.getContainerManager();
ContainerInfo containerInfo =
scmContainerManager
.allocateContainer(RatisReplicationConfig.getInstance(ONE), "test");
.allocateContainer(RatisReplicationConfig.getInstance(ONE), "testEmptyMissingContainer");
Copy link
Contributor

@devmadhuu devmadhuu Mar 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are increasing datanodes, then better to keep replication factor also as THREE

Copy link
Contributor Author

@raju-balpande raju-balpande Mar 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried changing it to HddsProtos.ReplicationFactor.THREE but it seems having problem with number of pipelines..
java.io.IOException: Could not allocate container. Cannot get any matching pipeline for replicationConfig: RATIS/THREE, State:PipelineState.OPEN

at org.apache.hadoop.hdds.scm.container.ContainerManagerImpl.allocateContainer(ContainerManagerImpl.java:202)
at org.apache.hadoop.ozone.recon.TestReconTasks.testEmptyMissingContainerDownNode(TestReconTasks.java:236)
at java.lang.reflect.Method.invoke(Method.java:498)

Copy link
Contributor

@devmadhuu devmadhuu Mar 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, this might be because of not meeting the criteria of sufficient healthy nodes because default minRatisVolumeSizeBytes is 1 GB and containerSizeBytes is 5GB. For test case it is okay then to use ReplicationFactor.ONE

long containerID = containerInfo.getContainerID();

Pipeline pipeline =
Expand All @@ -230,8 +240,8 @@ public void testEmptyMissingContainerDownNode() throws Exception {
runTestOzoneContainerViaDataNode(containerID, client);

// Make sure Recon got the container report with new container.
assertEquals(scmContainerManager.getContainers(),
reconContainerManager.getContainers());
assertArrayEquals(scmContainerManager.getContainers().toArray(),
reconContainerManager.getContainers().toArray());

// Bring down the Datanode that had the container replica.
cluster.shutdownHddsDatanode(pipeline.getFirstNode());
Expand All @@ -243,7 +253,7 @@ public void testEmptyMissingContainerDownNode() throws Exception {
ContainerSchemaDefinition.UnHealthyContainerStates.
EMPTY_MISSING,
0, 1000);
return (allEmptyMissingContainers.size() == 1);
return (allEmptyMissingContainers.size() >= 1);
});

// Now add a container to key mapping count as 3. This data is used to
Expand All @@ -261,7 +271,7 @@ public void testEmptyMissingContainerDownNode() throws Exception {
.getUnhealthyContainers(
ContainerSchemaDefinition.UnHealthyContainerStates.MISSING,
0, 1000);
return (allMissingContainers.size() == 1);
return (allMissingContainers.size() >= 1);
});

LambdaTestUtils.await(25000, 1000, () -> {
Expand Down