Skip to content

HADOOP-19311: [ABFS] Implement Backoff and Read Footer metrics using IOStatistics Class #7122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
anujmodi2021 merged 38 commits into from
Feb 6, 2025
Merged

HADOOP-19311: [ABFS] Implement Backoff and Read Footer metrics using IOStatistics Class #7122

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
f2e8409
AbfsBackoffMetrics new implementation
Sep 16, 2024
10906cd
ABFS Backoff metrics sample usecase implementation
Sep 18, 2024
db9ec05
ABFS Backoff metrics using IOStatistics
Sep 19, 2024
6531e83
AbfsBackoff metrics changes
Sep 24, 2024
7c6e0ff
Move AbfsBackoffMetrics to service directory
Oct 1, 2024
b7f7fbe
Hadoop-19311 Abfs Backoff and read footer metrics using IOStatistics
Oct 17, 2024
dabc171
Added license to newly added file
Oct 17, 2024
3692134
pulled latest trunk code
Oct 17, 2024
c24f0eb
Revert AbfsClient changes
Oct 17, 2024
8a87b39
Abfs Client format fix
Oct 17, 2024
fcaf09f
Files refactor
Oct 17, 2024
04c45a6
Fix checkstyle and added license
Oct 18, 2024
53b6623
Fix checkstyle errors
Oct 18, 2024
231a434
Added java doc for each classes and methods
Oct 18, 2024
e56e2e0
Added test case for back off metrics
Oct 18, 2024
dfbc356
Fix magic value checkstyle error
Oct 19, 2024
26ec83e
abfs client unnecessary changes reverted
bhattmanish98 Oct 21, 2024
5ecce29
Merge branch 'trunk' of https://github.com/bhattmanish98/hadoop into …
Oct 23, 2024
361af92
Test case fix
Oct 23, 2024
8feccf7
Merge branch 'trunk' of https://github.com/apache/hadoop into HADOOP-…
Nov 8, 2024
327a0a6
Comments for read footer metrics toString method
Nov 8, 2024
449b2b6
Changes after review
Nov 18, 2024
e4b680a
Checkstype fixes
Nov 18, 2024
3cc57f7
Add private constructor in StringUtils class
Nov 18, 2024
24e49f8
Merged latest trunk code
Dec 9, 2024
f636f41
Changes based on comments given
Dec 9, 2024
154a4ae
Use of Mean statistics in read footer metrics
Dec 9, 2024
7d80bd2
Rename update read method name
Dec 10, 2024
4feaac7
Fixed test cases and checkstyle failures
Dec 10, 2024
bc6f9cb
Read Length correction in test case
Dec 10, 2024
98c8716
Added metrics config in account template
Dec 11, 2024
bd4b131
Move constant to metrics constant file
Dec 17, 2024
b9ec4aa
Created common method for getPrecisionMetrics
bhattmanish98 Dec 18, 2024
3702356
Created common method for getPrecisionMetrics
bhattmanish98 Dec 18, 2024
d4f1e09
Removed entire package name from comments in retryValue and fileType …
bhattmanish98 Dec 18, 2024
65b64a9
Add commit to retrigger yetus build
bhattmanish98 Dec 18, 2024
ef2b169
Merge branch 'trunk' of https://github.com/apache/hadoop into HADOOP-…
bhattmanish98 Jan 6, 2025
5462967
Changes as well comments given
bhattmanish98 Jan 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
312 changes: 0 additions & 312 deletions ...op-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsBackoffMetrics.java
View file
Open in desktop

This file was deleted.

12 changes: 5 additions & 7 deletions hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import java.util.concurrent.atomic.AtomicLong;

import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.fs.azurebfs.services.AbfsBackoffMetrics;
import org.apache.hadoop.fs.azurebfs.services.AbfsCounters;
import org.apache.hadoop.fs.azurebfs.services.AbfsReadFooterMetrics;
import org.apache.hadoop.fs.azurebfs.utils.MetricFormat;
Expand Down Expand Up @@ -69,6 +70,7 @@
import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.SEND_REQUESTS;
import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.SERVER_UNAVAILABLE;
import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.WRITE_THROTTLES;
import static org.apache.hadoop.fs.azurebfs.enums.AbfsBackoffMetricsEnum.TOTAL_NUMBER_OF_REQUESTS;
import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.iostatisticsStore;
import static org.apache.hadoop.util.Time.now;

Expand Down Expand Up @@ -324,18 +326,14 @@ public DurationTracker trackDuration(String key) {
public String toString() {
String metric = "";
if (abfsBackoffMetrics != null) {
long totalNoRequests = getAbfsBackoffMetrics().getTotalNumberOfRequests();
long totalNoRequests = getAbfsBackoffMetrics().getMetricValue(TOTAL_NUMBER_OF_REQUESTS);
if (totalNoRequests > 0) {
metric += "#BO:" + getAbfsBackoffMetrics().toString();
}
}
if (abfsReadFooterMetrics != null) {
Map<String, AbfsReadFooterMetrics> metricsMap = getAbfsReadFooterMetrics().getMetricsMap();
if (metricsMap != null && !(metricsMap.isEmpty())) {
String readFooterMetric = getAbfsReadFooterMetrics().toString();
if (!readFooterMetric.equals("")) {
metric += "#FO:" + getAbfsReadFooterMetrics().toString();
}
if (getAbfsReadFooterMetrics().getTotalFiles() > 0) {
metric += "#FO:" + getAbfsReadFooterMetrics().toString();
}
}
return metric;
Expand Down
Loading
Loading