Skip to content

HDDS-11251. Deprecate definitions and remove listTrash and recoverTrash APIs #7060

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ivandika3 merged 16 commits into from
Aug 27, 2024
Merged

HDDS-11251. Deprecate definitions and remove listTrash and recoverTrash APIs #7060

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
17f7cef
Initial commit for listTrash and recoverTrash API deprecation
spacemonkd Aug 12, 2024
2be1dfe
Fixed .proto deprecation for v2
spacemonkd Aug 12, 2024
b3f38f4
Removed unused import
spacemonkd Aug 13, 2024
86aafd5
Marked recoverTrash as deprecated in ClientProtocol
spacemonkd Aug 13, 2024
6d9118c
Restore files and mark as deprecated
spacemonkd Aug 17, 2024
b645e54
Fixed blank spaces
spacemonkd Aug 17, 2024
35922e1
Removed listTrash and recoverTrash definitions
spacemonkd Aug 18, 2024
4a46f70
Fixed unused imports
spacemonkd Aug 18, 2024
8c97ac5
Removed trash key property from ozone-default.xml
spacemonkd Aug 19, 2024
21290b7
Added deprecated comment markers to listTrash and recoverTrash enum
spacemonkd Aug 19, 2024
aa5e8b6
Removed trashKey and trashKeyFail metrics
spacemonkd Aug 21, 2024
96529cc
Addressed build failures
spacemonkd Aug 21, 2024
76c77cb
Addressed build failures
spacemonkd Aug 21, 2024
163c552
Fixed keyOps count
spacemonkd Aug 22, 2024
817ec78
Fixed numKeyOps test
spacemonkd Aug 26, 2024
3172f3c
Addressed review comments
spacemonkd Aug 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions ...op-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -527,7 +527,10 @@ List<OzoneKey> listKeys(String volumeName, String bucketName,
* the cluster level set by admins.
* @return The list of keys that are deleted from the deleted table.
* @throws IOException
* @deprecated HDDS-3367 implements a new way to store trashed files.
* Please avoid using this as it will be removed in a future release
*/
@Deprecated
List<RepeatedOmKeyInfo> listTrash(String volumeName, String bucketName,
String startKeyName, String keyPrefix,
int maxKeys)
Expand All @@ -542,7 +545,10 @@ List<RepeatedOmKeyInfo> listTrash(String volumeName, String bucketName,
* @param destinationBucket - The bucket user want to recover to.
* @return The result of recovering operation is success or not.
* @throws IOException
* @deprecated HDDS-3367 implements a new way to store trashed files.
* Please avoid using this as it will be removed in a future release
*/
@Deprecated
boolean recoverTrash(String volumeName, String bucketName, String keyName,
String destinationBucket) throws IOException;

Expand Down
13 changes: 4 additions & 9 deletions hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1762,20 +1762,15 @@ public List<OzoneKey> listKeys(String volumeName, String bucketName,
@Override
public List<RepeatedOmKeyInfo> listTrash(String volumeName, String bucketName,
String startKeyName, String keyPrefix, int maxKeys) throws IOException {

Preconditions.checkNotNull(volumeName);
Preconditions.checkNotNull(bucketName);

return ozoneManagerClient.listTrash(volumeName, bucketName, startKeyName,
keyPrefix, maxKeys);
// listTrash is deprecated
throw new UnsupportedOperationException();
Comment thread
spacemonkd marked this conversation as resolved.
Outdated
}

@Override
public boolean recoverTrash(String volumeName, String bucketName,
String keyName, String destinationBucket) throws IOException {

return ozoneManagerClient.recoverTrash(volumeName, bucketName, keyName,
destinationBucket);
//recoverTrash is deprecated
throw new UnsupportedOperationException();
Comment thread
spacemonkd marked this conversation as resolved.
Outdated
}

@Override
Expand Down
2 changes: 0 additions & 2 deletions hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,6 @@ public static boolean isReadOnly(
case LookupKey:
case ListKeys:
case ListKeysLight:
case ListTrash:
case ServiceList:
case ListOpenFiles:
case ListMultiPartUploadParts:
Expand Down Expand Up @@ -302,7 +301,6 @@ public static boolean isReadOnly(
case SetAcl:
case AddAcl:
case PurgeKeys:
case RecoverTrash:
case FinalizeUpgrade:
case Prepare:
case CancelPrepare:
Expand Down
6 changes: 6 additions & 0 deletions ...-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1068,7 +1068,10 @@ DBUpdates getDBUpdates(
* the cluster level set by admins.
* @return The list of keys that are deleted from the deleted table.
* @throws IOException
* @deprecated HDDS-3367 implements a new way to store trashed files.
* Please avoid using this as it will be removed in a future release
*/
@Deprecated
List<RepeatedOmKeyInfo> listTrash(String volumeName, String bucketName,
String startKeyName, String keyPrefix, int maxKeys) throws IOException;

Expand All @@ -1081,7 +1084,10 @@ List<RepeatedOmKeyInfo> listTrash(String volumeName, String bucketName,
* @param destinationBucket - The bucket user want to recover to.
* @return The result of recovering operation is success or not.
* @throws IOException
* @deprecated HDDS-3367 implements a new way to store trashed files.
* Please avoid using this as it will be removed in a future release
*/
@Deprecated
default boolean recoverTrash(String volumeName, String bucketName,
String keyName, String destinationBucket) throws IOException {
return false;
Expand Down
76 changes: 4 additions & 72 deletions ...ava/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,6 @@
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ListStatusResponse;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ListTenantRequest;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ListTenantResponse;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ListTrashRequest;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ListTrashResponse;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ListVolumeRequest;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ListVolumeResponse;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.LookupFileRequest;
Expand Down Expand Up @@ -181,8 +179,6 @@
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.RangerBGSyncResponse;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.RecoverLeaseRequest;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.RecoverLeaseResponse;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.RecoverTrashRequest;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.RecoverTrashResponse;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.RefetchSecretKeyRequest;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.RefetchSecretKeyResponse;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.RemoveAclRequest;
Expand Down Expand Up @@ -2443,79 +2439,15 @@ public List<OzoneFileStatus> listStatus(OmKeyArgs args, boolean recursive,
public List<RepeatedOmKeyInfo> listTrash(String volumeName,
String bucketName, String startKeyName, String keyPrefix, int maxKeys)
throws IOException {

Preconditions.checkArgument(Strings.isNullOrEmpty(volumeName),
"The volume name cannot be null or " +
"empty. Please enter a valid volume name or use '*' as a wild card");

Preconditions.checkArgument(Strings.isNullOrEmpty(bucketName),
"The bucket name cannot be null or " +
"empty. Please enter a valid bucket name or use '*' as a wild card");

ListTrashRequest trashRequest = ListTrashRequest.newBuilder()
.setVolumeName(volumeName)
.setBucketName(bucketName)
.setStartKeyName(startKeyName)
.setKeyPrefix(keyPrefix)
.setMaxKeys(maxKeys)
.build();

OMRequest omRequest = createOMRequest(Type.ListTrash)
.setListTrashRequest(trashRequest)
.build();

ListTrashResponse trashResponse =
handleError(submitRequest(omRequest)).getListTrashResponse();

List<RepeatedOmKeyInfo> deletedKeyList =
new ArrayList<>(trashResponse.getDeletedKeysCount());

List<RepeatedOmKeyInfo> list = new ArrayList<>();
for (OzoneManagerProtocolProtos.RepeatedKeyInfo
repeatedKeyInfo : trashResponse.getDeletedKeysList()) {
RepeatedOmKeyInfo fromProto =
RepeatedOmKeyInfo.getFromProto(repeatedKeyInfo);
list.add(fromProto);
}
deletedKeyList.addAll(list);

return deletedKeyList;
// listTrash is deprecated
throw new UnsupportedOperationException();
}

@Override
public boolean recoverTrash(String volumeName, String bucketName,
String keyName, String destinationBucket) throws IOException {

Preconditions.checkArgument(Strings.isNullOrEmpty(volumeName),
"The volume name cannot be null or empty. " +
"Please enter a valid volume name.");

Preconditions.checkArgument(Strings.isNullOrEmpty(bucketName),
"The bucket name cannot be null or empty. " +
"Please enter a valid bucket name.");

Preconditions.checkArgument(Strings.isNullOrEmpty(keyName),
"The key name cannot be null or empty. " +
"Please enter a valid key name.");

Preconditions.checkArgument(Strings.isNullOrEmpty(destinationBucket),
"The destination bucket name cannot be null or empty. " +
"Please enter a valid destination bucket name.");

RecoverTrashRequest.Builder req = RecoverTrashRequest.newBuilder()
.setVolumeName(volumeName)
.setBucketName(bucketName)
.setKeyName(keyName)
.setDestinationBucket(destinationBucket);

OMRequest omRequest = createOMRequest(Type.RecoverTrash)
.setRecoverTrashRequest(req)
.build();

RecoverTrashResponse recoverResponse =
handleError(submitRequest(omRequest)).getRecoverTrashResponse();

return recoverResponse.getResponse();
//recoverTrash is deprecated
throw new UnsupportedOperationException();
}

@Override
Expand Down
34 changes: 19 additions & 15 deletions hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,8 +232,8 @@ message OMRequest {
optional UpdateGetS3SecretRequest updateGetS3SecretRequest = 82;
optional ListMultipartUploadsRequest listMultipartUploadsRequest = 83;

optional ListTrashRequest listTrashRequest = 91;
optional RecoverTrashRequest RecoverTrashRequest = 92;
optional ListTrashRequest listTrashRequest = 91 [deprecated = true];
optional RecoverTrashRequest RecoverTrashRequest = 92 [deprecated = true];

optional RevokeS3SecretRequest RevokeS3SecretRequest = 93;

Expand Down Expand Up @@ -360,8 +360,8 @@ message OMResponse {

optional ListMultipartUploadsResponse listMultipartUploadsResponse = 82;

optional ListTrashResponse listTrashResponse = 91;
optional RecoverTrashResponse RecoverTrashResponse = 92;
optional ListTrashResponse listTrashResponse = 91 [deprecated = true];
optional RecoverTrashResponse RecoverTrashResponse = 92 [deprecated = true];
optional PurgePathsResponse purgePathsResponse = 93 [deprecated = true];
optional PurgeDirectoriesResponse purgeDirectoriesResponse = 108;

Expand Down Expand Up @@ -547,15 +547,17 @@ enum Status {
in the deleted table on Ozone Manager.
*/
message ListTrashRequest {
required string volumeName = 1;
required string bucketName = 2;
optional string startKeyName = 3;
optional string keyPrefix = 4;
optional int32 maxKeys = 5;
// option deprecated = true;
required string volumeName = 1 [deprecated = true];
required string bucketName = 2 [deprecated = true];
optional string startKeyName = 3 [deprecated = true];
optional string keyPrefix = 4 [deprecated = true];
optional int32 maxKeys = 5 [deprecated = true];
}

message ListTrashResponse {
repeated RepeatedKeyInfo deletedKeys = 1;
// option deprecated = true;
repeated RepeatedKeyInfo deletedKeys = 1 [deprecated = true];
}

/**
Expand All @@ -564,14 +566,16 @@ message ListTrashResponse {
*/

message RecoverTrashRequest {
required string volumeName = 1;
required string bucketName = 2;
required string keyName = 3;
required string destinationBucket = 4;
// option deprecated = true;
required string volumeName = 1 [deprecated = true];
required string bucketName = 2 [deprecated = true];
required string keyName = 3 [deprecated = true];
required string destinationBucket = 4 [deprecated = true];
}

message RecoverTrashResponse {
required bool response = 1;
// option deprecated = true;
required bool response = 1 [deprecated = true];
}

message VolumeInfo {
Expand Down
6 changes: 6 additions & 0 deletions ...p-ozone/interface-storage/src/main/java/org/apache/hadoop/ozone/om/OMMetadataManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,10 @@ ListKeysResult listKeys(String volumeName,
* the cluster level set by admins.
* @return The list of keys that are deleted from the deleted table.
* @throws IOException
* @deprecated HDDS-3367 implements a new way to store trashed files.
* Please avoid using this as it will be removed in a future release
*/
@Deprecated
List<RepeatedOmKeyInfo> listTrash(String volumeName, String bucketName,
String startKeyName, String keyPrefix, int maxKeys) throws IOException;

Expand Down Expand Up @@ -312,7 +315,10 @@ ListSnapshotResponse listSnapshot(
* @param keyName - The key user want to recover.
* @param destinationBucket - The bucket user want to recover to.
* @return The result of recovering operation is success or not.
* @deprecated HDDS-3367 implements a new way to store trashed files.
* Please avoid using this as it will be removed in a future release
*/
@Deprecated
boolean recoverTrash(String volumeName, String bucketName,
String keyName, String destinationBucket) throws IOException;

Expand Down
3 changes: 3 additions & 0 deletions hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,10 @@ ListKeysResult listKeys(String volumeName, String bucketName, String startKey,
* the cluster level set by admins.
* @return The list of keys that are deleted from the deleted table.
* @throws IOException
* @deprecated HDDS-3367 implements a new way to store trashed files.
* Please avoid using this as it will be removed in a future release
*/
@Deprecated
List<RepeatedOmKeyInfo> listTrash(String volumeName, String bucketName,
String startKeyName, String keyPrefix, int maxKeys) throws IOException;

Expand Down
11 changes: 2 additions & 9 deletions hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -664,15 +664,8 @@ public ListKeysResult listKeys(String volumeName, String bucketName,
public List<RepeatedOmKeyInfo> listTrash(String volumeName,
String bucketName, String startKeyName, String keyPrefix,
int maxKeys) throws IOException {

Preconditions.checkNotNull(volumeName);
Preconditions.checkNotNull(bucketName);
Preconditions.checkArgument(maxKeys <= listTrashKeysMax,
"The max keys limit specified is not less than the cluster " +
"allowed maximum limit.");

return metadataManager.listTrash(volumeName, bucketName,
startKeyName, keyPrefix, maxKeys);
// listTrash is deprecated
throw new UnsupportedOperationException();
}

@Override
Expand Down
14 changes: 4 additions & 10 deletions ...p-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1389,9 +1389,8 @@ public ListKeysResult listKeys(String volumeName, String bucketName,
@Override
public List<RepeatedOmKeyInfo> listTrash(String volumeName, String bucketName,
String startKeyName, String keyPrefix, int maxKeys) throws IOException {

List<RepeatedOmKeyInfo> deletedKeys = new ArrayList<>();
return deletedKeys;
// listTrash is deprecated
throw new UnsupportedOperationException();
}

@Override
Expand Down Expand Up @@ -1473,13 +1472,8 @@ public ListSnapshotResponse listSnapshot(
@Override
public boolean recoverTrash(String volumeName, String bucketName,
String keyName, String destinationBucket) throws IOException {

/* TODO: HDDS-2425 and HDDS-2426
core logic stub would be added in later patch.
*/

boolean recoverOperation = true;
return recoverOperation;
//recoverTrash is deprecated
throw new UnsupportedOperationException();
}

/**
Expand Down
29 changes: 2 additions & 27 deletions hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2946,33 +2946,8 @@ public ListKeysLightResult listKeysLight(String volumeName,
public List<RepeatedOmKeyInfo> listTrash(String volumeName,
String bucketName, String startKeyName, String keyPrefix, int maxKeys)
throws IOException {
boolean auditSuccess = true;
Comment thread
spacemonkd marked this conversation as resolved.
Map<String, String> auditMap = buildAuditMap(volumeName);
auditMap.put(OzoneConsts.BUCKET, bucketName);
auditMap.put(OzoneConsts.START_KEY, startKeyName);
auditMap.put(OzoneConsts.KEY_PREFIX, keyPrefix);
auditMap.put(OzoneConsts.MAX_KEYS, String.valueOf(maxKeys));
try {
if (isAclEnabled) {
omMetadataReader.checkAcls(ResourceType.BUCKET,
StoreType.OZONE, ACLType.LIST,
volumeName, bucketName, keyPrefix);
}
metrics.incNumTrashKeyLists();
Comment thread
spacemonkd marked this conversation as resolved.
return keyManager.listTrash(volumeName, bucketName,
startKeyName, keyPrefix, maxKeys);
} catch (IOException ex) {
metrics.incNumTrashKeyListFails();
auditSuccess = false;
AUDIT.logReadFailure(buildAuditMessageForFailure(OMAction.LIST_TRASH,
auditMap, ex));
throw ex;
} finally {
if (auditSuccess) {
AUDIT.logReadSuccess(buildAuditMessageForSuccess(OMAction.LIST_TRASH,
auditMap));
}
}
// listTrash is deprecated
throw new UnsupportedOperationException();
}

@Override
Expand Down
Loading