Skip to content

[HUDI-4516] fix Task not serializable error when run HoodieCleaner after one failure #6265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nsivabalan merged 2 commits into from
Aug 2, 2022
Merged

[HUDI-4516] fix Task not serializable error when run HoodieCleaner after one failure #6265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0de7657
[HUDI-4516] fix task not serializable error
Aug 1, 2022
9737231
[HUDI-4516] minor update
Aug 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion hudi-common/src/main/java/org/apache/hudi/common/util/DefaultSizeEstimator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,14 @@

package org.apache.hudi.common.util;

import java.io.Serializable;

/**
* Default implementation of size-estimator that uses Twitter's ObjectSizeCalculator.
*
* @param <T>
*/
public class DefaultSizeEstimator<T> implements SizeEstimator<T> {
public class DefaultSizeEstimator<T> implements SizeEstimator<T>, Serializable {

@Override
public long sizeEstimate(T t) {
Expand Down
4 changes: 3 additions & 1 deletion hudi-common/src/main/java/org/apache/hudi/common/util/HoodieRecordSizeEstimator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,14 @@
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;

import java.io.Serializable;

/**
* Size Estimator for Hoodie record payload.
*
* @param <T>
*/
public class HoodieRecordSizeEstimator<T extends HoodieRecordPayload> implements SizeEstimator<HoodieRecord<T>> {
public class HoodieRecordSizeEstimator<T extends HoodieRecordPayload> implements SizeEstimator<HoodieRecord<T>>, Serializable {

private static final Logger LOG = LogManager.getLogger(HoodieRecordSizeEstimator.class);

Expand Down
2 changes: 1 addition & 1 deletion hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
* map may occupy more memory than is available, resulting in OOM. However, if the spill threshold is too low, we spill
* frequently and incur unnecessary disk writes.
*/
public class ExternalSpillableMap<T extends Serializable, R extends Serializable> implements Map<T, R> {
public class ExternalSpillableMap<T extends Serializable, R extends Serializable> implements Map<T, R>, Serializable {

// Find the actual estimated payload size after inserting N records
private static final int NUMBER_OF_RECORDS_TO_ESTIMATE_PAYLOAD_SIZE = 100;
Expand Down
4 changes: 2 additions & 2 deletions hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBBasedMap.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,12 @@
/**
* A map's implementation based on RocksDB.
*/
public final class RocksDBBasedMap<K extends Serializable, R extends Serializable> implements Map<K, R> {
public final class RocksDBBasedMap<K extends Serializable, R extends Serializable> implements Map<K, R>, Serializable {

private static final String COL_FAMILY_NAME = "map_handle";

private final String rocksDbStoragePath;
private RocksDBDAO rocksDBDAO;
private transient RocksDBDAO rocksDBDAO;
private final String columnFamilyName;

public RocksDBBasedMap(String rocksDbStoragePath) {
Expand Down