[HUDI-3993] Replacing UDF in Bulk Insert w/ RDD transformation #5470

alexeykudinkin · 2022-04-30T00:16:04Z

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request

Replacing UDF in Bulk Insert w/ RDD transformation.

Brief change log

TBD

Verify this pull request

This pull request is already covered by existing tests, such as (please describe tests).
This change added tests and can be verified as follows:

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

...-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java

nsivabalan · 2022-05-04T14:48:43Z

...asource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala

+      "Key-generator class name is required")
+
+    val prependedRdd: RDD[InternalRow] =
+      df.queryExecution.toRdd.mapPartitions { iter =>


does this toRdd incur any perf hit? if yes, can you do some benchmark w/ udfs based vs this and report what do you see. Alternatively you can also, run a benchmark w/ raw parquet write w/ bulk insert row writer non partitioned and no sort mode and ensure we see comparable nos w/ this patch.

I am seeing some perf hit w/ this code change. will wait to sync up with Alexey on this.

toRdd is how Datasets are getting executed in Spark eventually. There's no perf hit by using it.

This PR should only be applied in conjunction w/ this one #5523

Can you confirm this issue has been resolved in Master branch

I think issue @nsivabalan is referring to is the fact that this PR shouldn't be measured in isolation but only together with #5523 (which is landed as well)

nsivabalan · 2022-05-04T14:52:51Z

...asource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala

+    val updatedDF = HoodieUnsafeRDDUtils.createDataFrame(df.sparkSession, prependedRdd, updatedSchema)
+
+    if (!populateMetaFields) {
+      updatedDF


probably it was a gap before. but we may not have to support dropPartitionColumns even with virtual key code path. can we fix that please

nsivabalan · 2022-05-04T15:20:41Z

...asource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala

+  }
+
+  private def dedupeRows(df: DataFrame, preCombineFieldRef: String, isGlobalIndex: Boolean): DataFrame = {
+    val recordKeyMetaFieldOrd = df.schema.fieldIndex(HoodieRecord.RECORD_KEY_METADATA_FIELD)


this may also need to be fixed for virtual key path, or we can call it out that its not supported for now. even prior to this patch, we did not have support for de-duping in virtual key flow in row writer.

Yes, virtual keys de-duping isn't supported currently

nsivabalan · 2022-05-04T15:24:23Z

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/BuiltinKeyGenerator.java

+  @Override
+  @PublicAPIMethod(maturity = ApiMaturityLevel.EVOLVING)
+  public String getRecordKey(InternalRow internalRow, StructType schema) {
+    try {


with the changes in my other patch, we don't need to deserialize to Row to fetch the value. Can you take a look

These are temporary changes, that are addressed in #5523

are resolution on this? Did you end up backing out the temporary changes

Correct. These are revisited

nsivabalan · 2022-05-04T15:30:45Z

...-client/hudi-spark-client/src/test/scala/org/apache/spark/sql/TestHoodieUnsafeRowUtils.scala

+
+  @Test
+  def testGetNestedRowValue(): Unit = {
+    val schema = StructType(Seq(


minor: if you intend to use the same schema for many tests, we can make this an instance variable and not declare in every test. its immutable and so we could even make static final.

nsivabalan · 2022-05-04T15:36:06Z

...-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java

+      // NOTE: Helper keeps track of [[lastKnownPartitionPath]] as [[UTF8String]] to avoid
+      //       conversion from Catalyst internal representation into a [[String]]
+      partitionPath = row.getString(
+          HoodieRecord.HOODIE_META_COLUMNS_NAME_TO_POS.get(HoodieRecord.PARTITION_PATH_METADATA_FIELD));


we can directly use 3 here instead of looking up in hashmap

nsivabalan · 2022-05-04T18:37:49Z

...lient/hudi-spark-client/src/main/java/org/apache/hudi/keygen/NonpartitionedKeyGenerator.java

  }

+  @Override
+  public String getRecordKey(Row row) {


shouldn't we migrate this fix to SimpleKeyGen, if you feel existing impl in SimpleKeyGen could be fixed? why making changes just to NonPartitionedKeyGen only.

This is addressed in #5523

...nt/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowCreateHandle.java

nsivabalan · 2022-05-10T20:41:20Z

...nt/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowCreateHandle.java

+
+    String writeToken = getWriteToken(taskPartitionId, taskId, taskEpochId);
+    String fileName = FSUtils.makeDataFileName(instantTime, writeToken, fileId,
+        table.getMetaClient().getTableConfig().getBaseFileFormat().getFileExtension());


table.getBaseFileExtension()

yihua

A few notable improvements and refactoring are from this PR. @alexeykudinkin have you benchmarked the performance from this set of changes?

yihua · 2022-07-18T23:24:45Z

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/model/HoodieInternalRow.java

+    };
+
    this.row = row;
+    this.containsMetaFields = containsMetaFields;


if containsMetaFields is false, should the length of metaFields be 0?

There's some confusion: containsMetaFields relates to whether inner row contains the meta-fields itself. However, HIR will always override the meta-fields by overlaying on top of whatever the source row contains (this is necessary b/c UnsafeRow can't be updated)

Am gonna update the docs to make it more clear

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/model/HoodieInternalRow.java

yihua · 2022-07-18T23:30:47Z

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/model/HoodieInternalRow.java

+  private int rebaseOrdinal(int ordinal) {
+    // NOTE: In cases when source row does not contain meta fields, we will have to
+    //       rebase ordinal onto its indexes
+    return containsMetaFields ? ordinal : ordinal - metaFields.length;


If the source row does not contain meta fields (containsMetaFields is false), and assuming metaFields is empty, the logic here for adjusting the ordinal is not necessary?

Please check my comments above -- we always overlay meta-fields, since we need them to be mutable (they're being updated dynamically in writer)

yihua · 2022-07-18T23:33:15Z

...i-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieInternalRowFileWriter.java

   * @throws IOException on any exception while writing.
   */
-  void writeRow(String key, InternalRow row) throws IOException;
+  void writeRow(UTF8String key, InternalRow row) throws IOException;


Is the usage of UTF8String type for performance?

Correct -- to avoid conversion b/w String and UTF8String

yihua · 2022-07-18T23:38:13Z

hudi-common/src/main/java/org/apache/hudi/common/util/HoodieTimer.java

+  public HoodieTimer(boolean shouldStart) {
+    if (shouldStart) {
+      startTimer();
+    }
+  }


This is not obvious (timer = new HoodieTime(true)) compared to exsiting way (timer = new HoodieTimer().startTimer()). Should we revert the change?

Old semantic is still preserved: it works as it have been, and just adds new way when you don't need to invoke startTimer explicitly

Understood. I'm saying that timer = new HoodieTimer().startTimer() looks obvious for starting the timer instead of looking into what the boolean represents.

Fair enough

yihua · 2022-07-19T00:36:20Z

...-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java

-    } else {
-      minRecordKey = recordKey;
+  public void add(UTF8String recordKey) {
+    this.bloomFilter.add(recordKey.getBytes());


So the content or the byte array of the String and UTF8String instances should be the same here, right? So that the bloom filter lookup is not affected based on the String key.

Bloom filter always ingest UTF8 (Java by default encodes in UTF16)

Sg, want to make sure there is no gap between Spark UTF8String and UTF8 encoding in Java, since this is going to affect the Bloom Index.

BloomFilter add does . So we seem to be fine. it's good to trust-but-verify once though that recordKey.getBytes() is equal to string.getBytes(StandardCharsets.UTF_8). @alexeykudinkin you probably checked that during development?

yihua · 2022-07-19T00:40:49Z

...-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java

+    if (keyGeneratorOpt.isPresent() && keyGeneratorOpt.get() instanceof SimpleKeyGenerator) {
+      this.simpleKeyGen = true;
+      this.simplePartitionFieldIndex = (Integer) structType.getFieldIndex(keyGeneratorOpt.get().getPartitionPathFields().get(0)).get();
+      this.simplePartitionFieldDataType = structType.fields()[simplePartitionFieldIndex].dataType();
+    } else {
+      this.simpleKeyGen = false;
+      this.simplePartitionFieldIndex = -1;
+      this.simplePartitionFieldDataType = null;


The question is out-of-scope for this PR, but why do we need special-case handling for the simple key generator here?

We actually do not (it was done for perf reasons before). This is addressed in #5523

yihua · 2022-07-19T00:45:39Z

...asource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala

+
+import scala.collection.JavaConverters.asScalaBufferConverter
+
+object HoodieDatasetBulkInsertHelper extends Logging {


Is this refactored based on hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/HoodieDatasetBulkInsertHelper.java?

Correct. It's a simplified version converted into Scala (to handle RDDs)

Trying to understand this better. Why did this need to be in scala?

To avoid back-n-forth Java/Scala conversions

yihua · 2022-07-19T00:47:56Z

...asource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala

+        }
+      }
+
+    val metaFields = Seq(


nit: use HoodieRecord.HOODIE_META_COLUMNS and transformation to form the meta fields?

Good call. The reason i didn't do it in the first place was b/c order is critical here, and even though we're using a list, i didn't want this constraint to be instead obscured in other class (where order actually might not matter at all)

That's fair. But shouldn't all the places use the same order so that we can maintain the order in one place like HoodieRecord.HOODIE_META_COLUMNS to avoid discrepancy?

Fair point. Problem is that ordering only matters in a handful of contexts (compared to all usages of this list) and it harder to justify why ordering matters when you're looking at just the HoodieRecord class

OK, for this one we can keep it. Sth to think of for consistency to avoid bugs.

We should follow up and consolidate into one list in HoodieRecord. +1. Unless the other usages break with or . different ordering, I don't see any reason why we won't

yihua · 2022-07-19T00:55:12Z

...asource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala

+        }
+        // NOTE: It's critical whenever we keep the reference to the row, to make a copy
+        //       since Spark might be providing us with a mutable copy (updated during the iteration)
+        (rowKey, row.copy())


is row.copy() needed here for reduceByKey?

We only can get away w/o copying when we do one-pass (streaming-like) processing. If at any point we need to hold a reference to it -- we will have to make a copy (it's gonna fail otherwise)

OK, later on, I think we need to revisit this pattern of copy() in the DAG to make sure they are needed. Could you create a ticket?

Note, what i'm saying is only applicable to InternalRow which don't copy by default and instead point into shared, mutable underlying buffer (actually holding what's been read)

What exact scenarios cause Spark to fail without copy. Could you please expand on that?

This exact code will fail if we remove copy, b/c often InternalRow is a mutable copy that Spark changes during iteration, which is safe when we access just the one under the pointer, but in the subsequent reduceByKey we access 2 rows at the same time

…rHelper`

…irectly

…line the code and avoid unnecessary conversions

Added more tests;

Tidying up;

- Removing incorrect exclusions - Adding "hadoop-hdfs" test dependency

…verride `InternalRow` methods; Fixed tests;

…w` to make sure Avro/Spark key-gens are in-sync; Added `RowKeyGenUtils`;

alexeykudinkin · 2022-07-20T00:39:22Z

@hudi-bot run azure

alexeykudinkin · 2022-07-20T20:50:10Z

@hudi-bot run azure

…unnecessary `DataFrame` > `RDD` conversions

yihua

LGTM

yihua · 2022-07-21T04:15:38Z

...asource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala

+        }
+        // NOTE: It's critical whenever we keep the reference to the row, to make a copy
+        //       since Spark might be providing us with a mutable copy (updated during the iteration)
+        (rowKey, row.copy())


hudi-bot · 2022-07-21T09:59:36Z

CI report:

b4573ac Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

vinothchandar

@alexeykudinkin could you please respond!follow up on the minor comments.

this PR by itself seems to incur RDD conversion. Will look into #5523 as pointed out in review responses.

vinothchandar · 2022-07-25T00:33:25Z

...-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java

-    } else {
-      minRecordKey = recordKey;
+  public void add(UTF8String recordKey) {
+    this.bloomFilter.add(recordKey.getBytes());


BloomFilter add does . So we seem to be fine. it's good to trust-but-verify once though that recordKey.getBytes() is equal to string.getBytes(StandardCharsets.UTF_8). @alexeykudinkin you probably checked that during development?

vinothchandar · 2022-07-25T00:34:25Z

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/BuiltinKeyGenerator.java

 import org.apache.spark.sql.catalyst.InternalRow;
 import org.apache.spark.sql.types.DataType;
 import org.apache.spark.sql.types.StructType;
+import scala.Function1;


don't really like scala imports in Java (becomes an issue - for us one day when we want to shrink scala spread in code). Any way we can avoid this

This is removed in #5523

vinothchandar · 2022-07-25T00:35:06Z

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/BuiltinKeyGenerator.java

+  @Override
+  @PublicAPIMethod(maturity = ApiMaturityLevel.EVOLVING)
+  public String getRecordKey(InternalRow internalRow, StructType schema) {
+    try {


are resolution on this? Did you end up backing out the temporary changes

vinothchandar · 2022-07-25T11:20:17Z

...asource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala

+
+import scala.collection.JavaConverters.asScalaBufferConverter
+
+object HoodieDatasetBulkInsertHelper extends Logging {


Trying to understand this better. Why did this need to be in scala?

vinothchandar · 2022-07-25T11:25:18Z

...asource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala

+        }
+      }
+
+    val metaFields = Seq(


We should follow up and consolidate into one list in HoodieRecord. +1. Unless the other usages break with or . different ordering, I don't see any reason why we won't

vinothchandar · 2022-07-25T11:26:49Z

...asource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala

+      "Key-generator class name is required")
+
+    val prependedRdd: RDD[InternalRow] =
+      df.queryExecution.toRdd.mapPartitions { iter =>


Can you confirm this issue has been resolved in Master branch

vinothchandar · 2022-07-25T11:30:52Z

...asource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala

+        }
+        // NOTE: It's critical whenever we keep the reference to the row, to make a copy
+        //       since Spark might be providing us with a mutable copy (updated during the iteration)
+        (rowKey, row.copy())


What exact scenarios cause Spark to fail without copy. Could you please expand on that?

vinothchandar · 2022-07-25T14:01:02Z

I need to convince myself of the RDD conversion in place. So this is marked - "major concerns" until then

vinothchandar · 2022-07-25T16:50:47Z

@alexeykudinkin Want to get my understanding straight, as well make sure we have an explanation for how these factors play out with the new changes.

The original row writer impl originated in overhead from doing df.queryExecution.toRdd here, done before Avro record conversion. We traced this into a code in Spark, that makes an additional pass (almost) to materialize the Rows with a schema to be used by the iterator.
I see that in 0.11.1 we were just processing the dataframe as DataSet<Row> and ergo the use of UDFs for the other functionality. This is what's been fixed in 0.12 now.

I want to understand how we are avoiding the RDD conversion costs, in the current approach? This cost becomes obvious when you do records with large number of columns (due to overhead per record)

alexeykudinkin · 2022-07-25T17:06:37Z

@vinothchandar TL;DR is the difference b/w Row and InternalRow:

When you do df.rdd you invoke deserializer which will deserialize internal binary representation (UnsafeRow) into a Row holding Java native types (it also holds the schema)
df.queryExecution.toRdd is an internal API that returns you an RDD of InternalRows avoiding such conversion (that’s the primary reason for introduction of many utilities in HoodieUnsafeUtils to be able to access private Spark APIs)

alexeykudinkin commented Apr 30, 2022

View reviewed changes

...-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java Show resolved Hide resolved

alexeykudinkin force-pushed the ak/spk-blk-ins-opt branch from 1ba983d to b14a25c Compare April 30, 2022 00:58

alexeykudinkin changed the title ~~[HUDI-3993][Perf] Replacing UDF in Bulk Insert w/ RDD transformation~~ [WIP][HUDI-3993][Perf] Replacing UDF in Bulk Insert w/ RDD transformation Apr 30, 2022

alexeykudinkin force-pushed the ak/spk-blk-ins-opt branch from b14a25c to 1489d37 Compare April 30, 2022 04:40

yihua added area:performance Performance optimizations writer-core labels May 3, 2022

nsivabalan reviewed May 4, 2022

View reviewed changes

alexeykudinkin force-pushed the ak/spk-blk-ins-opt branch 2 times, most recently from dba2edf to 9c7e7ea Compare May 5, 2022 02:41

alexeykudinkin changed the title ~~[WIP][HUDI-3993][Perf] Replacing UDF in Bulk Insert w/ RDD transformation~~ [HUDI-3993][Perf][Stacked on 5497] Replacing UDF in Bulk Insert w/ RDD transformation May 7, 2022

alexeykudinkin changed the title ~~[HUDI-3993][Perf][Stacked on 5497] Replacing UDF in Bulk Insert w/ RDD transformation~~ [HUDI-3993][Stacked on 5497] Replacing UDF in Bulk Insert w/ RDD transformation May 7, 2022

nsivabalan added the priority:critical Production degraded; pipelines stalled label May 10, 2022

nsivabalan reviewed May 10, 2022

View reviewed changes

nsivabalan force-pushed the ak/spk-blk-ins-opt branch from 9c7e7ea to dc91261 Compare May 11, 2022 03:32

codope added priority:blocker Production down; release blocker and removed priority:critical Production degraded; pipelines stalled labels Jul 9, 2022

codope assigned codope and yihua Jul 11, 2022

TengHuo mentioned this pull request Jul 12, 2022

[SUPPORT]'hoodie.datasource.write.hive_style_partitioning':'true' does not take effect in hudi-0.11.1 & spark 3.2.1 #6070

Closed

alexeykudinkin changed the title ~~[HUDI-3993][Stacked on 5497] Replacing UDF in Bulk Insert w/ RDD transformation~~ [HUDI-3993] Replacing UDF in Bulk Insert w/ RDD transformation Jul 14, 2022

alexeykudinkin force-pushed the ak/spk-blk-ins-opt branch 2 times, most recently from 0600a70 to 6dfc22b Compare July 15, 2022 20:49

yihua reviewed Jul 19, 2022

View reviewed changes

Alexey Kudinkin added 7 commits July 18, 2022 21:27

Minor optimizations

514ac5d

Cleaned up BulkInsertDataInternalWriterHelper init seq

a74d0b3

Cleaning up partition path extraction in `BulkInsertDataInternalWrite…

b577aa7

…rHelper`

Added getRecordKey to generate record key ingesting InternalRow d…

b7a0c82

…irectly

Revisited HoodieDatasetBulkInsertHelper to avoid using UDFs, stream…

ace872c

…line the code and avoid unnecessary conversions

Fixed tests

9dd0c7a

Fixing typo

fd8bdc9

Alexey Kudinkin added 10 commits July 18, 2022 21:27

Tidying up;

26677d9

Added more tests;

Avoding unnecessary hash-map lookups;

ce7321b

Tidying up;

Fixing compilation (restoring changes)

5db8ac0

Fixing NPE

34ef154

Fixing "hudi-spark3" POM:

84c6842

- Removing incorrect exclusions - Adding "hadoop-hdfs" test dependency

Fixing copy-paste

4a1fa79

Fixed pseudo-random gen to generate non-repeating sequences

116e752

Fixed ComplexKeyGenerator, GlobalDeleteKeyGenerator to properly o…

bb3ffa8

…verride `InternalRow` methods; Fixed tests;

Fixed deserialization of some logical types when handling `InternalRo…

44b1f08

…w` to make sure Avro/Spark key-gens are in-sync; Added `RowKeyGenUtils`;

Fixing compilation

34b0127

alexeykudinkin force-pushed the ak/spk-blk-ins-opt branch from d02c06e to 34b0127 Compare July 19, 2022 04:31

Alexey Kudinkin added 2 commits July 19, 2022 10:48

Fixing tests

e803911

Replace copy w/ clone to avoid unnecessary underlying buffer copying

8f60e6f

Refactored HoodieDatasetBulkInsertHelper to dedupe RDDs to avoid …

b4573ac

…unnecessary `DataFrame` > `RDD` conversions

alexeykudinkin force-pushed the ak/spk-blk-ins-opt branch from 505ee48 to b4573ac Compare July 20, 2022 21:11

alexeykudinkin mentioned this pull request Jul 21, 2022

[HUDI-4039] Make sure all builtin KeyGenerators properly implement Spark specific APIs #5523

Merged

5 tasks

yihua approved these changes Jul 21, 2022

View reviewed changes

apache deleted a comment from hudi-bot Jul 21, 2022

yihua merged commit a33bdd3 into apache:master Jul 21, 2022

codope mentioned this pull request Jul 25, 2022

[HUDI-4374][WIP] Support row writing path for bulk insert via streaming sink #6099

Closed

5 tasks

vinothchandar reviewed Jul 25, 2022

View reviewed changes


		import scala.collection.JavaConverters.asScalaBufferConverter

		object HoodieDatasetBulkInsertHelper extends Logging {

[HUDI-3993] Replacing UDF in Bulk Insert w/ RDD transformation #5470

[HUDI-3993] Replacing UDF in Bulk Insert w/ RDD transformation #5470

Uh oh!

Conversation

alexeykudinkin commented Apr 30, 2022

Tips

What is the purpose of the pull request

Brief change log

Verify this pull request

Committer checklist

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan May 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

nsivabalan May 4, 2022 •

edited

Loading