diff --git a/docs/en/architecture/api-design/catalog-table.md b/docs/en/architecture/api-design/catalog-table.md new file mode 100644 index 000000000000..01702956a7a1 --- /dev/null +++ b/docs/en/architecture/api-design/catalog-table.md @@ -0,0 +1,677 @@ +--- +sidebar_position: 4 +title: CatalogTable and Metadata Management +--- + +# CatalogTable and Metadata Management + +## 1. Overview + +### 1.1 Problem Background + +Data integration requires explicit schema management: + +- **Schema Definition**: How to define and validate table schemas? +- **Schema Propagation**: How to pass schema through Source → Transform → Sink? +- **Schema Evolution**: How to handle runtime DDL changes (ADD/DROP columns)? +- **Type Mapping**: How to map types between different data sources? +- **Metadata Completeness**: How to capture complete table metadata (constraints, partitions)? + +### 1.2 Design Goals + +SeaTunnel's metadata management aims to: + +1. **Type Safety**: Explicit schema validation at job submission +2. **Completeness**: Capture all table metadata (columns, constraints, partitions, options) +3. **Evolution Support**: Handle runtime schema changes (DDL synchronization) +4. **Engine Independence**: Schema representation independent of execution engine +5. **Ease of Use**: Simple API for schema creation and transformation + +## 2. Core Concepts + +### 2.1 CatalogTable + +Complete representation of a table with all metadata. + +```java +public class CatalogTable implements Serializable { + // Table identifier + private final TableIdentifier tableId; + + // Schema definition + private final TableSchema tableSchema; + + // Table options (connector-specific configuration) + private final Map options; + + // Partition keys + private final List partitionKeys; + + // Comment + private final String comment; + + // Catalog name + private final String catalogName; +} +``` + +**Key Components**: +- `TableIdentifier`: Unique table identity (`catalog.database[.schema].table`) +- `TableSchema`: Schema with columns, primary key, constraints +- `options`: Connector-specific settings (e.g., Kafka topic, JDBC table name) +- `partitionKeys`: Partition columns for partitioned tables + +### 2.2 TableSchema + +Schema definition with columns and constraints. + +```java +public class TableSchema implements Serializable { + // Column definitions + private final List columns; + + // Primary key + private final PrimaryKey primaryKey; + + // Unique/foreign key constraints + private final List constraintKeys; +} +``` + +### 2.3 Column + +Column definition with type and constraints. + +```java +public class Column implements Serializable { + private final String name; + private final SeaTunnelDataType dataType; + private final String comment; + + // Column options + private final Map options; + + // Constraints + private final boolean nullable; + private final Object defaultValue; +} +``` + +### 2.4 SeaTunnelDataType + +Unified type system across connectors. + +**Basic Types**: +```java +// Numeric +DataTypes.TINYINT() +DataTypes.SMALLINT() +DataTypes.INT() +DataTypes.BIGINT() +DataTypes.FLOAT() +DataTypes.DOUBLE() +DataTypes.DECIMAL(precision, scale) + +// String +DataTypes.STRING() +DataTypes.CHAR(length) +DataTypes.VARCHAR(length) + +// Binary +DataTypes.BYTES() + +// Date/Time +DataTypes.DATE() +DataTypes.TIME() +DataTypes.TIMESTAMP() + +// Boolean +DataTypes.BOOLEAN() +``` + +**Complex Types**: +```java +// Array +DataTypes.ARRAY(elementType) + +// Map +DataTypes.MAP(keyType, valueType) + +// Row (Struct) +DataTypes.ROW(fields) +``` + +## 3. Schema Creation + +### 3.1 Builder Pattern + +```java +CatalogTable catalogTable = CatalogTable.of( + TableIdentifier.of("my_catalog", "my_db", "my_table"), + TableSchema.builder() + .column("id", DataTypes.BIGINT()) + .column("name", DataTypes.STRING()) + .column("age", DataTypes.INT()) + .column("created_at", DataTypes.TIMESTAMP()) + .primaryKey("id") + .build(), + Map.of("connector", "jdbc"), + Collections.emptyList(), // No partitions + "User table" +); +``` + +### 3.2 Column Builder + +```java +Column column = Column.builder() + .name("user_id") + .dataType(DataTypes.BIGINT()) + .nullable(false) + .defaultValue(0L) + .comment("User identifier") + .build(); +``` + +### 3.3 Primary Key and Constraints + +```java +TableSchema schema = TableSchema.builder() + .column("id", DataTypes.BIGINT()) + .column("email", DataTypes.STRING()) + .column("username", DataTypes.STRING()) + + // Primary key + .primaryKey("id") + + // Unique constraint + .constraint(ConstraintKey.of( + ConstraintKey.ConstraintType.UNIQUE_KEY, + "uk_email", + Arrays.asList( + ConstraintKey.ConstraintKeyColumn.of("email", null) + ) + )) + + .build(); +``` + +## 4. Schema Propagation + +### 4.1 Source → Transform → Sink Flow + +``` +┌──────────────┐ +│ Source │ +│ │ +│ produces │ +│ CatalogTable │ +└──────┬───────┘ + │ + ▼ (Input Schema) +┌──────────────┐ +│ Transform │ +│ │ +│ modifies │ +│ CatalogTable │ +└──────┬───────┘ + │ + ▼ (Output Schema) +┌──────────────┐ +│ Sink │ +│ │ +│ validates │ +│ CatalogTable │ +└──────────────┘ +``` + +### 4.2 Source Schema Production + +```java +public class JdbcSource implements SeaTunnelSource<...> { + @Override + public List getProducedCatalogTables() { + // Read schema from database metadata + DatabaseMetaData metaData = connection.getMetaData(); + ResultSet columns = metaData.getColumns(null, schema, table, null); + String database = "..."; + + // Build schema + TableSchema.Builder builder = TableSchema.builder(); + while (columns.next()) { + String columnName = columns.getString("COLUMN_NAME"); + int jdbcType = columns.getInt("DATA_TYPE"); + SeaTunnelDataType type = JdbcTypeConverter.convert(jdbcType); + + builder.column(columnName, type); + } + + return Collections.singletonList( + CatalogTable.of( + TableIdentifier.of(catalog, database, schema, table), + builder.build() + ) + ); + } +} +``` + +### 4.3 Transform Schema Transformation + +```java +public class SqlTransform implements SeaTunnelTransform { + @Override + public CatalogTable getProducedCatalogTable() { + CatalogTable inputTable = getInputCatalogTable(); + + // Parse SQL to infer output schema + // Example: SELECT id, UPPER(name) as name_upper, age FROM input + TableSchema outputSchema = TableSchema.builder() + .column("id", inputTable.getColumn("id").getDataType()) + .column("name_upper", DataTypes.STRING()) // Transformed + .column("age", inputTable.getColumn("age").getDataType()) + .build(); + + return inputTable.copy(outputSchema); + } +} +``` + +### 4.4 Sink Schema Validation + +```java +public class JdbcSink implements SeaTunnelSink<...> { + @Override + public Optional getWriteCatalogTable() { + // Validate input schema matches target table + CatalogTable inputTable = getInputCatalogTable(); + CatalogTable targetTable = readTargetTableSchema(); + + // Check column compatibility + for (Column inputColumn : inputTable.getColumns()) { + Column targetColumn = targetTable.getColumn(inputColumn.getName()); + if (targetColumn == null) { + throw new SchemaException("Column not found: " + inputColumn.getName()); + } + + if (!isCompatible(inputColumn.getDataType(), targetColumn.getDataType())) { + throw new SchemaException("Incompatible types for " + inputColumn.getName()); + } + } + + return Optional.of(targetTable); + } +} +``` + +## 5. Schema Evolution + +### 5.1 SchemaChangeEvent + +Represents DDL changes captured by CDC sources. + +```java +public abstract class SchemaChangeEvent implements Serializable { + private final TableIdentifier tableId; +} + +public class AlterTableAddColumnEvent extends SchemaChangeEvent { + private final Column column; +} + +public class AlterTableDropColumnEvent extends SchemaChangeEvent { + private final String columnName; +} + +public class AlterTableModifyColumnEvent extends SchemaChangeEvent { + private final Column column; +} +``` + +### 5.2 CDC Source Schema Evolution + +```java +public class MysqlCDCSource { + private void handleDDL(String ddl) { + // Parse DDL statement + if (ddl.contains("ADD COLUMN")) { + Column newColumn = parseDDL(ddl); + + // Create schema change event + SchemaChangeEvent event = new AlterTableAddColumnEvent( + tableId, + newColumn + ); + + // Emit event downstream + collector.collect(event); + } + } +} +``` + +### 5.3 Transform Schema Evolution Mapping + +```java +public class SqlTransform { + @Override + public SchemaChangeEvent mapSchemaChangeEvent(SchemaChangeEvent event) { + if (event instanceof AlterTableAddColumnEvent) { + AlterTableAddColumnEvent addEvent = (AlterTableAddColumnEvent) event; + + // Map column through transform logic + Column transformedColumn = transformColumn(addEvent.getColumn()); + + return new AlterTableAddColumnEvent( + event.getTableId(), + transformedColumn + ); + } + + return event; // Pass through + } +} +``` + +### 5.4 Sink Schema Evolution Application + +```java +public class JdbcSink { + private void applySchemaChange(SchemaChangeEvent event) { + if (event instanceof AlterTableAddColumnEvent) { + AlterTableAddColumnEvent addEvent = (AlterTableAddColumnEvent) event; + Column column = addEvent.getColumn(); + + // Generate DDL + String ddl = String.format( + "ALTER TABLE %s ADD COLUMN %s %s", + event.getTableId().getTableName(), + column.getName(), + toSqlType(column.getDataType()) + ); + + // Execute DDL + statement.execute(ddl); + + LOG.info("Applied schema change: {}", ddl); + } + } +} +``` + +## 6. Type Mapping + +### 6.1 JDBC Type Mapping + +```java +public class JdbcTypeConverter { + public static SeaTunnelDataType convert(int jdbcType) { + switch (jdbcType) { + case Types.TINYINT: + return DataTypes.TINYINT(); + case Types.SMALLINT: + return DataTypes.SMALLINT(); + case Types.INTEGER: + return DataTypes.INT(); + case Types.BIGINT: + return DataTypes.BIGINT(); + case Types.FLOAT: + case Types.REAL: + return DataTypes.FLOAT(); + case Types.DOUBLE: + return DataTypes.DOUBLE(); + case Types.DECIMAL: + case Types.NUMERIC: + return DataTypes.DECIMAL(precision, scale); + case Types.CHAR: + return DataTypes.CHAR(length); + case Types.VARCHAR: + return DataTypes.VARCHAR(length); + case Types.LONGVARCHAR: + return DataTypes.STRING(); + case Types.DATE: + return DataTypes.DATE(); + case Types.TIME: + return DataTypes.TIME(); + case Types.TIMESTAMP: + return DataTypes.TIMESTAMP(); + case Types.BOOLEAN: + return DataTypes.BOOLEAN(); + case Types.BINARY: + case Types.VARBINARY: + case Types.LONGVARBINARY: + return DataTypes.BYTES(); + default: + throw new UnsupportedTypeException("Unsupported JDBC type: " + jdbcType); + } + } +} +``` + +### 6.2 Kafka (Avro) Type Mapping + +```java +public class AvroTypeConverter { + public static SeaTunnelDataType convert(Schema avroSchema) { + switch (avroSchema.getType()) { + case INT: + return DataTypes.INT(); + case LONG: + return DataTypes.BIGINT(); + case FLOAT: + return DataTypes.FLOAT(); + case DOUBLE: + return DataTypes.DOUBLE(); + case BOOLEAN: + return DataTypes.BOOLEAN(); + case STRING: + return DataTypes.STRING(); + case BYTES: + return DataTypes.BYTES(); + case ARRAY: + return DataTypes.ARRAY(convert(avroSchema.getElementType())); + case MAP: + return DataTypes.MAP( + DataTypes.STRING(), + convert(avroSchema.getValueType()) + ); + case RECORD: + // Convert to ROW type + List fields = new ArrayList<>(); + for (Schema.Field field : avroSchema.getFields()) { + fields.add(new Column( + field.name(), + convert(field.schema()) + )); + } + return DataTypes.ROW(fields); + default: + throw new UnsupportedTypeException("Unsupported Avro type: " + avroSchema.getType()); + } + } +} +``` + +## 7. Partitioned Tables + +### 7.1 Partition Definition + +```java +CatalogTable catalogTable = CatalogTable.of( + tableId, + schema, + options, + Arrays.asList("year", "month", "day"), // Partition keys + comment +); +``` + +### 7.2 Partition-Aware Source + +```java +public class HiveSource { + @Override + public CatalogTable getProducedCatalogTable() { + // Read Hive table metadata + Table hiveTable = hiveMetastore.getTable(dbName, tableName); + + // Extract partition keys + List partitionKeys = hiveTable.getPartitionKeys().stream() + .map(FieldSchema::getName) + .collect(Collectors.toList()); + + return CatalogTable.of( + tableId, + schema, + options, + partitionKeys, + comment + ); + } +} +``` + +### 7.3 Partition-Aware Sink + +```java +public class IcebergSink { + private void write(SeaTunnelRow row, CatalogTable table) { + // Extract partition values from row + Map partitionValues = new HashMap<>(); + for (String partitionKey : table.getPartitionKeys()) { + int index = table.getSchema().indexOf(partitionKey); + partitionValues.put(partitionKey, row.getField(index)); + } + + // Write to correct partition + PartitionSpec spec = PartitionSpec.builderFor(schema) + .identity("year") + .identity("month") + .identity("day") + .build(); + + DataFile dataFile = writeToPartition(partitionValues, row); + icebergTable.newAppend().appendFile(dataFile).commit(); + } +} +``` + +## 8. Best Practices + +### 8.1 Schema Definition + +**Prefer Explicit Schema**: +```java +// ✅ GOOD: Explicit schema +TableSchema schema = TableSchema.builder() + .column("id", DataTypes.BIGINT()) + .column("name", DataTypes.STRING()) + .build(); + +// ❌ BAD: Implicit schema (relies on inference) +// Schema inferred from first row - risky! +``` + +**Use Appropriate Types**: +```java +// ✅ GOOD: Use specific types +.column("price", DataTypes.DECIMAL(10, 2)) +.column("created_at", DataTypes.TIMESTAMP()) + +// ❌ BAD: Overly generic types +.column("price", DataTypes.STRING()) // Should be DECIMAL +.column("created_at", DataTypes.STRING()) // Should be TIMESTAMP +``` + +### 8.2 Schema Validation + +**Validate Early**: +```java +// In Source +@Override +public void open() { + CatalogTable catalogTable = getProducedCatalogTables().get(0); + validateSchema(catalogTable); // Fail fast +} + +// In Sink +@Override +public void open() { + CatalogTable inputTable = getInputCatalogTable(); + CatalogTable targetTable = getWriteCatalogTable().orElseThrow(IllegalStateException::new); + validateCompatibility(inputTable, targetTable); // Fail fast +} +``` + +### 8.3 Type Compatibility + +**Type Widening (Safe)**: +```java +// INT → BIGINT (safe) +// FLOAT → DOUBLE (safe) +// VARCHAR(10) → VARCHAR(20) (safe) +``` + +**Type Narrowing (Unsafe)**: +```java +// BIGINT → INT (may overflow) +// DOUBLE → FLOAT (precision loss) +// VARCHAR(20) → VARCHAR(10) (truncation) +``` + +## 9. Configuration + +### 9.1 Schema Override + +```hocon +source { + JDBC { + url = "..." + query = "SELECT * FROM users" + + # Override inferred schema + schema { + fields { + id = "BIGINT" + name = "STRING" + age = "INT" + } + } + } +} +``` + +### 9.2 Schema Evolution Control + +```hocon +sink { + JDBC { + url = "..." + + # Schema evolution options + schema-evolution { + enabled = true + auto-create-table = true + auto-add-column = true + auto-drop-column = false # Dangerous! + } + } +} +``` + +## 10. Related Resources + +- [Source Architecture](source-architecture.md) +- [Sink Architecture](sink-architecture.md) +- [Schema Evolution](../../introduction/concepts/schema-evolution.md) +- [Schema Feature](../../introduction/concepts/schema-feature.md) + +## 11. References + +### Key Source Files + +- [CatalogTable.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/CatalogTable.java) +- [TableSchema.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/TableSchema.java) +- [Column.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/Column.java) +- [SeaTunnelDataType.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/table/type/SeaTunnelDataType.java) +- [SchemaChangeEvent.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/table/event/SchemaChangeEvent.java) diff --git a/docs/en/architecture/api-design/sink-architecture.md b/docs/en/architecture/api-design/sink-architecture.md new file mode 100644 index 000000000000..ac3a8597e86b --- /dev/null +++ b/docs/en/architecture/api-design/sink-architecture.md @@ -0,0 +1,1014 @@ +--- +sidebar_position: 3 +title: Sink Architecture +--- + +# Sink Architecture + +## 1. Overview + +### 1.1 Problem Background + +Writing data to external systems in distributed environments presents critical challenges: + +- **Exactly-Once Guarantee**: How to ensure each record is written exactly once, not zero or multiple times? +- **Transactional Consistency**: How to commit writes atomically across multiple parallel writers? +- **Fault Tolerance**: How to recover from failures without data loss or duplication? +- **Backpressure**: How to handle slow sinks without overwhelming the system? +- **Idempotency**: How to make retries safe? + +### 1.2 Design Goals + +SeaTunnel's Sink API aims to: + +1. **Provide Verifiable Consistency Semantics**: With checkpoint boundaries + 2PC, achieve exactly-once when the external sink supports transactional/idempotent commit +2. **Support Parallel Writes**: Scale throughput with multiple writer instances +3. **Enable Global Coordination**: Coordinate commits across distributed writers +4. **Ensure Fault Tolerance**: Recover from failures without data inconsistency +5. **Provide Flexibility**: Support various commit strategies (per-writer, aggregated, none) + +### 1.3 Applicable Scenarios + +- Transactional databases (JDBC with XA transactions) +- Message queues (Kafka with transactions) +- File systems (atomic file rename) +- Data lakes (Iceberg, Hudi, Delta Lake with table transactions) +- Search engines (Elasticsearch with versioning) + +## 2. Architecture Design + +### 2.1 Overall Architecture + +``` +┌────────────────────────────────────────────────────────────────┐ +│ TaskExecutionService (Worker Side) │ +│ │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ SinkWriter │ │ +│ │ │ │ +│ │ • Receive records from upstream │ │ +│ │ • Buffer and write data │ │ +│ │ • Produce commitInfo at checkpoint boundary │ │ +│ │ • Snapshot writer state │ │ +│ │ • Cleanup/rollback on failure (engine-dependent) │ │ +│ └──────────────────────────────────────────────────────┘ │ +│ │ │ +└────────────────────────────┼─────────────────────────────────────┘ + │ (CommitInfo) + ▼ +┌────────────────────────────────────────────────────────────────┐ +│ Coordinator Side (control plane, engine-dependent) │ +│ │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ SinkCommitter (Optional) │ │ +│ │ │ │ +│ │ • Receive commit infos from multiple writers │ │ +│ │ • Commit each writer's changes independently │ │ +│ │ • Retry failed commits │ │ +│ │ • Must be idempotent │ │ +│ └──────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ (Optional: AggregatedCommitInfo) │ +│ ▼ │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ SinkAggregatedCommitter │ │ +│ │ (Optional) │ │ +│ │ │ │ +│ │ • Aggregate commit infos from all writers │ │ +│ │ • Perform single global commit operation │ │ +│ │ • Single-threaded, global coordinator │ │ +│ └──────────────────────────────────────────────────────┘ │ +│ │ +└──────────────────────────────────────────────────────────────────┘ + │ + ▼ + External Data Sink + (Database / File / Message Queue) +``` + +### 2.2 Core Components + +#### SeaTunnelSink (Factory Interface) + +The top-level interface that serves as a factory for creating writers and committers. + +```java +public interface SeaTunnelSink + extends Serializable { + + /** + * Create SinkWriter (called on worker) + */ + SinkWriter createWriter(SinkWriter.Context context) + throws IOException; + + /** + * Restore SinkWriter from checkpoint (called on worker) + */ + default SinkWriter restoreWriter( + SinkWriter.Context context, + List states) throws IOException { + return createWriter(context); + } + + /** + * Serializer for writer state (optional). + */ + default Optional> getWriterStateSerializer() { + return Optional.empty(); + } + + /** + * Create SinkCommitter (optional, trigger location depends on execution engine) + */ + default Optional> createCommitter() throws IOException { + return Optional.empty(); + } + + /** + * Serializer for commit info (optional). + */ + default Optional> getCommitInfoSerializer() { + return Optional.empty(); + } + + /** + * Create SinkAggregatedCommitter (optional). + */ + default Optional> + createAggregatedCommitter() throws IOException { + return Optional.empty(); + } + + /** + * Serializer for aggregated commit info (optional). + */ + default Optional> getAggregatedCommitInfoSerializer() { + return Optional.empty(); + } + + /** + * Get input schema. + */ + default Optional getWriteCatalogTable() { + return Optional.empty(); + } +} +``` + +**Key Design Points**: +- Three-tier commit architecture: Writer → Committer → AggregatedCommitter +- Committer and AggregatedCommitter are optional (depends on sink requirements) +- Writer is always required (performs actual data writing) + +### 2.3 Interaction Flow + +#### Normal Write Flow (with Two-Phase Commit) + +```mermaid +sequenceDiagram + participant CP as CheckpointCoordinator + participant Writer1 as SinkWriter 1 + participant Writer2 as SinkWriter 2 + participant Committer as SinkCommitter + participant Sink as External Sink + + Writer1->>Writer1: write(record) + Writer2->>Writer2: write(record) + + CP->>Writer1: triggerBarrier(checkpointId) + CP->>Writer2: triggerBarrier(checkpointId) + + Writer1->>Writer1: prepareCommit(checkpointId) + Writer1->>CP: ack(commitInfo1) + Writer2->>Writer2: prepareCommit(checkpointId) + Writer2->>CP: ack(commitInfo2) + + CP->>CP: All writers acked + CP->>CP: Persist checkpoint + + CP->>Committer: commit([commitInfo1, commitInfo2]) + Committer->>Sink: Commit writer1 changes + Committer->>Sink: Commit writer2 changes + Committer->>CP: ack() + + Note over Writer1,Writer2: Framework may notify checkpoint completion for cleanup (engine-dependent) +``` + +#### Failure and Retry Flow + +```mermaid +sequenceDiagram + participant CP as CheckpointCoordinator + participant Writer as SinkWriter + participant Committer as SinkCommitter + participant Sink as External Sink + + Writer->>Writer: prepareCommit(checkpointId) + Writer->>CP: ack(commitInfo) + + CP->>Writer: [Failure - writer crashes] + + CP->>CP: Checkpoint fails + CP->>CP: Restore from previous checkpoint + + CP->>Writer: restoreWriter(previousState) + Writer->>Writer: Replay records from checkpoint + + Writer->>Writer: prepareCommit(checkpointId) + Writer->>CP: ack(commitInfo) + + CP->>Committer: commit([commitInfo]) + Committer->>Sink: Commit (idempotent) + Committer-->>Sink: [Commit fails due to network] + Committer->>Committer: Retry + Committer->>Sink: Commit (idempotent) + Sink-->>Committer: Success + + Note over Writer,Committer: Framework may notify checkpoint completion for cleanup (engine-dependent) +``` + +## 3. Key Implementations + +### 3.1 SinkWriter Interface + +The writer runs on workers and performs actual data writing. + +```java +public interface SinkWriter { + + /** + * Write single record + */ + void write(IN element) throws IOException; + + /** + * Prepare commit info during checkpoint. + * + * Guideline: do not make data externally visible in this phase. + */ + Optional prepareCommit(long checkpointId) throws IOException; + + /** + * Abort prepared commit if checkpoint fails + */ + void abortPrepare(); + + /** + * Snapshot writer state for checkpoint + */ + List snapshotState(long checkpointId) throws IOException; + + /** + * Close writer + */ + void close() throws IOException; + + /** + * Context for interacting with framework + */ + interface Context { + int getIndexOfSubtask(); + MetricsContext getMetricsContext(); + } +} +``` + +**Critical Requirements**: +- `prepareCommit(checkpointId)` should not make data externally visible (commit is done in `SinkCommitter` / `SinkAggregatedCommitter`) +- `prepareCommit(checkpointId)` returns commit info that will be passed to committer +- State returned by `snapshotState()` must capture all uncommitted writes +- `abortPrepare()` is only used by Spark when `prepareCommit(...)` fails by throwing an exception + +**Implementation Example (JDBC with XA Transactions)**: + +```java +public class JdbcExactlyOnceSinkWriter implements SinkWriter { + + private final XAConnection xaConnection; + private final XAResource xaResource; + private final Connection connection; + private final PreparedStatement statement; + private final List pendingXids = new ArrayList<>(); + + @Override + public void write(SeaTunnelRow element) throws IOException { + try { + // Start XA transaction if needed + if (currentXid == null) { + currentXid = generateXid(); + xaResource.start(currentXid, XAResource.TMNOFLAGS); + } + + // Execute INSERT (buffered in transaction) + setParameters(statement, element); + statement.executeUpdate(); + + } catch (SQLException e) { + throw new IOException("Failed to write record", e); + } + } + + @Override + public Optional prepareCommit(long checkpointId) throws IOException { + if (currentXid == null) { + return Optional.empty(); // No data written + } + + try { + // End XA transaction + xaResource.end(currentXid, XAResource.TMSUCCESS); + + // Prepare XA transaction (FIRST PHASE - no side effects yet) + xaResource.prepare(currentXid); + + // Return XID for committer + XidInfo xidInfo = new XidInfo(currentXid); + pendingXids.add(currentXid); + currentXid = null; + + return Optional.of(xidInfo); + + } catch (XAException e) { + throw new IOException("Failed to prepare XA transaction", e); + } + } + + @Override + public void abortPrepare() { + // Rollback prepared transaction + if (currentXid != null) { + try { + xaResource.rollback(currentXid); + } catch (XAException e) { + LOG.error("Failed to rollback XA transaction", e); + } + } + } + + @Override + public List snapshotState(long checkpointId) { + // For XA, state is managed by database + return Collections.emptyList(); + } +} +``` + +**Implementation Example (File Sink with Atomic Rename)**: + +```java +public class FileSinkWriter implements SinkWriter { + + private final String tempFilePath; + private final String finalFilePath; + private final OutputStream outputStream; + private long bytesWritten = 0; + + @Override + public void write(SeaTunnelRow element) throws IOException { + // Write to temporary file + byte[] bytes = serialize(element); + outputStream.write(bytes); + bytesWritten += bytes.length; + } + + @Override + public Optional prepareCommit(long checkpointId) throws IOException { + // Flush and close temp file (no rename yet!) + outputStream.flush(); + outputStream.close(); + + // Return commit info for committer to rename file + return Optional.of(new FileCommitInfo(tempFilePath, finalFilePath)); + } + + @Override + public void abortPrepare() { + // Delete temporary file + new File(tempFilePath).delete(); + } + + @Override + public List snapshotState(long checkpointId) { + // Save current write position + return Collections.singletonList(new FileWriterState(bytesWritten)); + } +} +``` + +### 3.2 SinkCommitter Interface + +The committer runs on master and coordinates commits from multiple writers. + +```java +public interface SinkCommitter extends Closeable { + + /** + * Commit multiple commit infos (from multiple writers or retries) + * MUST be idempotent - may be called multiple times with same commitInfo + */ + List commit(List commitInfos) throws IOException; + + /** + * Abort commit infos (optional) + */ + default void abort(List commitInfos) throws IOException {} + + /** + * Close committer + */ + void close() throws IOException; +} +``` + +**Critical Requirements**: +- `commit()` **MUST** be idempotent (calling twice with same commitInfo should be safe) +- Returns list of **failed** commitInfos (will be retried) +- Should handle partial failures gracefully + +**Implementation Example (JDBC XA Committer)**: + +```java +public class JdbcSinkCommitter implements SinkCommitter { + + private final XADataSource xaDataSource; + + @Override + public List commit(List commitInfos) throws IOException { + List failed = new ArrayList<>(); + + for (XidInfo xidInfo : commitInfos) { + try { + XAConnection xaConn = xaDataSource.getXAConnection(); + XAResource xaResource = xaConn.getXAResource(); + + // SECOND PHASE: Commit prepared transaction + xaResource.commit(xidInfo.getXid(), false); + + xaConn.close(); + + } catch (XAException e) { + if (e.errorCode == XAException.XAER_NOTA) { + // Transaction already committed (idempotent) + LOG.info("XA transaction already committed: {}", xidInfo.getXid()); + } else { + // Commit failed, will retry + LOG.error("Failed to commit XA transaction: {}", xidInfo.getXid(), e); + failed.add(xidInfo); + } + } + } + + return failed; // Framework will retry failed commits + } + + @Override + public void abort(List commitInfos) { + // Rollback prepared transactions + for (XidInfo xidInfo : commitInfos) { + try { + XAConnection xaConn = xaDataSource.getXAConnection(); + xaConn.getXAResource().rollback(xidInfo.getXid()); + xaConn.close(); + } catch (Exception e) { + LOG.error("Failed to rollback XA transaction", e); + } + } + } +} +``` + +**Implementation Example (File Committer with Atomic Rename)**: + +```java +public class FileSinkCommitter implements SinkCommitter { + + private final FileSystem fileSystem; + + @Override + public List commit(List commitInfos) { + List failed = new ArrayList<>(); + + for (FileCommitInfo commitInfo : commitInfos) { + try { + Path tempPath = new Path(commitInfo.getTempFilePath()); + Path finalPath = new Path(commitInfo.getFinalFilePath()); + + // Atomic rename (commit) + if (fileSystem.exists(finalPath)) { + // File already committed (idempotent) + LOG.info("File already exists, skipping: {}", finalPath); + fileSystem.delete(tempPath, false); // Clean up temp file + } else { + boolean success = fileSystem.rename(tempPath, finalPath); + if (!success) { + failed.add(commitInfo); + } + } + + } catch (IOException e) { + LOG.error("Failed to commit file: {}", commitInfo, e); + failed.add(commitInfo); + } + } + + return failed; + } +} +``` + +### 3.3 SinkAggregatedCommitter Interface + +The aggregated committer performs single global commit for all writers. + +```java +public interface SinkAggregatedCommitter + extends Closeable { + + /** + * Combine commit infos from multiple writers into single aggregated info + */ + AggregatedCommitInfoT combine(List commitInfos); + + /** + * Commit aggregated info (single global operation) + * MUST be idempotent + */ + List commit(List aggregatedCommitInfos) + throws IOException; + + /** + * Abort aggregated commit infos + */ + default void abort(List aggregatedCommitInfos) throws IOException {} + + /** + * Restore committer state from checkpoint + */ + default void restoreCommit(List aggregatedCommitInfos) + throws IOException {} + + /** + * Close committer + */ + void close() throws IOException; +} +``` + +**Use Cases**: +- Hive table commit (single COMMIT TRANSACTION for all partitions) +- Iceberg table commit (single table snapshot) +- Global index updates (update index once for all writes) + +**Implementation Example (Hive Sink)**: + +```java +public class HiveAggregatedCommitter + implements SinkAggregatedCommitter { + + @Override + public HiveCommitInfo combine(List commitInfos) { + // Collect all written files across all writers + List allFiles = new ArrayList<>(); + for (HiveWriteInfo writeInfo : commitInfos) { + allFiles.addAll(writeInfo.getWrittenFiles()); + } + return new HiveCommitInfo(allFiles); + } + + @Override + public List commit(List aggregatedCommitInfos) { + List failed = new ArrayList<>(); + + for (HiveCommitInfo commitInfo : aggregatedCommitInfos) { + try { + // Single global commit for entire table + hiveMetastore.beginTransaction(); + + for (String file : commitInfo.getAllFiles()) { + hiveMetastore.addPartitionFile(tableName, file); + } + + hiveMetastore.commitTransaction(); // Global atomic commit + + } catch (Exception e) { + LOG.error("Failed to commit to Hive", e); + hiveMetastore.rollbackTransaction(); + failed.add(commitInfo); + } + } + + return failed; + } +} +``` + +### 3.4 Code References + +**API Interfaces**: +- [SeaTunnelSink.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SeaTunnelSink.java) +- [SinkWriter.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SinkWriter.java) +- [SinkCommitter.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SinkCommitter.java) +- [SinkAggregatedCommitter.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SinkAggregatedCommitter.java) + +**Example Implementations**: +- JDBC Sink: `seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/` +- Kafka Sink: `seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/` +- File Sink: `seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/` + +## 4. Design Considerations + +### 4.1 Design Trade-offs + +#### Two-Phase Commit + +**Pros**: +- Strong consistency guarantee (exactly-once) +- Automatic failure recovery +- Clear separation between prepare and commit + +**Cons**: +- Increased latency (data visible only after commit) +- Requires transactional support in sink +- Additional state for commit info +- More complex implementation + +**When to Use**: +- Financial transactions, billing, audit logs +- Any scenario requiring exactly-once guarantee + +**When Not to Use**: +- At-least-once is acceptable (logging, metrics) +- Sink doesn't support transactions +- Ultra-low latency required + +#### Three-Tier vs Two-Tier Commit + +**Two-Tier (Writer → Committer)**: +- Each writer's commit handled independently +- Parallel commit operations +- Suitable for most sinks + +**Three-Tier (Writer → Committer → AggregatedCommitter)**: +- All writers' commits aggregated into single operation +- Single global commit point +- Required for table-level transactions (Hive, Iceberg) + +### 4.2 Performance Considerations + +#### Batch Writing + +```java +public class BatchSinkWriter { + private final List batch = new ArrayList<>(); + private static final int BATCH_SIZE = 1000; + + @Override + public void write(SeaTunnelRow element) { + batch.add(element); + if (batch.size() >= BATCH_SIZE) { + flushBatch(); + } + } + + private void flushBatch() { + // Write entire batch in single operation + statement.executeBatch(); + batch.clear(); + } +} +``` + +**Benefits**: +- Amortize per-record overhead +- Reduce network round-trips +- Better throughput + +#### Async Writes + +```java +public class AsyncSinkWriter { + private final BlockingQueue> pendingWrites = new LinkedBlockingQueue<>(); + + @Override + public void write(SeaTunnelRow element) { + CompletableFuture future = CompletableFuture.runAsync(() -> { + // Async write operation + actualWrite(element); + }, executorService); + + pendingWrites.add(future); + } + + @Override + public Optional prepareCommit(long checkpointId) { + // Wait for all pending writes to complete + for (CompletableFuture future : pendingWrites) { + future.join(); + } + pendingWrites.clear(); + + return Optional.of(createCommitInfo()); + } +} +``` + +#### Connection Pooling + +```java +public class JdbcSinkWriter { + private final HikariDataSource dataSource; + + @Override + public void write(SeaTunnelRow element) { + try (Connection conn = dataSource.getConnection()) { + // Reuse pooled connections + PreparedStatement stmt = conn.prepareStatement(sql); + stmt.executeUpdate(); + } + } +} +``` + +### 4.3 Idempotency Patterns + +#### 1. Natural Idempotency (Upsert) + +```java +// INSERT ON DUPLICATE KEY UPDATE (MySQL) +String sql = "INSERT INTO table (id, name) VALUES (?, ?) " + + "ON DUPLICATE KEY UPDATE name = VALUES(name)"; + +// MERGE INTO (Oracle, SQL Server) +String sql = "MERGE INTO table USING (SELECT ? as id, ? as name FROM dual) src " + + "ON (table.id = src.id) " + + "WHEN MATCHED THEN UPDATE SET table.name = src.name " + + "WHEN NOT MATCHED THEN INSERT (id, name) VALUES (src.id, src.name)"; +``` + +#### 2. Deduplication Key + +```java +public class KafkaSinkWriter { + @Override + public void write(SeaTunnelRow element) { + ProducerRecord record = new ProducerRecord<>( + topic, + element.getField(0).toString(), // Key for deduplication + element.toString() + ); + + // Kafka deduplicates based on (topic, partition, offset, idempotent producer) + producer.send(record); + } +} +``` + +#### 3. External Deduplication Table + +```java +public class JdbcCommitter { + @Override + public List commit(List commitInfos) { + for (XidInfo xidInfo : commitInfos) { + String xidString = xidInfo.getXid().toString(); + + // Check if already committed + boolean exists = checkCommitTable(xidString); + if (exists) { + LOG.info("XID already committed: {}", xidString); + continue; // Idempotent + } + + // Commit transaction + xaResource.commit(xidInfo.getXid(), false); + + // Record commit + insertCommitTable(xidString, System.currentTimeMillis()); + } + } +} +``` + +## 5. Best Practices + +### 5.1 Usage Recommendations + +**1. Choose Appropriate Commit Level** + +```java +// Simple sink: Writer only (at-least-once) +public class SimpleSink implements SeaTunnelSink<...> { + SinkWriter createWriter(...) { return new SimpleWriter(); } + // No committer - data written directly +} + +// Transactional sink: Writer + Committer (exactly-once) +public class TransactionalSink implements SeaTunnelSink<...> { + SinkWriter createWriter(...) { return new TransactionalWriter(); } + Optional createCommitter() { return Optional.of(new Committer()); } +} + +// Table sink: Writer + Committer + AggregatedCommitter +public class TableSink implements SeaTunnelSink<...> { + SinkWriter createWriter(...) { return new TableWriter(); } + Optional createCommitter() { return Optional.of(new Committer()); } + Optional createAggregatedCommitter() { + return Optional.of(new AggregatedCommitter()); + } +} +``` + +**2. Proper State Management** + +```java +public class StatefulSinkWriter { + private long recordsWritten = 0; + private long bytesWritten = 0; + + @Override + public List snapshotState(long checkpointId) { + return Collections.singletonList( + new WriterState(recordsWritten, bytesWritten) + ); + } + + public StatefulSinkWriter restoreState(List states) { + if (!states.isEmpty()) { + WriterState state = states.get(0); + this.recordsWritten = state.getRecordsWritten(); + this.bytesWritten = state.getBytesWritten(); + } + return this; + } +} +``` + +**3. Resource Management** + +```java +@Override +public void close() throws IOException { + // Close in reverse order of creation + if (statement != null) statement.close(); + if (connection != null) connection.close(); + if (dataSource != null) dataSource.close(); +} +``` + +### 5.2 Common Pitfalls + +**1. Side Effects in prepareCommit(checkpointId)** + +```java +// ❌ BAD: Actual commit in prepareCommit(checkpointId) +public Optional prepareCommit(long checkpointId) { + connection.commit(); // WRONG! This is a side effect! + return Optional.of(new CommitInfo()); +} + +// ✅ GOOD: Only prepare, no side effects +public Optional prepareCommit(long checkpointId) { + xaResource.end(xid, XAResource.TMSUCCESS); + xaResource.prepare(xid); // Prepare only, no commit yet + return Optional.of(new XidInfo(xid)); +} +``` + +**2. Non-Idempotent Commit** + +```java +// ❌ BAD: Direct INSERT (not idempotent) +public List commit(List commitInfos) { + for (CommitInfo info : commitInfos) { + executeInsert(info); // May fail if called twice! + } +} + +// ✅ GOOD: UPSERT (idempotent) +public List commit(List commitInfos) { + for (CommitInfo info : commitInfos) { + executeUpsert(info); // Safe to call multiple times + } +} +``` + +**3. Large State** + +```java +// ❌ BAD: Buffer all records in state +public class BadWriter { + private List bufferedRows = new ArrayList<>(); // May be huge! + + public List snapshotState() { + return Collections.singletonList(new State(bufferedRows)); + } +} + +// ✅ GOOD: Flush before checkpoint, track metadata only +public class GoodWriter { + private long lastCommittedOffset = 0; + + public Optional prepareCommit(long checkpointId) { + flushBufferedRows(); // Write to external system + return Optional.of(new CommitInfo(lastCommittedOffset)); + } +} +``` + +### 5.3 Debugging Tips + +**1. Enable XA Transaction Logging** + +```java +// Log XA operations for debugging +LOG.info("Starting XA transaction: {}", xid); +xaResource.start(xid, XAResource.TMNOFLAGS); + +LOG.info("Preparing XA transaction: {}", xid); +xaResource.prepare(xid); + +LOG.info("Committing XA transaction: {}", xid); +xaResource.commit(xid, false); +``` + +**2. Track Commit Progress** + +```java +public class MonitoredCommitter { + private final Counter commitAttempts = metricGroup.counter("commit_attempts"); + private final Counter commitSuccesses = metricGroup.counter("commit_successes"); + private final Counter commitFailures = metricGroup.counter("commit_failures"); + + public List commit(List commitInfos) { + commitAttempts.inc(commitInfos.size()); + + List failed = new ArrayList<>(); + for (CommitInfo info : commitInfos) { + try { + doCommit(info); + commitSuccesses.inc(); + } catch (Exception e) { + commitFailures.inc(); + failed.add(info); + } + } + return failed; + } +} +``` + +**3. Test Failure Scenarios** + +```java +@Test +public void testCheckpointFailureRecovery() { + // Write data + writer.write(row1); + writer.write(row2); + + // Prepare commit + Optional commitInfo = writer.prepareCommit(checkpointId); + + // Simulate checkpoint failure + writer.abortPrepare(); + + // Verify no data committed + assertFalse(dataExistsInSink()); + + // Restore and retry + writer.write(row1); + writer.write(row2); + commitInfo = writer.prepareCommit(checkpointId); + + // Commit should succeed + committer.commit(Collections.singletonList(commitInfo.get())); + assertTrue(dataExistsInSink()); +} +``` + +## 6. Related Resources + +- [Architecture Overview](../overview.md) +- [Design Philosophy](../design-philosophy.md) +- [Source Architecture](source-architecture.md) +- [Checkpoint Mechanism](../fault-tolerance/checkpoint-mechanism.md) +- [Exactly-Once Semantics](../fault-tolerance/exactly-once.md) + +## 7. References + +### Example Connectors + +- **Simple Sink**: ConsoleSink (logs to stdout) +- **File Sink**: FileSink (atomic file rename) +- **Database Sink**: JdbcSink (XA transactions) +- **Streaming Sink**: KafkaSink (Kafka transactions) +- **Table Sink**: IcebergSink (table commits) + +### Further Reading + +- [Two-Phase Commit Protocol](https://en.wikipedia.org/wiki/Two-phase_commit_protocol) +- [XA Transactions](https://www.oracle.com/java/technologies/xa-transactions.html) +- [Kafka Transactions](https://kafka.apache.org/documentation/#semantics) +- [Iceberg Table Format](https://iceberg.apache.org/spec/) diff --git a/docs/en/architecture/api-design/source-architecture.md b/docs/en/architecture/api-design/source-architecture.md new file mode 100644 index 000000000000..471bfe502fbf --- /dev/null +++ b/docs/en/architecture/api-design/source-architecture.md @@ -0,0 +1,827 @@ +--- +sidebar_position: 2 +title: Source Architecture +--- + +# Source Architecture + +## 1. Overview + +### 1.1 Problem Background + +Data sources in distributed systems present several challenges: + +- **Parallelism**: How to read data in parallel from a single source? +- **Fault Tolerance**: How to resume from where we left off after failures? +- **Dynamic Assignment**: How to handle worker failures and redistribute work? +- **Bounded vs Unbounded**: How to unify batch and streaming sources? +- **Backpressure**: How to handle slow downstream processing? + +### 1.2 Design Goals + +SeaTunnel's Source API aims to: + +1. **Enable Parallel Reading**: Support split-based parallelism for scalability +2. **Ensure Fault Tolerance**: Checkpoint split state for exactly-once processing +3. **Separate Coordination from Execution**: Enumerator (master) and Reader (worker) separation +4. **Support Dynamic Assignment**: Reassign splits on failures or imbalance +5. **Unify Batch and Streaming**: Single API for both bounded and unbounded sources + +### 1.3 Applicable Scenarios + +- File-based sources (local files, HDFS, S3, OSS) +- Database sources (MySQL, PostgreSQL, Oracle, JDBC-compatible) +- Message queue sources (Kafka, Pulsar, RabbitMQ) +- CDC sources (MySQL CDC, PostgreSQL CDC, Oracle CDC) +- Stream sources (Socket, HTTP, custom protocols) + +## 2. Architecture Design + +### 2.1 Overall Architecture + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Coordinator (master/coordinator side) │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ SourceSplitEnumerator │ │ +│ │ │ │ +│ │ • Discover/generate splits in run() (impl-defined) │ │ +│ │ • Assign splits to readers │ │ +│ │ • Handle reader registration │ │ +│ │ • Handle split requests │ │ +│ │ • Reclaim splits from failed readers │ │ +│ │ • Snapshot enumerator state │ │ +│ │ • Send/receive custom events │ │ +│ └────────────────────────────────────────────────────┘ │ +│ │ │ +└────────────────────────────┼───────────────────────────────────┘ + │ (Split Assignment) + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ TaskExecutionService (Worker Side) │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ SourceReader │ │ +│ │ │ │ +│ │ • Receive assigned splits │ │ +│ │ • Read data from splits │ │ +│ │ • Emit records downstream │ │ +│ │ • Snapshot reader state (split progress) │ │ +│ │ • Handle split completion │ │ +│ │ • Send/receive custom events │ │ +│ └────────────────────────────────────────────────────┘ │ +│ │ │ +└────────────────────────────┼───────────────────────────────────┘ + │ + ▼ + SeaTunnelRow + (to Transform/Sink) +``` + +### 2.2 Core Components + +#### SeaTunnelSource (Factory Interface) + +The top-level interface that serves as a factory for creating readers and enumerators. + +```java +public interface SeaTunnelSource + extends Serializable { + + /** + * Get source boundedness (BOUNDED for batch, UNBOUNDED for streaming) + */ + Boundedness getBoundedness(); + + /** + * Create SourceReader (called on worker) + */ + SourceReader createReader(SourceReader.Context readerContext) throws Exception; + + /** + * Split serializer used for network transfer and checkpointing. + */ + Serializer getSplitSerializer(); + + /** + * Create SourceSplitEnumerator (called on master) + */ + SourceSplitEnumerator createEnumerator( + SourceSplitEnumerator.Context enumeratorContext) throws Exception; + + /** + * Restore SourceSplitEnumerator from checkpoint (called on master) + */ + SourceSplitEnumerator restoreEnumerator( + SourceSplitEnumerator.Context enumeratorContext, + StateT checkpointState) throws Exception; + + /** + * Enumerator-state serializer used for checkpointing. + */ + Serializer getEnumeratorStateSerializer(); + + /** + * Get output schema (CatalogTable list, supports multi-table) + */ + List getProducedCatalogTables(); +} +``` + +**Key Methods**: +- `getBoundedness()`: Indicates if source is bounded (batch) or unbounded (stream) +- `createReader()`: Factory for reader instances (one per worker task) +- `createEnumerator()`: Factory for enumerator (single instance on master) +- `restoreEnumerator()`: Restore enumerator from checkpoint state +- `getProducedCatalogTables()`: Defines output schema (supports multi-table) +- `getSplitSerializer()` / `getEnumeratorStateSerializer()`: Split/enumerator-state serializers for network transfer and checkpointing + +#### SourceSplit (Minimal Serializable Unit) + +Represents a partitionable unit of data. + +```java +public interface SourceSplit extends Serializable { + /** + * Unique identifier for this split + */ + String splitId(); +} +``` + +**Implementation Examples**: + +```java +// File-based split +public class FileSplit implements SourceSplit { + private final String splitId; + private final String filePath; + private final long startOffset; + private final long length; +} + +// JDBC-based split (query range) +public class JdbcSourceSplit implements SourceSplit { + private final String splitId; + private final String query; + private final Object[] queryParams; +} + +// Kafka-based split (partition) +public class KafkaSourceSplit implements SourceSplit { + private final String splitId; + private final String topic; + private final int partition; + private final long startOffset; +} +``` + +**Design Notes**: +- Splits must be serializable for network transfer +- Split state (e.g., current offset) stored separately in reader state +- Splits can be reassigned to different readers + +### 2.3 Interaction Flow + +#### Initial Startup Flow + +```mermaid +sequenceDiagram + participant Coord as Coordinator + participant Enum as SourceSplitEnumerator + participant Worker as TaskExecutionService + participant Reader as SourceReader + + Coord->>Enum: createEnumerator(context) + Enum->>Enum: open() + + Worker->>Reader: createReader(context) + Reader->>Reader: open() + + Coord->>Enum: registerReader(subtaskId) + Enum->>Enum: run() (discover/generate splits, impl-defined) + + Reader->>Enum: context.sendSplitRequest() + Enum->>Enum: handleSplitRequest(subtaskId) + Enum->>Reader: assignSplit(splits) + + Reader->>Reader: addSplits(splits) + Reader->>Reader: pollNext(collector) + Reader->>Worker: collect(record) +``` + +#### Checkpoint Flow + +```mermaid +sequenceDiagram + participant CP as CheckpointCoordinator + participant Enum as SourceSplitEnumerator + participant Reader as SourceReader + + CP->>Reader: triggerBarrier(checkpointId) + Reader->>Reader: snapshotState(checkpointId) + Reader->>CP: ack(readerState) + + CP->>Enum: snapshotState(checkpointId) + Enum->>Enum: snapshot enumerator state + Enum->>CP: ack(enumeratorState) + + CP->>CP: All acks received + CP->>CP: Persist checkpoint +``` + +#### Failure Recovery Flow + +```mermaid +sequenceDiagram + participant Coord as Coordinator + participant Enum as SourceSplitEnumerator + participant OldReader as Failed Reader + participant NewReader as New Reader + + OldReader->>OldReader: [Failure] + Coord->>Enum: addSplitsBack(splits, subtaskId) + Enum->>Enum: Mark splits as pending + + Coord->>NewReader: Deploy on new worker + NewReader->>NewReader: Restore from checkpoint (reader state) + Coord->>Enum: registerReader(subtaskId) + + Enum->>NewReader: assignSplit(recovered splits) + NewReader->>NewReader: Resume from checkpointed offset +``` + +## 3. Key Implementations + +### 3.1 SourceSplitEnumerator Interface + +The enumerator runs on the master side and coordinates split assignment. + +```java +public interface SourceSplitEnumerator + extends AutoCloseable, CheckpointListener { + + /** + * Called when enumerator starts + */ + void open(); + + /** + * Executes split discovery and background coordination logic. + * + * Note: run() and snapshotState() may be invoked concurrently by different threads. + */ + void run() throws Exception; + + /** + * Add a split back to the enumerator for reassignment (typically after reader failure). + */ + void addSplitsBack(List splits, int subtaskId); + + /** + * Current number of unassigned splits. + */ + int currentUnassignedSplitSize(); + + /** + * Called when a reader requests more splits. + */ + void handleSplitRequest(int subtaskId); + + /** + * Called when a reader registers. + */ + void registerReader(int subtaskId); + + /** + * Snapshot enumerator state for checkpoint + */ + StateT snapshotState(long checkpointId) throws Exception; + + /** + * Handle custom event from reader + */ + default void handleSourceEvent(int subtaskId, SourceEvent sourceEvent) {} + + /** + * Close enumerator + */ + void close() throws IOException; + + /** + * Context for interacting with framework + */ + interface Context { + int currentParallelism(); + Set registeredReaders(); + void assignSplit(int subtaskId, List splits); + void signalNoMoreSplits(int subtaskId); + void sendEventToSourceReader(int subtaskId, SourceEvent event); + } +} +``` + +**Key Responsibilities**: +- **Split Discovery**: Generate splits from data source (files, partitions, shards) +- **Assignment Strategy**: Decide which splits go to which readers +- **Dynamic Handling**: Handle reader registration, split requests, failures +- **State Management**: Snapshot remaining splits and assignment state + +**Implementation Example**: + +```java +public class JdbcSourceSplitEnumerator implements SourceSplitEnumerator { + + private final Queue pendingSplits = new LinkedList<>(); + private final Set assignedSplits = new HashSet<>(); + private final Context context; + + @Override + public void run() throws Exception { + // Discover splits by querying database metadata + List splits = generateSplitsByPartition(); + pendingSplits.addAll(splits); + } + + @Override + public void handleSplitRequest(int subtaskId) { + // Assign next available split + JdbcSourceSplit split = pendingSplits.poll(); + if (split != null) { + context.assignSplit(subtaskId, Collections.singletonList(split)); + assignedSplits.add(split.splitId()); + } else { + context.signalNoMoreSplits(subtaskId); + } + } + + @Override + public void addSplitsBack(List splits, int subtaskId) { + // Reclaim splits from failed reader + pendingSplits.addAll(splits); + splits.forEach(split -> assignedSplits.remove(split.splitId())); + } + + @Override + public JdbcSourceState snapshotState(long checkpointId) { + // Save remaining splits and assignment info + return new JdbcSourceState(new ArrayList<>(pendingSplits), assignedSplits); + } +} +``` + +### 3.2 SourceReader Interface + +The reader runs on workers and performs actual data reading. + +```java +public interface SourceReader + extends AutoCloseable, CheckpointListener { + + /** + * Called when reader starts + */ + void open() throws Exception; + + /** + * Poll next batch of records (non-blocking or timeout) + */ + void pollNext(Collector output) throws Exception; + + /** + * Snapshot reader state for checkpoint (typically the current splits/positions). + */ + List snapshotState(long checkpointId) throws Exception; + + /** + * Add newly assigned splits. + */ + void addSplits(List splits); + + /** + * Signal no more splits will be assigned. + */ + void handleNoMoreSplits(); + + /** + * Handle custom event from enumerator + */ + default void handleSourceEvent(SourceEvent sourceEvent) {} + + /** + * Close reader + */ + void close() throws IOException; + + /** + * Context for interacting with framework + */ + interface Context { + int getIndexOfSubtask(); + Boundedness getBoundedness(); + void signalNoMoreElement(); + void sendSplitRequest(); + void sendSourceEventToEnumerator(SourceEvent sourceEvent); + } +} +``` + +**Key Responsibilities**: +- **Data Reading**: Pull records from assigned splits +- **Progress Tracking**: Track offset/position within each split +- **State Management**: Snapshot split progress for recovery +- **Split Management**: Handle split assignment, completion, and removal + +**Implementation Example**: + +```java +public class JdbcSourceReader implements SourceReader { + + private final Queue pendingSplits = new LinkedList<>(); + private JdbcSourceSplit currentSplit; + private ResultSet currentResultSet; + + @Override + public void pollNext(Collector output) throws Exception { + if (currentResultSet == null) { + // Fetch next split + currentSplit = pendingSplits.poll(); + if (currentSplit == null) { + context.sendSplitRequest(); // Request more splits + return; + } + // Execute query for current split + currentResultSet = executeQuery(currentSplit); + } + + // Read batch of rows + int count = 0; + while (currentResultSet.next() && count++ < BATCH_SIZE) { + SeaTunnelRow row = convertToRow(currentResultSet); + output.collect(row); + } + + // Check if split completed + if (!currentResultSet.next()) { + currentResultSet.close(); + currentResultSet = null; + currentSplit = null; + } + } + + @Override + public void addSplits(List splits) { + pendingSplits.addAll(splits); + } + + @Override + public List snapshotState(long checkpointId) { + // Save current split and offset + List states = new ArrayList<>(); + if (currentSplit != null) { + states.add(new JdbcSourceState(currentSplit, currentRow)); + } + pendingSplits.forEach(split -> + states.add(new JdbcSourceState(split, 0))); + return states; + } +} +``` + +### 3.3 SourceEvent (Custom Communication) + +Allows enumerator and reader to exchange custom messages. + +```java +public interface SourceEvent extends Serializable { +} + +// Example: Reader notifies enumerator of discovered partitions +public class PartitionDiscoveredEvent implements SourceEvent { + private final List newPartitions; +} + +// Example: Enumerator notifies reader of configuration change +public class ConfigChangeEvent implements SourceEvent { + private final Map newConfig; +} +``` + +**Use Cases**: +- Dynamic partition discovery (Kafka, HDFS) +- Runtime configuration changes +- Custom coordination logic + +### 3.4 Code References + +**API Interfaces**: +- [SeaTunnelSource.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/source/SeaTunnelSource.java) +- [SourceSplitEnumerator.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/source/SourceSplitEnumerator.java) +- [SourceReader.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/source/SourceReader.java) +- [SourceSplit.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/source/SourceSplit.java) + +**Example Implementations**: +- JDBC Source: `seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/` +- Kafka Source: `seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/` +- File Source: `seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/` + +## 4. Design Considerations + +### 4.1 Design Trade-offs + +#### Enumerator-Reader Separation + +**Pros**: +- Clean separation of coordination (master) and execution (worker) +- Enumerator can reassign splits without reader knowledge +- Centralized coordination simplifies split assignment logic +- Fault tolerance: enumerator and reader fail independently + +**Cons**: +- Additional network communication (split assignment messages) +- More complex API for connector developers +- Potential bottleneck if enumerator is slow + +**Mitigation**: +- Asynchronous split assignment +- Batch split requests/assignments +- Lazy split discovery + +#### Split Granularity + +**Coarse-grained splits** (few large splits): +- **Pro**: Less coordination overhead +- **Con**: Poor load balancing, longer recovery time + +**Fine-grained splits** (many small splits): +- **Pro**: Better load balancing, faster recovery +- **Con**: Higher coordination overhead + +**Guideline**: Choose split granularity based on source capabilities, expected parallelism, and checkpoint/recovery cost. + +### 4.2 Performance Considerations + +#### Batch Reading + +```java +@Override +public void pollNext(Collector output) throws Exception { + // Read batch instead of single record + for (int i = 0; i < BATCH_SIZE && hasNext(); i++) { + output.collect(readNextRow()); + } +} +``` + +**Benefits**: +- Amortize per-record overhead +- Better CPU cache utilization +- Reduce lock contention + +#### Non-blocking Poll + +```java +@Override +public void pollNext(Collector output) throws Exception { + // Return immediately if no data available + if (!hasNext()) { + return; // Framework will call again later + } + output.collect(readNextRow()); +} +``` + +**Benefits**: +- Avoid blocking worker thread +- Enable backpressure handling +- Better resource utilization + +#### Connection Pooling + +```java +public class JdbcSourceReader { + private final HikariDataSource dataSource; // Connection pool + + @Override + public void pollNext(Collector output) { + try (Connection conn = dataSource.getConnection()) { + // Reuse pooled connections + } + } +} +``` + +### 4.3 Extensibility + +#### Custom Split Assignment Strategy + +```java +public class CustomEnumerator implements SourceSplitEnumerator<...> { + + @Override + public void handleSplitRequest(int subtaskId) { + // Custom logic: assign splits based on data locality + JdbcSourceSplit split = findClosestSplit(subtaskId); + context.assignSplit(subtaskId, Collections.singletonList(split)); + } + + private JdbcSourceSplit findClosestSplit(int subtaskId) { + // Check worker location and assign split on same rack/region + WorkerLocation location = getWorkerLocation(subtaskId); + return pendingSplits.stream() + .filter(split -> split.location().equals(location)) + .findFirst() + .orElse(pendingSplits.poll()); + } +} +``` + +#### Dynamic Split Discovery + +```java +public class KafkaSourceSplitEnumerator { + + @Override + public void run() throws Exception { + // Discover initial partitions + discoverPartitions(); + + // Periodically check for new partitions + scheduledExecutor.scheduleAtFixedRate( + this::discoverPartitions, + 60, 60, TimeUnit.SECONDS + ); + } + + private void discoverPartitions() { + List newPartitions = kafkaAdmin.listPartitions(); + // Assign new partitions to readers + assignNewPartitions(newPartitions); + } +} +``` + +## 5. Best Practices + +### 5.1 Usage Recommendations + +**1. Split Sizing** +- Files: split by file/offset ranges according to file format and I/O characteristics +- Databases: split by primary key / partition key ranges (or other stable predicates) +- Message queues: use native partitions (e.g., Kafka partitions) + +**2. State Management** +- Keep split/reader state small and stable across versions +- Use offsets/positions instead of buffered data +- Serialize efficiently (Kryo, Protobuf) + +**3. Error Handling** +```java +@Override +public void pollNext(Collector output) throws Exception { + try { + // Read data + } catch (TransientException e) { + // Retry transient errors + Thread.sleep(1000); + retry(); + } catch (FatalException e) { + // Fatal errors should propagate + throw e; + } +} +``` + +**4. Resource Management** +```java +@Override +public void close() throws IOException { + // Always close resources + if (resultSet != null) resultSet.close(); + if (connection != null) connection.close(); + if (dataSource != null) dataSource.close(); +} +``` + +### 5.2 Common Pitfalls + +**1. Blocking pollNext()** +```java +// ❌ BAD: Blocks indefinitely +public void pollNext(Collector output) { + while (true) { + Record record = queue.take(); // Blocks until data available + output.collect(record); + } +} + +// ✅ GOOD: Non-blocking or timeout +public void pollNext(Collector output) { + Record record = queue.poll(100, TimeUnit.MILLISECONDS); + if (record != null) { + output.collect(record); + } +} +``` + +**2. Large State** +```java +// ❌ BAD: Buffer entire split in state +public class BadReaderState { + private List bufferedRows; // May be huge! +} + +// ✅ GOOD: Only track offset +public class GoodReaderState { + private long currentOffset; // Small and efficient +} +``` + +**3. Forgetting to Request Splits** +```java +// ❌ BAD: Reader never gets splits +public void pollNext(Collector output) { + if (pendingSplits.isEmpty()) { + return; // Oops, should request more splits! + } +} + +// ✅ GOOD: Explicitly request splits +public void pollNext(Collector output) { + if (pendingSplits.isEmpty()) { + context.sendSplitRequest(); + return; + } +} +``` + +### 5.3 Debugging Tips + +**1. Enable Debug Logging** +```java +private static final Logger LOG = LoggerFactory.getLogger(JdbcSourceReader.class); + +public void pollNext(Collector output) { + LOG.debug("Polling split: {}, offset: {}", currentSplit.splitId(), currentOffset); + // ... +} +``` + +**2. Track Metrics** +```java +public class JdbcSourceReader { + private long recordsRead = 0; + private long bytesRead = 0; + + public void pollNext(Collector output) { + SeaTunnelRow row = readRow(); + recordsRead++; + bytesRead += row.getBytesSize(); + output.collect(row); + } +} +``` + +**3. Test Split Reassignment** +```java +// Simulate reader failure to test split recovery +@Test +public void testSplitReassignment() { + // Assign splits to reader 0 + enumerator.handleSplitRequest(0); + + // Simulate reader 0 failure + enumerator.addSplitsBack(assignedSplits, 0); + + // New reader 1 should get those splits + enumerator.registerReader(1); + enumerator.handleSplitRequest(1); + + // Verify splits were reassigned + assertThat(assignedSplits).isNotEmpty(); +} +``` + +## 6. Related Resources + +- [Architecture Overview](../overview.md) +- [Design Philosophy](../design-philosophy.md) +- [Sink Architecture](sink-architecture.md) +- [Checkpoint Mechanism](../fault-tolerance/checkpoint-mechanism.md) +- [How to Create Your Connector](../../developer/how-to-create-your-connector.md) + +## 7. References + +### Example Connectors + +- **Simple Source**: FakeSource (generates test data) +- **File Source**: FileSource (local/HDFS/S3 files) +- **Database Source**: JdbcSource (JDBC-compatible databases) +- **Streaming Source**: KafkaSource (Apache Kafka) +- **CDC Source**: MySQLCDCSource (MySQL binlog) + +### Further Reading + +- Apache Flink FLIP-27: ["Refactored Source API"](https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface) +- Kafka Consumer: [Consumer Groups and Partition Assignment](https://kafka.apache.org/documentation/#consumerconfigs) diff --git a/docs/en/architecture/api-design/translation-layer.md b/docs/en/architecture/api-design/translation-layer.md new file mode 100644 index 000000000000..58c03da83a6e --- /dev/null +++ b/docs/en/architecture/api-design/translation-layer.md @@ -0,0 +1,817 @@ +--- +sidebar_position: 1 +title: Translation Layer +--- + +# Translation Layer Architecture + +## 1. Overview + +### 1.1 Problem Background + +SeaTunnel provides a unified connector API, but jobs need to run on different execution engines: + +- **Engine Diversity**: Flink, Spark, SeaTunnel Engine (Zeta) have different APIs +- **Code Duplication**: Without translation, each connector needs 3 implementations +- **Maintenance Burden**: Bug fixes require changes in all implementations +- **API Evolution**: Engine API changes break connectors +- **User Experience**: Users want consistent behavior across engines + +### 1.2 Design Goals + +SeaTunnel's translation layer aims to: + +1. **Enable Portability**: Same connector runs on any engine +2. **Hide Complexity**: Connector developers only learn SeaTunnel API +3. **Maintain Fidelity**: Preserve semantic guarantees across engines +4. **Minimize Overhead**: Keep translation overhead low (depends on connectors and type conversions) +5. **Support Evolution**: Isolate connectors from engine API changes + +### 1.3 Architecture Overview + +``` +┌──────────────────────────────────────────────────────────────┐ +│ SeaTunnel API Layer │ +│ (Engine-Independent Connector Interface) │ +│ │ +│ SeaTunnelSource SeaTunnelSink SeaTunnelTransform │ +└──────────────────────────────────────────────────────────────┘ + │ + │ Translation Layer + ┌─────────────┼─────────────┐ + ▼ ▼ ▼ +┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ +│ Flink Adapter │ │ Spark Adapter │ │ Zeta (Native) │ +│ │ │ │ │ │ +│ FlinkSource │ │ SparkSource │ │ Direct │ +│ FlinkSink │ │ SparkSink │ │ Execution │ +└──────────────────┘ └──────────────────┘ └──────────────────┘ + │ │ │ + ▼ ▼ ▼ +┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ +│ Apache Flink │ │ Apache Spark │ │ SeaTunnel Engine │ +│ Runtime │ │ Runtime │ │ (Zeta) │ +└──────────────────┘ └──────────────────┘ └──────────────────┘ +``` + +## 2. Flink Translation Layer + +### 2.1 FlinkSource Adapter + +Adapts `SeaTunnelSource` to Flink's `Source` interface. + +```java +public class FlinkSource + implements Source, EnumeratorStateWrapper> { + + // Wrapped SeaTunnel source + private final SeaTunnelSource seaTunnelSource; + + @Override + public Boundedness getBoundedness() { + // Delegate to SeaTunnel source + return seaTunnelSource.getBoundedness() == Boundedness.BOUNDED + ? Boundedness.BOUNDED + : Boundedness.CONTINUOUS_UNBOUNDED; + } + + @Override + public SourceReader> createReader( + SourceReaderContext readerContext + ) { + // Create SeaTunnel reader with adapted context + org.apache.seatunnel.api.source.SourceReader seaTunnelReader = + seaTunnelSource.createReader(new FlinkSourceReaderContext(readerContext)); + + // Wrap in Flink adapter + return new FlinkSourceReader<>(seaTunnelReader, readerContext); + } + + @Override + public SplitEnumerator, EnumeratorStateWrapper> + createEnumerator(SplitEnumeratorContext> context) { + + // Create SeaTunnel enumerator with adapted context + SourceSplitEnumerator seaTunnelEnumerator = + seaTunnelSource.createEnumerator( + new FlinkSourceSplitEnumeratorContext<>(context) + ); + + // Wrap in Flink adapter + return new FlinkSourceEnumerator<>(seaTunnelEnumerator, context); + } + + @Override + public SimpleVersionedSerializer> getSplitSerializer() { + // Adapt SeaTunnel serializer to Flink serializer + return new FlinkSimpleVersionedSerializer<>( + seaTunnelSource.getSplitSerializer() + ); + } +} +``` + +### 2.2 FlinkSourceReader Adapter + +```java +public class FlinkSourceReader + implements SourceReader> { + + private final org.apache.seatunnel.api.source.SourceReader seaTunnelReader; + private final SourceReaderContext flinkContext; + + @Override + public void start() { + // Delegate to SeaTunnel reader + try { + seaTunnelReader.open(); + } catch (Exception e) { + throw new FlinkRuntimeException("Failed to open SeaTunnel reader", e); + } + } + + @Override + public InputStatus pollNext(ReaderOutput output) { + try { + // Adapt output collector + CollectorAdapter collector = new CollectorAdapter<>(output); + + // Poll from SeaTunnel reader + seaTunnelReader.pollNext(collector); + + if (collector.hasRecords()) { + return InputStatus.MORE_AVAILABLE; + } else { + return InputStatus.NOTHING_AVAILABLE; + } + } catch (Exception e) { + throw new FlinkRuntimeException("Failed to poll from SeaTunnel reader", e); + } + } + + @Override + public void addSplits(List> splits) { + // Unwrap and delegate + List unwrappedSplits = splits.stream() + .map(SplitWrapper::getSplit) + .collect(Collectors.toList()); + + seaTunnelReader.addSplits(unwrappedSplits); + } + + @Override + public void notifyCheckpointComplete(long checkpointId) { + try { + seaTunnelReader.notifyCheckpointComplete(checkpointId); + } catch (Exception e) { + throw new FlinkRuntimeException("Failed to notify checkpoint complete", e); + } + } + + @Override + public List> snapshotState(long checkpointId) { + try { + List state = seaTunnelReader.snapshotState(checkpointId); + + // Wrap splits for Flink + return state.stream() + .map(SplitWrapper::new) + .collect(Collectors.toList()); + } catch (Exception e) { + throw new FlinkRuntimeException("Failed to snapshot state", e); + } + } +} +``` + +### 2.3 FlinkSourceEnumerator Adapter + +```java +public class FlinkSourceEnumerator + implements SplitEnumerator, EnumeratorStateWrapper> { + + private final SourceSplitEnumerator seaTunnelEnumerator; + private final SplitEnumeratorContext> flinkContext; + + @Override + public void start() { + try { + seaTunnelEnumerator.open(); + seaTunnelEnumerator.run(); + } catch (Exception e) { + throw new FlinkRuntimeException("Failed to start enumerator", e); + } + } + + @Override + public void handleSplitRequest(int subtaskId, @Nullable String requesterHostname) { + // Delegate to SeaTunnel enumerator + seaTunnelEnumerator.handleSplitRequest(subtaskId); + } + + @Override + public void addSplitsBack(List> splits, int subtaskId) { + // Unwrap and delegate + List unwrappedSplits = splits.stream() + .map(SplitWrapper::getSplit) + .collect(Collectors.toList()); + + seaTunnelEnumerator.addSplitsBack(unwrappedSplits, subtaskId); + } + + @Override + public void addReader(int subtaskId) { + seaTunnelEnumerator.addReader(subtaskId); + } + + @Override + public EnumeratorStateWrapper snapshotState(long checkpointId) { + try { + StateT state = seaTunnelEnumerator.snapshotState(checkpointId); + return new EnumeratorStateWrapper<>(state); + } catch (Exception e) { + throw new FlinkRuntimeException("Failed to snapshot enumerator state", e); + } + } +} +``` + +### 2.4 Context Adapters + +**FlinkSourceReaderContext**: +```java +public class FlinkSourceReaderContext + implements org.apache.seatunnel.api.source.SourceReader.Context { + + private final SourceReaderContext flinkContext; + + @Override + public int getIndexOfSubtask() { + return flinkContext.getIndexOfThisSubtask(); + } + + @Override + public void sendSplitRequest() { + // Flink automatically handles split requests + // No explicit API needed + } + + @Override + public void sendSourceEventToEnumerator(SourceEvent event) { + flinkContext.sendSourceEventToCoordinator( + new SourceEventWrapper(event) + ); + } +} +``` + +**FlinkSourceSplitEnumeratorContext**: +```java +public class FlinkSourceSplitEnumeratorContext + implements SourceSplitEnumerator.Context { + + private final SplitEnumeratorContext> flinkContext; + + @Override + public int currentParallelism() { + return flinkContext.currentParallelism(); + } + + @Override + public Set registeredReaders() { + return flinkContext.registeredReaders().keySet(); + } + + @Override + public void assignSplit(int subtaskId, List splits) { + // Wrap and delegate + List> wrappedSplits = splits.stream() + .map(SplitWrapper::new) + .collect(Collectors.toList()); + + flinkContext.assignSplits(new SplitsAssignment<>( + Collections.singletonMap(subtaskId, wrappedSplits) + )); + } + + @Override + public void signalNoMoreSplits(int subtaskId) { + flinkContext.signalNoMoreSplits(subtaskId); + } + + @Override + public void sendEventToSourceReader(int subtaskId, SourceEvent event) { + flinkContext.sendEventToSourceReader(subtaskId, new SourceEventWrapper(event)); + } +} +``` + +### 2.5 FlinkSink Adapter + +```java +public class FlinkSink + implements Sink { + + private final SeaTunnelSink seaTunnelSink; + + @Override + public SinkWriter createWriter(InitContext context) { + // Create SeaTunnel writer with adapted context + org.apache.seatunnel.api.sink.SinkWriter seaTunnelWriter = + seaTunnelSink.createWriter(new FlinkSinkWriterContext(context)); + + // Wrap in Flink adapter + return new FlinkSinkWriter<>(seaTunnelWriter); + } + + @Override + public Optional> createCommitter() { + return seaTunnelSink.createCommitter() + .map(FlinkCommitter::new); + } + + @Override + public Optional> createGlobalCommitter() { + return seaTunnelSink.createAggregatedCommitter() + .map(FlinkGlobalCommitter::new); + } + + @Override + public Optional> getCommittableSerializer() { + return seaTunnelSink.getCommitInfoSerializer() + .map(FlinkSimpleVersionedSerializer::new); + } + + @Override + public Optional> getWriterStateSerializer() { + return seaTunnelSink.getWriterStateSerializer() + .map(FlinkSimpleVersionedSerializer::new); + } +} +``` + +### 2.6 FlinkSinkWriter Adapter + +```java +public class FlinkSinkWriter + implements SinkWriter { + + private final org.apache.seatunnel.api.sink.SinkWriter seaTunnelWriter; + private long checkpointId; + + @Override + public void write(IN element, Context context) throws IOException { + // Delegate to SeaTunnel writer + seaTunnelWriter.write(element); + } + + @Override + public List prepareCommit(boolean flush) throws IOException { + Optional commitInfo = seaTunnelWriter.prepareCommit(checkpointId); + return commitInfo.map(Collections::singletonList) + .orElse(Collections.emptyList()); + } + + @Override + public List snapshotState(long checkpointId) throws IOException { + return seaTunnelWriter.snapshotState(checkpointId); + } + + @Override + public void close() throws Exception { + seaTunnelWriter.close(); + } +} +``` + +## 3. Spark Translation Layer + +Note: Spark 2.4 and Spark 3.x use different datasource APIs. SeaTunnel maintains separate Spark translation modules/adapters per Spark major version, so the exact adapter types and lifecycle hooks may differ. + +### 3.1 SparkSource Adapter + +Adapts `SeaTunnelSource` to Spark's `DataSourceReader` interface. + +```java +public class SparkSource + implements DataSourceReader { + + private final SeaTunnelSource seaTunnelSource; + + @Override + public StructType readSchema() { + // Convert SeaTunnel schema to Spark schema + CatalogTable catalogTable = seaTunnelSource.getProducedCatalogTables().get(0); + return SparkTypeConverter.convert(catalogTable.getTableSchema()); + } + + @Override + public List> planInputPartitions() { + // Create enumerator and generate splits + SourceSplitEnumerator enumerator = + seaTunnelSource.createEnumerator(new SparkEnumeratorContext()); + + try { + enumerator.open(); + enumerator.run(); + + // Collect all splits + List splits = collectAllSplits(enumerator); + + // Wrap each split as Spark InputPartition + return splits.stream() + .map(split -> new SparkInputPartition<>(seaTunnelSource, split)) + .collect(Collectors.toList()); + + } catch (Exception e) { + throw new RuntimeException("Failed to plan input partitions", e); + } + } +} +``` + +### 3.2 SparkInputPartition + +```java +public class SparkInputPartition + implements InputPartition { + + private final SeaTunnelSource seaTunnelSource; + private final SplitT split; + + @Override + public InputPartitionReader createPartitionReader() { + // Create SeaTunnel reader + org.apache.seatunnel.api.source.SourceReader seaTunnelReader = + seaTunnelSource.createReader(new SparkReaderContext()); + + // Wrap in Spark adapter + return new SparkPartitionReader<>(seaTunnelReader, split); + } +} +``` + +### 3.3 SparkPartitionReader + +```java +public class SparkPartitionReader + implements InputPartitionReader { + + private final org.apache.seatunnel.api.source.SourceReader seaTunnelReader; + private final Queue buffer = new LinkedList<>(); + + public SparkPartitionReader( + org.apache.seatunnel.api.source.SourceReader reader, + SplitT split + ) { + this.seaTunnelReader = reader; + + try { + seaTunnelReader.open(); + seaTunnelReader.addSplits(Collections.singletonList(split)); + } catch (Exception e) { + throw new RuntimeException("Failed to open reader", e); + } + } + + @Override + public boolean next() throws IOException { + if (!buffer.isEmpty()) { + return true; + } + + // Poll from SeaTunnel reader + try { + seaTunnelReader.pollNext(new Collector() { + @Override + public void collect(T record) { + // Convert to Spark InternalRow + InternalRow row = SparkTypeConverter.convert(record); + buffer.offer(row); + } + }); + + return !buffer.isEmpty(); + + } catch (Exception e) { + throw new IOException("Failed to poll next", e); + } + } + + @Override + public InternalRow get() { + return buffer.poll(); + } + + @Override + public void close() throws IOException { + try { + seaTunnelReader.close(); + } catch (Exception e) { + throw new IOException("Failed to close reader", e); + } + } +} +``` + +### 3.4 SparkSink Adapter + +```java +public class SparkSink + implements DataSourceWriter { + + private final SeaTunnelSink seaTunnelSink; + + @Override + public DataWriterFactory createWriterFactory() { + return new SparkDataWriterFactory<>(seaTunnelSink); + } + + @Override + public boolean useCommitCoordinator() { + // Use commit coordinator if sink has committer + return seaTunnelSink.createCommitter().isPresent(); + } + + @Override + public void commit(WriterCommitMessage[] messages) { + Optional> committerOpt = seaTunnelSink.createCommitter(); + + if (committerOpt.isPresent()) { + SinkCommitter committer = committerOpt.get(); + + // Extract commit infos from messages + List commitInfos = Arrays.stream(messages) + .map(msg -> ((SparkCommitMessage) msg).getCommitInfo()) + .collect(Collectors.toList()); + + // Commit + try { + List failed = committer.commit(commitInfos); + if (!failed.isEmpty()) { + throw new IOException("Some commits failed: " + failed); + } + } catch (IOException e) { + throw new RuntimeException("Failed to commit", e); + } + } + } + + @Override + public void abort(WriterCommitMessage[] messages) { + // Handle abort + Optional> committerOpt = seaTunnelSink.createCommitter(); + + if (committerOpt.isPresent()) { + SinkCommitter committer = committerOpt.get(); + + List commitInfos = Arrays.stream(messages) + .map(msg -> ((SparkCommitMessage) msg).getCommitInfo()) + .collect(Collectors.toList()); + + try { + committer.abort(commitInfos); + } catch (IOException e) { + throw new RuntimeException("Failed to abort", e); + } + } + } +} +``` + +## 4. Serialization Adapters + +### 4.1 FlinkSimpleVersionedSerializer + +```java +public class FlinkSimpleVersionedSerializer + implements SimpleVersionedSerializer { + + private final org.apache.seatunnel.api.serialization.Serializer seaTunnelSerializer; + + @Override + public int getVersion() { + // Delegate to SeaTunnel serializer + return seaTunnelSerializer.getVersion(); + } + + @Override + public byte[] serialize(T obj) throws IOException { + return seaTunnelSerializer.serialize(obj); + } + + @Override + public T deserialize(int version, byte[] serialized) throws IOException { + return seaTunnelSerializer.deserialize(serialized); + } +} +``` + +## 5. Type Conversion + +### 5.1 Spark Type Conversion + +```java +public class SparkTypeConverter { + public static StructType convert(TableSchema schema) { + List fields = new ArrayList<>(); + + for (Column column : schema.getColumns()) { + StructField field = new StructField( + column.getName(), + convertDataType(column.getDataType()), + column.isNullable(), + Metadata.empty() + ); + fields.add(field); + } + + return new StructType(fields.toArray(new StructField[0])); + } + + private static DataType convertDataType(SeaTunnelDataType seaTunnelType) { + switch (seaTunnelType.getSqlType()) { + case TINYINT: + return DataTypes.ByteType; + case SMALLINT: + return DataTypes.ShortType; + case INT: + return DataTypes.IntegerType; + case BIGINT: + return DataTypes.LongType; + case FLOAT: + return DataTypes.FloatType; + case DOUBLE: + return DataTypes.DoubleType; + case DECIMAL: + DecimalType decimalType = (DecimalType) seaTunnelType; + return DataTypes.createDecimalType( + decimalType.getPrecision(), + decimalType.getScale() + ); + case STRING: + return DataTypes.StringType; + case BOOLEAN: + return DataTypes.BooleanType; + case DATE: + return DataTypes.DateType; + case TIMESTAMP: + return DataTypes.TimestampType; + case BYTES: + return DataTypes.BinaryType; + case ARRAY: + ArrayType arrayType = (ArrayType) seaTunnelType; + return DataTypes.createArrayType( + convertDataType(arrayType.getElementType()) + ); + case MAP: + MapType mapType = (MapType) seaTunnelType; + return DataTypes.createMapType( + convertDataType(mapType.getKeyType()), + convertDataType(mapType.getValueType()) + ); + default: + throw new UnsupportedOperationException( + "Unsupported type: " + seaTunnelType); + } + } +} +``` + +## 6. Performance Considerations + +### 6.1 Translation Overhead + +Translation overhead depends on connector implementations, serialization, and type conversion complexity. Prefer measuring in your own workload rather than relying on fixed numbers. + +### 6.2 Optimization Techniques + +**Batch Type Conversion**: +```java +// ❌ BAD: Convert per record +public void collect(SeaTunnelRow record) { + InternalRow sparkRow = convertToSparkRow(record); + output.collect(sparkRow); +} + +// ✅ GOOD: Batch convert (amortize overhead) +public void collect(List records) { + InternalRow[] sparkRows = batchConvertToSparkRows(records); + for (InternalRow row : sparkRows) { + output.collect(row); + } +} +``` + +**Avoid Unnecessary Wrapping**: +```java +// If Split already serializable, don't wrap +public class SplitWrapper { + private final T split; + + // Lazy wrapping: only wrap when needed for serialization + public byte[] serialize() { + if (split instanceof Serializable) { + return directSerialize(split); // No wrapping overhead + } else { + return wrapAndSerialize(split); // Fallback + } + } +} +``` + +## 7. Limitations and Workarounds + +### 7.1 Engine-Specific Features + +**Problem**: Some engine features have no SeaTunnel equivalent. + +**Example**: Flink's `WatermarkStrategy` +```java +// Flink-specific watermark strategy cannot be expressed in SeaTunnel API +WatermarkStrategy watermarkStrategy = WatermarkStrategy + .forBoundedOutOfOrderness(Duration.ofSeconds(5)); +``` + +**Workaround**: Provide engine-specific configuration +```hocon +source { + Kafka { + # SeaTunnel config + topic = "my_topic" + + # Engine-specific config (for Flink only) + flink.watermark.strategy = "bounded-out-of-orderness" + flink.watermark.max-out-of-orderness = "5s" + } +} +``` + +### 7.2 Type System Differences + +**Problem**: Type systems don't fully align. + +**Example**: Spark has `TimestampType`, Flink has `LocalZonedTimestampType` and `TimestampType`. + +**Workaround**: Use least common denominator +```java +// SeaTunnel uses generic TIMESTAMP +// Translation layer maps to appropriate engine type based on config +``` + +## 8. Best Practices + +### 8.1 Connector Development + +**DO**: +- Implement SeaTunnel API only +- Test with multiple engines +- Use SeaTunnel types + +**DON'T**: +- Reference engine-specific APIs in connector code +- Assume specific engine behavior +- Use engine-specific optimizations + +### 8.2 Testing + +**Test on All Engines**: +```java +@RunWith(Parameterized.class) +public class ConnectorTest { + @Parameters + public static Collection engines() { + return Arrays.asList(new Object[][]{ + {"flink"}, + {"spark"}, + {"seatunnel"} + }); + } + + @Test + public void testExactlyOnce(String engine) { + // Run same test on different engines + runJobOnEngine(engine, jobConfig); + verifyResults(); + } +} +``` + +## 9. Related Resources + +- [Source Architecture](../api-design/source-architecture.md) +- [Sink Architecture](../api-design/sink-architecture.md) +- [Design Philosophy](../design-philosophy.md) + +## 10. References + +### Key Source Files + +- Flink Translation: `seatunnel-translation/seatunnel-translation-flink/` +- Spark Translation: `seatunnel-translation/seatunnel-translation-spark/` +- Base Interfaces: `seatunnel-api/src/main/java/org/apache/seatunnel/api/` + +### Further Reading + +- [Apache Flink Source API](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/sources/) +- [Apache Spark Data Source V2](https://spark.apache.org/docs/latest/sql-data-sources.html) diff --git a/docs/en/architecture/design-philosophy.md b/docs/en/architecture/design-philosophy.md new file mode 100644 index 000000000000..ff4a1f3d5473 --- /dev/null +++ b/docs/en/architecture/design-philosophy.md @@ -0,0 +1,507 @@ +--- +sidebar_position: 2 +title: Design Philosophy +--- + +# SeaTunnel Design Philosophy + +## 1. Overview + +This document explains the core design principles, philosophies, and trade-offs that shaped SeaTunnel's architecture. Understanding these principles helps contributors make consistent design decisions and users understand the system's strengths and limitations. + +## 2. Core Design Principles + +### 2.1 Engine Independence + +**Principle**: Decouple connector logic from execution engines. + +**Motivation**: +- Users may have existing infrastructure investments (Flink, Spark clusters) +- Different engines suit different scenarios (batch vs streaming, resource constraints) +- Connector developers shouldn't need to understand multiple engine APIs + +**Implementation**: +- Unified SeaTunnel API layer abstracts engine-specific details +- Translation layer adapts SeaTunnel API to engine-specific APIs +- Aim for maximum connector reuse across engines (some engine-specific adaptation may still be required via the translation layer) + +**Trade-offs**: +- **Pro**: High reusability - write once, run across engines via adapters +- **Pro**: Easier connector development - single API to learn +- **Con**: Cannot leverage engine-specific optimizations +- **Con**: Additional translation overhead +- **Mitigation**: Translation layer is thin and optimized; most overhead is in I/O, not translation + +**Example**: Connectors only implement SeaTunnel API abstractions (Source/Sink/Transform), and different execution engines complete adaptation through the translation layer; thus connector logic is decoupled from engine API changes. + +### 2.2 Separation of Coordination and Execution + +**Principle**: Separate control logic (coordination) from data processing (execution). + +**Motivation**: +- Coordination logic is single-threaded and lightweight +- Execution logic is parallel and resource-intensive +- Fault tolerance requires independent state management for each + +**Implementation Principle**: + +**Coordination Layer (Master-side)**: +- Location: Runs on master nodes with global view +- Core Responsibilities: Resource discovery, work distribution, failure detection, state coordination +- Characteristics: Single-threaded, lightweight, no actual data processing +- Managed State: Assignment plan, pending work units, global progress tracking + +**Execution Layer (Worker-side)**: +- Location: Runs on worker nodes with independent parallel execution +- Core Responsibilities: Local data processing, progress reporting, checkpoint participation +- Characteristics: Multi-threaded, resource-intensive, handles large data volumes +- Managed State: Local processing progress, buffered data, execution context + +**Communication Mechanism**: +- Coordination layer → Execution layer: Dispatches work via events (e.g., assign new data splits) +- Execution layer → Coordination layer: Reports progress via messages (e.g., split completed, request new work) +- During checkpoints: Each layer snapshots its own state independently + +**Trade-offs**: +- **Pro**: Clear separation of concerns +- **Pro**: Enumerator can reassign splits on failures +- **Pro**: Committer enables global transaction coordination +- **Con**: Additional communication overhead +- **Con**: More complex API for connector developers +- **Mitigation**: Reasonable defaults; simple connectors can use trivial enumerators/committers + +**Example**: +- Master side: Responsible for "discovering/generating work units (splits) + assignment + reclamation + state snapshots" +- Worker side: Responsible for "executing reads/writes + progress reporting + checkpoint participation" + +The key reason for this design: Fault tolerance requires distinguishing between "control state" (assigned/pending splits) and "execution progress" (offset/position per split) to enable precise recovery and fast reassignment after failures. + +### 2.3 Split-based Parallelism + +**Principle**: Divide data sources into independently processable splits. + +**Motivation**: +- Enable parallel processing without tight coordination +- Support dynamic load balancing and fault recovery +- Provide checkpoint granularity (per-split progress) + +**Implementation**: +- Data sources divided into splits (file blocks, DB partitions, Kafka partitions, etc.) +- Enumerator generates splits lazily or eagerly +- Readers process splits independently +- Unprocessed splits can be reassigned on failure + +**Trade-offs**: +- **Pro**: Excellent scalability - add workers to process more splits +- **Pro**: Fine-grained fault recovery - only failed splits need reprocessing +- **Pro**: Dynamic load balancing - assign more splits to idle workers +- **Con**: Split generation overhead for some sources +- **Con**: Requires state tracking per split +- **Mitigation**: Lazy split generation; split state is lightweight + +**Example**: +```java +// JDBC Source: Split by partition or chunk +class JdbcSourceSplit implements SourceSplit { + private final String splitId; + private final String query; // SELECT * FROM table WHERE id >= ? AND id < ? + private final long startOffset; + private final long endOffset; +} + +// File Source: Split by file or byte range +class FileSplit implements SourceSplit { + private final String filePath; + private final long startOffset; + private final long length; +} +``` + +### 2.4 Exactly-Once Semantics through Two-Phase Commit + +**Principle**: Guarantee exactly-once end-to-end data delivery. + +**Motivation**: +- Data integration must not lose or duplicate data +- Failures can occur at any time (network, process crashes) +- External systems require transactional guarantees + +**Implementation Principle**: + +Two-phase commit protocol separates data writing into two independent phases: + +1. **Prepare Phase**: + - Timing: Triggered when checkpoint barrier arrives + - Action: Writer generates "committable but not yet committed" credentials (e.g., transaction ID, temp file path) + - Constraint: No externally visible side effects (data not visible to external systems) + - State: Credential information persisted with checkpoint + +2. **Commit Phase**: + - Timing: After checkpoint completes successfully + - Action: Coordinator atomically commits changes using credentials (e.g., commit transaction, move files) + - Effect: Data becomes visible to external systems + - Guarantee: Idempotent - repeated commits have no side effects + +3. **Abort Handling**: + - Timing: When checkpoint fails or times out + - Action: Clean up temporary resources from prepare phase (e.g., rollback transaction, delete temp files) + - Effect: Ensures no partial writes or inconsistent state + +**Trade-offs**: +- **Pro**: Strong consistency guarantee +- **Pro**: Automatic recovery from failures +- **Con**: Requires transactional support in sinks (or idempotent operations) +- **Con**: Increased latency (data visible only after commit) +- **Con**: Additional state for commit info +- **Mitigation**: Optional feature; at-least-once mode available for non-transactional sinks + +**Example**: A typical exactly-once implementation follows this pattern: "the writer first generates committable credentials (commit info), and after checkpoint succeeds, the coordinator performs the final commit". This approach delays side effects (visible changes to external systems) until after checkpoint success, avoiding duplicate visible writes during failure recovery. + +### 2.5 Schema as First-Class Citizen + +**Principle**: Treat schema as explicit, typed metadata propagated through pipelines. + +**Motivation**: +- Data integration requires schema transformation and validation +- Schema evolution (DDL changes) must be handled explicitly +- Type mismatches should be caught early + +**Implementation**: +- `CatalogTable` encapsulates complete table metadata +- `TableSchema` defines structure (columns, primary key, constraints) +- Schema propagated through Source → Transform → Sink +- `SchemaChangeEvent` represents DDL changes (ADD/DROP/MODIFY columns) + +**Trade-offs**: +- **Pro**: Type safety - validate schema at job submission +- **Pro**: Schema evolution - handle DDL changes at runtime +- **Pro**: Better error messages - schema mismatches detected early +- **Con**: Additional complexity for schema-less sources +- **Con**: Schema discovery overhead for some sources +- **Mitigation**: Schema inference helpers; optional schema override + +**Example**: +```java +// Source produces typed schema +CatalogTable catalogTable = CatalogTable.of( + tableId, + TableSchema.builder() + .column("id", DataTypes.BIGINT()) + .column("name", DataTypes.STRING()) + .primaryKey("id") + .build() +); + +// Transform validates and modifies schema +public CatalogTable getProducedCatalogTable() { + return inputCatalogTable.copy( + TableSchema.builder() + .column("id", DataTypes.BIGINT()) + .column("name_upper", DataTypes.STRING()) // Transformed + .build() + ); +} +``` + +### 2.6 Plugin Architecture with Class Loader Isolation + +**Principle**: Connectors are plugins loaded dynamically with isolated dependencies. + +**Motivation**: +- Avoid dependency conflicts (e.g., multiple JDBC driver versions) +- Enable hot-pluggable connectors without core rebuild +- Reduce core distribution size + +**Implementation**: +- Java SPI for connector discovery +- Each connector has isolated class loader +- Shade plugin dependencies to avoid conflicts +- Factory pattern for instantiation + +**Trade-offs**: +- **Pro**: Dependency isolation - no version conflicts +- **Pro**: Smaller core distribution +- **Pro**: Easy to add third-party connectors +- **Con**: Class loader complexity +- **Con**: Some shared libraries (e.g., Guava) may have issues +- **Mitigation**: Careful shading; shared common libraries in core + +**Example**: +``` +seatunnel-engine/lib/ # Core libraries +connector-jdbc/lib/ # JDBC driver (isolated) +connector-kafka/lib/ # Kafka client (isolated) + +# Each connector loaded by separate ClassLoader +ConnectorClassLoader(connector-jdbc) -> loads mysql-connector-java-8.0.26.jar +ConnectorClassLoader(connector-kafka) -> loads kafka-clients-3.0.0.jar +``` + +### 2.7 State Management with Checkpoint Storage Abstraction + +**Principle**: Decouple state management from storage implementation. + +**Motivation**: +- Different deployments need different storage (HDFS, S3, local, OSS) +- State size varies widely (KBs to TBs) +- Storage durability and performance requirements differ + +**Implementation**: +- `CheckpointStorage` abstraction (FileSystem, HDFS, S3, OSS) +- Pluggable serialization for state +- Incremental checkpoint support +- Automatic state cleanup + +**Trade-offs**: +- **Pro**: Flexibility - choose storage based on deployment +- **Pro**: Incremental checkpoints reduce overhead +- **Con**: Storage performance impacts checkpoint latency +- **Con**: Requires distributed file system for production +- **Mitigation**: Async checkpoint upload; configurable intervals + +### 2.8 Multi-Table Synchronization + +**Principle**: Support synchronizing multiple tables in a single job. + +**Motivation**: +- Database migration often involves hundreds of tables +- Creating one job per table wastes resources +- Schema evolution must apply to all tables + +**Implementation**: +- `MultiTableSource` / `MultiTableSink` wrap individual table sources/sinks +- `TablePath` routes records to correct table +- Schema changes propagated per table +- Replica support for throughput + +**Trade-offs**: +- **Pro**: Resource efficiency - one job instead of hundreds +- **Pro**: Consistent snapshot across tables +- **Pro**: Centralized monitoring +- **Con**: One table failure can affect others +- **Con**: More complex error handling +- **Mitigation**: Configurable error tolerance; per-table metrics + +## 3. Architectural Trade-offs + +### 3.1 Simplicity vs Performance + +**Choice**: Favor simplicity and correctness over extreme performance optimization. + +**Rationale**: +- Data integration is I/O-bound, not CPU-bound +- Correct semantics (exactly-once) more critical than raw speed +- Simple code is maintainable and debuggable + +**Evidence**: +- Network and disk I/O dominate processing time (> 90%) +- Translation layer overhead is negligible (< 1%) +- Code readability prioritized (e.g., clear state machine, no micro-optimizations) + +### 3.2 Flexibility vs Ease of Use + +**Choice**: Provide reasonable defaults while allowing advanced customization. + +**Rationale**: +- Most users want simple configuration +- Power users need fine-grained control +- Both needs can be met with layered API + +**Implementation**: +- High-level config for common cases (e.g., `jdbc://host:port/db`) +- Low-level options for experts (e.g., connection pool tuning) +- Sensible defaults (parallelism, checkpoint interval, buffer size) + +### 3.3 Generality vs Specialization + +**Choice**: General-purpose API with specialized implementations. + +**Rationale**: +- Unified API simplifies learning and usage +- Different sources have unique characteristics (bounded vs unbounded, splitability) +- Specialization happens in connector implementations, not API + +**Example**: +- `SourceSplitEnumerator` general enough for files, databases, and message queues +- File connector uses file-based splits +- Kafka connector uses partition-based splits +- JDBC connector uses query-based splits + +### 3.4 Strong Consistency vs Latency + +**Choice**: Offer both exactly-once (high latency) and at-least-once (low latency) modes. + +**Rationale**: +- Some applications require strong consistency (financial, billing) +- Other applications tolerate duplicates for lower latency (logging, metrics) +- Let users choose based on requirements + +**Configuration**: +```hocon +env { + checkpoint.mode = "EXACTLY_ONCE" # or "AT_LEAST_ONCE" + checkpoint.interval = 60000 # ms +} +``` + +## 4. Evolution from V1 to V2 + +### 4.1 V1 Limitations + +SeaTunnel V1 (pre-2.3.0) had significant architectural limitations: + +1. **Engine-Specific Connectors**: Separate implementations for Spark and Flink +2. **No Unified API**: No abstraction layer, tight coupling to engines +3. **Limited Fault Tolerance**: Relied entirely on engine checkpointing +4. **No Schema Management**: Schema implicit, no evolution support +5. **Single-Table Only**: Multi-table synchronization not supported + +### 4.2 V2 Improvements + +SeaTunnel V2 (2.3.0+) redesigned the architecture: + +| Aspect | V1 | V2 | +|--------|----|----| +| **API** | Engine-specific | Unified SeaTunnel API | +| **Connectors** | Duplicated code | Single implementation | +| **Fault Tolerance** | Engine-dependent | Explicit checkpoint protocol | +| **Schema** | Implicit | Explicit CatalogTable | +| **Multi-Table** | Not supported | Native support | +| **Engine Support** | Spark, Flink | Spark, Flink, Zeta | +| **Exactly-Once** | Partial | End-to-end with 2PC | + +### 4.3 Migration Path + +V1 and V2 connectors coexist but use different APIs: +- V1 connectors: `seatunnel-connectors/` (deprecated) +- V2 connectors: `seatunnel-connectors-v2/` (recommended) + +V2 is the future; V1 is in maintenance mode. + +## 5. Key Design Decisions + +### 5.1 Why Separate Enumerator and Reader? + +**Alternative**: Single component handles both split generation and reading. + +**Decision**: Separate components. + +**Reasoning**: +- Split generation is coordination logic (should run on master) +- Data reading is execution logic (should run on workers) +- Failure of one shouldn't affect the other +- Allows split reassignment without reader restart + +### 5.2 Why Three-Level Sink Commit (Writer → Committer → AggregatedCommitter)? + +**Alternative**: Two-level (Writer → Committer) or direct Writer commit. + +**Decision**: Optional three-level commit. + +**Reasoning**: +- **Writer**: Parallel, stateful, per-task +- **Committer**: Parallel, stateless, aggregates per-writer commits +- **AggregatedCommitter**: Single-threaded, stateful, global coordinator + +Many sinks only need Writer + Committer; AggregatedCommitter is for complex cases (e.g., Hive table commit requiring single global operation). + +### 5.3 Why LogicalDag → PhysicalPlan Separation? + +**Alternative**: Directly generate physical execution plan from config. + +**Decision**: Two-stage planning. + +**Reasoning**: +- LogicalDag represents user intent (portable, engine-independent) +- PhysicalPlan represents execution strategy (engine-specific, optimized) +- Separation enables: + - Cross-engine portability (same LogicalDag, different PhysicalPlans) + - Optimization passes (fusion, split reassignment) + - Testing (validate logical plan separately) + +### 5.4 Why Pipeline-based Execution? + +**Alternative**: Single global task graph. + +**Decision**: Jobs divided into pipelines. + +**Reasoning**: +- Independent checkpoint coordination per pipeline +- Clearer failure boundaries +- Easier to reason about data flow +- Supports complex DAGs (multiple sources/sinks) + +### 5.5 Why Not Use Engine-Native Checkpoint? + +**Alternative**: Rely entirely on Flink/Spark checkpoint mechanisms. + +**Decision**: Explicit SeaTunnel checkpoint protocol. + +**Reasoning**: +- Engine independence - need consistent semantics across engines +- Zeta engine wouldn't have checkpointing otherwise +- More control over exactly-once semantics +- Unified monitoring and observability + +However, for Flink translation, SeaTunnel checkpoints align with Flink checkpoints to avoid duplication. + +## 6. Future Directions + +### 6.1 Planned Enhancements + +- **Dynamic Scaling**: Add/remove workers during job execution +- **Adaptive Batch Size**: Auto-tune batch sizes based on throughput +- **Query Pushdown**: Push filters/projections to sources +- **Vectorized Execution**: Process batches of rows (columnar) +- **Speculative Execution**: Mitigate stragglers + +### 6.2 Research Directions + +- **Machine Learning Integration**: ML-based optimization (split sizing, parallelism) +- **Unified Batch and Streaming**: True unified processing model +- **Global Query Optimization**: Cross-pipeline optimization + +## 7. Lessons Learned + +### 7.1 What Worked Well + +1. **Engine Independence**: Validated by successful Zeta engine addition without API changes +2. **Split-based Parallelism**: Scales well to 1000+ parallel tasks +3. **Explicit Schema**: Caught many bugs early, enabled schema evolution +4. **Two-Phase Commit**: Reliable exactly-once semantics + +### 7.2 What Could Be Better + +1. **API Complexity**: Enumerator/Committer adds learning curve for simple connectors +2. **Class Loader Issues**: Occasional conflicts with shaded dependencies +3. **Checkpoint Latency**: Large state causes checkpoint delays +4. **Documentation Gaps**: Architecture docs lagged behind code + +### 7.3 If Starting Over + +1. **Simplify API**: Provide higher-level abstractions for simple sources/sinks +2. **Async I/O Support**: First-class async API for non-blocking connectors +3. **Built-in Metrics**: Standardized metrics collection in API +4. **Schema Registry Integration**: Tighter integration with external schema registries + +## 8. Conclusion + +SeaTunnel's architecture reflects careful trade-offs between competing concerns: +- Engine independence vs engine-specific optimization +- Simplicity vs flexibility +- Consistency vs latency +- Generality vs specialization + +The V2 redesign addressed major V1 limitations while establishing principles for long-term evolution. Understanding these design philosophies helps contributors make consistent decisions and users understand SeaTunnel's strengths and appropriate use cases. + +## 9. References + +- [Architecture Overview](overview.md) +- [Source Architecture](api-design/source-architecture.md) +- [Sink Architecture](api-design/sink-architecture.md) +- [Checkpoint Mechanism](fault-tolerance/checkpoint-mechanism.md) + +### Academic Papers + +- Chandy-Lamport: ["Distributed Snapshots: Determining Global States of Distributed Systems"](https://lamport.azurewebsites.net/pubs/chandy.pdf) +- Flink: ["Apache Flink: Stream and Batch Processing in a Single Engine"](https://asterios.katsifodimos.com/assets/publications/flink-deb.pdf) diff --git a/docs/en/architecture/engine/dag-execution.md b/docs/en/architecture/engine/dag-execution.md new file mode 100644 index 000000000000..be7c8edea232 --- /dev/null +++ b/docs/en/architecture/engine/dag-execution.md @@ -0,0 +1,758 @@ +--- +sidebar_position: 2 +title: DAG Execution Model +--- + +# DAG Execution Model + +## 1. Overview + +### 1.1 Problem Background + +Distributed data processing requires transforming user intentions into executable distributed tasks: + +- **Abstraction Levels**: How to separate logical intent from physical execution? +- **Optimization**: How to optimize task placement and data shuffling? +- **Pipeline**: How to execute complex DAGs with multiple sources/sinks? +- **Parallelism**: How to determine task parallelism and distribution? +- **Fault Isolation**: How to limit failure impact to affected components? + +### 1.2 Design Goals + +SeaTunnel's DAG execution model aims to: + +1. **Separate Concerns**: Logical planning (user intent) vs physical execution (runtime details) +2. **Enable Optimization**: Task fusion, pipeline分割, resource allocation +3. **Support Complex Topologies**: Multiple sources, sinks, branches, joins +4. **Facilitate Fault Tolerance**: Clear failure boundaries with independent checkpoints +5. **Maximize Parallelism**: Efficient parallel execution with minimal coordination + +### 1.3 Execution Model Overview + +``` +User Config (HOCON) + │ + ▼ +┌─────────────────────┐ +│ LogicalDag │ Logical Plan (What to do) +│ • LogicalVertex │ - Source/Transform/Sink actions +│ • LogicalEdge │ - Data dependencies +│ • Parallelism │ - Logical parallelism +└─────────────────────┘ + │ (Plan Generation) + ▼ +┌─────────────────────┐ +│ PhysicalPlan │ Physical Plan (How to execute) +│ • SubPlan[] │ - Multiple pipelines +│ • Resources │ - Resource requirements +│ • Scheduling │ - Deployment strategy +└─────────────────────┘ + │ (Pipeline Split) + ▼ +┌─────────────────────┐ +│ SubPlan (Pipeline) │ Independent Execution Unit +│ • PhysicalVertex[] │ - Parallel task instances +│ • CheckpointCoord │ - Independent checkpointing +│ • PipelineLocation │ - Unique identifier +└─────────────────────┘ + │ (Task Deployment) + ▼ +┌─────────────────────┐ +│ PhysicalVertex │ Deployed Task Group +│ • TaskGroup │ - Co-located tasks (fusion) +│ • SlotProfile │ - Assigned resource slot +│ • ExecutionState │ - Running state +└─────────────────────┘ + │ (Execution) + ▼ +┌─────────────────────┐ +│ SeaTunnelTask │ Actual Execution +│ • Source/Transform │ - Data processing +│ • /Sink Logic │ - State management +└─────────────────────┘ +``` + +## 2. LogicalDag: User Intent + +### 2.1 Structure + +LogicalDag represents the user's job configuration in an engine-independent way. + +```java +public class LogicalDag { + // Vertices: Source, Transform, Sink actions + private final Map logicalVertexMap; + + // Edges: Data flow dependencies + private final Set edges; + + // Job configuration + private final JobConfig jobConfig; +} +``` + +### 2.2 LogicalVertex + +Represents a single action (Source/Transform/Sink) with parallelism. + +```java +public class LogicalVertex { + private final long vertexId; + private final Action action; // SourceAction, TransformChainAction, SinkAction + private final int parallelism; // Number of parallel instances +} +``` + +**Action Types**: +- **SourceAction**: Wraps `SeaTunnelSource`, produces `CatalogTable` +- **TransformChainAction**: Chain of `SeaTunnelTransform`, transforms schema +- **SinkAction**: Wraps `SeaTunnelSink`, consumes `CatalogTable` + +**Example**: +```java +// From config: +// source { JDBC { ... parallelism = 4 } } +// transform { Sql { ... parallelism = 8 } } +// sink { Elasticsearch { ... parallelism = 2 } } + +LogicalVertex sourceVertex = new LogicalVertex( + vertexId: 1, + action: new SourceAction(jdbcSource), + parallelism: 4 +); + +LogicalVertex transformVertex = new LogicalVertex( + vertexId: 2, + action: new TransformChainAction(sqlTransform), + parallelism: 8 +); + +LogicalVertex sinkVertex = new LogicalVertex( + vertexId: 3, + action: new SinkAction(esSink), + parallelism: 2 +); +``` + +### 2.3 LogicalEdge + +Represents data flow between actions. + +```java +public class LogicalEdge { + private final long inputVertexId; // Upstream vertex + private final long targetVertexId; // Downstream vertex +} +``` + +**Example**: +```java +// Source → Transform edge +LogicalEdge edge1 = new LogicalEdge( + inputVertexId: 1, // JDBC Source + targetVertexId: 2 // SQL Transform +); + +// Transform → Sink edge +LogicalEdge edge2 = new LogicalEdge( + inputVertexId: 2, // SQL Transform + targetVertexId: 3 // Elasticsearch Sink +); +``` + +### 2.4 LogicalDag Creation + +Built from user configuration: + +```java +// JobMaster creates LogicalDag +LogicalDag logicalDag = LogicalDagGenerator.generate(jobConfig); +``` + +**Process**: +1. Parse HOCON config (source, transform, sink sections) +2. Create `Action` objects for each configured component +3. Infer data flow from config structure +4. Validate schema compatibility +5. Build `LogicalDag` object + +**Example Config → LogicalDag**: +```hocon +env { + parallelism = 4 +} + +source { + JDBC { + url = "jdbc:mysql://..." + query = "SELECT * FROM orders" + } +} + +transform { + Sql { + query = "SELECT order_id, SUM(amount) FROM this GROUP BY order_id" + } +} + +sink { + Elasticsearch { + hosts = ["es-host:9200"] + index = "orders_summary" + } +} +``` + +Generated LogicalDag: +``` +Vertex 1 (JDBC Source, parallelism=4) + │ + ▼ +Vertex 2 (SQL Transform, parallelism=4) + │ + ▼ +Vertex 3 (Elasticsearch Sink, parallelism=4) +``` + +## 3. PhysicalPlan: Execution Strategy + +### 3.1 Structure + +PhysicalPlan describes how to execute the LogicalDag on distributed workers. + +```java +public class PhysicalPlan { + // Multiple pipelines (SubPlans) + private final List pipelineList; + + // Immutable job information + private final JobImmutableInformation jobImmutableInformation; + + // Distributed state (Hazelcast IMap) + private final IMap runningJobStateIMap; + private final IMap runningJobStateTimestampsIMap; + + // Job completion future + private final CompletableFuture jobEndFuture; +} +``` + +### 3.2 Pipeline Splitting + +A LogicalDag is split into multiple **Pipelines** (SubPlans) by the current `PipelineGenerator` implementation: + +1. **Unrelated Subgraphs**: Disconnected parts of the DAG become independent pipelines +2. **Multiple-Input Vertices**: If a connected subgraph contains a vertex with multiple upstream inputs, the generator splits the subgraph into multiple linear pipelines along each source→sink path and clones vertices where needed + +**Note**: Multiple sinks (branching) do not necessarily create multiple pipelines. When there is no multiple-input vertex, a branching graph is usually kept as a single pipeline. + +**Example 1: Simple Linear Pipeline**: +```hocon +source { JDBC { } } +transform { Sql { } } +sink { Elasticsearch { } } +``` + +Generated: **1 Pipeline** +``` +Pipeline 1: [JDBC Source] → [SQL Transform] → [Elasticsearch Sink] +``` + +**Example 2: Multiple Sources**: +```hocon +source { + JDBC { plugin_output = "orders" } + Kafka { plugin_output = "events" } +} + +transform { + Sql { query = "SELECT * FROM orders UNION SELECT * FROM events" } +} + +sink { + Elasticsearch { } +} +``` + +Generated: **2 Pipelines** +``` +Pipeline 1: [JDBC Source] → [SQL Transform] → [Elasticsearch Sink] +Pipeline 2: [Kafka Source] → [SQL Transform] → [Elasticsearch Sink] +``` + +**Example 3: Multiple Sinks**: +```hocon +source { + MySQL-CDC { } +} + +sink { + Elasticsearch { plugin_input = "MySQL-CDC" } + JDBC { plugin_input = "MySQL-CDC" } +} +``` + +Generated: **1 Pipeline** +``` +Pipeline 1: [MySQL-CDC Source] → ([Elasticsearch Sink], [JDBC Sink]) +``` + +### 3.3 PhysicalPlan Generation + +```java +// In JobMaster +PhysicalPlan physicalPlan = new PhysicalPlanGenerator(logicalDag, resourceManager) + .generate(); +``` + +**Steps**: +1. **Analyze LogicalDag**: Identify sources, sinks, and dependencies +2. **Split into Pipelines**: Create SubPlan for each pipeline +3. **Generate PhysicalVertices**: Create parallel instances for each action +4. **Allocate Resources**: Request slots from ResourceManager +5. **Assign Tasks**: Map PhysicalVertices to slots +6. **Create Coordinators**: Setup CheckpointCoordinator per pipeline + +## 4. SubPlan (Pipeline) + +### 4.1 Structure + +SubPlan represents an independently executing pipeline. + +```java +public class SubPlan { + private final int pipelineId; + private final PipelineLocation pipelineLocation; + + // All task instances in this pipeline + private final List physicalVertexList; + + // Coordinator tasks (Enumerator, Committer) + private final List coordinatorVertexList; + + // Checkpoint coordinator for this pipeline + private final CheckpointCoordinator checkpointCoordinator; + + // Execution state + private PipelineStatus pipelineStatus; +} +``` + +### 4.2 PhysicalVertex List + +Each LogicalVertex with parallelism N generates N PhysicalVertices. + +**Example**: +``` +LogicalVertex: JDBC Source (parallelism = 4) + ↓ +PhysicalVertices: + - PhysicalVertex (subtask 0, slot 1) + - PhysicalVertex (subtask 1, slot 2) + - PhysicalVertex (subtask 2, slot 3) + - PhysicalVertex (subtask 3, slot 4) +``` + +### 4.3 Coordinator Vertices + +Special vertices for coordination tasks: + +- **SourceSplitEnumerator**: Runs on master, assigns splits to readers +- **SinkCommitter**: Runs on master, coordinates commits +- **SinkAggregatedCommitter**: Runs on master, global commit coordination + +**Example**: +``` +SubPlan for JDBC → Transform → Elasticsearch: + physicalVertexList: + - JdbcSourceTask (4 instances) + - TransformTask (4 instances) + - ElasticsearchSinkTask (4 instances) + + coordinatorVertexList: + - JdbcSourceSplitEnumerator (1 instance, master) + - ElasticsearchSinkCommitter (1 instance, master) +``` + +### 4.4 Independent Checkpointing + +Each pipeline has its own `CheckpointCoordinator`: + +**Benefits**: +- Independent checkpoint intervals +- Isolated failure domains +- Reduced coordination overhead +- Simpler barrier alignment + +**Example**: +``` +Pipeline 1 (JDBC → ES): + CheckpointCoordinator triggers every 60s + Manages checkpoints for JDBC and ES tasks only + +Pipeline 2 (Kafka → JDBC): + CheckpointCoordinator triggers every 30s (different interval) + Manages checkpoints for Kafka and JDBC tasks only +``` + +## 5. PhysicalVertex: Deployed Task + +### 5.1 Structure + +PhysicalVertex represents a deployed task instance. + +```java +public class PhysicalVertex { + private final TaskGroupLocation taskGroupLocation; + private final TaskGroupDefaultImpl taskGroup; + + // Assigned resource slot + private final SlotProfile slotProfile; + + // Execution state (CREATED, RUNNING, FAILED, etc.) + private ExecutionState currentExecutionState; + + // Plugin jars (for class loader isolation) + private final List> pluginJarsUrls; +} +``` + +### 5.2 TaskGroup: Task Fusion + +Multiple tasks can be fused into a single `TaskGroup` for efficiency. + +```java +public class TaskGroupDefaultImpl implements TaskGroup { + private final TaskGroupLocation taskGroupLocation; + + // Multiple tasks in this group + private final Set tasks; + + // Shared thread pool + private final ExecutorService executorService; + + // Shared network buffers + private final Map>> internalChannels; +} +``` + +**Fusion Conditions**: +1. Same parallelism +2. Sequential dependency (A → B) +3. No data shuffle required + +**Example (with fusion)**: +``` +LogicalDag: + Source (parallelism=4) → Transform (parallelism=4) → Sink (parallelism=4) + +Without Fusion: + 12 separate tasks (4 + 4 + 4) + Network overhead for Source → Transform and Transform → Sink + +With Fusion: + 4 TaskGroups, each containing: + [SourceTask → TransformTask → SinkTask] (single thread, shared memory) +``` + +**Benefits**: +- Reduced network serialization/deserialization +- Better CPU cache locality +- Lower memory footprint +- Simplified deployment + +### 5.3 Slot Assignment + +Each PhysicalVertex is assigned a `SlotProfile`: + +```java +public class SlotProfile { + private final long slotID; + private final Address workerAddress; + private final ResourceProfile resourceProfile; // CPU, memory +} +``` + +**Assignment Process**: +1. JobMaster requests slots from ResourceManager +2. ResourceManager selects workers based on strategy (random, slot ratio, load) +3. ResourceManager allocates slots and returns SlotProfiles +4. JobMaster assigns SlotProfiles to PhysicalVertices +5. JobMaster deploys tasks via `DeployTaskOperation` + +## 6. Task Deployment and Execution + +### 6.1 Deployment Flow + +```mermaid +sequenceDiagram + participant JM as JobMaster + participant RM as ResourceManager + participant Worker as Worker Node + participant Task as SeaTunnelTask + + JM->>JM: Generate PhysicalPlan + JM->>RM: applyResources(resourceProfiles) + RM->>RM: Allocate slots + RM-->>JM: Return SlotProfiles + + JM->>JM: Assign slots to PhysicalVertices + + loop For each PhysicalVertex + JM->>Worker: DeployTaskOperation(taskGroup) + Worker->>Task: Create SeaTunnelTask + Task->>Task: INIT → WAITING_RESTORE + Task->>JM: Report ready + end + + JM->>Worker: Start execution + Worker->>Task: READY_START → STARTING → RUNNING +``` + +### 6.2 Task Execution + +Each `SeaTunnelTask` executes its assigned action: + +**SourceSeaTunnelTask**: +```java +while (isRunning()) { + // Poll data from SourceReader + sourceReader.pollNext(collector); + + // Handle checkpoint barriers + if (checkpointTriggered) { + triggerBarrier(checkpointId); + } +} +``` + +**TransformSeaTunnelTask**: +```java +while (isRunning()) { + // Read from input queue + Record record = inputQueue.take(); + + // Apply transform + Record transformed = transform.map(record); + + // Write to output queue + outputQueue.put(transformed); +} +``` + +**SinkSeaTunnelTask**: +```java +while (isRunning()) { + // Read from input queue + Record record = inputQueue.take(); + + // Write to sink + sinkWriter.write(record); + + // Handle checkpoint barriers + if (barrierReceived) { + commitInfo = sinkWriter.prepareCommit(checkpointId); + snapshotState(checkpointId); + } +} +``` + +## 7. Optimization Strategies + +### 7.1 Task Fusion + +**When to Fuse**: +- Same parallelism +- Sequential operators (no branching) +- No shuffle boundary + +**When NOT to Fuse**: +- Different parallelism (e.g., source=4, sink=8) +- Branching DAG (one source, multiple sinks) +- Shuffle required (e.g., GROUP BY, JOIN) + +Task fusion behavior and controls are engine-implementation specific. Avoid relying on undocumented `env.job.mode` values in architecture examples. + +### 7.2 Parallelism Inference + +Parallelism resolution (SeaTunnel Engine / Zeta): + +- If an action/connector config specifies `parallelism`, it takes precedence +- Otherwise use `env.parallelism` (default is `1`) + +**Example**: +```hocon +env { parallelism = 1 } + +source { + JDBC { parallelism = 4 } # Explicit +} + +transform { + Sql { } # Inferred: 4 (from source) +} + +sink { + Elasticsearch { } # Inferred: 4 (from transform) +} +``` + +### 7.3 Resource Allocation + +**Slot Calculation**: +``` +Required Slots = Sum of all task parallelism + +Example: + Source (parallelism=4) + Transform (parallelism=4) + Sink (parallelism=2) + = 10 slots required + +With Fusion: + TaskGroup (parallelism=4, fusion[Source+Transform]) + Sink (parallelism=2) + = 6 slots required +``` + +**Resource Profile**: +```java +ResourceProfile profile = + new ResourceProfile( + CPU.of(1), // 1 CPU core + Memory.of(512 * 1024 * 1024L) // 512MB heap (bytes) + ); +``` + +## 8. Failure Handling + +### 8.1 Task Failure + +**Detection**: +- Task throws exception +- Heartbeat timeout + +**Recovery**: +1. Mark task as FAILED +2. Fail entire pipeline (conservative) +3. Restore from latest checkpoint +4. Reallocate resources +5. Redeploy and restart pipeline + +### 8.2 Pipeline Failure Isolation + +**Key Insight**: Pipeline failures are isolated. + +**Example**: +``` +Job with 2 pipelines: + Pipeline 1: JDBC → ES (RUNNING) + Pipeline 2: Kafka → JDBC (FAILED) + +Result: + Pipeline 2 restarts from checkpoint + Pipeline 1 continues unaffected +``` + +**Benefits**: +- Reduced blast radius +- Faster recovery (only failed pipeline) +- Better resource utilization + +## 9. Monitoring and Observability + +### 9.1 Key Metrics + +**Pipeline-Level**: +- Pipeline status and lifecycle transitions (CREATED / RUNNING / FINISHED / FAILED) +- Task counts and placement across workers/slots +- Checkpoint progress (latest checkpoint id, duration, failures) + +**Task-Level**: +- Task status and restart counters +- Record/byte throughput (in/out) +- Backpressure / queueing indicators (engine-dependent) + +### 9.2 Visualization + +``` +Job: mysql-to-es +│ +├── Pipeline 1 (mysql-cdc → elasticsearch) +│ ├── PhysicalVertex 0 [RUNNING] @ worker-1:slot-1 +│ ├── PhysicalVertex 1 [RUNNING] @ worker-2:slot-1 +│ ├── PhysicalVertex 2 [RUNNING] @ worker-3:slot-1 +│ └── PhysicalVertex 3 [RUNNING] @ worker-4:slot-1 +│ +└── Pipeline 2 (mysql-cdc → jdbc) + ├── PhysicalVertex 0 [RUNNING] @ worker-1:slot-2 + └── PhysicalVertex 1 [RUNNING] @ worker-2:slot-2 +``` + +## 10. Best Practices + +### 10.1 Parallelism Configuration + +**Rule of Thumb**: +``` +Parallelism = min( + data partitions, + available slots, + target throughput / single-task throughput +) +``` + +**Examples**: +- **JDBC Source**: Set to number of DB partitions (e.g., 8 partitions → parallelism=8) +- **Kafka Source**: Set to number of partitions (e.g., 32 partitions → parallelism=32) +- **File Source**: Set to number of files or file splits +- **CPU-Intensive Transform**: Set to number of CPU cores +- **I/O-Intensive Sink**: Set based on target system capacity + +### 10.2 Pipeline Design + +**Keep Pipelines Simple**: +- Prefer linear pipelines (Source → Transform → Sink) +- Avoid complex branching when possible +- Use multiple jobs for completely independent workflows + +**Use Multiple Jobs When**: +- Different checkpoint intervals needed +- Different resource requirements +- Independent failure domains desired + +### 10.3 Troubleshooting + +**Problem**: Tasks not starting + +**Check**: +1. Enough available slots? (`required_slots <= available_slots`) +2. Resource profile reasonable? (not requesting 100 CPU cores) +3. Tag filters correct? (if using tag-based assignment) + +**Problem**: Low throughput + +**Check**: +1. Parallelism too low? (increase parallelism) +2. Task fusion disabled? (enable for better performance) +3. Checkpoint interval too short? (increase interval) + +## 11. Related Resources + +- [Engine Architecture](engine-architecture.md) +- [Resource Management](resource-management.md) +- [Checkpoint Mechanism](../fault-tolerance/checkpoint-mechanism.md) +- [Architecture Overview](../overview.md) + +## 12. References + +### Key Source Files + +- [LogicalDag.java](../../../seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/dag/logical/LogicalDag.java) +- [PhysicalPlan.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/PhysicalPlan.java) +- [SubPlan.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java) +- [PhysicalVertex.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/PhysicalVertex.java) +- [TaskGroupDefaultImpl.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/task/group/TaskGroupDefaultImpl.java) + +### Further Reading + +- [Google Borg Paper](https://research.google/pubs/pub43438/) - Task scheduling inspiration +- [Apache Flink JobGraph](https://nightlies.apache.org/flink/flink-docs-stable/docs/internals/job_scheduling/) +- [Spark DAG Scheduler](https://spark.apache.org/docs/latest/job-scheduling.html) diff --git a/docs/en/architecture/engine/engine-architecture.md b/docs/en/architecture/engine/engine-architecture.md new file mode 100644 index 000000000000..5c666fb42c30 --- /dev/null +++ b/docs/en/architecture/engine/engine-architecture.md @@ -0,0 +1,700 @@ +--- +sidebar_position: 1 +title: Engine Architecture +--- + +# SeaTunnel Engine (Zeta) Architecture + +## 1. Overview + +### 1.1 Problem Background + +Data integration engines must solve fundamental distributed systems challenges: + +- **Distributed Execution**: How to execute jobs across multiple machines? +- **Resource Management**: How to allocate and schedule tasks efficiently? +- **Fault Tolerance**: How to recover from worker/master failures? +- **Coordination**: How to synchronize distributed tasks (checkpoints, commits)? +- **Scalability**: How to handle increasing workloads? + +### 1.2 Design Goals + +SeaTunnel Engine (Zeta) is designed as a native execution engine with: + +1. **Lightweight**: Minimal dependencies, fast startup, low resource overhead +2. **High Performance**: Optimized for data synchronization workloads +3. **Fault Tolerance**: Checkpoint-based recovery with exactly-once semantics +4. **Resource Efficiency**: Slot-based resource management with fine-grained control +5. **Engine Independence**: Supports same connector API as Flink/Spark translations + +### 1.3 Architecture Comparison + +| Feature | SeaTunnel Zeta | Apache Flink | Apache Spark | +|---------|---------------|--------------|--------------| +| **Primary Use Case** | Data sync, CDC | Stream processing | Batch + ML | +| **Resource Model** | Slot-based | Slot-based | Executor-based | +| **State Backend** | Pluggable (HDFS/S3/Local) | RocksDB/Heap | In-memory/Disk | +| **Checkpoint** | Distributed snapshots | Chandy-Lamport | RDD lineage | +| **Operational Complexity** | Lower (engine-native) | Higher | Higher | + +## 2. Overall Architecture + +### 2.1 Master-Worker Architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Master Node │ +│ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ CoordinatorService │ │ +│ │ • Manages all running jobs │ │ +│ │ • Job submission and lifecycle management │ │ +│ │ • Maintains job state (IMap) │ │ +│ │ • Resource manager factory │ │ +│ └───────────────────────────────────────────────────────┘ │ +│ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ JobMaster (one per job) │ │ +│ │ • Generates physical execution plan │ │ +│ │ • Requests resources from ResourceManager │ │ +│ │ • Deploys tasks to workers │ │ +│ │ • Coordinates checkpoints │ │ +│ │ • Handles failover and recovery │ │ +│ └───────────────────────────────────────────────────────┘ │ +│ │ │ │ +│ │ (Task Deploy) │ (Resource Request) │ +│ ▼ ▼ │ +│ ┌─────────────────┐ ┌────────────────────────────┐ │ +│ │ CheckpointManager│ │ ResourceManager │ │ +│ │ (per pipeline) │ │ • Slot allocation │ │ +│ └─────────────────┘ │ • Worker registration │ │ +│ │ • Load balancing │ │ +│ └────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ + │ + │ (Hazelcast Cluster) + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Worker Nodes │ +│ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ TaskExecutionService │ │ +│ │ • Deploys and executes tasks │ │ +│ │ • Manages task lifecycle │ │ +│ │ • Reports heartbeat │ │ +│ │ • Slot resource management │ │ +│ └───────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ SeaTunnelTask (multiple per worker) │ │ +│ │ │ │ +│ │ ┌─────────────────────────────────────────────┐ │ │ +│ │ │ SourceFlowLifeCycle │ │ │ +│ │ │ • SourceReader │ │ │ +│ │ │ • SeaTunnelSourceCollector │ │ │ +│ │ └─────────────────────────────────────────────┘ │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ ┌─────────────────────────────────────────────┐ │ │ +│ │ │ TransformFlowLifeCycle │ │ │ +│ │ │ • Transform chain │ │ │ +│ │ └─────────────────────────────────────────────┘ │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ ┌─────────────────────────────────────────────┐ │ │ +│ │ │ SinkFlowLifeCycle │ │ │ +│ │ │ • SinkWriter │ │ │ +│ │ └─────────────────────────────────────────────┘ │ │ +│ └───────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### 2.2 Core Components + +#### CoordinatorService + +Centralized service managing all jobs in the cluster. + +**Responsibilities**: +- Accept job submissions +- Create JobMaster for each job +- Maintain job state in distributed IMap +- Provide job query and management APIs +- Handle job lifecycle events + +**Key Data Structures**: +```java +// Running job state (distributed IMap backed by Hazelcast) +IMap runningJobInfoIMap; +IMap runningJobStateIMap; +IMap runningJobStateTimestampsIMap; + +// Completed job history +IMap completedJobInfoIMap; +``` + +**Code Reference**: +- [CoordinatorService.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java) + +#### JobMaster + +Manages single job execution lifecycle. + +**Responsibilities**: +- Parse configuration → generate LogicalDag +- Generate PhysicalPlan from LogicalDag +- Request resources (slots) from ResourceManager +- Deploy tasks to workers +- Coordinate pipeline checkpoints +- Handle task failures and reschedule + +**Lifecycle**: +``` +Created → Initialized → Scheduled → Running → Finished/Failed/Canceled +``` + +**Key Operations**: +1. `init()`: Generate physical plan, create checkpoint coordinators +2. `run()`: Request resources, deploy tasks, start execution +3. `handleFailure()`: Restart failed tasks, restore from checkpoint + +**Code Reference**: +- [JobMaster.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobMaster.java) + +#### ResourceManager + +Manages worker resources and slot allocation. + +**Responsibilities**: +- Track worker registration and heartbeat +- Maintain worker resource profiles (CPU, memory) +- Allocate slots based on strategies (random, slot ratio, load-based) +- Release slots after task completion +- Handle worker failures + +**Slot Allocation Strategies**: +```java +// 1. Random: Random selection among available workers +// 2. SlotRatio: Prefer workers with more available slots +// 3. SystemLoad: Prefer workers with lower CPU/memory usage +``` + +**Code Reference**: +- [ResourceManager.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/resourcemanager/ResourceManager.java) +- [AbstractResourceManager.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/resourcemanager/AbstractResourceManager.java) + +## 3. DAG Execution Model + +### 3.1 Execution Plan Transformation + +``` +User Config (HOCON) + │ + ▼ +┌───────────────┐ +│ LogicalDag │ • Logical vertices (Source/Transform/Sink) +│ │ • Logical edges (data flow) +│ │ • Parallelism (per vertex) +└───────────────┘ + │ (JobMaster.generatePhysicalPlan()) + ▼ +┌───────────────┐ +│ PhysicalPlan │ • List of SubPlan (pipelines) +│ │ • JobImmutableInformation +│ │ • Resource requirements +└───────────────┘ + │ + ▼ +┌───────────────┐ +│ SubPlan │ • Pipeline (independent execution unit) +│ (Pipeline) │ • List of PhysicalVertex +│ │ • CheckpointCoordinator +└───────────────┘ + │ + ▼ +┌───────────────┐ +│PhysicalVertex │ • TaskGroup (co-located tasks) +│ │ • Assigned SlotProfile +│ │ • ExecutionState +└───────────────┘ + │ + ▼ +┌───────────────┐ +│ TaskGroup │ • Multiple SeaTunnelTask instances +│ │ • Shared network buffer +│ │ • Thread pool +└───────────────┘ + │ + ▼ +┌───────────────┐ +│ SeaTunnelTask │ • Single task execution +│ │ • Source/Transform/Sink lifecycle +│ │ • Task state machine +└───────────────┘ +``` + +### 3.2 LogicalDag + +Represents user's intent in engine-independent way. + +```java +public class LogicalDag { + private final Map logicalVertexMap; + private final Set edges; + private final JobConfig jobConfig; +} + +public class LogicalVertex { + private final long vertexId; + private final Action action; // SourceAction / TransformChainAction / SinkAction + private final int parallelism; +} + +public class LogicalEdge { + private final long inputVertexId; + private final long targetVertexId; +} +``` + +**Creation**: +```java +// From user config +LogicalDag logicalDag = LogicalDagBuilder.build(jobConfig); +``` + +### 3.3 PhysicalPlan + +Represents actual execution plan with resource allocation. + +```java +public class PhysicalPlan { + private final List pipelineList; + private final JobImmutableInformation jobImmutableInformation; + private final CompletableFuture jobEndFuture; +} + +public class SubPlan { + private final int pipelineId; + private final List physicalVertexList; + private final List coordinatorVertexList; + private final CheckpointCoordinator checkpointCoordinator; +} + +public class PhysicalVertex { + private final TaskGroupLocation taskGroupLocation; + private final TaskGroupDefaultImpl taskGroup; + private final SlotProfile slotProfile; // Assigned slot + private final ExecutionState currentExecutionState; +} +``` + +**Generation**: +```java +PhysicalPlan physicalPlan = jobMaster.getPhysicalPlan(); +// JobMaster internally: +// 1. Split LogicalDag into pipelines +// 2. Generate PhysicalVertex for each parallel instance +// 3. Create CheckpointCoordinator per pipeline +``` + +### 3.4 Pipeline Execution + +Jobs are divided into **Pipelines** (SubPlans) for independent execution: + +**Example**: +```hocon +# Config with multiple sources/sinks +env { ... } + +source { + MySQL-CDC { table = "orders" } + Kafka { topic = "events" } +} + +transform { + Sql { query = "SELECT * FROM orders JOIN events ON ..." } +} + +sink { + Elasticsearch { index = "orders" } + JDBC { table = "events" } +} +``` + +**Generated Pipelines**: +``` +Pipeline 1: MySQL-CDC → Transform → Elasticsearch +Pipeline 2: Kafka → Transform → JDBC +``` + +**Benefits**: +- Independent checkpoint coordination +- Isolated failure domains +- Parallel pipeline execution + +### 3.5 Task Fusion + +Multiple actions can be fused into single TaskGroup for efficiency: + +``` +Without Fusion: +[Source Task] → Network → [Transform Task] → Network → [Sink Task] + +With Fusion: +[TaskGroup: Source → Transform → Sink] (single thread, no network) +``` + +**Fusion Conditions**: +- Same parallelism +- Sequential dependency +- No shuffle required + +## 4. Task Lifecycle + +### 4.1 Task State Machine + +``` + [Created] + │ + ▼ + [INIT] ────────────────────────────────────┐ + │ │ + ▼ │ +[WAITING_RESTORE] (if recovering) │ + │ │ + ▼ │ + [READY_START] │ + │ │ + ▼ │ + [STARTING] ──────────────┐ │ + │ │ │ + ▼ ▼ ▼ + [RUNNING] ──────────> [FAILED] ─────> (Restart) + │ + ▼ +[PREPARE_CLOSE] + │ + ▼ + [CLOSED] + │ + ▼ + [CANCELED] (if job canceled) +``` + +**State Transitions**: +1. **CREATED → INIT**: Task created, initializing resources +2. **INIT → WAITING_RESTORE**: Recovering from checkpoint +3. **WAITING_RESTORE → READY_START**: State restored +4. **READY_START → STARTING**: Opening Source/Transform/Sink +5. **STARTING → RUNNING**: Data processing started +6. **RUNNING → PREPARE_CLOSE**: Normal completion +7. **PREPARE_CLOSE → CLOSED**: Resources cleaned up +8. **RUNNING → FAILED**: Exception occurred + +### 4.2 SeaTunnelTask Execution + +```java +public abstract class SeaTunnelTask implements Runnable { + private final TaskLocation taskLocation; + private final TaskExecutionContext executionContext; + private ExecutionState executionState; + + @Override + public void run() { + try { + init(); + restoreState(); // If recovering + open(); + + while (isRunning()) { + processData(); // Source: read, Transform: process, Sink: write + handleBarrier(); // Checkpoint barriers + } + + close(); + } catch (Exception e) { + handleException(e); + } + } +} +``` + +**Task Types**: +- **SourceSeaTunnelTask**: Runs SourceReader, emits data +- **SinkSeaTunnelTask**: Runs SinkWriter, consumes data +- **TransformSeaTunnelTask**: Runs Transform chain + +### 4.3 FlowLifeCycle Management + +Each task manages component lifecycle through FlowLifeCycle: + +```java +// Source task +public class SourceFlowLifeCycle implements FlowLifeCycle { + private final SourceReader sourceReader; + private final SeaTunnelSourceCollector collector; + + @Override + public void open() { + sourceReader.open(); + } + + @Override + public void collect() { + sourceReader.pollNext(collector); // Read data + } + + @Override + public void close() { + sourceReader.close(); + } +} + +// Sink task +public class SinkFlowLifeCycle implements FlowLifeCycle { + private final SinkWriter sinkWriter; + + @Override + public void collect() { + T record = inputQueue.poll(); + sinkWriter.write(record); // Write data + } +} +``` + +## 5. Checkpoint Coordination + +### 5.1 CheckpointCoordinator (per Pipeline) + +Each pipeline has independent checkpoint coordinator. + +**Responsibilities**: +- Trigger checkpoint periodically +- Inject checkpoint barriers into data flow +- Collect task acknowledgements +- Persist completed checkpoints +- Clean up old checkpoints + +**Key Data Structures**: +```java +public class CheckpointCoordinator { + private final CheckpointIDCounter checkpointIdCounter; + private final Map pendingCheckpoints; + private final ArrayDeque completedCheckpointIds; + private final CheckpointStorage checkpointStorage; +} +``` + +**Checkpoint Flow**: +1. Coordinator triggers checkpoint (periodic or manual) +2. Send barriers to all source tasks in pipeline +3. Barriers propagate through data flow +4. Each task snapshots state upon receiving barrier +5. Tasks send ACK back to coordinator +6. Coordinator waits for all ACKs +7. Create CompletedCheckpoint, persist to storage + +**Code Reference**: +- [CheckpointCoordinator.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointCoordinator.java) + +### 5.2 Checkpoint Barrier + +Special control message that flows with data: + +```java +public class Barrier { + private final long checkpointId; + private final long timestamp; + private final CheckpointType type; // CHECKPOINT or SAVEPOINT +} +``` + +**Barrier Alignment**: +- Tasks with multiple inputs wait for barrier from ALL inputs before snapshotting +- Ensures consistent snapshot across distributed tasks + +## 6. Resource Management + +### 6.1 Slot Model + +**SlotProfile**: +```java +public class SlotProfile { + private final int slotID; + private final Address worker; + private final ResourceProfile resourceProfile; // CPU, memory +} + +public class ResourceProfile { + private final CPU cpu; + private final Memory heapMemory; +} +``` + +**WorkerProfile**: +```java +public class WorkerProfile { + private final Address address; + private final ResourceProfile profile; + private final ResourceProfile unassignedResource; + private final SlotProfile[] assignedSlots; + private final SlotProfile[] unassignedSlots; + private final Map attributes; +} +``` + +### 6.2 Resource Allocation Flow + +```mermaid +sequenceDiagram + participant JM as JobMaster + participant RM as ResourceManager + participant Worker as Worker Node + + JM->>RM: applyResources(jobId, resourceProfiles) + RM->>RM: Select workers (strategy) + RM->>RM: Allocate slots + RM->>JM: Return slot profiles + + JM->>Worker: Deploy task (DeployTaskOperation) + Worker->>Worker: Create SeaTunnelTask + Worker->>JM: ACK + + JM->>JM: Task running +``` + +### 6.3 Tag-Based Slot Filtering + +Assign tasks to specific worker groups: + +```hocon +env { + # Job-level worker attribute filter (key/value full match) + tag_filter = { + zone = "db-zone" + } +} +``` + +**Usage**: +- Data locality (assign to workers near data source) +- Resource isolation (GPU workers for ML transforms) +- Multi-tenancy (different teams use different worker pools) + +## 7. Failure Handling + +### 7.1 Task Failure + +**Detection**: +- Task reports exception to JobMaster +- JobMaster monitors task heartbeat +- Timeout triggers failure detection + +**Recovery**: +1. Mark task as FAILED +2. Release task's slot +3. Retrieve latest successful checkpoint +4. Restart task with restored state +5. Reassign splits (for Source tasks) + +### 7.2 Worker Failure + +**Detection**: +- ResourceManager monitors worker heartbeat +- Hazelcast cluster detects member removal + +**Recovery**: +1. Mark all tasks on failed worker as FAILED +2. Trigger job failover +3. Restore from latest checkpoint +4. Reallocate slots on healthy workers +5. Redeploy tasks + +### 7.3 Master Failure + +**High Availability**: +- Multiple master nodes (Hazelcast cluster) +- Job state stored in distributed IMap (replicated) +- New master takes over from IMap state + +**Recovery**: +1. Detect master failure (Hazelcast) +2. Elect new master +3. New master reads job state from IMap +4. Reconnect to workers +5. Resume checkpoint coordination + +## 8. Design Considerations + +### 8.1 Why Pipeline-based Execution? + +**Alternative**: Single global DAG execution + +**Decision**: Divide into pipelines + +**Benefits**: +- Independent checkpoint coordination (less coordination overhead) +- Clear failure boundaries (one pipeline fails, others continue) +- Easier to reason about data flow +- Support complex DAGs (multiple sources/sinks) + +**Drawbacks**: +- Cannot fuse tasks across pipeline boundaries +- Potential data serialization between pipelines + +### 8.2 Why Hazelcast for Coordination? + +**Alternative**: Zookeeper, etcd, custom Raft implementation + +**Decision**: Hazelcast IMDG + +**Benefits**: +- In-memory distributed data structures (low latency) +- Built-in cluster management and failure detection +- Easy to embed (no external dependencies) +- Familiar API (Java Collections) + +**Drawbacks**: +- Memory overhead for large state +- Less battle-tested than Zookeeper for coordination + +### 8.3 Performance Optimizations + +**1. Task Fusion**: +- Reduce network overhead +- Improve CPU cache locality +- Lower serialization cost + +**2. Async Checkpoint**: +- Checkpoint upload doesn't block data processing +- Parallel checkpoint across tasks + +**3. Incremental Checkpoint**: +- Only upload changed state (future enhancement) + +**4. Zero-Copy Data Transfer**: +- Shared memory between co-located tasks +- Avoid unnecessary serialization + +## 9. Related Resources + +- [Architecture Overview](../overview.md) +- [Design Philosophy](../design-philosophy.md) +- [Checkpoint Mechanism](../fault-tolerance/checkpoint-mechanism.md) +- [Resource Management](resource-management.md) +- [DAG Execution](dag-execution.md) + +## 10. References + +### Key Source Files + +- Engine Core: `seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/` +- DAG: `seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/dag/` +- Checkpoint: `seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/` + +### Further Reading + +- [Hazelcast IMDG](https://docs.hazelcast.com/imdg/latest/) +- [Google Borg Paper](https://research.google/pubs/pub43438/) - Inspiration for resource management +- [Apache Flink Architecture](https://flink.apache.org/flink-architecture.html) diff --git a/docs/en/architecture/engine/resource-management.md b/docs/en/architecture/engine/resource-management.md new file mode 100644 index 000000000000..fb0a3b294f7c --- /dev/null +++ b/docs/en/architecture/engine/resource-management.md @@ -0,0 +1,534 @@ +--- +sidebar_position: 3 +title: Resource Management +--- + +# Resource Management + +## 1. Overview + +### 1.1 Problem Background + +Distributed execution engines must efficiently manage computing resources: + +- **Resource Allocation**: How to assign tasks to workers fairly and efficiently? +- **Load Balancing**: How to distribute workload evenly across workers? +- **Resource Isolation**: How to prevent resource contention between jobs? +- **Dynamic Scaling**: How to add/remove workers without disrupting jobs? +- **Heterogeneous Resources**: How to handle workers with different capabilities? + +### 1.2 Design Goals + +SeaTunnel's resource management system aims to: + +1. **Fine-Grained Control**: Slot-based allocation for precise resource management +2. **Flexible Strategies**: Multiple allocation strategies for different scenarios +3. **Tag-Based Filtering**: Assign tasks to specific worker groups +4. **High Availability**: Tolerate worker failures with automatic reassignment +5. **Observability**: Track resource usage and availability in real-time + +### 1.3 Architecture Overview + +``` +┌──────────────────────────────────────────────────────────────┐ +│ JobMaster │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ Request Resources │ │ +│ │ • Calculate required slots │ │ +│ │ • Specify resource profiles (CPU, memory) │ │ +│ │ • Apply tag filters (optional) │ │ +│ └────────────────────────────────────────────────────┘ │ +└──────────────────────────────┬───────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ ResourceManager │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ Worker Registry │ │ +│ │ • WorkerProfile (per worker) │ │ +│ │ - Total resources │ │ +│ │ - Available resources │ │ +│ │ - Assigned slots │ │ +│ │ - Unassigned slots │ │ +│ └────────────────────────────────────────────────────┘ │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ Allocation Strategies │ │ +│ │ • RandomStrategy / SlotRatioStrategy / SystemLoadStrategy │ +│ └────────────────────────────────────────────────────┘ │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ Slot Management │ │ +│ │ • Allocate slots │ │ +│ │ • Release slots │ │ +│ │ • Track slot usage │ │ +│ └────────────────────────────────────────────────────┘ │ +└──────────────────────────────┬───────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ Worker Nodes │ +│ │ +│ Worker 1 Worker 2 Worker N │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Slot 1 │ │ Slot 1 │ │ Slot 1 │ │ +│ │ Slot 2 │ │ Slot 2 │ │ Slot 2 │ │ +│ │ ... │ │ ... │ │ ... │ │ +│ └──────────┘ └──────────┘ └──────────┘ │ +└──────────────────────────────────────────────────────────────┘ +``` + +## 2. Core Concepts + +### 2.1 Slot + +A **Slot** is the fundamental unit of resource allocation. + +```java +public class SlotProfile { + // Unique slot identifier + private final int slotID; + + // Worker address where this slot resides + private final Address worker; + + // Resource capacity of this slot + private final ResourceProfile resourceProfile; +} +``` + +**Key Properties**: +- **Granular**: Each slot can host one or more tasks (task fusion) +- **Typed**: Slots have resource profiles (CPU, memory) +- **Stateful**: Slots track assignment status (assigned/unassigned) + +**Example**: +```java +SlotProfile slot = + new SlotProfile( + new Address("worker-1", 5801), + 1001, + new ResourceProfile(CPU.of(1), Memory.of(512 * 1024 * 1024L)), + "seq-1" + ); +``` + +### 2.2 ResourceProfile + +Describes resource requirements or capacity. + +```java +public class ResourceProfile { + private final CPU cpu; + private final Memory heapMemory; +} + +public class CPU { + private final int core; // Number of CPU cores +} + +public class Memory { + private final long bytes; // Heap memory in bytes +} +``` + +**Usage**: +- **Task Requirements**: JobMaster specifies required resources per task +- **Slot Capacity**: Each slot advertises its available resources +- **Matching**: ResourceManager matches task requirements to slot capacity + +### 2.3 WorkerProfile + +Represents a worker node's resources and slot inventory. + +```java +public class WorkerProfile { + // Worker address + private final Address address; + + // Total resources (all slots combined) + private final ResourceProfile profile; + + // Currently available resources + private final ResourceProfile unassignedResource; + + // Slots assigned to jobs + private final SlotProfile[] assignedSlots; + + // Slots available for assignment + private final SlotProfile[] unassignedSlots; + + // Worker attributes (used by job-level tag_filter) + private final Map attributes; + + // Optional system load info (for SystemLoadStrategy) + private final SystemLoadInfo systemLoadInfo; +} +``` + +**Lifecycle**: +1. **Registration**: Worker registers with ResourceManager on startup +2. **Heartbeat**: Worker sends periodic heartbeats with updated resource info +3. **Allocation**: ResourceManager assigns slots from unassigned pool +4. **Release**: Completed tasks free slots, moving them back to unassigned pool +5. **Deregistration**: Worker leaves cluster (graceful or failure) + +## 3. Resource Manager + +### 3.1 Interface + +```java +public interface ResourceManager { + /** + * Apply for resources (called by JobMaster) + */ + CompletableFuture> applyResources( + long jobId, + List resourceProfiles, + Map tagFilter + ) throws NoEnoughResourceException; + + /** + * Release resources (called by JobMaster after task completion) + */ + CompletableFuture releaseResources(long jobId, List slots); + + /** + * Worker heartbeat (called by TaskExecutionService) + */ + void heartbeat(WorkerProfile workerProfile); + + /** + * Handle worker removal (failure or graceful shutdown) + */ + void memberRemoved(MembershipServiceEvent event); +} +``` + +### 3.2 Implementation: AbstractResourceManager + +```java +public abstract class AbstractResourceManager implements ResourceManager { + // Registered workers + protected final ConcurrentMap registerWorker; + + // Worker selection strategy (RandomStrategy / SlotRatioStrategy / SystemLoadStrategy) + protected final SlotAllocationStrategy slotAllocationStrategy; + + @Override + public CompletableFuture> applyResources( + long jobId, + List resourceProfiles, + Map tagFilter + ) throws NoEnoughResourceException { + // 1. Filter workers by tagFilter (match worker attributes) + Map candidates = filterWorkerByTag(tagFilter); + + // 2. For each requested profile, select a worker by strategy and pick an unassigned slot + // (actual slot selection/marking is implementation-defined) + return requestSlots(jobId, resourceProfiles, candidates, slotAllocationStrategy); + } +} +``` + +## 4. Slot Allocation Strategies + +In SeaTunnel Engine / Zeta, allocation typically consists of: +1. Select a candidate worker (strategy) +2. Pick an unassigned slot from that worker + +### 4.1 RandomStrategy + +Randomly selects a worker from the available candidates. + +```java +public class RandomStrategy implements SlotAllocationStrategy { + @Override + public Optional selectWorker(List availableWorkers) { + Collections.shuffle(availableWorkers); + return availableWorkers.stream().findFirst(); + } +} +``` + +### 4.2 SlotRatioStrategy + +Selects the worker with the lowest slot usage ratio (prefers workers with more available slots). + +### 4.3 SystemLoadStrategy + +Selects the worker with the lowest system load (based on heartbeat-reported load information). + +## 5. Tag-Based Slot Filtering + +### 5.1 Use Cases + +**Data Locality**: +```hocon +env { + # Job-level worker attribute filter (full key/value match) + tag_filter = { + zone = "us-west-1" + } +} +``` + +**Resource Specialization**: +```hocon +env { + tag_filter = { + resource = "gpu" + } +} +``` + +**Multi-Tenancy**: +```hocon +env { + job.name = "tenant-a-job" + tag_filter = { + tenant = "a" + } +} +``` + +### 5.2 Matching Semantics + +The engine matches `env.tag_filter` against worker `attributes` (key/value full match). If no worker matches, resource allocation fails. + +## 6. Resource Allocation Flow + +### 6.1 Normal Allocation + +```mermaid +sequenceDiagram + participant JM as JobMaster + participant RM as ResourceManager + participant Worker as Worker Node + + JM->>JM: Generate PhysicalPlan + JM->>JM: Calculate required resources + + JM->>RM: applyResources(profiles, tags) + + RM->>RM: Filter workers by tags + RM->>RM: Select workers by strategy + RM->>RM: Allocate slots + + RM-->>JM: Return SlotProfiles + + JM->>JM: Assign slots to PhysicalVertices + + loop For each task + JM->>Worker: DeployTaskOperation(task, slot) + Worker->>Worker: Execute task in slot + Worker-->>JM: ACK + end +``` + +### 6.2 Insufficient Resources + +```mermaid +sequenceDiagram + participant JM as JobMaster + participant RM as ResourceManager + + JM->>RM: applyResources(100 slots) + + RM->>RM: Check available slots + Note over RM: Only 50 slots available + + RM-->>JM: NoEnoughResourceException + + JM->>JM: Retry with backoff + Note over JM: Wait for resources to free up + + JM->>RM: applyResources(100 slots) + RM-->>JM: Success (after resources freed) +``` + +### 6.3 Resource Release + +```mermaid +sequenceDiagram + participant Task as SeaTunnelTask + participant JM as JobMaster + participant RM as ResourceManager + + Task->>Task: Task completes/fails + + Task->>JM: Task finished + + JM->>RM: releaseResources(slots) + + RM->>RM: Mark slots as unassigned + RM->>RM: Update WorkerProfile + + Note over RM: Slots available for
new allocations +``` + +## 7. Failure Handling + +### 7.1 Worker Failure + +**Detection**: +- Heartbeat timeout (default: 60 seconds) +- Hazelcast member removed event + +**Recovery**: +```java +@Override +public void memberRemoved(MembershipEvent event) { + Address failedWorker = event.getMember().getAddress(); + + // 1. Remove worker from registry + WorkerProfile failed = registerWorker.remove(failedWorker); + + // 2. Notify JobMasters of slot losses + List lostSlots = failed.getAssignedSlots(); + for (SlotProfile slot : lostSlots) { + long jobId = getJobIdForSlot(slot); + JobMaster jobMaster = getJobMaster(jobId); + + // 3. Trigger job failover + jobMaster.notifySlotLost(slot); + } +} +``` + +**JobMaster Response**: +1. Mark tasks on failed slots as FAILED +2. Restore from latest checkpoint +3. Request new slots from ResourceManager +4. Redeploy tasks + +### 7.2 ResourceManager Failure + +**High Availability**: +- ResourceManager state is stateless (worker registry rebuilt from heartbeats) +- New ResourceManager instance starts on master failover +- Workers re-register via heartbeat mechanism + +**Recovery**: +- Worker liveness is determined by heartbeat updates and cluster membership events (exact timeout/threshold is implementation/config-dependent) + +## 8. Configuration + +### 8.1 Slot Configuration + +Example (`config/seatunnel.yaml`, SeaTunnel Engine / Zeta): + +```yaml +seatunnel: + engine: + slot-service: + dynamic-slot: true + slot-num: 16 + slot-allocate-strategy: RANDOM # RANDOM / SLOT_RATIO / SYSTEM_LOAD +``` + +## 9. Monitoring and Metrics + +### 9.1 Key Metrics + +**Cluster-Level**: +- Worker count and liveness (registered vs active) +- Slot inventory and utilization (assigned vs unassigned) + +**Per-Worker**: +- CPU/memory utilization (if reported) +- Slots assigned/unassigned + +**Per-Job**: +- Slots requested/allocated +- Resource wait time (if available) + +### 9.2 Observability + +**Resource Dashboard Example**: +``` +Cluster Resources: + Workers: 10 (all healthy) + Total Slots: 20 + Available Slots: 8 + Utilization: 60% + +Top Resource Consumers: + job-123: 6 slots (mysql-cdc → elasticsearch) + job-456: 4 slots (kafka → jdbc) + job-789: 2 slots (file → s3) + +Worker Distribution: + worker-1: 2/2 slots (100%) + worker-2: 1/2 slots (50%) + worker-3: 2/2 slots (100%) + ... +``` + +## 10. Best Practices + +### 10.1 Slot Sizing + +Slot sizing (slots per worker, heap per slot, etc.) depends on workload characteristics and deployment constraints. Avoid treating formulas in architecture docs as mandatory defaults. + +### 10.2 Strategy Selection + +**Use RandomStrategy when**: +- Homogeneous cluster (all workers identical) +- Simple deployments +- Fast allocation more important than perfect balance + +**Use SlotRatioStrategy when**: +- Need good load balancing +- Mixed job sizes +- Moderate cluster size (< 100 workers) + +**Use SystemLoadStrategy when**: +- Heterogeneous cluster +- Workers have varying CPU/memory +- Optimizing resource utilization is critical + +### 10.3 Tag Usage + +**Data Locality**: +```hocon +env { + # Match worker attributes, e.g., zone=us-west-1a + tag_filter = { + zone = "us-west-1a" + } +} +``` + +**Resource Isolation**: +```hocon +env { + job.name = "critical-job" + tag_filter = { + priority = "high" + } +} +``` + +## 11. Related Resources + +- [Engine Architecture](engine-architecture.md) +- [DAG Execution](dag-execution.md) +- [Architecture Overview](../overview.md) + +## 12. References + +### Key Source Files + +- [ResourceManager.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/resourcemanager/ResourceManager.java) +- [AbstractResourceManager.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/resourcemanager/AbstractResourceManager.java) +- [SlotProfile.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/resourcemanager/resource/SlotProfile.java) +- [WorkerProfile.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/resourcemanager/worker/WorkerProfile.java) + +### Further Reading + +- [Google Borg](https://research.google/pubs/pub43438/) - Large-scale cluster management +- [Apache YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html) - Resource management in Hadoop +- [Kubernetes](https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/) - Container orchestration and scheduling diff --git a/docs/en/architecture/fault-tolerance/checkpoint-mechanism.md b/docs/en/architecture/fault-tolerance/checkpoint-mechanism.md new file mode 100644 index 000000000000..7902873b388d --- /dev/null +++ b/docs/en/architecture/fault-tolerance/checkpoint-mechanism.md @@ -0,0 +1,759 @@ +--- +sidebar_position: 1 +title: Checkpoint Mechanism +--- + +# Checkpoint Mechanism + +## 1. Overview + +### 1.1 Problem Background + +Distributed data processing systems face critical challenges for fault tolerance: + +- **State Loss**: How to preserve processing state across failures? +- **Exactly-Once**: How to ensure each record is processed exactly once? +- **Distributed Consistency**: How to create consistent snapshots across distributed tasks? +- **Performance**: How to checkpoint without blocking data processing? +- **Recovery**: How to efficiently restore state after failures? + +### 1.2 Design Goals + +SeaTunnel's checkpoint mechanism aims to: + +1. **Guarantee Exactly-Once Semantics**: Consistent state snapshots + two-phase commit +2. **Minimize Overhead**: Asynchronous checkpoint, no data processing blocking +3. **Fast Recovery**: Restore from latest checkpoint in seconds +4. **Distributed Coordination**: Coordinate checkpoints across hundreds of tasks +5. **Pluggable Storage**: Support multiple storage backends (HDFS, S3, Local, OSS) + +### 1.3 Theoretical Foundation + +SeaTunnel's checkpoint is based on the **Chandy-Lamport distributed snapshot algorithm**: + +**Key Idea**: Insert special markers (barriers) into data streams. When a task receives barrier: +1. Snapshot its local state +2. Forward barrier downstream +3. Continue processing + +Result: Globally consistent snapshot without pausing entire system. + +**Reference**: ["Distributed Snapshots: Determining Global States of Distributed Systems"](https://lamport.azurewebsites.net/pubs/chandy.pdf) (Chandy & Lamport, 1985) + +## 2. Architecture Design + +### 2.1 Checkpoint Architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ JobMaster (per job) │ +│ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ CheckpointCoordinator (per pipeline) │ │ +│ │ │ │ +│ │ • Trigger checkpoint (periodic/manual) │ │ +│ │ • Generate checkpoint ID │ │ +│ │ • Track pending checkpoints │ │ +│ │ • Collect task acknowledgements │ │ +│ │ • Persist completed checkpoints │ │ +│ │ • Cleanup old checkpoints │ │ +│ └───────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ (Trigger Barrier) │ +│ ▼ │ +└─────────────────────────────────────────────────────────────────┘ + │ + │ (CheckpointBarrier) + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Worker Nodes │ +│ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ SourceTask 1 │ │ SourceTask 2 │ │ SourceTask N │ │ +│ │ │ │ │ │ │ │ +│ │ 1. Receive │ │ 1. Receive │ │ 1. Receive │ │ +│ │ Barrier │ │ Barrier │ │ Barrier │ │ +│ │ 2. Snapshot │ │ 2. Snapshot │ │ 2. Snapshot │ │ +│ │ State │ │ State │ │ State │ │ +│ │ 3. ACK │ │ 3. ACK │ │ 3. ACK │ │ +│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ +│ │ │ │ │ +│ │ (Barrier Propagation) │ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ Transform 1 │ │ Transform 2 │ │ Transform N │ │ +│ │ │ │ │ │ │ │ +│ │ 1. Receive │ │ 1. Receive │ │ 1. Receive │ │ +│ │ Barrier │ │ Barrier │ │ Barrier │ │ +│ │ 2. Snapshot │ │ 2. Snapshot │ │ 2. Snapshot │ │ +│ │ State │ │ State │ │ State │ │ +│ │ 3. ACK │ │ 3. ACK │ │ 3. ACK │ │ +│ │ 4. Forward │ │ 4. Forward │ │ 4. Forward │ │ +│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ SinkTask 1 │ │ SinkTask 2 │ │ SinkTask N │ │ +│ │ │ │ │ │ │ │ +│ │ 1. Receive │ │ 1. Receive │ │ 1. Receive │ │ +│ │ Barrier │ │ Barrier │ │ Barrier │ │ +│ │ 2. Prepare │ │ 2. Prepare │ │ 2. Prepare │ │ +│ │ Commit │ │ Commit │ │ Commit │ │ +│ │ 3. Snapshot │ │ 3. Snapshot │ │ 3. Snapshot │ │ +│ │ State │ │ State │ │ State │ │ +│ │ 4. ACK │ │ 4. ACK │ │ 4. ACK │ │ +│ └──────────────┘ └──────────────┘ └──────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ + │ + │ (All ACKs received) + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ CheckpointStorage │ +│ (HDFS / S3 / Local / OSS) │ +│ │ +│ CompletedCheckpoint { │ +│ checkpointId: 123 │ +│ taskStates: { │ +│ SourceTask-1: { splits: [...], offsets: [...] } │ +│ SinkTask-1: { commitInfo: XidInfo(...) } │ +│ ... │ +│ } │ +│ } │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### 2.2 Key Data Structures + +#### CheckpointCoordinator + +```java +public class CheckpointCoordinator { + // Checkpoint ID generator + private final CheckpointIDCounter checkpointIdCounter; + + // Checkpoint execution plan + private final CheckpointPlan checkpointPlan; + + // Pending checkpoints (in progress) + private final Map pendingCheckpoints; + + // Completed checkpoints (success) + private final ArrayDeque completedCheckpointIds; + + // Latest completed checkpoint + private CompletedCheckpoint latestCompletedCheckpoint; + + // Checkpoint storage + private final CheckpointStorage checkpointStorage; + + // Configuration + private final long checkpointInterval; // Trigger interval (ms) + private final long checkpointTimeout; // Timeout (ms) + private final int minPauseBetweenCheckpoints; // Min pause (ms) +} +``` + +#### PendingCheckpoint + +Represents in-progress checkpoint. + +```java +public class PendingCheckpoint { + private final long checkpointId; + private final CheckpointType checkpointType; // CHECKPOINT or SAVEPOINT + private final long triggerTimestamp; + + // Tasks that haven't acknowledged yet + private final Set notYetAcknowledgedTasks; + + // Collected action states (from task ACKs) + private final Map actionStates; + + // Task statistics (records processed, bytes, etc.) + private final Map taskStatistics; + + // Future completed when all tasks ACK + private final CompletableFuture completableFuture; + + /** + * Called when task acknowledges checkpoint + */ + public void acknowledgeTask(long taskId, List states, + TaskStatistics statistics) { + notYetAcknowledgedTasks.remove(taskId); + + // Collect states + for (ActionSubtaskState state : states) { + actionStates.computeIfAbsent(state.getKey(), k -> new ActionState()) + .putSubtaskState(state); + } + + // Collect statistics + taskStatistics.put(taskId, statistics); + + // Check if all tasks acknowledged + if (notYetAcknowledgedTasks.isEmpty()) { + completeCheckpoint(); + } + } + + private void completeCheckpoint() { + CompletedCheckpoint completed = new CompletedCheckpoint( + checkpointId, actionStates, taskStatistics, System.currentTimeMillis() + ); + completableFuture.complete(completed); + } +} +``` + +#### CompletedCheckpoint + +Persisted checkpoint data. + +```java +public class CompletedCheckpoint implements Serializable { + private final long checkpointId; + private final Map taskStates; + private final Map taskStatistics; + private final long completedTimestamp; +} + +public class ActionState implements Serializable { + private final ActionStateKey key; // (pipelineId, actionId) + private final Map subtaskStates; +} + +public class ActionSubtaskState implements Serializable { + private final int subtaskIndex; + private final byte[] state; // Serialized state +} +``` + +### 2.3 CheckpointStorage + +Abstraction for checkpoint persistence. + +```java +public interface CheckpointStorage { + /** + * Store completed checkpoint + */ + void storeCheckpoint(CompletedCheckpoint checkpoint) throws IOException; + + /** + * Get latest checkpoint + */ + Optional getLatestCheckpoint() throws IOException; + + /** + * Get specific checkpoint by ID + */ + Optional getCheckpoint(long checkpointId) throws IOException; + + /** + * Delete old checkpoint + */ + void deleteCheckpoint(long checkpointId) throws IOException; +} +``` + +**Implementations**: +- `LocalFileStorage`: Local file system (testing) +- `HdfsStorage`: Hadoop FileSystem-based backend; can work with HDFS/S3A/etc depending on Hadoop configuration + +Note: S3 and OSS support are provided through Hadoop FileSystem configuration (e.g., `fs.s3a.impl`) rather than separate CheckpointStorage implementations. + +## 3. Checkpoint Flow + +### 3.1 Trigger Checkpoint + +```mermaid +sequenceDiagram + participant Timer as Periodic Timer + participant Coord as CheckpointCoordinator + participant Plan as CheckpointPlan + + Timer->>Coord: Trigger (every 60s) + Coord->>Coord: Generate checkpointId (123) + + Coord->>Coord: Check conditions + Note over Coord: • Min pause elapsed?
• Max concurrent not exceeded?
• Previous checkpoint complete? + + Coord->>Coord: Create PendingCheckpoint(123) + Coord->>Plan: Get starting tasks + + loop For each starting task + Coord->>Task: Send CheckpointBarrierTriggerOperation(123) + end + + Coord->>Coord: Start timeout timer (10 minutes) +``` + +**Trigger Conditions**: +1. Checkpoint interval elapsed (e.g., 60 seconds) +2. Minimum pause between checkpoints elapsed (e.g., 10 seconds) +3. Number of concurrent checkpoints < max (e.g., 1) +4. No checkpoint in progress (for single concurrent) + +### 3.2 Barrier Propagation + +```mermaid +sequenceDiagram + participant Coord as Coordinator + participant Source as SourceTask + participant Transform as TransformTask + participant Sink as SinkTask + + Coord->>Source: Trigger barrier(123) + + Source->>Source: Receive barrier + Source->>Source: snapshotState() → splits, offsets + Source->>Coord: ACK(state) + Source->>Transform: Forward barrier(123) + + Transform->>Transform: Receive barrier + Transform->>Transform: snapshotState() → transform state + Transform->>Coord: ACK(state) + Transform->>Sink: Forward barrier(123) + + Sink->>Sink: Receive barrier + Sink->>Sink: prepareCommit(checkpointId) → commitInfo + Sink->>Sink: snapshotState() → writer state + Sink->>Coord: ACK(commitInfo + state) + + Coord->>Coord: All ACKs received + Coord->>Coord: Create CompletedCheckpoint +``` + +**Barrier Flow Rules**: +1. **Source Tasks**: Start of pipeline, receive barrier from coordinator +2. **Transform Tasks**: Receive from upstream, snapshot, forward downstream +3. **Sink Tasks**: End of pipeline, receive from upstream, snapshot, no forward + +**Barrier Alignment** (for tasks with multiple inputs): +```java +// Task with 2 inputs +Input 1: ──data──data──[barrier-123]──data──data── + │ Wait! +Input 2: ──data──data──data──data──[barrier-123]── + │ + ▼ + Both barriers received, snapshot state +``` + +### 3.3 State Snapshot + +Each task type snapshots different state: + +**SourceTask**: +```java +@Override +public void triggerBarrier(long checkpointId) { + // 1. Snapshot SourceReader state (splits + offsets) + List states = sourceFlowLifeCycle.snapshotState(checkpointId); + + // 2. Create ActionSubtaskState + ActionSubtaskState state = new ActionSubtaskState(subtaskIndex, states); + + // 3. Send ACK to coordinator + sendAcknowledgement(checkpointId, Collections.singletonList(state)); + + // 4. Forward barrier downstream + forwardBarrierToDownstream(checkpointId); +} +``` + +**TransformTask**: +```java +@Override +public void triggerBarrier(long checkpointId) { + // 1. Snapshot Transform state (usually stateless, empty state) + List states = transformFlowLifeCycle.snapshotState(checkpointId); + + // 2. Create ActionSubtaskState + ActionSubtaskState state = new ActionSubtaskState(subtaskIndex, states); + + // 3. Send ACK + sendAcknowledgement(checkpointId, Collections.singletonList(state)); + + // 4. Forward barrier + forwardBarrierToDownstream(checkpointId); +} +``` + +**SinkTask**: +```java +@Override +public void triggerBarrier(long checkpointId) { + // 1. Prepare commit (TWO-PHASE COMMIT) + Optional commitInfo = sinkWriter.prepareCommit(checkpointId); + + // 2. Snapshot writer state + List writerStates = sinkWriter.snapshotState(checkpointId); + + // 3. Create ActionSubtaskState (includes both commit info and state) + ActionSubtaskState state = new ActionSubtaskState( + subtaskIndex, + serialize(writerStates), + commitInfo.orElse(null) + ); + + // 4. Send ACK (NO forwarding - end of pipeline) + sendAcknowledgement(checkpointId, Collections.singletonList(state)); +} +``` + +### 3.4 Checkpoint Completion + +```mermaid +sequenceDiagram + participant Coord as CheckpointCoordinator + participant Pending as PendingCheckpoint + participant Storage as CheckpointStorage + participant Committer as SinkCommitter + participant Tasks as All Tasks + + Pending->>Pending: All tasks ACKed + + Pending->>Coord: notifyCheckpointComplete() + + Coord->>Coord: Create CompletedCheckpoint + Coord->>Storage: Persist checkpoint + Storage-->>Coord: Success + + Coord->>Committer: commit(commitInfos) + Committer-->>Coord: Success + + Coord->>Tasks: notifyCheckpointComplete(123) + Tasks->>Tasks: Cleanup resources + + Coord->>Storage: Delete old checkpoints +``` + +**Completion Steps**: +1. All tasks acknowledged +2. Create `CompletedCheckpoint` from `PendingCheckpoint` +3. Persist checkpoint to storage +4. Trigger sink commit (two-phase commit) +5. Notify all tasks of completion +6. Cleanup old checkpoints (retain last N) + +### 3.5 Checkpoint Timeout + +```java +// CheckpointCoordinator +private void startCheckpointTimeout(long checkpointId, long timeoutMs) { + scheduledExecutor.schedule(() -> { + PendingCheckpoint pending = pendingCheckpoints.get(checkpointId); + if (pending != null && !pending.isCompleted()) { + LOG.warn("Checkpoint {} timeout after {}ms, {} tasks not yet acknowledged", + checkpointId, timeoutMs, pending.getNotYetAcknowledgedTasks()); + + // Fail checkpoint + pending.abort(); + pendingCheckpoints.remove(checkpointId); + + // Trigger job failover if needed + handleCheckpointFailure(checkpointId); + } + }, timeoutMs, TimeUnit.MILLISECONDS); +} +``` + +**Timeout Handling**: +- Default timeout: 10 minutes +- If timeout, checkpoint fails +- Job continues with previous checkpoint +- Next checkpoint will be triggered per schedule + +## 4. Recovery Process + +### 4.1 Restore from Checkpoint + +```mermaid +sequenceDiagram + participant JM as JobMaster + participant Storage as CheckpointStorage + participant Source as SourceTask + participant Sink as SinkTask + + JM->>Storage: getLatestCheckpoint() + Storage-->>JM: CompletedCheckpoint(123) + + JM->>JM: Extract states per task + + JM->>Source: Deploy with NotifyTaskRestoreOperation + activate Source + Source->>Source: restoreState(splits, offsets) + Source->>Source: Seek to checkpointed offset + Source-->>JM: Ready + deactivate Source + + JM->>Sink: Deploy with NotifyTaskRestoreOperation + activate Sink + Sink->>Sink: restoreWriter(writerState) + Sink->>Sink: Restore uncommitted transactions + Sink-->>JM: Ready + deactivate Sink + + JM->>Source: Start execution + JM->>Sink: Start execution +``` + +**Restore Steps**: +1. JobMaster retrieves latest `CompletedCheckpoint` from storage +2. Extract state for each task (by ActionStateKey and subtaskIndex) +3. Deploy tasks with `NotifyTaskRestoreOperation` containing state +4. Tasks restore state: + - **SourceReader**: Restore splits and offsets, seek to position + - **Transform**: Restore transform state (usually none) + - **SinkWriter**: Restore writer state, may have uncommitted transactions +5. Tasks transition to READY_START state +6. Job resumes execution + +**Example: JDBC Source Recovery**: +```java +public class JdbcSourceReader { + @Override + public void restoreState(List states) { + for (JdbcSourceState state : states) { + JdbcSourceSplit split = state.getSplit(); + long offset = state.getCurrentOffset(); + + // Restore split with offset + pendingSplits.add(split); + + // When processing split, start from offset + String query = split.getQuery() + " OFFSET " + offset; + } + } +} +``` + +### 4.2 Exactly-Once Recovery + +Combination of checkpoint restore + sink two-phase commit ensures exactly-once: + +``` +Checkpoint N (completed): + Source offsets: [100, 200, 300] + Sink prepared commits: [XID-1, XID-2, XID-3] + Sink committer commits XID-1, XID-2, XID-3 + + ↓ [Failure] + +Recovery from Checkpoint N: + 1. Restore source offsets: [100, 200, 300] + 2. Sources start reading from offset 100, 200, 300 + 3. Sink writers restore state (may have uncommitted XIDs) + 4. Sink committer retries committing XIDs (idempotent) + +Result: Records 0-99, 100-199, 200-299 committed exactly once + Records from 100+ reprocessed but not duplicated (idempotent commit) +``` + +## 5. Configuration and Tuning + +### 5.1 Checkpoint Configuration + +```hocon +env { + # Enable checkpoint + checkpoint.interval = 60000 # Trigger every 60 seconds + + # Checkpoint timeout + checkpoint.timeout = 600000 # 10 minutes + + # Min pause between checkpoints + min-pause = 10000 # 10 seconds +} +``` + +Checkpoint storage is configured on the engine side (e.g., `config/seatunnel.yaml` under `seatunnel.engine.checkpoint.storage`), rather than as job-level `env` options. + +### 5.2 Tuning Guidelines + +**Checkpoint Interval**: +- **Shorter interval**: Faster recovery, higher overhead +- **Longer interval**: Lower overhead, slower recovery + +**Trade-offs**: +- Shorter interval → More frequent I/O → Higher storage cost +- Longer interval → Less overhead → Longer recovery time + +**Rule of Thumb**: Set interval to tolerable recovery time (data loss window). + +**Checkpoint Timeout**: +- Should be >> checkpoint interval +- Depends on state size and storage speed +- Choose based on end-to-end latency, state size, and checkpoint storage throughput + +**Storage Selection (SeaTunnel Engine)**: +- `localfile` (LocalFileStorage): local filesystem, non-HA +- `hdfs` (HdfsStorage): Hadoop FileSystem-based backend; can work with HDFS/S3A/etc depending on Hadoop configuration + +## 6. Performance Optimization + +### 6.1 Async Checkpoint + +State snapshot doesn't block data processing: + +```java +public class AsyncSnapshotSupport { + @Override + public void snapshotState(long checkpointId) { + // 1. Create snapshot of current state (fast, in-memory copy) + StateSnapshot snapshot = createSnapshot(); + + // 2. Continue data processing (doesn't wait for serialization/upload) + // ... + + // 3. Async serialize and upload + CompletableFuture.runAsync(() -> { + byte[] serialized = serialize(snapshot); + checkpointStorage.upload(checkpointId, serialized); + }, executorService); + } +} +``` + +### 6.2 Incremental Checkpoint (Future) + +Only checkpoint changed state: + +```java +// Full checkpoint (first) +Checkpoint 1: State = 1GB → Upload 1GB + +// Incremental checkpoints (subsequent) +Checkpoint 2: State = 1.1GB → Upload 100MB (delta) +Checkpoint 3: State = 1.05GB → Upload 0MB (deletion doesn't upload) +``` + +**Benefits**: +- Reduce checkpoint time +- Lower storage I/O +- Faster checkpoint completion + +**Challenges**: +- More complex state management +- Need to track state changes +- Restore requires chain of deltas + +### 6.3 Local State Backend (Future) + +Store hot state locally, checkpoint only summary: + +```java +// RocksDB local state backend +class RocksDBStateBackend { + private final RocksDB rocksDB; // Fast local SSD + + @Override + public void put(String key, byte[] value) { + rocksDB.put(key.getBytes(), value); // Local write (fast) + } + + @Override + public byte[] snapshotState() { + // Only checkpoint RocksDB snapshot reference + return rocksDB.createCheckpoint().getBytes(); + } +} +``` + +## 7. Best Practices + +### 7.1 State Size Optimization + +**1. Keep State Small**: +```java +// ❌ BAD: Buffer entire dataset +class BadSourceReader { + private List bufferedRows = new ArrayList<>(); // May be huge! + + List snapshotState() { + return serialize(bufferedRows); // Huge state + } +} + +// ✅ GOOD: Track offset only +class GoodSourceReader { + private long currentOffset = 0; + + List snapshotState() { + return serialize(currentOffset); // Small state + } +} +``` + +**2. Use Efficient Serialization**: +- Prefer Protobuf, Kryo over Java serialization +- Compress large state (gzip, snappy) + +### 7.2 Monitoring + +**Key Metrics**: +- `checkpoint_duration`: Time from trigger to completion +- `checkpoint_size`: Size of persisted checkpoint +- `checkpoint_failure_rate`: Percentage of failed checkpoints +- `checkpoint_alignment_duration`: Time spent aligning barriers + +**Alerting**: +- Alert if `checkpoint_duration` > threshold (e.g., 5 minutes) +- Alert if `checkpoint_failure_rate` > 10% +- Alert if no checkpoint completed in 2x interval + +### 7.3 Troubleshooting + +**Problem**: Checkpoint timeout + +**Possible Causes**: +1. Task stuck (slow data processing) +2. Large state (slow serialization/upload) +3. Slow storage (network/disk I/O) +4. Barrier alignment slow (skewed data) + +**Solutions**: +- Increase checkpoint timeout +- Optimize state size +- Use faster storage +- Tune parallelism + +**Problem**: High checkpoint overhead + +**Possible Causes**: +1. Checkpoint interval too short +2. Large state size +3. Slow storage + +**Solutions**: +- Increase checkpoint interval +- Optimize state size +- Enable incremental checkpoint (when available) + +## 8. Related Resources + +- [Architecture Overview](../overview.md) +- [Design Philosophy](../design-philosophy.md) +- [Engine Architecture](../engine/engine-architecture.md) +- [Sink Architecture](../api-design/sink-architecture.md) +- [Exactly-Once Semantics](exactly-once.md) + +## 9. References + +### Key Source Files + +- [CheckpointCoordinator.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointCoordinator.java) +- [PendingCheckpoint.java](../../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/PendingCheckpoint.java) +- [CheckpointStorage.java](../../../seatunnel-engine/seatunnel-engine-storage/checkpoint-storage-api/src/main/java/org/apache/seatunnel/engine/checkpoint/storage/api/CheckpointStorage.java) + +### Academic Papers + +- Chandy, K. M., & Lamport, L. (1985). ["Distributed Snapshots: Determining Global States of Distributed Systems"](https://lamport.azurewebsites.net/pubs/chandy.pdf) +- Carbone, P., et al. (2017). ["State Management in Apache Flink"](http://www.vldb.org/pvldb/vol10/p1718-carbone.pdf) + +### Further Reading + +- [Apache Flink Checkpointing](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/fault-tolerance/checkpointing/) +- [Spark Structured Streaming Checkpointing](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing) diff --git a/docs/en/architecture/fault-tolerance/exactly-once.md b/docs/en/architecture/fault-tolerance/exactly-once.md new file mode 100644 index 000000000000..420ef34381c9 --- /dev/null +++ b/docs/en/architecture/fault-tolerance/exactly-once.md @@ -0,0 +1,780 @@ +--- +sidebar_position: 2 +title: Exactly-Once Semantics +--- + +# Exactly-Once Semantics + +## 1. Overview + +### 1.1 Problem Background + +Distributed data processing faces fundamental delivery guarantees challenges: + +- **At-Most-Once**: Records may be lost (unacceptable for critical data) +- **At-Least-Once**: Records may be duplicated (causes counting errors, double charges) +- **Exactly-Once**: Each record processed exactly once (ideal but complex) + +**Real-World Impact**: +``` +Scenario: Financial transaction processing + +At-Least-Once: + Transaction $100 processed twice → User charged $200 ❌ + +Exactly-Once: + Transaction $100 processed once → User charged $100 ✅ +``` + +### 1.2 Design Goals + +SeaTunnel's exactly-once semantics aims to: + +1. **Verifiable End-to-End Consistency**: With checkpoint boundaries + sink transactional/idempotent commits, avoid data loss/duplication under the documented failure model +2. **Transparent Implementation**: Framework handles complexity, users configure minimally +3. **Performance Efficiency**: Minimize overhead while maintaining guarantee +4. **Failure Resilience**: Maintain guarantee across task/worker/master failures +5. **Broad Applicability**: Support transactional sinks and also provide practical semantics for non-transactional sinks (e.g., idempotent writes / at-least-once) + +### 1.3 Consistency Levels + +| Level | Guarantee | Use Cases | Implementation | +|-------|-----------|-----------|----------------| +| **At-Most-Once** | No duplicates, may lose | Non-critical logs | No retry | +| **At-Least-Once** | No loss, may duplicate | Idempotent processing | Retry without transaction | +| **Exactly-Once** | No loss, no duplicates | Financial, billing, audit | Checkpoint + 2PC | + +## 2. Theoretical Foundation + +### 2.1 Chandy-Lamport Algorithm + +**Concept**: Distributed snapshot without stopping the entire system. + +**Mechanism**: +1. Coordinator injects **barriers** (markers) into data streams +2. Upon receiving barrier, each operator: + - Snapshots its local state + - Forwards barrier downstream +3. When all operators snapshot, we have a **consistent global snapshot** + +**Key Property**: Snapshot represents a consistent cut across distributed system state. + +### 2.2 Two-Phase Commit Protocol + +**Concept**: Atomic commitment across distributed participants. + +**Phases**: +1. **Prepare Phase**: All participants prepare (avoid making changes externally visible) +2. **Commit Phase**: Coordinator decides commit/abort, all participants execute + +**In SeaTunnel**: +- **Prepare**: `SinkWriter.prepareCommit(checkpointId)` during checkpoint +- **Commit**: `SinkCommitter.commit()` after checkpoint completes + +## 3. Architecture for Exactly-Once + +### 3.1 End-to-End Pipeline + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Source │ +│ • Read from external system │ +│ • Track offsets/positions │ +│ • Snapshot offsets in checkpoint │ +└──────────────────────────┬───────────────────────────────────┘ + │ + ▼ Checkpoint Barrier +┌──────────────────────────────────────────────────────────────┐ +│ Transform │ +│ • Process records │ +│ • Snapshot transform state (if any) │ +└──────────────────────────┬───────────────────────────────────┘ + │ + ▼ Checkpoint Barrier +┌──────────────────────────────────────────────────────────────┐ +│ Sink Writer │ +│ • Buffer writes │ +│ • prepareCommit(checkpointId) → Generate CommitInfo (PHASE 1)│ +│ • Snapshot writer state │ +└──────────────────────────┬───────────────────────────────────┘ + │ + │ CommitInfo + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ CheckpointCoordinator │ +│ • Collect all CommitInfos │ +│ • Persist CompletedCheckpoint │ +│ • Trigger commit phase │ +└──────────────────────────┬───────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ Sink Committer │ +│ • commit(CommitInfos) → Apply changes (PHASE 2) │ +│ • Must be idempotent │ +└──────────────────────────┬───────────────────────────────────┘ + │ + ▼ + External Sink + (Changes visible) +``` + +### 3.2 Key Components + +**Source Offset Management**: +```java +public class KafkaSourceReader { + private Map currentOffsets; + + @Override + public void pollNext(Collector output) { + ConsumerRecords records = consumer.poll(timeout); + for (ConsumerRecord record : records) { + // Process record + output.collect(convert(record)); + + // Track offset + currentOffsets.put( + new TopicPartition(record.topic(), record.partition()), + record.offset() + ); + } + } + + @Override + public List snapshotState(long checkpointId) { + // Snapshot offsets (will be committed after checkpoint completes) + return Collections.singletonList(new KafkaSourceState(currentOffsets)); + } + + @Override + public void notifyCheckpointComplete(long checkpointId) { + // Commit offsets to Kafka (idempotent) + consumer.commitSync(currentOffsets); + } +} +``` + +**Sink Two-Phase Commit**: +```java +public class JdbcExactlyOnceSinkWriter { + private XAConnection xaConnection; + private Xid currentXid; + + @Override + public void write(SeaTunnelRow element) { + if (currentXid == null) { + // Start XA transaction + currentXid = generateXid(); + xaConnection.getXAResource().start(currentXid, XAResource.TMNOFLAGS); + } + + // Execute INSERT (buffered in XA transaction) + statement.executeUpdate(toSQL(element)); + } + + @Override + public Optional prepareCommit(long checkpointId) { + if (currentXid == null) { + return Optional.empty(); + } + + // PHASE 1: Prepare (no side effects) + xaConnection.getXAResource().end(currentXid, XAResource.TMSUCCESS); + xaConnection.getXAResource().prepare(currentXid); + + // Return XID for committer + XidInfo xidInfo = new XidInfo(currentXid); + currentXid = null; + return Optional.of(xidInfo); + } +} + +public class JdbcSinkCommitter { + @Override + public List commit(List commitInfos) { + List failed = new ArrayList<>(); + + for (XidInfo xidInfo : commitInfos) { + try { + // PHASE 2: Commit (side effects now visible) + xaConnection.getXAResource().commit(xidInfo.getXid(), false); + } catch (XAException e) { + if (e.errorCode == XAException.XAER_NOTA) { + // Already committed (idempotent) + LOG.info("XID already committed: {}", xidInfo); + } else { + failed.add(xidInfo); + } + } + } + + return failed; + } +} +``` + +## 4. Implementation Patterns + +### 4.1 Transactional Sinks (XA) + +**Supported Systems**: MySQL, PostgreSQL, Oracle, SQL Server + +**Implementation**: +```java +public class JdbcExactlyOnceSink implements SeaTunnelSink<...> { + @Override + public SinkWriter<...> createWriter(Context context) { + // Enable XA transactions + XADataSource xaDataSource = createXADataSource(); + return new JdbcExactlyOnceSinkWriter(xaDataSource); + } + + @Override + public Optional> createCommitter() { + return Optional.of(new JdbcSinkCommitter(xaDataSource)); + } +} +``` + +**Pros**: +- Strong consistency guarantee +- Automatic rollback on failure + +**Cons**: +- Requires database XA support +- Higher latency (2PC overhead) +- Lock contention during prepare phase + +### 4.2 Idempotent Sinks (Upsert) + +**Supported Systems**: Key-value stores, Elasticsearch (with doc ID) + +**Implementation**: +```java +public class ElasticsearchSinkWriter { + @Override + public void write(SeaTunnelRow element) { + // Use deterministic document ID + String docId = extractPrimaryKey(element); + + IndexRequest request = new IndexRequest("my_index") + .id(docId) // Idempotent key + .source(toJson(element)); + + bulkProcessor.add(request); + } + + @Override + public Optional prepareCommit(long checkpointId) { + // Flush bulk processor + bulkProcessor.flush(); + + // No explicit commit needed (operations are idempotent) + return Optional.empty(); + } +} +``` + +**Key**: Same primary key → same document → idempotent updates + +**Pros**: +- No transaction overhead +- Lower latency + +**Cons**: +- Requires unique key +- Cannot handle complex transactions + +### 4.3 Log-Based Sinks (Kafka) + +**Implementation**: +```java +public class KafkaSinkWriter { + private KafkaProducer producer; + private String transactionId; + + public KafkaSinkWriter() { + // Enable Kafka transactions + Properties props = new Properties(); + props.put("transactional.id", generateTransactionalId()); + props.put("enable.idempotence", "true"); + + producer = new KafkaProducer<>(props); + producer.initTransactions(); + } + + @Override + public void write(SeaTunnelRow element) { + if (!transactionStarted) { + producer.beginTransaction(); + transactionStarted = true; + } + + ProducerRecord record = convert(element); + producer.send(record); + } + + @Override + public Optional prepareCommit(long checkpointId) { + // PHASE 1: Prepare (flush, but don't commit) + producer.flush(); + + // Return transaction info + return Optional.of(new KafkaCommitInfo(transactionId)); + } +} + +public class KafkaSinkCommitter { + @Override + public List commit(List commitInfos) { + for (KafkaCommitInfo info : commitInfos) { + // PHASE 2: Commit transaction + producer.commitTransaction(); + + // Start new transaction for next checkpoint + producer.beginTransaction(); + } + return Collections.emptyList(); + } +} +``` + +### 4.4 File Sinks (Atomic Rename) + +**Implementation**: +```java +public class FileSinkWriter { + private String tempFilePath; + private String finalFilePath; + private OutputStream outputStream; + + @Override + public void write(SeaTunnelRow element) { + // Write to temporary file + byte[] bytes = serialize(element); + outputStream.write(bytes); + } + + @Override + public Optional prepareCommit(long checkpointId) { + // PHASE 1: Close temp file (no rename yet) + outputStream.close(); + + return Optional.of(new FileCommitInfo(tempFilePath, finalFilePath)); + } +} + +public class FileSinkCommitter { + @Override + public List commit(List commitInfos) { + List failed = new ArrayList<>(); + + for (FileCommitInfo info : commitInfos) { + // PHASE 2: Atomic rename (file becomes visible) + boolean success = fileSystem.rename( + new Path(info.getTempFilePath()), + new Path(info.getFinalFilePath()) + ); + + if (!success) { + failed.add(info); + } + } + + return failed; + } +} +``` + +**Key**: Atomic rename ensures file is either fully visible or not visible. + +## 5. Failure Scenarios and Recovery + +### 5.1 Task Failure Before Checkpoint + +``` +Timeline: + t0: Checkpoint N completed + t1: Process records [1000-2000] + t2: Task fails ❌ + t3: Restore from Checkpoint N + t4: Reprocess records [1000-2000] + +Result: + ✅ No data loss (records reprocessed) + ✅ No duplication (nothing committed before failure) +``` + +### 5.2 Task Failure After prepareCommit + +``` +Timeline: + t0: Checkpoint N in progress + t1: SinkWriter.prepareCommit(checkpointId) → XID-123 prepared + t2: Task fails ❌ (before commit) + t3: Restore from Checkpoint N-1 + t4: Reprocess records + t5: New prepareCommit(checkpointId) → XID-124 prepared + t6: Committer commits XID-124 + +Result: + ✅ XID-123 never committed (automatically rolled back after timeout) + ✅ XID-124 committed (correct data) +``` + +### 5.3 Committer Failure During Commit + +``` +Timeline: + t0: Checkpoint N completed + t1: Committer starts committing [XID-100, XID-101, XID-102] + t2: Commits XID-100 ✅ + t3: Committer fails ❌ (XID-101, XID-102 not committed) + t4: New committer retries [XID-100, XID-101, XID-102] + t5: Commits XID-100 (already committed, idempotent) ✅ + t6: Commits XID-101 ✅ + t7: Commits XID-102 ✅ + +Result: + ✅ All XIDs eventually committed + ✅ No duplication (idempotent commit) +``` + +### 5.4 Network Partition + +``` +Timeline: + t0: SinkWriter prepares XID-200 + t1: Checkpoint completes + t2: Committer sends commit(XID-200) + t3: Network partition ⚠️ (commit success, but ACK lost) + t4: Committer retries commit(XID-200) + t5: XID-200 already committed (idempotent) + +Result: + ✅ Data committed exactly once + ✅ Idempotency prevents duplication +``` + +## 6. Idempotency Requirements + +### 6.1 Why Idempotency Matters + +**Problem**: Network failures, retries, and failover can cause duplicate commit attempts. + +**Solution**: Committer operations must be idempotent. + +```java +// ❌ BAD: Non-idempotent (calling twice inserts twice) +void commit(CommitInfo info) { + statement.execute("INSERT INTO table VALUES (1, 'data')"); +} + +// ✅ GOOD: Idempotent (calling twice has same effect as once) +void commit(CommitInfo info) { + statement.execute( + "INSERT INTO table VALUES (1, 'data') " + + "ON DUPLICATE KEY UPDATE data = VALUES(data)" + ); +} +``` + +### 6.2 Implementing Idempotency + +**Strategy 1: Check-then-Execute** +```java +public List commit(List commitInfos) { + for (XidInfo xid : commitInfos) { + // Check if already committed + if (isCommitted(xid)) { + LOG.info("XID already committed: {}", xid); + continue; // Idempotent + } + + // Commit and record + xaResource.commit(xid, false); + recordCommit(xid); + } +} +``` + +**Strategy 2: Database-Level Idempotency** +```sql +-- Unique constraint ensures idempotency +CREATE TABLE commits ( + xid VARCHAR(255) PRIMARY KEY, + committed_at TIMESTAMP +); + +-- Idempotent insert +INSERT IGNORE INTO commits (xid, committed_at) +VALUES ('XID-123', NOW()); +``` + +**Strategy 3: Natural Idempotency (XA)** +```java +try { + xaResource.commit(xid, false); +} catch (XAException e) { + if (e.errorCode == XAException.XAER_NOTA) { + // Transaction not found = already committed + return; // Idempotent + } + throw e; +} +``` + +## 7. Performance Considerations + +### 7.1 Checkpoint Interval Trade-offs + +``` +Short Interval (10-30s): + ✅ Fast recovery (less reprocessing) + ❌ Higher overhead (frequent snapshots) + ❌ More commit operations + +Long Interval (5-10min): + ✅ Lower overhead (less frequent snapshots) + ❌ Slower recovery (more reprocessing) + ✅ Fewer commit operations +``` + +**Recommendation**: 60-120 seconds for most workloads + +### 7.2 Batch Size Optimization + +```java +public class OptimizedSinkWriter { + private static final int BATCH_SIZE = 1000; + private List buffer = new ArrayList<>(); + + @Override + public void write(SeaTunnelRow element) { + buffer.add(element); + + if (buffer.size() >= BATCH_SIZE) { + // Batch insert (amortize overhead) + statement.executeBatch(); + buffer.clear(); + } + } +} +``` + +**Impact**: 1000x batch → ~10x throughput improvement + +### 7.3 Async Checkpoint + +```java +public List snapshotState(long checkpointId) { + // Quick: Copy state snapshot (in-memory) + StateSnapshot snapshot = state.copy(); + + // Async: Serialize and upload + CompletableFuture.runAsync(() -> { + byte[] serialized = serialize(snapshot); + checkpointStorage.upload(checkpointId, serialized); + }); + + return snapshot; +} +``` + +**Impact**: Data processing continues while snapshot uploads + +## 8. Configuration + +### 8.1 Enable Exactly-Once + +```hocon +env { + # Checkpoint configuration + checkpoint.interval = 60000 # 60 seconds + checkpoint.timeout = 600000 # 10 minutes + + # Exactly-once mode (vs at-least-once) + # This is implicit when using transactional sinks +} +``` + +### 8.2 Source Configuration + +**Kafka**: +```hocon +source { + Kafka { + bootstrap.servers = "localhost:9092" + topic = "my_topic" + + # Kafka consumer offset commit + commit_on_checkpoint = true # Commit offsets after checkpoint + } +} +``` + +**JDBC**: +```hocon +source { + JDBC { + url = "jdbc:mysql://..." + + # Query-based source (idempotent reprocessing) + query = "SELECT * FROM table WHERE id >= ? AND id < ?" + } +} +``` + +### 8.3 Sink Configuration + +**JDBC (XA)**: +```hocon +sink { + JDBC { + url = "jdbc:mysql://..." + + # Enable XA transactions + xa_data_source_class_name = "com.mysql.cj.jdbc.MysqlXADataSource" + is_exactly_once = true + } +} +``` + +**Kafka (Transactions)**: +```hocon +sink { + Kafka { + bootstrap.servers = "localhost:9092" + topic = "output_topic" + + # Kafka transactions + transaction.id = "seatunnel-kafka-sink" + enable.idempotence = true + } +} +``` + +## 9. Testing Exactly-Once + +### 9.1 Functional Test + +```java +@Test +public void testExactlyOnce() { + // 1. Insert 1000 records + insertRecords(1000); + + // 2. Trigger checkpoint + coordinator.triggerCheckpoint(); + + // 3. Simulate failure + task.fail(); + + // 4. Restore and continue + task.restore(checkpointId); + insertRecords(1000); // Same records reprocessed + + // 5. Verify: Should have exactly 1000 records (no duplicates) + assertEquals(1000, countRecordsInSink()); +} +``` + +### 9.2 Chaos Testing + +```java +@Test +public void testExactlyOnceUnderChaos() { + ChaosMonkey chaos = new ChaosMonkey() + .killTaskRandomly(probability = 0.1) + .injectNetworkDelay(maxDelayMs = 5000) + .pauseCheckpointRandomly(probability = 0.05); + + // Run for 10 minutes with chaos + runJobWithChaos(duration = 10 * 60 * 1000, chaos); + + // Verify: Input count == Output count + assertEquals(countSource(), countSink()); +} +``` + +### 9.3 Monitoring Verification + +``` +Metrics to Track: + +source.records_read = 1,000,000 +sink.records_written = 1,000,000 +sink.records_committed = 1,000,000 + +✅ All counts match → Exactly-once verified +``` + +## 10. Best Practices + +### 10.1 Choose Appropriate Sink + +**Use Transactional Sinks (XA) for**: +- Financial transactions +- Billing systems +- Audit logs +- Critical data + +**Use Idempotent Sinks for**: +- High-throughput scenarios +- Eventual consistency acceptable +- No transaction support + +### 10.2 Handle Poisoned Records + +```java +@Override +public void write(SeaTunnelRow element) { + try { + statement.executeUpdate(toSQL(element)); + } catch (SQLException e) { + // Log poisoned record + LOG.error("Failed to write record: {}", element, e); + + // Send to dead letter queue + deadLetterQueue.send(element); + + // Don't fail entire checkpoint + } +} +``` + +### 10.3 Monitor Checkpoint Health + +**Key Metrics**: +- `checkpoint.duration`: Should be < 10% of interval +- `checkpoint.failure_rate`: Should be < 1% +- `checkpoint.size`: Monitor growth over time + +**Alerts**: +``` +Alert if checkpoint.duration > 300s +Alert if checkpoint.failure_rate > 5% +Alert if no checkpoint in 2x interval +``` + +## 11. Related Resources + +- [Checkpoint Mechanism](checkpoint-mechanism.md) +- [Sink Architecture](../api-design/sink-architecture.md) +- [Source Architecture](../api-design/source-architecture.md) +- [Engine Architecture](../engine/engine-architecture.md) + +## 12. References + +### Academic Papers + +- Chandy & Lamport (1985): ["Distributed Snapshots"](https://lamport.azurewebsites.net/pubs/chandy.pdf) +- Gray & Lamport (2006): ["Consensus on Transaction Commit"](https://lamport.azurewebsites.net/pubs/paxos-commit.pdf) +- Carbone et al. (2017): ["State Management in Apache Flink"](http://www.vldb.org/pvldb/vol10/p1718-carbone.pdf) + +### Further Reading + +- [Two-Phase Commit Protocol](https://en.wikipedia.org/wiki/Two-phase_commit_protocol) +- [XA Transactions](https://pubs.opengroup.org/onlinepubs/009680699/toc.pdf) +- [Kafka Exactly-Once](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/) diff --git a/docs/en/architecture/features/multi-table.md b/docs/en/architecture/features/multi-table.md new file mode 100644 index 000000000000..0c3e97bfbe94 --- /dev/null +++ b/docs/en/architecture/features/multi-table.md @@ -0,0 +1,757 @@ +--- +sidebar_position: 3 +title: Multi-Table Synchronization +--- + +# Multi-Table Synchronization Architecture + +## 1. Overview + +### 1.1 Problem Background + +Database migration and CDC scenarios often require synchronizing hundreds of tables: + +- **Resource Efficiency**: How to avoid creating one job per table? +- **Consistent Snapshot**: How to ensure all tables start from same point in time? +- **Schema Routing**: How to route data to correct target tables? +- **Independent Schemas**: How to handle different schemas per table? +- **Parallel Writing**: How to maximize throughput for multiple tables? + +### 1.2 Design Goals + +SeaTunnel's multi-table synchronization aims to: + +1. **Single Job, Multiple Tables**: Synchronize hundreds of tables in one job +2. **Resource Efficiency**: Share resources across tables +3. **Schema Independence**: Each table maintains its own schema +4. **Dynamic Routing**: Route records to correct sink based on table identity +5. **Horizontal Scalability**: Support replica writers for high throughput + +### 1.3 Use Cases + +**Database Migration**: +```hocon +source { + MySQL-CDC { + # Capture all tables in database + database-name = "my_db" + table-name = ".*" # Regex: all tables + } +} + +sink { + JDBC { + # Write to PostgreSQL + url = "jdbc:postgresql://..." + } +} +``` + +**Multi-Table CDC**: +```hocon +source { + MySQL-CDC { + table-name = "order_.*|user_.*|product_.*" # Multiple table patterns + } +} + +sink { + Elasticsearch { + # Different indices per table + } +} +``` + +## 2. Core Abstractions + +### 2.1 TablePath + +Unique identifier for routing records to tables. + +```java +public class TablePath implements Serializable { + private final String databaseName; + private final String schemaName; + private final String tableName; + + // Unique string representation + public String getFullName() { + return String.join(".", databaseName, schemaName, tableName); + } +} +``` + +**Example**: +```java +TablePath orderTable = TablePath.of("my_db", "public", "orders"); +TablePath userTable = TablePath.of("my_db", "public", "users"); +``` + +### 2.2 SeaTunnelRow with TableId + +Records carry table identity for routing. + +```java +public class SeaTunnelRow { + private final String tableId; // TablePath serialized + private final SeaTunnelRowKind rowKind; // INSERT, UPDATE, DELETE + private final Object[] fields; + + public TablePath getTablePath() { + return TablePath.deserialize(tableId); + } +} +``` + +### 2.3 SinkIdentifier + +Unique identifier for sink writers (table + replica index). + +```java +public class SinkIdentifier implements Serializable { + private final TableIdentifier tableIdentifier; + private final int index; // Replica index + + // For multi-table: one identifier per table per replica + // Example: (orders, 0), (orders, 1), (users, 0), (users, 1) +} +``` + +## 3. MultiTableSource Architecture + +### 3.1 Structure + +```java +public class MultiTableSource + implements SeaTunnelSource { + + // Underlying sources (one per table) + private final Map> sources; + + // Produced catalog tables + private final List catalogTables; +} +``` + +### 3.2 Creation + +```java +// From configuration +MultiTableSource multiSource = + MultiTableSource.builder() + .addSource(orderTablePath, orderSource) + .addSource(userTablePath, userSource) + .addSource(productTablePath, productSource) + .build(); +``` + +### 3.3 Enumerator: Unified Split Assignment + +```java +public class MultiTableSourceSplitEnumerator { + private final Map enumerators; + + @Override + public void handleSplitRequest(int subtaskId) { + // Round-robin across table enumerators + for (Map.Entry entry : enumerators.entrySet()) { + TablePath tablePath = entry.getKey(); + SourceSplitEnumerator enumerator = entry.getValue(); + + // Request split from table enumerator + enumerator.handleSplitRequest(subtaskId); + } + } + + @Override + public void addReader(int subtaskId) { + // Register reader with all table enumerators + for (SourceSplitEnumerator enumerator : enumerators.values()) { + enumerator.addReader(subtaskId); + } + } +} +``` + +### 3.4 Reader: Multi-Table Data Reading + +```java +public class MultiTableSourceReader { + private final Map readers; + private final Queue readOrder; // Round-robin queue + + @Override + public void pollNext(Collector output) { + if (readOrder.isEmpty()) { + return; + } + + // Round-robin read from tables + TablePath currentTable = readOrder.poll(); + SourceReader reader = readers.get(currentTable); + + // Read from current table + reader.pollNext(new Collector() { + @Override + public void collect(SeaTunnelRow row) { + // Tag row with table path + row.setTableId(currentTable.serialize()); + output.collect(row); + } + }); + + // Re-add to queue for next round + readOrder.offer(currentTable); + } + + @Override + public void addSplits(List splits) { + // Route splits to correct table readers + for (SplitT split : splits) { + TablePath tablePath = extractTablePath(split); + SourceReader reader = readers.get(tablePath); + reader.addSplits(Collections.singletonList(split)); + + // Add table to read order if not present + if (!readOrder.contains(tablePath)) { + readOrder.offer(tablePath); + } + } + } +} +``` + +## 4. MultiTableSink Architecture + +### 4.1 Structure + +```java +public class MultiTableSink + implements SeaTunnelSink { + + // Underlying sinks (one per table) + private final Map sinks; + + // Number of writer replicas per table + private final int replicaNum; + + // Input catalog tables + private final List catalogTables; +} +``` + +### 4.2 Writer: Multi-Table Writing with Replicas + +```java +public class MultiTableSinkWriter + implements SinkWriter { + + // Writers per table (multiple replicas per table) + private final Map> writers; + + // Replica count per table + private final int replicaNum; + + // Context + private final int writerIndex; // This writer's global index + + @Override + public void write(IN element) throws IOException { + SeaTunnelRow row = (SeaTunnelRow) element; + + // 1. Determine target table + TablePath tablePath = row.getTablePath(); + + // 2. Select replica for this table (load balancing) + int replicaIndex = selectReplica(tablePath, row); + + // 3. Get writer for (table, replica) + SinkIdentifier identifier = new SinkIdentifier( + new TableIdentifier(tablePath), + replicaIndex + ); + + SinkWriter writer = writers.get(identifier); + + // 4. Write to selected writer + writer.write(element); + } + + private int selectReplica(TablePath tablePath, SeaTunnelRow row) { + // If primary key is available, route stably by primary key hash. + Optional primaryKey = extractPrimaryKeyIfPresent(row); + if (primaryKey.isPresent()) { + return Math.abs(primaryKey.get().hashCode()) % replicaNum; + } + + // Otherwise, distribute across replicas (no stable routing guarantee). + return (int) (System.nanoTime() % replicaNum); + } + + @Override + public Optional prepareCommit(long checkpointId) throws IOException { + // Collect commit info from all writers + List allCommitInfos = new ArrayList<>(); + + for (SinkWriter writer : writers.values()) { + Optional commitInfo = writer.prepareCommit(checkpointId); + commitInfo.ifPresent(allCommitInfos::add); + } + + // Wrap in multi-table commit info + return Optional.of((CommitInfoT) new MultiTableCommitInfo(allCommitInfos)); + } + + @Override + public List snapshotState(long checkpointId) throws IOException { + // Snapshot all writers + List allStates = new ArrayList<>(); + + for (Map.Entry entry : writers.entrySet()) { + List states = entry.getValue().snapshotState(checkpointId); + + // Tag states with sink identifier for recovery + for (StateT state : states) { + allStates.add(wrapWithIdentifier(entry.getKey(), state)); + } + } + + return allStates; + } +} +``` + +### 4.3 Committer: Multi-Table Commit Coordination + +```java +public class MultiTableSinkCommitter + implements SinkCommitter { + + // Committers per table + private final Map> committers; + + @Override + public List commit(List commitInfos) throws IOException { + List failed = new ArrayList<>(); + + // Group commit infos by table + Map> groupedInfos = groupByTable(commitInfos); + + // Commit per table + for (Map.Entry> entry : groupedInfos.entrySet()) { + TablePath tablePath = entry.getKey(); + List tableCommitInfos = entry.getValue(); + + SinkCommitter committer = committers.get(tablePath); + + // Commit for this table + List tableFailed = committer.commit(tableCommitInfos); + failed.addAll(tableFailed); + } + + return failed; + } + + private Map> groupByTable(List commitInfos) { + Map> grouped = new HashMap<>(); + + for (CommitInfoT commitInfo : commitInfos) { + TablePath tablePath = extractTablePath(commitInfo); + grouped.computeIfAbsent(tablePath, k -> new ArrayList<>()).add(commitInfo); + } + + return grouped; + } +} +``` + +## 5. Replica Mechanism + +### 5.1 Why Replicas? + +**Problem**: Single writer per table becomes bottleneck for high-throughput tables. + +**Solution**: Multiple replica writers per table for parallel writing. + +``` +Without Replicas: + orders table (1000 writes/sec) → [Single Writer] → Bottleneck + +With Replicas (replicaNum=4): + orders table (1000 writes/sec) → [Writer 0] (250 writes/sec) + → [Writer 1] (250 writes/sec) + → [Writer 2] (250 writes/sec) + → [Writer 3] (250 writes/sec) +``` + +### 5.2 Replica Configuration + +```hocon +sink { + JDBC { + url = "..." + + # Multi-table configuration + multi_table_sink_replica = 4 # replicas per table (applies to all tables) + } +} +``` + +### 5.3 Replica Selection Strategies + +**Hash-Based (when primary key is available)**: +```java +// Ensures same primary key always goes to same replica (order preservation) +int replica = Math.abs(primaryKey.hashCode()) % replicaNum; +``` + +**Random (when primary key is not available)**: +```java +// Distributes load across replicas (no stable routing guarantee) +int replica = (int) (System.nanoTime() % replicaNum); +``` + +## 6. Schema Management in Multi-Table + +### 6.1 Independent Schemas + +Each table maintains its own schema: + +```java +public class MultiTableSink { + // Schema per table + private final Map catalogTables; + + public CatalogTable getCatalogTable(TablePath tablePath) { + return catalogTables.get(tablePath); + } +} +``` + +### 6.2 Schema Evolution Routing + +```java +public class MultiTableSinkWriter { + public void handleSchemaChange(SchemaChangeEvent event) { + // Route schema change to correct table writer + TablePath tablePath = event.getTableId().toTablePath(); + + // Apply to all replicas of this table + for (int i = 0; i < replicaNum; i++) { + SinkIdentifier identifier = new SinkIdentifier( + new TableIdentifier(tablePath), + i + ); + + SinkWriter writer = writers.get(identifier); + writer.applySchemaChange(event); + } + } +} +``` + +## 7. Data Flow Example + +### 7.1 Full Pipeline + +``` +┌──────────────────────────────────────────────────────────────┐ +│ MySQL CDC Source │ +│ • Captures changes from 100 tables │ +│ • Tags each row with TablePath │ +└──────────────────────────┬───────────────────────────────────┘ + │ + ▼ + ┌─────────────────────────────────────┐ + │ SeaTunnelRow (with TablePath) │ + │ tableId: "my_db.public.orders" │ + │ fields: [1, "order-001", 99.99] │ + └─────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ MultiTableSinkWriter │ +│ • Extracts TablePath from row │ +│ • Selects replica (hash or random) │ +│ • Routes to correct writer │ +└──────────────────────────┬───────────────────────────────────┘ + │ + ┌──────────────────┼──────────────────┐ + ▼ ▼ ▼ +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ orders │ │ users │ │ products │ +│ Writer 0 │ │ Writer 0 │ │ Writer 0 │ +│ Writer 1 │ │ Writer 1 │ │ Writer 1 │ +│ Writer 2 │ │ │ │ │ +│ Writer 3 │ │ │ │ │ +└──────────────┘ └──────────────┘ └──────────────┘ + │ │ │ + ▼ ▼ ▼ +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ PostgreSQL │ │ PostgreSQL │ │ PostgreSQL │ +│ orders │ │ users │ │ products │ +└──────────────┘ └──────────────┘ └──────────────┘ +``` + +### 7.2 Write Flow + +```mermaid +sequenceDiagram + participant Source as MySQL CDC + participant Writer as MultiTableSinkWriter + participant OrderWriter as Order Writer (Replica 0) + participant UserWriter as User Writer (Replica 0) + participant PG as PostgreSQL + + Source->>Writer: Row(tableId="orders", data=[...]) + Writer->>Writer: Extract TablePath("orders") + Writer->>Writer: Select replica (hash) → 0 + Writer->>OrderWriter: write(row) + OrderWriter->>PG: INSERT INTO orders ... + + Source->>Writer: Row(tableId="users", data=[...]) + Writer->>Writer: Extract TablePath("users") + Writer->>Writer: Select replica (hash) → 0 + Writer->>UserWriter: write(row) + UserWriter->>PG: INSERT INTO users ... +``` + +### 7.3 Checkpoint Flow + +```mermaid +sequenceDiagram + participant CP as CheckpointCoordinator + participant Writer as MultiTableSinkWriter + participant W1 as Order Writer 0 + participant W2 as Order Writer 1 + participant W3 as User Writer 0 + + CP->>Writer: triggerBarrier(checkpointId) + + Writer->>W1: prepareCommit(checkpointId) + W1-->>Writer: CommitInfo(orders, replica=0) + + Writer->>W2: prepareCommit(checkpointId) + W2-->>Writer: CommitInfo(orders, replica=1) + + Writer->>W3: prepareCommit(checkpointId) + W3-->>Writer: CommitInfo(users, replica=0) + + Writer->>CP: ACK([CommitInfo1, CommitInfo2, CommitInfo3]) +``` + +## 8. Performance Optimization + +### 8.1 Replica Sizing + +**Rule of Thumb**: +``` +replicaNum = ceil(Table Write Rate / Single Writer Throughput) + +Example: + orders: 10,000 writes/sec + Single writer: 2,500 writes/sec + replicaNum = ceil(10,000 / 2,500) = 4 +``` + +### 8.2 Table-Specific Replicas + +```java +// Future enhancement: different replicas per table +Map replicaConfig = Map.of( + TablePath.of("orders"), 4, // High-throughput table + TablePath.of("users"), 2, // Medium-throughput + TablePath.of("config"), 1 // Low-throughput +); +``` + +### 8.3 Batch Writing + +```java +public class MultiTableSinkWriter { + private final Map> buffers; + private static final int BATCH_SIZE = 1000; + + @Override + public void write(SeaTunnelRow row) { + SinkIdentifier identifier = selectWriter(row); + + List buffer = buffers.computeIfAbsent( + identifier, + k -> new ArrayList<>() + ); + + buffer.add(row); + + if (buffer.size() >= BATCH_SIZE) { + flushBuffer(identifier, buffer); + } + } +} +``` + +## 9. Monitoring and Observability + +### 9.1 Key Metrics + +**Per-Table Metrics**: +- `table.{tableName}.records_written`: Records written per table +- `table.{tableName}.bytes_written`: Bytes written per table +- `table.{tableName}.write_latency`: Write latency per table + +**Per-Replica Metrics**: +- `table.{tableName}.replica.{index}.records`: Records per replica +- `table.{tableName}.replica.{index}.utilization`: Replica utilization + +**Global Metrics**: +- `multitable.tables.total`: Total number of tables +- `multitable.writers.total`: Total number of writers (tables × replicas) +- `multitable.throughput`: Aggregate throughput + +### 9.2 Monitoring Dashboard + +``` +Multi-Table Job: mysql-to-postgres + +Tables: 100 +Writers: 250 (avg 2.5 replicas per table) +Throughput: 50,000 records/sec + +Top Tables by Throughput: + 1. orders: 15,000 rec/sec (4 replicas) + 2. events: 10,000 rec/sec (4 replicas) + 3. users: 5,000 rec/sec (2 replicas) + ... + +Replica Distribution: + orders: + Replica 0: 3,750 rec/sec (25%) + Replica 1: 3,800 rec/sec (25.3%) + Replica 2: 3,700 rec/sec (24.7%) + Replica 3: 3,750 rec/sec (25%) +``` + +## 10. Best Practices + +### 10.1 Table Selection + +Table include/exclude patterns are connector-specific. Please refer to the specific Source connector documentation for the supported option keys and formats. + +### 10.2 Replica Configuration + +**Start Conservative**: +```hocon +sink { + JDBC { + # Start with 1 replica, increase if bottleneck + multi_table_sink_replica = 1 + } +} +``` + +**Monitor and Tune**: +```bash +# Check if single replica is bottleneck +# If write latency high → increase replicas +multi_table_sink_replica = 2 # Double capacity +``` + +### 10.3 Schema Management + +**Pre-create Target Tables**: +```sql +-- Better: pre-create all target tables +CREATE TABLE orders (...); +CREATE TABLE users (...); +CREATE TABLE products (...); +``` + +**Enable Auto-Create (Carefully)**: +```hocon +sink { + JDBC { + # Auto-create missing tables + schema-evolution { + enabled = true + auto-create-table = true + } + } +} +``` + +### 10.4 Error Handling + +Error tolerance and retry policies are typically connector-specific. Avoid relying on undocumented `multi-table.*` option keys unless they are defined by the connector you use. + +## 11. Limitations and Considerations + +### 11.1 Current Limitations + +**Shared Parallelism**: +- All tables share same parallelism +- Cannot set different parallelism per table + +**Fixed Replicas**: +- Same replica count for all tables +- High-throughput and low-throughput tables treated equally + +**Memory Overhead**: +- Each writer maintains separate buffer +- 100 tables × 4 replicas = 400 writers in memory + +### 11.2 Workarounds + +**High-Throughput Tables**: +```hocon +# Option 1: Separate job for hot tables +job-1 { source { table-name = "orders" } } # Dedicated job + +job-2 { source { table-name = "user_.*|product_.*" } } # Rest +``` + +**Memory Optimization**: +```hocon +# Reduce buffer size per writer +sink { + JDBC { + batch-size = 500 # Smaller batches + } +} +``` + +## 12. Future Enhancements + +### 12.1 Dynamic Replicas + +Per-table replica overrides are not supported by the current `multi_table_sink_replica` option (it applies to all tables). If you need per-table replicas, it requires additional connector/framework capabilities. + +### 12.2 Adaptive Replicas + +```java +// Auto-adjust replicas based on throughput +if (table.getWriteRate() > threshold) { + increaseReplicas(table); +} else if (table.getWriteRate() < lowThreshold) { + decreaseReplicas(table); +} +``` + +## 13. Related Resources + +- [CatalogTable and Metadata](../api-design/catalog-table.md) +- [Sink Architecture](../api-design/sink-architecture.md) +- [DAG Execution](../engine/dag-execution.md) +- [Schema Evolution](../../introduction/concepts/schema-evolution.md) + +## 14. References + +### Key Source Files + +- [MultiTableSink.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/MultiTableSink.java) +- [SinkIdentifier.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SinkIdentifier.java) +- [TablePath.java](../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/TablePath.java) + +### Example Implementations + +- MySQL CDC Source: `seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/` +- JDBC Sink: `seatunnel-connectors-v2/connector-jdbc/` diff --git a/docs/en/architecture/overview.md b/docs/en/architecture/overview.md new file mode 100644 index 000000000000..53fc699637f7 --- /dev/null +++ b/docs/en/architecture/overview.md @@ -0,0 +1,456 @@ +--- +sidebar_position: 1 +title: Architecture Overview +--- + +# SeaTunnel Architecture Overview + +## 1. Introduction + +### 1.1 Design Goals + +SeaTunnel is designed as a distributed data integration platform with the following core objectives: + +- **Engine Independence**: Decouple connector logic from execution engines, enabling the same connectors to run on SeaTunnel Engine (Zeta), Apache Flink, or Apache Spark +- **High Performance**: Support large-scale data synchronization with high throughput and low latency +- **Fault Tolerance**: Provide exactly-once semantics through distributed snapshots and two-phase commit +- **Ease of Use**: Offer simple configuration and a rich connector ecosystem +- **Extensibility**: Plugin-based architecture allowing easy addition of new connectors and transforms + +### 1.2 Target Use Cases + +- **Batch Data Synchronization**: Large-scale batch data migration between heterogeneous data sources +- **Real-time Data Integration**: Stream data capture and synchronization with CDC support +- **Data Lake/Warehouse Ingestion**: Efficient data loading to data lakes (Iceberg, Hudi, Delta Lake) and warehouses +- **Multi-table Synchronization**: Synchronizing multiple tables in a single job with schema evolution support + +## 2. Overall Architecture + +SeaTunnel adopts a layered architecture that separates concerns and enables flexibility: + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ User Configuration Layer │ +│ (HOCON Config / SQL / Web UI) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ SeaTunnel API Layer │ +│ (Source API / Sink API / Transform API / Table API) │ +│ │ +│ • SeaTunnelSource • CatalogTable │ +│ • SeaTunnelSink • TableSchema │ +│ • SeaTunnelTransform • SchemaChangeEvent │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Connector Ecosystem │ +│ │ +│ [Jdbc] [Kafka] [MySQL-CDC] [Elasticsearch] [Iceberg] ... │ +│ (Connector Ecosystem) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Translation Layer │ +│ (Adapts SeaTunnel API to Engine-Specific API) │ +│ │ +│ • FlinkSource/FlinkSink • SparkSource/SparkSink │ +│ • Context Adapters • Serialization Adapters │ +└─────────────────────────────────────────────────────────────────┘ + │ + ┌─────────────────────┼─────────────────────┐ + ▼ ▼ ▼ +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ SeaTunnel │ │ Apache │ │ Apache │ +│ Engine (Zeta)│ │ Flink │ │ Spark │ +│ │ │ │ │ │ +│ • Master │ │ • JobManager │ │ • Driver │ +│ • Worker │ │ • TaskManager│ │ • Executor │ +│ • Checkpoint │ │ • State │ │ • RDD/DS │ +└──────────────┘ └──────────────┘ └──────────────┘ +``` + +### 2.1 Layer Responsibilities + +| Layer | Responsibility | Key Components | +|-------|---------------|----------------| +| **Configuration Layer** | Job definition, parameter configuration | HOCON parser, SQL parser, config validation | +| **API Layer** | Unified abstraction for connectors | Source/Sink/Transform interfaces, CatalogTable | +| **Connector Layer** | Data source/sink implementations | Various connectors (JDBC, Kafka, CDC, etc.) | +| **Translation Layer** | Engine-specific adaptation | Flink/Spark adapters, context wrappers | +| **Engine Layer** | Job execution and resource management | Scheduling, fault tolerance, state management | + +## 3. Core Components + +### 3.1 SeaTunnel API + +The API layer provides engine-independent abstractions: + +#### Source API +- **SeaTunnelSource**: Factory interface for creating readers and enumerators +- **SourceSplitEnumerator**: Master-side component for split generation and assignment +- **SourceReader**: Worker-side component for reading data from splits +- **SourceSplit**: Minimal serializable unit representing a data partition + +**Key Design**: Separation of coordination (Enumerator) and execution (Reader) enables efficient parallel processing and fault tolerance. + +**Code Reference**: +- [seatunnel-api/.../SeaTunnelSource.java](../../seatunnel-api/src/main/java/org/apache/seatunnel/api/source/SeaTunnelSource.java) +- [seatunnel-api/.../SourceSplitEnumerator.java](../../seatunnel-api/src/main/java/org/apache/seatunnel/api/source/SourceSplitEnumerator.java) + +#### Sink API +- **SeaTunnelSink**: Factory interface for creating writers and committers +- **SinkWriter**: Worker-side component for writing data +- **SinkCommitter**: Coordinator for commit operations from multiple writers +- **SinkAggregatedCommitter**: Global coordinator for aggregated commits + +**Key Design**: Two-phase commit protocol (prepareCommit → commit) ensures exactly-once semantics. + +**Code Reference**: +- [seatunnel-api/.../SeaTunnelSink.java](../../seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SeaTunnelSink.java) +- [seatunnel-api/.../SinkWriter.java](../../seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SinkWriter.java) + +#### Transform API +- **SeaTunnelTransform**: Data transformation interface +- **SeaTunnelMapTransform**: 1:1 transformation +- **SeaTunnelFlatMapTransform**: 1:N transformation + +**Code Reference**: +- [seatunnel-api/.../SeaTunnelTransform.java](../../seatunnel-api/src/main/java/org/apache/seatunnel/api/transform/SeaTunnelTransform.java) + +#### Table API +- **CatalogTable**: Complete table metadata (schema, partition keys, options) +- **TableSchema**: Schema definition (columns, primary key, constraints) +- **SchemaChangeEvent**: Represents DDL changes for schema evolution + +**Code Reference**: +- [seatunnel-api/.../CatalogTable.java](../../seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/CatalogTable.java) + +### 3.2 SeaTunnel Engine (Zeta) + +The native execution engine provides: + +#### Master Components +- **CoordinatorService**: Manages all running JobMasters +- **JobMaster**: Manages single job lifecycle, generates physical plans, coordinates checkpoints +- **CheckpointCoordinator**: Coordinates distributed snapshots per pipeline +- **ResourceManager**: Manages worker resources and slot allocation + +#### Worker Components +- **TaskExecutionService**: Deploys and executes tasks +- **SeaTunnelTask**: Executes Source/Transform/Sink logic +- **FlowLifeCycle**: Manages lifecycle of Source/Transform/Sink components + +#### Execution Model +``` +LogicalDag → PhysicalPlan → SubPlan (Pipeline) → PhysicalVertex → TaskGroup → SeaTunnelTask +``` + +**Code Reference**: +- [seatunnel-engine/.../server/CoordinatorService.java](../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java) +- [seatunnel-engine/.../server/master/JobMaster.java](../../seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobMaster.java) + +### 3.3 Translation Layer + +Enables engine portability through adapter pattern: + +- **FlinkSource/FlinkSink**: Adapts SeaTunnel API to Flink's Source/Sink interfaces +- **SparkSource/SparkSink**: Adapts SeaTunnel API to Spark's RDD/Dataset interfaces +- **Context Adapters**: Wraps engine-specific contexts (SourceReaderContext, SinkWriterContext) +- **Serialization Adapters**: Bridges SeaTunnel and engine serialization mechanisms + +**Code Reference**: +- [seatunnel-translation/.../flink/source/FlinkSource.java](../../seatunnel-translation/seatunnel-translation-flink/seatunnel-translation-flink-common/src/main/java/org/apache/seatunnel/translation/flink/source/FlinkSource.java) + +### 3.4 Connector Ecosystem + +All connectors follow a standardized structure: + +``` +connector-[name]/ +├── src/main/java/.../ +│ ├── [Name]Source.java # Implements SeaTunnelSource +│ ├── [Name]SourceReader.java # Implements SourceReader +│ ├── [Name]SourceSplitEnumerator.java +│ ├── [Name]SourceSplit.java +│ ├── [Name]Sink.java # Implements SeaTunnelSink +│ ├── [Name]SinkWriter.java # Implements SinkWriter +│ └── config/[Name]Config.java +└── src/main/resources/META-INF/services/ + ├── org.apache.seatunnel.api.table.factory.TableSourceFactory + └── org.apache.seatunnel.api.table.factory.TableSinkFactory +``` + +**Discovery Mechanism**: Java SPI (Service Provider Interface) for dynamic connector loading. + +## 4. Data Flow Model + +### 4.1 Source Data Flow + +``` +Data Source + │ + ▼ +┌─────────────────────┐ +│ SourceSplitEnumerator│ (Master Side) +│ • Generate Splits │ +│ • Assign to Readers │ +└─────────────────────┘ + │ (Split Assignment) + ▼ +┌─────────────────────┐ +│ SourceReader │ (Worker Side) +│ • Read from Split │ +│ • Emit Records │ +└─────────────────────┘ + │ + ▼ + SeaTunnelRow + │ + ▼ + Transform Chain (Optional) + │ + ▼ + SeaTunnelRow + │ + ▼ +┌─────────────────────┐ +│ SinkWriter │ (Worker Side) +│ • Buffer Records │ +│ • Prepare Commit │ +└─────────────────────┘ + │ (CommitInfo) + ▼ +┌─────────────────────┐ +│ SinkCommitter │ (Coordinator) +│ • Commit Changes │ +└─────────────────────┘ + │ + ▼ +Data Sink +``` + +### 4.2 Split-based Parallelism + +- Data sources are divided into **Splits** (e.g., file blocks, database partitions, Kafka partitions) +- Each **SourceReader** processes one or more splits independently +- Dynamic split assignment enables load balancing and fault recovery +- Split state is checkpointed for exactly-once processing + +### 4.3 Pipeline Execution + +Jobs are divided into **Pipelines** (SubPlans): + +``` +Pipeline 1: [Source A] → [Transform 1] → [Sink A] + ↓ +Pipeline 2: [Source B] ───────→ [Transform 2] → [Sink B] +``` + +Each pipeline: +- Has independent parallelism configuration +- Maintains its own checkpoint coordinator +- Can execute concurrently or sequentially + +## 5. Job Execution Flow + +### 5.1 Submission Phase + +```mermaid +sequenceDiagram + participant Client + participant CoordinatorService + participant JobMaster + participant ResourceManager + + Client->>CoordinatorService: Submit Job Config + CoordinatorService->>CoordinatorService: Parse Config → LogicalDag + CoordinatorService->>JobMaster: Create JobMaster + JobMaster->>JobMaster: Generate PhysicalPlan + JobMaster->>ResourceManager: Request Resources + ResourceManager->>JobMaster: Allocate Slots + JobMaster->>TaskExecutionService: Deploy Tasks +``` + +### 5.2 Execution Phase + +1. **Task Initialization** + - Deploy tasks to allocated slots + - Initialize Source/Transform/Sink components + - Restore state from checkpoint (if recovering) + +2. **Data Processing** + - SourceReader pulls data from splits + - Data flows through transform chain + - SinkWriter buffers and writes data + +3. **Checkpoint Coordination** + - CheckpointCoordinator triggers checkpoint + - Checkpoint barriers flow through data pipeline + - Tasks snapshot their state + - Coordinator collects acknowledgements + +4. **Commit Phase** + - SinkWriter prepares commit information + - SinkCommitter coordinates commits + - State persisted to checkpoint storage + +### 5.3 State Machine + +**Task State Transitions**: +``` +CREATED → INIT → WAITING_RESTORE → READY_START → STARTING → RUNNING + ↓ + FAILED ← ─────────────────────── → PREPARE_CLOSE → CLOSED + ↓ + CANCELED +``` + +**Job State Transitions**: +``` +CREATED → SCHEDULED → RUNNING → FINISHED + ↓ ↓ + FAILED CANCELING → CANCELED +``` + +## 6. Key Features + +### 6.1 Fault Tolerance + +**Checkpoint Mechanism**: +- Distributed snapshots inspired by Chandy-Lamport algorithm +- Checkpoint barriers propagate through data streams +- State stored in pluggable checkpoint storage (HDFS, S3, local) +- Automatic recovery from latest successful checkpoint + +**Failover Strategy**: +- Task-level failover: Restart failed task and related pipeline +- Region-based failover: Minimize impact on unaffected tasks +- Split reassignment: Failed splits redistributed to healthy workers + +### 6.2 Exactly-Once Semantics + +**Two-Phase Commit Protocol**: +1. **Prepare Phase**: SinkWriter prepares commit info during checkpoint +2. **Commit Phase**: SinkCommitter commits after checkpoint completes +3. **Abort Handling**: Roll back on failure before commit + +**Idempotency**: SinkCommitter operations must be idempotent to handle retries + +### 6.3 Dynamic Resource Management + +- **Slot-based Allocation**: Fine-grained resource management +- **Tag-based Filtering**: Assign tasks to specific worker groups +- **Load Balancing**: Multiple strategies (random, slot ratio, system load) +- **Dynamic Scaling**: Add/remove workers without job restart (future) + +### 6.4 Schema Evolution + +- **DDL Propagation**: Capture schema changes from source (ADD/DROP/MODIFY columns) +- **Schema Mapping**: Transform schema changes through pipeline +- **Dynamic Application**: Apply schema changes to sink tables +- **Compatibility Checks**: Validate schema changes before application + +### 6.5 Multi-Table Support + +- **Single Job, Multiple Tables**: Synchronize hundreds of tables in one job +- **Table Routing**: Route records to correct sink based on TablePath +- **Independent Schemas**: Each table maintains its own schema +- **Replica Support**: Multiple writer replicas per table for higher throughput + +## 7. Module Structure + +``` +seatunnel/ +├── seatunnel-api/ # Core API definitions +│ ├── source/ # Source API +│ ├── sink/ # Sink API +│ ├── transform/ # Transform API +│ └── table/ # Table and Schema API +│ +├── seatunnel-connectors-v2/ # Connector implementations +│ ├── connector-jdbc/ # JDBC connector +│ ├── connector-kafka/ # Kafka connector +│ ├── connector-cdc-mysql/ # MySQL CDC connector +│ └── ... # connectors +│ +├── seatunnel-transforms-v2/ # Transform implementations +│ ├── transform-sql/ # SQL transform +│ ├── transform-filter/ # Filter transform +│ └── ... +│ +├── seatunnel-engine/ # SeaTunnel Engine (Zeta) +│ ├── seatunnel-engine-core/ # Core execution logic +│ ├── seatunnel-engine-server/ # Server components (Master/Worker) +│ └── seatunnel-engine-storage/ # Checkpoint storage +│ +├── seatunnel-translation/ # Engine translation layers +│ ├── seatunnel-translation-flink/ +│ └── seatunnel-translation-spark/ +│ +├── seatunnel-formats/ # Data format handlers +│ ├── seatunnel-format-json/ +│ ├── seatunnel-format-avro/ +│ └── ... +│ +├── seatunnel-core/ # Job submission and CLI +└── seatunnel-e2e/ # End-to-end tests +``` + +## 8. Design Principles + +### 8.1 Separation of Concerns + +- **API vs Implementation**: Clean API boundaries enable multiple implementations +- **Coordination vs Execution**: Enumerator/Committer (master) separate from Reader/Writer (worker) +- **Logical vs Physical**: LogicalDag (user intent) separate from PhysicalPlan (execution details) + +### 8.2 Plugin Architecture + +- **SPI-based Discovery**: Connectors loaded dynamically via Java SPI +- **Class Loader Isolation**: Each connector uses isolated class loader +- **Hot Pluggable**: Add connectors without rebuilding core + +### 8.3 Engine Independence + +- **Unified API**: Same connector code runs on any engine +- **Translation Layer**: Adapts API to engine specifics +- **No Engine Leakage**: Connector developers don't need engine knowledge + +### 8.4 Scalability + +- **Horizontal Scaling**: Add workers to increase throughput +- **Split-based Parallelism**: Fine-grained parallel processing +- **Stateless Workers**: Workers can be added/removed dynamically + +### 8.5 Reliability + +- **Distributed Checkpoints**: Consistent snapshots across distributed tasks +- **Incremental State**: Optimize checkpoint size for large state +- **Exactly-Once Guarantee**: End-to-end consistency + +## 9. Next Steps + +To dive deeper into specific architectural components: + +- [Design Philosophy](design-philosophy.md) - Core design principles and trade-offs +- [Source Architecture](api-design/source-architecture.md) - Deep dive into Source API design +- [Sink Architecture](api-design/sink-architecture.md) - Deep dive into Sink API design +- [Engine Architecture](engine/engine-architecture.md) - SeaTunnel Engine internals +- [Checkpoint Mechanism](fault-tolerance/checkpoint-mechanism.md) - Fault tolerance implementation + +For practical guides: + +- [How to Create Your Connector](../developer/how-to-create-your-connector.md) +- [Quick Start](../getting-started/locally/quick-start-seatunnel-engine.md) + +## 10. References + +### 10.1 Related Concepts + +- [Apache Flink](https://flink.apache.org/) - Inspiration for checkpoint and state management +- [Apache Kafka](https://kafka.apache.org/) - Consumer group model influenced split assignment +- [Chandy-Lamport Algorithm](https://en.wikipedia.org/wiki/Chandy-Lamport_algorithm) - Distributed snapshot algorithm diff --git a/docs/en/developer/how-to-create-your-connector.md b/docs/en/developer/how-to-create-your-connector.md index b99bc85d9990..6a6412ed1e88 100644 --- a/docs/en/developer/how-to-create-your-connector.md +++ b/docs/en/developer/how-to-create-your-connector.md @@ -1,3 +1,15 @@ # Develop Your Own Connector -If you want to develop your own connector for the new SeaTunnel connector API (Connector V2), please check [here](https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/README.md). \ No newline at end of file +If you want to develop your own connector for the new SeaTunnel connector API (Connector V2), please check [here](https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/README.md). + +## Architecture Reference + +For detailed information on SeaTunnel's API design and engine architecture, see: + +- [Architecture Overview](../architecture/overview.md) - Overall architecture and design principles +- [Source Architecture](../architecture/api-design/source-architecture.md) - Deep dive into Source API design +- [Sink Architecture](../architecture/api-design/sink-architecture.md) - Deep dive into Sink API design +- [Translation Layer](../architecture/api-design/translation-layer.md) - How connectors work across different engines +- [Checkpoint Mechanism](../architecture/fault-tolerance/checkpoint-mechanism.md) - Fault tolerance and state management + +These documents will help you understand the underlying architecture and design patterns used in SeaTunnel connectors. diff --git a/docs/sidebars.js b/docs/sidebars.js index 17a1495cf778..c8043417da39 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -47,6 +47,48 @@ const sidebars = { } ] }, + { + "type": "category", + "label": "Architecture", + "items": [ + "architecture/overview", + "architecture/design-philosophy", + { + "type": "category", + "label": "API Design", + "items": [ + "architecture/api-design/source-architecture", + "architecture/api-design/sink-architecture", + "architecture/api-design/catalog-table", + "architecture/api-design/translation-layer" + ] + }, + { + "type": "category", + "label": "Engine", + "items": [ + "architecture/engine/engine-architecture", + "architecture/engine/dag-execution", + "architecture/engine/resource-management" + ] + }, + { + "type": "category", + "label": "Fault Tolerance", + "items": [ + "architecture/fault-tolerance/checkpoint-mechanism", + "architecture/fault-tolerance/exactly-once" + ] + }, + { + "type": "category", + "label": "Features", + "items": [ + "architecture/features/multi-table" + ] + } + ] + }, { "type": "category", "label": "Getting Started", diff --git a/docs/zh/architecture/api-design/catalog-table.md b/docs/zh/architecture/api-design/catalog-table.md new file mode 100644 index 000000000000..b7b6cb88bc8f --- /dev/null +++ b/docs/zh/architecture/api-design/catalog-table.md @@ -0,0 +1,389 @@ +--- +sidebar_position: 4 +title: CatalogTable 和元数据管理 +--- + +# CatalogTable 和元数据管理 + +## 1. 概述 + +### 1.1 问题背景 + +数据集成需要显式的模式管理: + +- **模式定义**: 如何定义和验证表模式? +- **模式传播**: 如何在数据源(source) → 转换器(transform) → 目标端(sink)之间传递模式? +- **模式演化**: 如何处理运行时 DDL 变更(添加/删除列)? +- **类型映射**: 如何在不同数据源之间映射类型? +- **元数据完整性**: 如何捕获完整的表元数据(约束、分区)? + +### 1.2 设计目标 + +SeaTunnel 的元数据管理旨在: + +1. **类型安全**: 在作业提交时进行显式模式验证 +2. **完整性**: 捕获所有表元数据(列、约束、分区、选项) +3. **支持演化**: 处理运行时模式变更(DDL 同步) +4. **引擎独立**: 模式表示独立于执行引擎 +5. **易用性**: 用于模式创建和转换的简单 API + +## 2. 核心概念 + +### 2.1 CatalogTable + +包含所有元数据的表的完整表示。 + +CatalogTable 是 SeaTunnel 对“表及其元数据”的统一表示，通常包含: +- **tableId**: 表标识(可定位到 catalog/database/schema/table) +- **tableSchema**: 模式定义(列、主键、约束等) +- **options**: 连接器/表级选项(如实际表名、topic、format 等) +- **partitionKeys**: 分区键(可选) +- **comment/catalogName**: 注释与归属 catalog 信息(可选) + +**关键组件**: +- `TableIdentifier`: 唯一表标识(catalog.database[.schema].table) +- `TableSchema`: 包含列、主键、约束的模式 +- `options`: 连接器特定设置(例如 Kafka 主题、JDBC 表名) +- `partitionKeys`: 分区表的分区列 + +### 2.2 TableSchema + +包含列和约束的模式定义。 + +TableSchema 关注“表有哪些列，以及这些列有哪些约束”: +- **columns**: 列定义列表(顺序敏感) +- **primaryKey**: 主键定义(可选) +- **constraintKeys**: 唯一键/外键等约束(可选) + +### 2.3 Column + +包含类型和约束的列定义。 + +Column 通常由以下信息构成: +- **name**: 列名 +- **dataType**: SeaTunnelDataType 统一类型 +- **nullable/defaultValue**: 空值与默认值语义 +- **comment/options**: 备注与连接器/列级扩展选项 + +### 2.4 SeaTunnelDataType + +跨连接器的统一类型系统。 + +**基本类型**(示例): +- 数值: TINYINT/SMALLINT/INT/BIGINT/FLOAT/DOUBLE/DECIMAL(precision, scale) +- 字符串: STRING/CHAR(length)/VARCHAR(length) +- 二进制: BYTES +- 日期/时间: DATE/TIME/TIMESTAMP +- 布尔: BOOLEAN + +**复杂类型**(示例): +- ARRAY(elementType) +- MAP(keyType, valueType) +- ROW(fields) + +## 3. 模式创建 + +### 3.1 构建器模式 + +推荐的构建步骤: +1. 明确 TableIdentifier(作业内唯一定位) +2. 通过 TableSchema.Builder 按顺序定义 columns +3. 若需要去重/更新语义，定义 primaryKey +4. 写入 options(连接器侧的物理映射信息) +5. 如为分区表，补充分区键 partitionKeys + +### 3.2 列构建器 + +列定义需要尽量显式: +- name/dataType 是必选 +- nullable/defaultValue 决定写入与 DDL 的语义 +- comment/options 用于补充连接器侧能力(例如精度、编码、额外属性) + +### 3.3 主键和约束 + +约束表达要点: +- primaryKey/uniqueKey 是“语义约束”，用于: + - 转换/下游写入侧的幂等键选择 + - schema 兼容性校验 + - 部分连接器的 DDL 自动生成 +- 外键等约束在跨系统同步时常受限于目标端能力与时序一致性，通常需要在“可用性/一致性”之间做权衡 + +## 4. 模式传播 + +### 4.1 数据源 → 转换器 → 目标端流程 + +``` +┌──────────────┐ +│数据源(source) │ +│ │ +│ 生产 │ +│ CatalogTable │ +└──────┬───────┘ + │ + ▼ (输入模式) +┌──────────────┐ +│ 转换器 │ +│ │ +│ 修改 │ +│ CatalogTable │ +└──────┬───────┘ + │ + ▼ (输出模式) +┌──────────────┐ +│ 目标端 │ +│ │ +│ 验证 │ +│ CatalogTable │ +└──────────────┘ +``` + +### 4.2 数据源模式生产 + +数据 Source 读取端的职责: +- 从外部系统读取元数据(列、类型、主键/唯一键、分区、注释等) +- 将外部类型映射为 SeaTunnelDataType +- 产出 CatalogTable，作为作业的“输入契约” + +常见失败模式: +- 元数据读取失败(权限/网络/超时) +- 类型无法映射(外部类型超出 SeaTunnel 统一类型系统) +- schema 漂移(运行中 DDL)导致“生产的 CatalogTable”与真实数据不一致 + +### 4.3 转换器模式转换 + +转换器端的职责: +- 根据转换逻辑(表达式/字段选择/重命名等)计算输出 schema +- 保证输出 CatalogTable 可被下游 sink 验证与消费 + +常见风险: +- schema 推断不精确(例如 UDF、动态字段) +- 类型提升/缩窄导致的精度或溢出问题 +- 字段重命名/删除导致下游找不到列 + +### 4.4 目标端模式验证 + +目标端侧的职责: +- 获取输入 CatalogTable(来自上游) +- 获取目标端的真实表/索引元数据(或根据配置选择 auto-create) +- 做兼容性校验: + - 列是否存在/是否允许自动新增 + - 类型是否兼容(是否允许安全扩展) + - 约束/主键是否满足写入语义(尤其是 upsert/exactly-once) + +推荐策略: +- 早期失败：在作业启动阶段就完成校验，避免运行中才暴露不可写入 +- 明确兼容规则：哪些类型扩展允许、哪些缩窄禁止、如何处理 nullability 变化 + +## 5. 模式演化 + +### 5.1 SchemaChangeEvent + +SchemaChangeEvent 表示 **CDC 数据源捕获到的 DDL/元数据变更**，用于在数据流中传递“表结构发生了什么变化”。 + +核心语义: +- 变更必须能定位到具体表（TableIdentifier/TablePath 等） +- 变更类型是可枚举的（如新增列、删除列、修改列、重命名、主键/约束变化等） +- 变更负载以“语义化描述”为主（列名、类型、nullable、默认值等），而不是下游可直接执行的 SQL + +为什么要事件化: +- 对上游 CDC 而言，结构变化是数据的一部分，必须被可靠传播 +- 对下游（Transform/Sink）而言，结构变化通常需要与“业务兼容性规则”共同决策（允许/禁止、自动/人工） + +失败模式与建议: +- 事件丢失：下游 schema 与数据不一致，建议将 schema 事件纳入 checkpoint/恢复语义（至少保证“数据与变更事件的相对顺序”可恢复） +- 顺序错乱：先收到数据后收到 DDL，建议在 Source 侧保证同一表内顺序一致，或在下游做缓冲与重放 +- 不可应用变更：例如删除列/缩窄类型导致不可写，建议启动阶段明确策略并在运行时可观测告警 + +### 5.2 CDC 数据源模式演化 + +CDC Source 的职责不是“执行 DDL”，而是 **把变更识别出来并以事件形式注入数据流**。 + +推荐工作流: +1. 捕获上游变更（binlog/redo log/DDL log/元数据快照差异） +2. 解析为结构化事件（新增/删除/修改列等） +3. 与数据事件一同向下游发出，保证同一表内的顺序可解释 +4. 在 checkpoint/恢复时保证：不会出现“数据前进但 schema 事件回退”的不可恢复状态 + +常见边界: +- DDL 批量发生：可能产生多个事件，应明确合并/拆分规则与顺序 +- 同名列重复/大小写规则：需与 Catalog/TableIdentifier 规范对齐 +- DDL 解析失败：建议降级为“停止作业 + 明确报错”，或按配置选择“跳过变更 + 记录告警”（默认不推荐） + +### 5.3 转换器模式演化映射 + +Transform 侧需要回答的问题是：**上游 schema 变化，在经过转换逻辑后，等价的下游变化是什么？** + +典型规则: +- 字段选择：如果下游不再保留该列，则“新增列事件”可被忽略；但“删除列事件”可能仍需要传播以便下游校验 +- 字段重命名：需要把事件中的列名同步映射 +- 类型转换：需要把“上游类型变化”映射为“下游类型变化”（例如 cast、精度变化） +- 表达式生成列：上游新增列不一定影响下游，但下游可能新增派生列（属于转换器内部 schema 变化） + +失败模式: +- 无法判定影响：例如 UDF 返回动态字段，建议显式配置输出 schema 或选择“禁止自动演化” +- 不可逆转换：例如精度缩窄/字符串解析失败，建议在演化阶段就拒绝或要求人工介入 + +### 5.4 目标端模式演化应用 + +Sink 侧的职责是 **对变更做兼容性决策并落地到目标系统**（如果启用自动演化）。 + +推荐处理流程: +1. 获取目标端当前表/索引元数据（可能来自 Catalog、JDBC 元数据、Hive Metastore 等） +2. 按策略判断是否允许该类变更（如自动建表、自动新增列、是否允许 drop/rename） +3. 将“语义事件”转换成目标系统的 DDL/元数据 API 调用 +4. 将变更落地动作纳入可恢复语义： + - 如果 sink 支持 2PC/事务，则尽量在 commit 阶段与数据提交协同 + - 如果目标端 DDL 不能事务化，至少保证幂等与可重试（例如“列已存在”视为成功） + +失败模式与建议: +- DDL 执行失败：目标端权限/锁冲突/存储限制，建议快速失败并输出明确告警，避免 silent skip +- 并发变更：多个并行 writer 同时尝试演化，建议统一到单点/串行执行（或使用外部锁） +- 演化与写入竞争：写入在 DDL 未生效时到达，建议在应用变更后再放行数据，或使用缓冲/重试 + +## 6. 类型映射 + +### 6.1 JDBC 类型映射 + +JDBC 类型映射的目标是把“目标系统类型”规范化为 SeaTunnel 内部类型（SeaTunnelDataType），从而让上游/下游对齐 schema 语义。 + +映射原则: +- 尽量保持语义而非字面：例如 `VARCHAR`/`LONGVARCHAR` 最终都可能落到 `STRING` +- 保留关键约束：长度、精度、scale、时区（如果目标系统支持） +- 明确不可映射类型的策略：快速失败 vs 降级为 `STRING/BYTES`（默认建议失败） + +兼容性与风险: +- 精度相关：`DECIMAL(p,s)` 的 `p/s` 需要完整保留，否则可能出现截断/溢出 +- 时间相关：`TIMESTAMP`/`TIMESTAMP WITH TIME ZONE` 的语义差异需要明确 +- 二进制相关：`BINARY/VARBINARY` 建议映射为 `BYTES`，不要静默转字符串 + +### 6.2 Kafka (Avro) 类型映射 + +Avro/Protobuf/JSON Schema 等“消息协议”通常是嵌套结构，映射时需要同时处理: +- 基础类型：int/long/string/bytes/bool 等 +- 复合类型：array/map/record（对应 SeaTunnel 的 ARRAY/MAP/ROW） +- 兼容性规则：新增字段、字段默认值、union/nullability + +推荐策略: +- 将 `record` 映射为 `ROW`，并保持字段顺序与名字稳定 +- 对 nullable：显式表达（而不是隐式 union） +- 对 schema registry：把 schema 版本作为可观测信息输出，便于排障与回滚 + +## 7. 分区表 + +### 7.1 分区定义 + +分区信息是 CatalogTable 的一部分：它把“表 schema”与“物理分布/组织方式”连接起来。 + +分区键的典型用途: +- 让 Source 能按分区裁剪（partition pruning），减少扫描范围 +- 让 Sink 能按分区写入，提高写入性能并避免热点 +- 让下游表管理系统（Hive/Iceberg/Hudi）正确理解数据布局 + +### 7.2 分区感知数据源 + +Source 侧的关键是：从外部元数据系统读取“分区键定义”并写入 Produced CatalogTable。 + +推荐能力: +- 支持分区过滤条件（按时间/范围），并明确过滤是在“枚举 split”阶段完成 +- 分区元数据缺失时快速失败，避免静默全表扫描 + +### 7.3 分区感知目标端 + +Sink 侧的关键是：把输入行映射到正确分区并以目标系统要求的方式提交。 + +常见失败模式: +- 分区键缺失/为空：需要明确处理策略（拒绝、写入默认分区、或降级为非分区写入） +- 分区字段类型不匹配：建议在启动阶段做 schema 校验 +- 并发写入同分区：需要考虑文件/小文件合并、提交冲突与幂等 + +## 8. 最佳实践 + +### 8.1 模式定义 + +**优先使用显式模式**: +- 推荐：在配置或作业定义阶段显式给出 schema（字段名、类型、nullable、精度等） +- 不推荐：完全依赖运行时推断（尤其是“取第一行推断”），容易在脏数据或字段漂移时产生不可恢复的问题 + +**选择合适类型**: +- 推荐：金额/计数等使用 `DECIMAL(p,s)`/`BIGINT` 等精确类型；时间使用 `DATE/TIME/TIMESTAMP` +- 不推荐：将所有字段降级为 `STRING`，会把错误推迟到下游并放大数据质量成本 + +### 8.2 模式验证 + +**早期验证**（快速失败）: +- Source：在 open/prepare 阶段确定 Produced CatalogTable，并完成“字段存在性/类型合法性/可投影性”等验证 +- Sink：在作业启动阶段完成“输入 schema 与目标表 schema”的兼容性校验，避免运行中才暴露不可写入 + +### 8.3 类型兼容性 + +**类型扩展（通常安全）**: +- `INT → BIGINT` +- `FLOAT → DOUBLE` +- `VARCHAR(10) → VARCHAR(20)` + +**类型缩窄（通常不安全）**: +- `BIGINT → INT`（溢出风险） +- `DOUBLE → FLOAT`（精度损失） +- `VARCHAR(20) → VARCHAR(10)`（截断风险） + +## 9. 配置 + +### 9.1 模式覆盖 + +```hocon +source { + Jdbc { + url = "..." + query = "SELECT * FROM users" + + # 覆盖推断的模式 + schema { + fields { + id = "BIGINT" + name = "STRING" + age = "INT" + } + } + } +} +``` + +### 9.2 模式演化控制 + +在 **CDC 场景**下，SeaTunnel 的模式演化通常由 **CDC Source 侧开关**控制：在 CDC 源启用 `schema-changes.enabled = true` 后，运行时 DDL/元数据变更会随数据流传播；下游 Sink 是否能自动应用变更取决于连接器是否支持 schema evolution。 + +下面给出一个“CDC → JDBC Sink”的最小可用示例（参数以各连接器文档为准）： + +```hocon +source { + MySQL-CDC { + url = "..." + table-names = ["db.table"] + + # 启用 CDC 模式变更事件（SchemaChangeEvent）传播 + schema-changes.enabled = true + } +} + +sink { + Jdbc { + url = "..." + + # 让 JDBC sink 能根据上游 schema 生成/刷新写入 SQL + generate_sink_sql = true + + # 作业启动阶段：若表不存在则创建（用于首次建表） + schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST" + } +} +``` + +> 说明：当前仓库中没有“schema-evolution 统一配置块”这一通用写法。 +> 新增/删除/重命名列等是否自动应用由具体 Sink 实现与目标端能力决定；其中 DROP/RENAME 属于高风险操作，建议在生产环境谨慎启用并做好灰度与回滚预案。 + +## 10. 相关资源 + +- [source 数据源架构](source-architecture.md) +- [sink 目标端架构](sink-architecture.md) +- [模式演化](../../introduction/concepts/schema-evolution.md) +- [模式特性](../../introduction/concepts/schema-feature.md) diff --git a/docs/zh/architecture/api-design/sink-architecture.md b/docs/zh/architecture/api-design/sink-architecture.md new file mode 100644 index 000000000000..a5c1872e4e7c --- /dev/null +++ b/docs/zh/architecture/api-design/sink-architecture.md @@ -0,0 +1,376 @@ +--- +sidebar_position: 3 +title: 数据写入 Sink 架构 +--- + +# 数据写入 Sink 架构 + +## 1. 概述 + +### 1.1 问题背景 + +在分布式环境中向外部系统写入数据面临关键挑战： + +- **精确一次保证**：如何确保每条记录精确写入一次，而不是零次或多次？ +- **事务一致性**：如何在多个并行写入器之间原子性地提交写入操作？ +- **容错**：如何从失败中恢复而不丢失数据或产生重复？ +- **反压**：如何处理慢速数据 Sink而不使系统过载？ +- **幂等性**：如何使重试操作安全？ + +### 1.2 设计目标 + +SeaTunnel 的数据 Sink 旨在： + +1. **提供可验证的一致性语义**：在外部系统支持事务/幂等提交的前提下，通过两阶段提交与检查点边界实现端到端一致性 +2. **支持并行写入**：通过多个写入器实例扩展吞吐量 +3. **启用全局协调**：协调分布式写入器之间的提交 +4. **确保容错**：从失败中恢复而不产生数据不一致 +5. **提供灵活性**：支持各种提交策略 + +### 1.3 适用场景 + +- 事务性数据库（JDBC 与 XA 事务） +- 消息队列（Kafka 与事务） +- 文件系统（原子文件重命名） +- 数据湖（Iceberg、Hudi、Paimon 与表事务） +- 搜索引擎（Elasticsearch 与版本控制） + +## 2. 架构设计 + +### 2.1 整体架构 + +``` +┌────────────────────────────────────────────────────────────────┐ +│ 执行引擎任务侧（数据面） │ +│ │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ SinkWriter │ │ +│ │ │ │ +│ │ • 从上游接收记录 │ │ +│ │ • 缓冲并写入数据 │ │ +│ │ • 在 checkpoint 边界产出 commitInfo │ │ +│ │ • 快照写入器状态 │ │ +│ └──────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ checkpoint 完成通知触发 │ +│ ▼ │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ SinkCommitter（可选） │ │ +│ │ │ │ +│ │ • 使 prepare 的变更对外可见 │ │ +│ │ • 失败可重试，要求幂等 │ │ +│ └──────────────────────────────────────────────────────┘ │ +│ │ +└────────────────────────────────────────────────────────────────┘ + │ + │ （可选：聚合提交任务，单实例） + ▼ +┌────────────────────────────────────────────────────────────────┐ +│ 执行引擎协调侧（控制面） │ +│ │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ SinkAggregatedCommitter（可选）│ │ +│ │ │ │ +│ │ • 聚合多个 writer 的 commitInfo │ │ +│ │ • 执行一次全局提交（单线程语义） │ +│ └──────────────────────────────────────────────────────┘ │ +│ │ +└────────────────────────────────────────────────────────────────┘ + │ + ▼ + 外部数据系统 + (数据库 / 文件 / 消息队列) +``` + +### 2.2 核心组件 + +#### SeaTunnelSink（工厂接口） + +作为创建写入器和提交器的工厂的顶层接口。 + +**契约要点（概念级）**： +- 创建 writer：在工作节点（Task）侧创建 `SinkWriter`，负责接收记录并写入 +- 恢复 writer：在 failover 后用 checkpoint 中的 writerState 恢复未完成写入 +- 创建 committer（可选）：当数据 Sink 需要两阶段提交时使用。它负责在 checkpoint 成功后提交 `prepareCommit(checkpointId)` 产生的提交信息；运行位置取决于执行引擎实现（例如在 SeaTunnel Engine 中由 Sink 任务在 `notifyCheckpointComplete` 回调中触发） +- 创建 aggregated committer（可选）：当外部系统需要“全局单点提交”（如表级提交/单次元数据提交）时使用。该提交器按单线程语义执行，通常与 committer 二选一；如果同时提供两者，需要确保语义不会重复提交/发生冲突 +- 描述写入 schema：通过 `CatalogTable` 明确输入字段、投影与类型约束 + +这组工厂方法的核心目的是把“写入（数据面）”与“提交（控制面）”解耦，使得 checkpoint 成为全局一致性边界。 + +**关键设计点**： +- 两阶段提交扩展点：写入器（必需）+（committer 或 aggregated committer，按需求选择） +- committer 与 aggregated committer 在很多场景下应视为互斥选项：前者提交每个 writer 的变更，后者先聚合再做一次全局提交 +- 写入器始终是必需的（执行实际的数据写入） + +### 2.3 交互流程 + +#### 正常写入流程（带两阶段提交） + +```mermaid +sequenceDiagram + participant CP as 框架（Checkpoint/回调） + participant Writer1 as SinkWriter 1 + participant Writer2 as SinkWriter 2 + participant Committer as SinkCommitter + participant Sink as 数据 Sink + + Writer1->>Writer1: write(record) + Writer2->>Writer2: write(record) + + CP->>Writer1: triggerBarrier(checkpointId) + CP->>Writer2: triggerBarrier(checkpointId) + + Writer1->>Writer1: prepareCommit(checkpointId) + Writer1->>CP: ack(commitInfo1) + Writer2->>Writer2: prepareCommit(checkpointId) + Writer2->>CP: ack(commitInfo2) + + CP->>CP: 所有写入器已确认 + CP->>CP: 持久化检查点 + + Note over CP,Committer: checkpoint 成功后，框架触发提交（触发点/运行位置取决于执行引擎实现） + CP->>Committer: commit([commitInfo1, commitInfo2]) + Committer->>Sink: 提交 writer1 的变更 + Committer->>Sink: 提交 writer2 的变更 + Committer->>CP: ack() + + CP->>Writer1: notifyCheckpointComplete(checkpointId) + CP->>Writer2: notifyCheckpointComplete(checkpointId) +``` + +#### 失败和重试流程 + +```mermaid +sequenceDiagram + participant CP as 框架（Checkpoint/回调） + participant Writer as SinkWriter + participant Committer as SinkCommitter + participant Sink as 数据 Sink + + Note over Writer: 写入进行中（事务/临时文件） + + CP->>Writer: triggerBarrier(checkpointId) + Writer->>Writer: prepareCommit(checkpointId) + Writer->>CP: ack(commitInfo) + + alt Checkpoint 成功 + Note over CP,Committer: checkpoint 成功后，框架/引擎会触发提交（触发点/运行位置取决于执行引擎实现） + CP->>Committer: commit([commitInfo]) + Committer->>Sink: 提交变更（幂等） + Committer->>CP: ack() + CP->>Writer: notifyCheckpointComplete(checkpointId) + else Checkpoint 失败/中止 + CP->>Writer: notifyCheckpointAborted(checkpointId) + Note over Writer,Committer: 引擎可选调用 commit/abort 相关回调进行清理；\n务必保证 commit 幂等，避免只依赖 abort 完成回滚 + end + + Note over Committer: commit 失败由框架重试\n必须保证幂等 +``` + +**核心职责**： +- `write(element)`：接收上游记录并写入外部系统的“临时/事务内”区域（避免对外可见） +- `prepareCommit(checkpointId)`：在 checkpoint 边界生成提交信息（commitInfo），要求“无副作用”（不让数据对外可见） +- `snapshotState(checkpointId)`：把“已写入但未提交”的可恢复状态写入 checkpoint（事务句柄、文件清单、位点等） +- `abortPrepare()`：用于回滚 `prepareCommit` 阶段产生的副作用（是否会被调用取决于执行引擎/实现路径） +- `notifyCheckpointAborted()`：checkpoint 失败/中止回调（若 writer 或运行时实现了 CheckpointListener，可在此做清理） +- `notifyCheckpointComplete()`：checkpoint 成功且提交完成后做清理（释放事务、删除临时文件/状态等） + +**关键要求**： +- `prepareCommit(...)` 必须无副作用；真正让数据对外可见的动作应发生在 committer 的 `commit()` 阶段 +- `snapshotState()` 必须覆盖所有“已写入但未提交”的中间结果，否则恢复会丢数据或重复写 +- 清理路径必须可重试且幂等：同一 checkpoint 的 abort/cleanup 可能被调用多次 + +**典型实现形态（不绑定具体源码）**： +- 事务型数据 Sink ：writer 在事务内写入，prepare 阶段产出事务句柄/提交 token，commit 阶段统一提交 +- 文件型数据 Sink ：writer 写临时文件并产出“文件清单/元数据”，commit 阶段做原子 rename/元数据提交 + +### 3.2 SinkCommitter 接口 + +提交器由执行引擎在 checkpoint 成功后触发执行，用于使本次 checkpoint 对应的“准备写入”对外可见（运行位置取决于具体执行引擎实现）。 + + +**契约要点**： +- `commit(commitInfos)`：对一批提交信息执行提交；必须支持重试，因此要求幂等 +- 返回值语义：返回“仍需重试/未完成”的提交信息集合（框架会在后续 checkpoint 或恢复路径中重试） +- `abort(commitInfos)`（可选）：放弃提交并做资源清理（例如回滚事务、删除临时文件） + +**关键要求**： +- `commit()` **必须**是幂等的（使用相同的 commitInfo 调用两次应该是安全的） +- 返回**失败的** commitInfos 列表（将被重试） +- 应优雅地处理部分失败 + +**实现提示**： +- 需要明确幂等键（例如事务 id、文件清单版本、外部系统的去重 key） +- 需要能区分“可重试失败”（网络抖动）与“不可重试失败”（权限/数据非法），避免无意义重试 + +### 3.3 SinkAggregatedCommitter 接口 + +聚合提交器为所有写入器执行单个全局提交。 + + +**契约要点**： +- `combine(commitInfos)`：把多个 writer 的提交信息聚合成“全局一次提交”所需的元数据 +- `commit(aggregatedCommitInfos)`：对聚合后的信息做全局提交；同样必须幂等 +- `restoreCommit(...)`：恢复聚合提交器状态，确保 failover 后仍可完成/重试“全局提交” + +**使用场景**： +- Hive 表提交（所有分区的单个 COMMIT TRANSACTION） +- Iceberg 表提交（单个表快照） +- 全局索引更新（为所有写入更新一次索引） + +**实现示例（语义级，以 Hive 为例）**： +- `combine`： Sink 总所有 writer 产生的文件/分区元数据，形成一次表级提交所需的“全量变更集” +- `commit`：对外部 metastore/表事务执行一次全局原子提交；失败后需要可重试且不重复（幂等） + +## 4. 设计考量 + +### 4.1 设计权衡 + +#### 两阶段提交 + +**优点**： +- 强一致性保证（精确一次） +- 自动失败恢复 +- 准备和提交之间的清晰分离 + +**缺点**： +- 增加延迟（数据仅在提交后可见） +- 需要数据 Sink 中的事务支持 +- 提交信息的额外状态 +- 更复杂的实现 + +**何时使用**： +- 金融交易、计费、审计日志 +- 外部系统支持事务/幂等提交，并且业务需要端到端精确一次的场景 + +**何时不使用**： +- 至少一次可接受（日志、指标） +- 数据 Sink 不支持事务 +- 需要超低延迟 + +#### 两层提交 vs 聚合提交 + +**两层（写入器 → 提交器）**： +- 每个写入器的提交独立处理 +- 并行提交操作 +- 适用于大多数数据 Sink + +**聚合提交（写入器 → 聚合提交器）**： +- 所有写入器的提交信息先被聚合 +- 执行一次全局提交操作（单线程语义） +- 适用于需要“单点表级提交/元数据提交”的外部系统（Hive、Iceberg 等） + +### 4.2 性能考量 + +#### 批量写入 + +将多条记录合并为一次外部写入（JDBC batch / bulk API / multi-put）。 + +**好处**： +- 摊销每条记录的开销 +- 减少网络往返 +- 更好的吞吐量 + +#### 异步写入 + +将外部 I/O 下沉到后台线程/异步客户端，以降低 `write()` 的尾延迟。但需要明确： +- 如果采用异步写入，`prepareCommit(...)` 需要等待所有“已接收记录”的异步写入完成，才能生成可靠的 commitInfo +- 需要有背压/限流策略，避免异步积压导致 OOM + +#### 连接池 + +对 JDBC/HTTP 等短连接成本高的外部系统，优先使用连接池/长连接以减少握手与认证开销。 + +### 4.3 幂等性模式 + +#### 1. 自然幂等性（Upsert） + +利用外部系统提供的 Upsert/Merge 语义，使“重复提交同一业务键”不会产生重复数据。 + +#### 2. 去重键 + +为每条写入生成可重复的幂等键（业务主键、事件 id、事务 id），并让外部系统/协议基于该键实现去重。 + +#### 3. 外部去重表 + +在外部系统维护“已提交记录表/去重索引”，提交前先检查是否已提交；这种方式通用但会引入额外写放大与一致性成本。 + +## 5. 最佳实践 + +### 5.1 使用建议 + +**1. 选择适当的提交级别** + +- 仅 writer：适合至少一次（数据写入立即可见，恢复会重放，需外部幂等） +- writer + committer：适合两阶段提交（checkpoint 边界产出 commitInfo，并在 checkpoint 成功后触发 commit；触发位置取决于执行引擎实现） +- writer + aggregated committer：适合表级事务/全局单点提交（先聚合多个 writer 的 commitInfo，再执行一次全局提交） + +**2. 正确的状态管理** + +- 状态里只放“恢复必需信息”（事务句柄/临时文件清单/最后一致性偏移量等），避免把大批数据放进状态 +- 恢复时要能把状态回放到 writer 内部，并确保 prepare/commit 的幂等性仍成立 + +**3. 资源管理** + +- 明确资源生命周期：writer/committer 的 `close()` 必须可重复调用且不抛出不可恢复异常 +- 尽量做到“按创建逆序关闭”，并确保失败时也能释放外部资源（连接/事务/临时文件） + +### 5.2 常见陷阱 + +**1. prepareCommit(...) 中的副作用** + +- `prepareCommit(...)` 只能生成“提交所需的凭据/元数据”，不能让数据对外可见 +- 一旦在 prepare 阶段产生外部副作用，failover 重放会导致重复写入 + +**2. 非幂等提交** + +- `commit()` 需要支持相同 commitInfo 的重复调用（网络抖动/主节点重启会发生） +- 优先依赖外部系统的幂等语义（upsert/merge/幂等事务 id），否则需要自建去重机制 + +**3. 大状态** + +- 避免把大量缓冲记录放进 checkpoint 状态，状态越大越容易导致 checkpoint 超时与恢复变慢 +- 把大数据留在外部系统（临时文件/事务日志），状态里只保留引用与必要元数据 + +### 5.3 调试技巧 + +**1. 启用 XA 事务日志** + +- 记录关键生命周期事件：事务开始/prepare/commit/rollback、checkpointId、writerIndex +- 避免记录敏感数据（凭据/明文 SQL/用户数据），以可追踪的事务 id 为主 + +**2. 跟踪提交进度** + +- 输出/采集提交指标：提交耗时、失败率、重试次数、单次提交大小 +- 重点关注“提交堆积”与“commitInfo 重试风暴”，它们通常意味着幂等设计或外部系统稳定性问题 + +**3. 测试失败场景** + +- 覆盖典型故障：writer 崩溃、committer 崩溃、commit 超时、重复提交、checkpoint 超时 +- 验证点：不丢数据、不重复可见（或重复可见但幂等）、恢复后可继续推进 checkpoint + +## 6. 相关资源 + +- [架构概览](../overview.md) +- [设计理念](../design-philosophy.md) +- [数据源架构](source-architecture.md) +- [检查点机制](../fault-tolerance/checkpoint-mechanism.md) +- [精确一次语义](../fault-tolerance/exactly-once.md) + +## 7. 参考资料 + +### 示例连接器 + +- **简单数据 Sink **：ConsoleSink（输出到标准输出） +- **文件数据 Sink **：FileSink（原子文件重命名） +- **数据库数据 Sink **：JdbcSink（XA 事务） +- **流式数据 Sink **：KafkaSink（Kafka 事务） +- **表数据 Sink **：IcebergSink（表提交） + +### 进一步阅读 + +- [两阶段提交协议](https://en.wikipedia.org/wiki/Two-phase_commit_protocol) +- [XA 事务](https://www.oracle.com/java/technologies/xa-transactions.html) +- [Kafka 事务](https://kafka.apache.org/documentation/#semantics) +- [Iceberg 表格式](https://iceberg.apache.org/spec/) diff --git a/docs/zh/architecture/api-design/source-architecture.md b/docs/zh/architecture/api-design/source-architecture.md new file mode 100644 index 000000000000..8ff8896beb4f --- /dev/null +++ b/docs/zh/architecture/api-design/source-architecture.md @@ -0,0 +1,438 @@ +--- +sidebar_position: 2 +title: 数据读取端 Source 架构 +--- + +# 数据 Source 端架构 + +## 1. 概述 + +### 1.1 问题背景 + +分布式系统中的数据源读取端面临几个挑战： + +- **并行度**：如何从单个 Sink 并行读取数据？ +- **容错**：失败后如何从中断处恢复？ +- **动态分配**：如何处理工作节点失败并重新分配工作？ +- **有界 vs 无界**：如何统一批处理和流式数据源？ +- **反压**：如何处理下游处理缓慢的情况？ + +### 1.2 设计目标 + +SeaTunnel 的源端 Source 端读取 API 旨在： + +1. **启用并行读取**：通过基于分片的并行度支持可扩展性 +2. **确保容错**：检查点分片状态以实现精确一次处理 +3. **分离协调与执行**：枚举器（主节点）和读取器（工作节点）分离 +4. **支持动态分配**：在失败或不平衡时重新分配分片 +5. **统一批处理和流处理**：有界和无界数据源的单一 API + +### 1.3 适用场景 + +- 基于文件的数据源（本地文件、HDFS、S3、OSS）等 +- 数据库数据源（MySQL、PostgreSQL、Oracle、JDBC 兼容）等 +- 消息队列数据源（Kafka、Pulsar、RabbitMQ）等 +- CDC 数据源（MySQL CDC、PostgreSQL CDC、Oracle CDC）等 +- 流式数据源（Socket、HTTP、自定义协议）等 + +## 2. 架构设计 + +### 2.1 整体架构 + +``` +┌──────────────────────────────────────────────────────────────┐ +│ 协调端（master/coordinator 侧） │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ SourceSplitEnumerator │ │ +│ │ │ │ +│ │ • 在 run() 中发现/生成分片（实现自定义） │ │ +│ │ • 分配分片给读取器 │ │ +│ │ • 处理读取器注册 │ │ +│ │ • 处理分片请求 │ │ +│ │ • 从失败的读取器回收分片 │ │ +│ │ • 快照枚举器状态 │ │ +│ │ • 发送/接收自定义事件 │ │ +│ └────────────────────────────────────────────────────┘ │ +│ │ │ +└────────────────────────────┼───────────────────────────────────┘ + │ (分片分配) + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ TaskExecutionService（工作节点侧） │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ SourceReader │ │ +│ │ │ │ +│ │ • 接收分配的分片 │ │ +│ │ • 从分片读取数据 │ │ +│ │ • 向下游发送记录 │ │ +│ │ • 快照读取器状态（分片进度） │ │ +│ │ • 处理分片完成 │ │ +│ │ • 发送/接收自定义事件 │ │ +│ └────────────────────────────────────────────────────┘ │ +│ │ │ +└────────────────────────────┼─────────────────────────────────┘ + │ + ▼ + SeaTunnelRow + (到转换/数据 Sink ) +``` + +### 2.2 核心组件 + +#### SeaTunnelSource（工厂接口） + +作为创建读取器和枚举器的工厂的顶层接口。 + +本节仅保留核心契约说明，完整签名以源码为准： + +**关键契约**： +- `getBoundedness()`：声明 BOUNDED/UNBOUNDED +- `createReader()`：创建运行在工作节点侧的 `SourceReader` +- `createEnumerator()` / `restoreEnumerator()`：创建/恢复运行在主节点侧的 `SourceSplitEnumerator` +- `getProducedCatalogTables()`：声明输出的表元数据（`CatalogTable` 列表，支持多表/模式信息） +- `getSplitSerializer()` / `getEnumeratorStateSerializer()`：split/枚举器状态序列化器（用于网络传输与 checkpoint） + +#### SourceSplit（最小可序列化单元） + +表示数据的可分区单元。 + +**核心约束**： +- **可独立处理**：split 表达一个可被单个 reader 独立读取的范围（例如文件片段、分区、主键范围）。 +- **可序列化传输**：split 需要能在主节点与工作节点之间传递。 +- **可重分配**：reader 失败时，未完成 split 必须可回收并分配给其他 reader。 + +**实现示例**： + +- 文件类：`(filePath, startOffset, length)` 或 “单文件一个 split” +- JDBC 类：`(queryRange / shardKeyRange / partition)` +- Kafka 类：`(topic, partition, startOffset)` + +**设计说明**： +- 分片必须可序列化以进行网络传输 +- 分片状态（例如，当前偏移量）单独存储在读取器状态中 +- 分片可以重新分配给不同的读取器 + +### 2.3 交互流程 + +#### 初始启动流程 + +```mermaid +sequenceDiagram + participant Coord as 框架（协调端） + participant Enum as SourceSplitEnumerator + participant Worker as TaskExecutionService + participant Reader as SourceReader + + Coord->>Enum: createEnumerator(context) + Enum->>Enum: open() + Enum->>Enum: run()\n（内部完成分片发现/生成） + + Worker->>Reader: createReader(context) + Coord->>Enum: registerReader(subtaskId) + + Reader->>Reader: context.sendSplitRequest() + Enum->>Enum: handleSplitRequest(subtaskId) + Enum->>Reader: assignSplit(splits) + + Reader->>Reader: addSplits(splits) + Reader->>Reader: pollNext(collector) + Reader->>Worker: collect(record) +``` + +#### 检查点流程 + +```mermaid +sequenceDiagram + participant CP as CheckpointCoordinator + participant Enum as SourceSplitEnumerator + participant Reader as SourceReader + + CP->>Reader: triggerBarrier(checkpointId) + Reader->>Reader: snapshotState(checkpointId) + Reader->>CP: ack(readerState) + + CP->>Enum: snapshotState(checkpointId) + Enum->>Enum: 快照枚举器状态 + Enum->>CP: ack(enumeratorState) + + CP->>CP: 收到所有确认 + CP->>CP: 持久化检查点 +``` + +#### 失败恢复流程 + +```mermaid +sequenceDiagram + participant Coord as 框架（协调端） + participant Enum as SourceSplitEnumerator + participant OldReader as 失败的读取器 + participant NewReader as 新读取器 + + OldReader->>OldReader: [失败] + Coord->>Enum: addSplitsBack(失败读取器的分片) + Enum->>Enum: 标记分片为待处理 + + Coord->>NewReader: 在新工作节点上部署 + NewReader->>NewReader: restoreState(checkpointedState) + Coord->>Enum: registerReader(subtaskId) + + Enum->>NewReader: assignSplit(恢复的分片) + NewReader->>NewReader: 从检查点偏移量恢复 +``` + +## 3. 关键实现 + +### 3.1 SourceSplitEnumerator 接口 + +枚举器在主节点侧运行并协调分片分配。 + +**关键契约（摘要）**： +- `run()`：枚举/发现分片并驱动分配逻辑 +- `registerReader(subtaskId)`：注册 reader（由引擎调用） +- `handleSplitRequest(subtaskId)`：处理 reader 请求分片 +- `addSplitsBack(splits, subtaskId)`：reader 失败时回收未完成分片 +- `snapshotState(checkpointId)`：快照枚举器状态（注意与 `run()` 的并发调用约束） + +**关键职责**： +- **分片发现**：从数据源生成分片（文件、分区、分片） +- **分配策略**：决定哪些分片分配给哪些读取器 +- **动态处理**：处理读取器注册、分片请求、失败 +- **状态管理**：快照剩余分片和分配状态 + +**典型实现思路（伪代码示意）**： + +``` +on run(): + pendingSplits += newlyDiscoveredSplits # 分片发现/生成逻辑由实现决定 + +on handleSplitRequest(subtaskId): + if pendingSplits not empty: + assignSplit(subtaskId, nextSplit) + else: + signalNoMoreSplits(subtaskId) + +on addSplitsBack(splits): + pendingSplits += splits +``` + +### 3.2 SourceReader 接口 + +读取器在工作节点上运行并执行实际的数据读取。 + +**关键契约（摘要）**： +- `pollNext(output)`：拉取下一批数据（建议非阻塞/可限时） +- `addSplits(splits)`：接收枚举器分配的 splits +- `snapshotState(checkpointId)`：返回 split checkpoint state（实际接口返回 `List`） +- `handleNoMoreSplits()`：收到无更多 split 的信号 +- `CheckpointListener` 回调：由框架触发 checkpoint 完成/中止通知 + +**关键职责**： +- **数据读取**：从分配的分片拉取记录 +- **进度跟踪**：跟踪每个分片内的偏移量/位置 +- **状态管理**：快照分片进度以进行恢复 +- **分片管理**：处理分片分配、完成和删除 + +**典型实现思路（伪代码示意）**： + +``` +pollNext(output): + if no active split: + request split if queue empty + else activate next split + read batch records from active split into output + +snapshotState(checkpointId): + return remaining/unconsumed splits (and progress via split内部状态或外部offset映射) +``` + +### 3.3 SourceEvent（自定义通信） + +允许枚举器和读取器交换自定义消息。 + +**核心约束**：事件需可序列化，用于 `SourceReader` 与 `SourceSplitEnumerator` 之间的自定义通信。 + +**使用场景**： +- 动态分区发现（Kafka、HDFS） +- 运行时配置更改 +- 自定义协调逻辑 + +## 4. 设计考量 + +### 4.1 设计权衡 + +#### 枚举器-读取器分离 + +**优点**： +- 清晰分离协调（主节点）和执行（工作节点） +- 枚举器可以在读取器不知情的情况下重新分配分片 +- 集中协调简化分片分配逻辑 +- 容错：枚举器和读取器独立失败 + +**缺点**： +- 额外的网络通信（分片分配消息） +- 连接器开发人员的 API 更复杂 +- 如果枚举器速度慢，可能成为瓶颈 + +**缓解措施**： +- 异步分片分配 +- 批量分片请求/分配 +- 延迟分片发现 + +#### 分片粒度 + +**粗粒度分片**（少量大分片）： +- **优点**：较少的协调开销 +- **缺点**：负载均衡差，恢复时间长 + +**细粒度分片**（许多小分片）： +- **优点**：更好的负载均衡，更快的恢复 +- **缺点**：更高的协调开销 + +**经验建议（仅供参考）**：按数据源特性与作业目标在“负载均衡/协调开销/恢复耗时”之间权衡分片粒度；不要在文档里把某个固定大小当作必然最佳值。 + +### 4.2 性能考量 + +#### 批量读取 + +建议批量读取而不是逐条读取，以摊销 I/O 与序列化开销。 + +**好处**： +- 摊销每条记录的开销 +- 更好的 CPU 缓存利用率 +- 减少锁竞争 + +#### 非阻塞轮询 + +建议在无可用数据时快速返回，由框架按调度节奏再次调用，避免阻塞工作线程。 + +**好处**： +- 避免阻塞工作线程 +- 启用反压处理 +- 更好的资源利用率 + +#### 连接池 + +数据库类 Source 建议使用连接池并控制并发连接数，避免对源端造成压垮式压力。 + +### 4.3 可扩展性 + +#### 自定义分片分配策略 + +自定义分配策略应基于可观测信号（负载、数据局部性、split 大小差异）并确保失败回收路径可用。 + +典型策略包括：按 split 大小做负载均衡、按数据局部性优先分配、对热点 reader 做节流等。 + +#### 动态分片发现 + +动态分片发现通常用于“分区会随时间变化”的数据源（如 Kafka、目录新增文件等）。推荐的设计方式是： + +1. **周期性发现**：枚举器按固定周期扫描新分区/新文件，并将其转换为新的 split。 +2. **增量分配**：新 split 作为增量加入待分配队列，由分配策略按负载分发给 reader。 +3. **一致性边界**：对“发现时点”与“开始消费时点”的关系做明确约束（例如：从发现时刻开始消费；或支持从指定 offset/时间戳消费）。 +4. **与 checkpoint 的关系**：必须确保“新 split 的出现”在故障恢复后可重放（通过枚举器状态快照或外部可重复发现的元数据源实现）。 + +## 5. 最佳实践 + +### 5.1 使用建议 + +**1. 分片大小** +- 文件：按文件系统与下游吞吐能力合理切分（例如按 block/文件/分区等天然边界） +- 数据库：按分片键范围/分页区间/分区等可独立读取的边界切分 +- 消息队列：通常使用原生分区（如 Kafka 分区）作为 split 边界 + +**2. 状态管理** +- 保持分片状态小（每个分片 < 1MB） +- 使用偏移量/位置而不是缓冲数据 +- 高效序列化（Kryo、Protobuf） + +**3. 错误处理** + +建议将错误分为两类并采用不同策略： +- **瞬态错误**（网络抖动、临时超时、可重试的限流）：允许有限次数重试，并使用退避策略（exponential backoff + jitter），同时把重试次数/最后错误输出到指标与日志。 +- **致命错误**（配置错误、权限不足、协议不兼容、数据不可解析且无法跳过）：应快速失败并把异常向框架上抛，触发作业失败或按作业级策略处理。 + +注意事项： +- 避免在工作线程里进行长时间 sleep；如果必须退避，优先采用非阻塞式调度或由框架驱动下一次 poll。 +- 对“可跳过的坏数据”要显式配置并记录（计数、采样、落盘/死信），默认不建议静默吞掉。 + +**4. 资源管理** + +资源管理建议： +- 对所有外部资源（连接、游标/ResultSet、文件句柄、线程池、缓冲区）建立“创建-使用-关闭”的明确生命周期，并保证 close 在异常路径也能执行。 +- 优先使用连接池并设置上限，避免并发 reader 放大源端压力。 +- 释放顺序建议与依赖关系一致（先游标/会话，后连接/池）。 + +### 5.2 常见陷阱 + +**1. 阻塞 pollNext()** + +反例：在 `pollNext()` 中无限期阻塞（例如等待队列/网络直到有数据），会占用工作线程并破坏框架调度。 + +推荐： +- 使用非阻塞或有超时的轮询，没数据时快速返回，让框架按节奏再次调用。 +- 把“等待数据”的职责交给外部组件（如有界队列 + 生产线程），但 reader 侧仍应遵循非阻塞/可中断原则。 + +**2. 大状态** + +反例：把整段数据缓冲进 checkpoint state，会导致状态膨胀、checkpoint 变慢、恢复时间不可控。 + +推荐： +- 状态只保存“可重放位置”（offset、游标位置、文件 path+position、分区+时间戳等）。 +- 把缓存留在内存并可丢弃，让恢复依赖可重复读取（replay）而不是依赖大状态。 + +**3. 忘记请求分片** + +反例：当本地没有可读 split 时直接返回，且没有向框架请求更多 split，会导致 reader 长期空转。 + +推荐： +- 当待处理 split 为空时，主动触发 split request（或进入“等待分片”的可调度状态）。 +- 同时输出指标（例如 pending split 数、空轮询次数），便于发现枚举器未分配/分配失衡问题。 + +### 5.3 调试技巧 + +**1. 启用调试日志** + +建议输出“可定位”的调试日志（并可按配置开关）： +- 当前 split 标识、消费位置（offset/position）、批大小 +- 上次 checkpoint 的 id/时间 +- 最近一次错误类型与重试次数 + +**2. 跟踪指标** + +建议最少暴露以下指标，便于容量规划与排障： +- 吞吐：records/s、bytes/s +- 延迟：端到端 lag（按时间戳/offset） +- backlog：待处理 split 数、每个 split 的剩余量 +- 可靠性：重试次数、失败次数、坏数据计数 + +**3. 测试分片重新分配** + +建议用“故障注入”的方式验证 split 回收与再分配： +- reader 异常退出/超时心跳 -> enumerator 回收其已分配但未完成的 splits +- 新 reader 加入 -> 能重新领取并从正确位置继续消费 +- 验证点：无重复消费（或重复可被幂等吸收）、无数据丢失、恢复耗时可接受 + +## 6. 相关资源 + +- [架构概览](../overview.md) +- [设计理念](../design-philosophy.md) +- [数据 Sink 架构](sink-architecture.md) +- [检查点机制](../fault-tolerance/checkpoint-mechanism.md) +- [如何创建您的连接器](../../developer/how-to-create-your-connector.md) + +## 7. 参考资料 + +### 示例连接器 + +- **简单数据源**：FakeSource（生成测试数据） +- **文件数据源**：FileSource（本地/HDFS/S3 文件） +- **数据库数据源**：JdbcSource（JDBC 兼容数据库） +- **流式数据源**：KafkaSource（Apache Kafka） +- **CDC 数据源**：MySQLCDCSource（MySQL binlog） + +### 进一步阅读 + +- Apache Flink FLIP-27：["Refactored Source API"](https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface) +- Kafka Consumer：[Consumer Groups and Partition Assignment](https://kafka.apache.org/documentation/#consumerconfigs) diff --git a/docs/zh/architecture/api-design/translation-layer.md b/docs/zh/architecture/api-design/translation-layer.md new file mode 100644 index 000000000000..54d25a541581 --- /dev/null +++ b/docs/zh/architecture/api-design/translation-layer.md @@ -0,0 +1,242 @@ +--- +sidebar_position: 1 +title: 转换层 +--- + +# 转换层架构 + +## 1. 概述 + +### 1.1 问题背景 + +SeaTunnel 提供统一的连接器 API,但作业需要在不同的执行引擎上运行: + +- **引擎多样性**: Flink、Spark、SeaTunnel Engine (Zeta) 具有不同的 API +- **代码重复**: 没有转换,每个连接器需要 3 个实现 +- **维护负担**: Bug 修复需要在所有实现中进行更改 +- **API 演化**: 引擎 API 变更会破坏连接器 +- **用户体验**: 用户希望跨引擎的一致行为 + +### 1.2 设计目标 + +SeaTunnel 的转换层旨在: + +1. **实现可移植性**: 相同的连接器可在任何引擎上运行 +2. **隐藏复杂性**: 连接器开发者只需学习 SeaTunnel API +3. **保持保真度**: 跨引擎保留语义保证 +4. **最小化开销**: 尽量降低转换对吞吐/延迟的影响（取决于 connector、类型转换与引擎实现） +5. **支持演化**: 将连接器与引擎 API 变更隔离 + +### 1.3 架构概览 + +``` +┌──────────────────────────────────────────────────────────────┐ +│ SeaTunnel API 层 │ +│ (引擎独立的连接器接口) │ +│ │ +│ SeaTunnelSource SeaTunnelSink SeaTunnelTransform │ +└──────────────────────────────────────────────────────────────┘ + │ + │ 转换层 + ┌─────────────┼─────────────┐ + ▼ ▼ ▼ +┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ +│ Flink 适配器 │ │ Spark 适配器 │ │ Zeta (原生) │ +│ │ │ │ │ │ +│ FlinkSource │ │ SparkSource │ │ 直接 │ +│ FlinkSink │ │ SparkSink │ │ 执行 │ +└──────────────────┘ └──────────────────┘ └──────────────────┘ + │ │ │ + ▼ ▼ ▼ +┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ +│ Apache Flink │ │ Apache Spark │ │ SeaTunnel Engine │ +│ 运行时 │ │ 运行时 │ │ (Zeta) │ +└──────────────────┘ └──────────────────┘ └──────────────────┘ +``` + +## 2. Flink 转换层 + +### 2.1 FlinkSource 适配器 + +将 `SeaTunnelSource` 适配到 Flink 的 `Source` 接口。 + +**适配点（语义级）**： +- **有界/无界语义**：把 SeaTunnel 的 boundedness 映射到 Flink 的 `Boundedness` +- **Reader 创建**：把 Flink `SourceReaderContext` 适配为 SeaTunnel reader context，并用 wrapper 把 SeaTunnel reader 包装成 Flink reader +- **Enumerator 创建**：把 Flink `SplitEnumeratorContext` 适配为 SeaTunnel enumerator context，并包装成 Flink enumerator +- **序列化器**：把 SeaTunnel 的 split/state 序列化器适配到 Flink 的 `SimpleVersionedSerializer` + +### 2.2 FlinkSourceReader 适配器 + +**适配点（语义级）**： +- `start/open`：把 Flink 的 reader 生命周期委托给 SeaTunnel reader +- `pollNext`：把 Flink `ReaderOutput` 适配为 SeaTunnel collector，并映射“有无数据可读”的返回语义 +- `addSplits`：把 Flink 的 split wrapper 解包为 SeaTunnel split 再下发 +- `snapshotState`：把 SeaTunnel reader 的快照结果包装为 Flink 侧可序列化的 split/state +- `notifyCheckpointComplete`：把 checkpoint 完成通知下沉到 SeaTunnel reader（用于清理/提交等） + +### 2.3 FlinkSourceEnumerator 适配器 + +**适配点（语义级）**： +- 生命周期：Flink enumerator 的 `start` 驱动 SeaTunnel enumerator 的 open/run +- 分片请求：Flink 的 split request 透传给 SeaTunnel enumerator 的分片分配逻辑 +- 分片回退：把回退 split 解包并回交给 SeaTunnel enumerator +- 状态快照：把 enumerator state 包装成 Flink 可持久化的 wrapper，以参与 checkpoint + +### 2.4 上下文适配器 + +**FlinkSourceReaderContext**: + +- 下标与并行度：把 Flink 的 subtask index 映射为 SeaTunnel reader 的 index +- 事件通道：把 SeaTunnel 的 SourceEvent 包装后发送到 Flink 的 coordinator/event channel +- 分片请求：Flink 会在运行时自动触发 split request，SeaTunnel 侧通常不需要显式触发 + +**FlinkSourceSplitEnumeratorContext**: + +- 并行度/注册 reader：把 Flink 的 runtime 信息暴露给 SeaTunnel enumerator +- 分片分配：把 SeaTunnel split 包装为 Flink split 并通过 Flink 的 assignment API 下发 +- no-more-splits：在有界场景下通知 reader 结束 +- 事件下发：把 SeaTunnel event 包装为 Flink event 并发送给指定 reader + +### 2.5 FlinkSink 适配器 + +**适配点（语义级）**： +- writer：把 Flink `InitContext` 适配为 SeaTunnel writer context 并创建 SeaTunnel `SinkWriter` +- committer/global committer：把 SeaTunnel 的两阶段提交组件包装为 Flink 的 committer 体系 +- serializer：把 SeaTunnel 的 commitInfo / writerState 序列化器适配为 Flink `SimpleVersionedSerializer` + +### 2.6 FlinkSinkWriter 适配器 + +**适配点（语义级）**： +- `write`：把 Flink sink writer 的写入请求委托给 SeaTunnel `SinkWriter.write` +- `prepareCommit`：把 SeaTunnel `prepareCommit()` 的可选 commitInfo 映射为 Flink 的 committable 列表 +- `snapshotState`：直接使用 SeaTunnel writer 的快照结果参与 Flink checkpoint +- `close`：委托关闭，确保释放外部资源 + +## 3. Spark 转换层 + +### 3.1 SparkSource 适配器 + +将 `SeaTunnelSource` 适配到 Spark 的数据源接口（Spark 2.4 与 Spark 3.x 使用的 DataSource API 形态不同，具体以对应版本适配模块实现为准）。 + +**适配点（语义级）**： +- `readSchema`：把 SeaTunnel `CatalogTable/TableSchema` 映射为 Spark `StructType` +- `planInputPartitions`：在 Spark 的批处理模型下，通常一次性生成全部 splits，并为每个 split 构造一个 `InputPartition` + +Spark 的执行模型偏“批式规划”，因此枚举器的职责更像是“规划阶段生成分片集合”，而不是长期运行的调度器。 + +### 3.2 SparkInputPartition + +**适配点（语义级）**： +- 每个 `InputPartition` 绑定一个 SeaTunnel split +- `createPartitionReader` 创建 SeaTunnel reader，注入该 split，并把输出转换为 Spark `InternalRow` + +### 3.3 SparkPartitionReader + +**适配点（语义级）**： +- 初始化：创建并打开 SeaTunnel reader，下发 split +- 读取循环：从 SeaTunnel reader 拉取记录并转换为 Spark `InternalRow`（必要时使用缓冲队列适配 pull-based API） +- 资源释放：关闭 reader 并释放外部资源 + +### 3.4 SparkSink 适配器 + +**适配点（语义级）**： +- writer factory：在 executor 侧创建写入器实例并接收 Spark `InternalRow` +- commit coordinator：当目标端存在提交器时启用 Spark 的提交协调路径 +- commit/abort：把 Spark 的提交消息转换为 SeaTunnel 的 commitInfo 列表，并交由 SeaTunnel `SinkCommitter` 执行（要求幂等/可重试） + +## 4. 序列化适配器 + +### 4.1 FlinkSimpleVersionedSerializer + +**适配点（语义级）**： +- 版本：将 SeaTunnel serializer 的版本号透传到 Flink 侧 +- 序列化/反序列化：直接委托给 SeaTunnel serializer，以保证跨引擎一致的状态编码 + +## 5. 类型转换 + +### 5.1 Spark 类型转换 + +**适配点（语义级）**： +- Schema：将 SeaTunnel `TableSchema` 映射为 Spark `StructType` +- DataType：按 `SqlType` 做一一映射（整数/浮点/decimal/string/boolean/date/timestamp/bytes/array/map 等） +- 兼容性：当引擎侧类型更细分时（例如 timestamp 语义差异），以 SeaTunnel 的“最小公分母”语义为准，并允许通过配置选择具体映射策略 + +## 6. 性能考虑 + +### 6.1 转换开销 + +转换层带来的开销主要来自上下文包装、类型转换、序列化/反序列化等。实际开销高度依赖具体 connector 的 I/O 特性与数据类型分布，因此本文不提供固定比例或吞吐数字，避免与真实环境产生偏差。 + +### 6.2 优化技术 + +**批量类型转换**: + +- 优先批量转换（向量化/批处理）以摊销 per-row 转换成本 +- 在不改变语义的前提下减少对象创建与复制（降低 GC 压力） + +**避免不必要的包装**: + +- 优先复用已有序列化能力，避免重复 wrapper 造成的额外拷贝 +- 在必须 wrapper 时采用惰性策略：仅在 checkpoint/网络传输时做包装 + +## 7. 限制和解决方法 + +### 7.1 引擎特定功能 + +**问题**: 某些引擎功能在 SeaTunnel 中没有等效项。 + +**示例**: Flink 的 `WatermarkStrategy` + +Flink 的 watermark/事件时间语义属于引擎特性，SeaTunnel 的连接器 API 默认不直接暴露该能力。 + +**解决方法**: 提供引擎特定配置 +```hocon +source { + Kafka { + # SeaTunnel 配置 + topic = "my_topic" + + # 引擎特定配置(仅用于 Flink) + flink.watermark.strategy = "bounded-out-of-orderness" + flink.watermark.max-out-of-orderness = "5s" + } +} +``` + +### 7.2 类型系统差异 + +**问题**: 类型系统不完全对齐。 + +**示例**: Spark 有 `TimestampType`,Flink 有 `LocalZonedTimestampType` 和 `TimestampType`。 + +**解决方法**: 使用最小公分母 + +SeaTunnel 侧使用统一抽象类型；转换层根据引擎能力与用户配置决定映射到哪一种引擎类型。 + +## 8. 最佳实践 + +### 8.1 连接器开发 + +**应该做的**: +- 仅实现 SeaTunnel API +- 在多个引擎上测试 +- 使用 SeaTunnel 类型 + +**不应该做的**: +- 在连接器代码中引用引擎特定 API +- 假设特定引擎行为 +- 使用引擎特定优化 + +### 8.2 测试 + +**在所有引擎上测试**: + +- 建议使用参数化/矩阵测试：同一套连接器用例在 Flink/Spark/Zeta 上跑 +- 覆盖语义一致性：exactly-once、checkpoint 恢复、schema 兼容、分片重新分配等 + +## 9. 相关资源 + +- [数据 Source 架构](../api-design/source-architecture.md) +- [目标端 Sink 架构](../api-design/sink-architecture.md) +- [设计理念](../design-philosophy.md) diff --git a/docs/zh/architecture/design-philosophy.md b/docs/zh/architecture/design-philosophy.md new file mode 100644 index 000000000000..e37445f5ccd2 --- /dev/null +++ b/docs/zh/architecture/design-philosophy.md @@ -0,0 +1,474 @@ +--- +sidebar_position: 2 +title: 设计理念 +--- + +# SeaTunnel 设计理念 + +## 1. 概述 + +本文档阐述了塑造 SeaTunnel 架构的核心设计原则、理念和权衡。理解这些原则有助于贡献者做出一致的设计决策，并帮助用户了解系统的优势和局限性。 + +## 2. 核心设计原则 + +### 2.1 引擎独立性 + +**原则**：将连接器逻辑与执行引擎解耦。 + +**动机**： +- 数据同步专用引擎 Zeta 出现之前，用户可能已有 Flink 或 Spark 集群 +- 不同引擎适用于不同场景（批处理 vs 流处理、资源约束） +- 连接器开发人员不应需要理解多个引擎 API + +**实现**： +- 统一的 SeaTunnel API 层抽象引擎特定细节 +- 转换层将 SeaTunnel API 适配到引擎特定 API +- 连接器逻辑尽量与执行引擎解耦；在转换层支持的前提下，同一套连接器实现可复用到不同引擎（具体可用性以连接器能力与引擎支持为准） + +**权衡**： +- **优点**：最大化可重用性 - 复用连接器逻辑，减少引擎适配重复开发 +- **优点**：更简单的连接器开发 - 只需学习单一 API +- **缺点**：无法利用引擎特定的优化 +- **缺点**：额外的转换开销 +- **缓解措施**：转换层轻薄且优化；大部分开销在 I/O 而非转换 + +**示例**：连接器仅实现 SeaTunnel API 的抽象（Source/Sink/Transform），不同执行引擎通过转换层完成适配；因此连接器逻辑与引擎 API 变更解耦。 + +### 2.2 协调与执行分离 + +**原则**：将控制逻辑（协调）与数据处理（执行）分离。 + +**动机**： +- 协调逻辑是单线程且轻量级的 +- 执行逻辑是并行且资源密集的 +- 容错需要为每个部分独立管理状态 + +**实现原理**： + +**协调层（Master 侧）**： +- 运行位置：主节点，维护全局视图 +- 核心职责：资源发现、工作分配、故障检测、状态协调 +- 运行特点：单线程、轻量级、不处理实际数据 +- 维护状态：分配计划、待处理工作单元、全局进度 + +**执行层（Worker 侧）**： +- 运行位置：工作节点，独立并行执行 +- 核心职责：本地数据处理、进度汇报、参与检查点 +- 运行特点：多线程、资源密集、处理大量数据 +- 维护状态：本地处理进度、缓冲数据、执行上下文 + +**通信机制**： +- 协调层 → 执行层：通过事件分发工作（如：分配新的数据分片） +- 执行层 → 协调层：通过消息汇报进度（如：完成分片、请求新工作） +- 检查点时：各自快照自己的状态，互不干扰 + +**权衡**： +- **优点**：清晰的关注点分离 +- **优点**：枚举器可以在失败时重新分配分片 +- **优点**：提交器实现全局事务协调 +- **缺点**：额外的通信开销 +- **缺点**：连接器开发人员的 API 更复杂 +- **缓解措施**：合理的默认值；简单连接器可以使用简单的枚举器/提交器 + +**示例**： +- 主节点侧：负责“发现/生成工作单元（split）+ 分配 + 回收 + 快照状态”。 +- 工作节点侧：负责“执行读取/写入 + 汇报进度 + 参与 checkpoint”。 + +这样设计的关键原因是：容错需要区分“控制状态”（分配/待处理 split）和“执行进度”（每个 split 的 offset/position），才能在失败后做到精准恢复与快速重分配。 + +### 2.3 基于分片的并行度 + +**原则**：将数据源划分为可独立处理的分片。 + +**动机**： +- 实现无需紧密协调的并行处理 +- 支持动态负载均衡和故障恢复 +- 提供检查点粒度（每个分片的进度） + +**实现**： +- 数据源划分为分片（文件块、DB 分区、Kafka 分区等） +- 枚举器延迟或急切地生成分片 +- 读取器独立处理分片 +- 未处理的分片可以在失败时重新分配 + +**权衡**： +- **优点**：出色的可扩展性 - 添加工作节点以处理更多分片 +- **优点**：细粒度故障恢复 - 仅需要重新处理失败的分片 +- **优点**：动态负载均衡 - 将更多分片分配给空闲的工作节点 +- **缺点**：某些数据源的分片生成开销 +- **缺点**：需要跟踪每个分片的状态 +- **缓解措施**：延迟分片生成；分片状态轻量级 + +**示例**： +- 数据库场景：split 通常表达“分片键范围/分页区间/分区”一类可独立读取的范围。 +- 文件场景：split 通常表达“文件 + 起始偏移 + 长度”或“单文件”。 + +这里不展示具体结构体代码，重点在于 split 的边界：必须能被独立处理、可序列化传输、可在失败后重新分配。 + +### 2.4 通过两阶段提交实现精确一次语义 + +**原则**：保证端到端精确一次数据传递。 + +**动机**： +- 数据集成不能丢失或重复数据 +- 失败可能在任何时候发生（网络、进程崩溃） +- 外部系统需要事务保证 + +**实现原理**： + +两阶段提交协议将数据写入过程分为两个独立阶段： + +1. **准备阶段（Prepare Phase）**： + - 时机：在检查点屏障到达时触发 + - 动作：写入端生成"可提交但未提交"的凭证（如事务 ID、临时文件路径） + - 约束：不对外部系统产生可见副作用（数据对外不可见） + - 状态：凭证信息随检查点一起持久化 + +2. **提交阶段（Commit Phase）**： + - 时机：检查点完整成功后 + - 动作：协调端使用凭证信息原子性地提交变更（如提交事务、移动文件） + - 效果：数据对外部系统可见 + - 保证：幂等性，重复提交不产生副作用 + +3. **中止处理（Abort Handling）**： + - 时机：检查点失败或超时 + - 动作：清理准备阶段产生的临时资源（如回滚事务、删除临时文件） + - 效果：保证不会产生部分写入或不一致状态 + +**权衡**： +- **优点**：强一致性保证 +- **优点**：自动从失败中恢复 +- **缺点**：需要数据 Sink 中的事务支持（或幂等操作） +- **缺点**：增加延迟（数据仅在提交后可见） +- **缺点**：提交信息的额外状态 +- **缓解措施**：可选特性；非事务性数据 Sink 可使用至少一次模式 + +**示例**：典型的 Exactly-Once 落地方式是“写入端先生成可提交凭证（commit info），checkpoint 成功后再由协调端执行最终提交”。这样做的原因是：把副作用（对外部系统的可见变更）延后到 checkpoint 成功之后，避免失败重启时产生重复可见写入。 + +### 2.5 模式作为一等公民 + +**原则**：将模式视为通过管道传播的显式、类型化的元数据。 + +**动机**： +- 数据集成需要模式转换和验证 +- 模式演化（DDL 变更）必须显式处理 +- 类型不匹配应该尽早捕获 + +**实现**： +- `CatalogTable` 封装完整的表元数据 +- `TableSchema` 定义结构（列、主键、约束） +- 模式通过数据源 → 转换 → 数据 Sink 传播 +- `SchemaChangeEvent` 表示 DDL 变更（ADD/DROP/MODIFY 列） + +**权衡**： +- **优点**：类型安全 - 在作业提交时验证模式 +- **优点**：模式演化 - 在运行时处理 DDL 变更 +- **优点**：更好的错误消息 - 尽早检测模式不匹配 +- **缺点**：无模式数据源的额外复杂性 +- **缺点**：某些数据源的模式发现开销 +- **缓解措施**：模式推断助手；可选的模式覆盖 + +**示例**：数据源产出“显式模式”（列、主键、约束、分区、选项等），转换对模式进行验证与映射，数据 Sink 在接收端再次校验。这样做的原因是：把“类型不匹配/缺列/主键冲突”等问题尽早暴露在提交阶段，而不是让它们在运行时以隐式的脏数据形式出现。 + +### 2.6 具有类加载器隔离的插件架构 + +**原则**：连接器是动态加载的插件，具有隔离的依赖。 + +**动机**： +- 避免依赖冲突（例如，多个 JDBC 驱动程序版本） +- 实现热插拔连接器，无需重新构建核心 +- 减少核心分发大小 + +**实现**： +- 用于连接器发现的 Java SPI +- 每个连接器具有隔离的类加载器 +- 遮蔽插件依赖以避免冲突 +- 用于实例化的工厂模式 + +**权衡**： +- **优点**：依赖隔离 - 无版本冲突 +- **优点**：更小的核心分发 +- **优点**：易于添加第三方连接器 +- **缺点**：类加载器复杂性 +- **缺点**：某些共享库（如 Guava）可能存在问题 +- **缓解措施**：谨慎遮蔽；核心中的共享通用库 + +**示例**： +``` +seatunnel-engine/lib/ # 核心库 +connector-jdbc/lib/ # JDBC 驱动程序（隔离） +connector-kafka/lib/ # Kafka 客户端（隔离） + +# 每个连接器由单独的 ClassLoader 加载 +ConnectorClassLoader(connector-jdbc) -> 加载 mysql-connector-java-8.0.26.jar +ConnectorClassLoader(connector-kafka) -> 加载 kafka-clients-3.0.0.jar +``` + +### 2.7 具有检查点存储抽象的状态管理 + +**原则**：将状态管理与存储实现解耦。 + +**动机**： +- 不同部署需要不同的存储（HDFS、S3、本地、OSS） +- 状态大小差异很大（KB 到 TB） +- 存储耐久性和性能要求不同 + +**实现**： +- 可插拔 checkpoint storage（例如 localfile/hdfs 等，取决于插件与配置） +- 状态的可插拔序列化 +- 增量检查点支持 +- 自动状态清理 + +**权衡**： +- **优点**：灵活性 - 根据部署选择存储 +- **优点**：增量检查点减少开销 +- **缺点**：存储性能影响检查点延迟 +- **缺点**：生产环境需要分布式文件系统 +- **缓解措施**：异步检查点上传；可配置间隔 + +### 2.8 多表同步 + +**原则**：支持在单个作业中同步多个表。 + +**动机**： +- 数据库迁移通常涉及数百个表 +- 为每个表创建一个作业浪费资源 +- 模式演化必须应用于所有表 + +**实现**： +- `MultiTableSource` / `MultiTableSink` 包装单个表数据源/Sink +- `TablePath` 将记录路由到正确的表 +- 按表传播模式变更 +- 支持副本以提高吞吐量 + +**权衡**： +- **优点**：资源效率 - 一个作业而不是数百个 +- **优点**：跨表一致快照 +- **优点**：集中监控 +- **缺点**：一个表失败可能影响其他表 +- **缺点**：更复杂的错误处理 +- **缓解措施**：可配置的错误容忍度；按表的指标 + +## 3. 架构权衡 + +### 3.1 简单性 vs 性能 + +**选择**：优先考虑简单性和正确性而非极端性能优化。 + +**理由**： +- 数据集成是 I/O 密集型的，而非 CPU 密集型 +- 正确的语义（精确一次）比原始速度更关键 +- 简单的代码易于维护和调试 + +**证据**： +- 网络和磁盘 I/O 主导处理时间（> 90%） +- 转换层开销可以忽略不计（< 1%） +- 代码可读性优先（例如，清晰的状态机，无微观优化） + +### 3.2 灵活性 vs 易用性 + +**选择**：提供合理的默认值，同时允许高级定制。 + +**理由**： +- 大多数用户想要简单的配置 +- 高级用户需要细粒度控制 +- 两种需求可以通过分层 API 满足 + +**实现**： +- 常见情况的高级配置（例如，`jdbc://host:port/db`） +- 专家的低级选项（例如，连接池调优） +- 合理的默认值（并行度、检查点间隔、缓冲区大小） + +### 3.3 通用性 vs 专业化 + +**选择**：通用 API 与专业化实现。 + +**理由**： +- 统一的 API 简化了学习和使用 +- 不同的数据源具有独特的特征（有界 vs 无界、可分片性） +- 专业化发生在连接器实现中，而非 API 中 + +**示例**： +- `SourceSplitEnumerator` 足够通用，可用于文件、数据库和消息队列 +- 文件连接器使用基于文件的分片 +- Kafka 连接器使用基于分区的分片 +- JDBC 连接器使用基于查询的分片 + +### 3.4 强一致性 vs 延迟 + +**选择**：提供精确一次（高延迟）和至少一次（低延迟）模式。 + +**理由**： +- 某些应用需要强一致性（金融、计费） +- 其他应用可以容忍重复以获得更低延迟（日志、指标） +- 让用户根据需求选择 + +**配置**： +```hocon +env { + checkpoint.mode = "EXACTLY_ONCE" # 或 "AT_LEAST_ONCE" + checkpoint.interval = 60000 # 毫秒 +} +``` + +## 4. 从 V1 到 V2 的演进 + +### 4.1 V1 的局限性 + +SeaTunnel V1（2.3.0 之前）存在重大架构局限性： + +1. **引擎特定连接器**：Spark 和 Flink 的单独实现 +2. **无统一 API**：无抽象层，与引擎紧密耦合 +3. **有限的容错**：完全依赖引擎检查点 +4. **无模式管理**：模式隐式，无演化支持 +5. **仅单表**：不支持多表同步 + +### 4.2 V2 改进 + +SeaTunnel V2（2.3.0+）重新设计了架构： + +| 方面 | V1 | V2 | +|-----|----|----| +| **API** | 引擎特定 | 统一的 SeaTunnel API | +| **连接器** | 重复代码 | 单一实现 | +| **容错** | 依赖引擎 | 显式检查点协议 | +| **模式** | 隐式 | 显式 CatalogTable | +| **多表** | 不支持 | 原生支持 | +| **引擎支持** | Spark、Flink | Spark、Flink、Zeta | +| **精确一次** | 部分 | 端到端 2PC | + +### 4.3 迁移路径 + +V1 和 V2 连接器共存但使用不同的 API： +- V1 连接器：`seatunnel-connectors/`（已弃用） +- V2 连接器：`seatunnel-connectors-v2/`（推荐） + +V2 是未来；V1 处于维护模式。 + +## 5. 关键设计决策 + +### 5.1 为什么分离枚举器和读取器？ + +**替代方案**：单个组件同时处理分片生成和读取。 + +**决策**：分离组件。 + +**理由**： +- 分片生成是协调逻辑（应在主节点上运行） +- 数据读取是执行逻辑（应在工作节点上运行） +- 一方的失败不应影响另一方 +- 允许在不重启读取器的情况下重新分配分片 + +### 5.2 为什么三级数据 Sink 提交（写入器 → 提交器 → 聚合提交器）？ + +**替代方案**：两级（写入器 → 提交器）或直接写入器提交。 + +**决策**：可选的三级提交。 + +**理由**： +- **写入器**：并行、有状态、每个任务 +- **提交器**：并行、无状态、聚合每个写入器的提交 +- **聚合提交器**：单线程、有状态、全局协调器 + +许多数据 Sink 只需要写入器 + 提交器；聚合提交器用于复杂情况（例如，需要单一全局操作的 Hive 表提交）。 + +### 5.3 为什么 LogicalDag → PhysicalPlan 分离？ + +**替代方案**：直接从配置生成物理执行计划。 + +**决策**：两阶段规划。 + +**理由**： +- LogicalDag 表示用户意图（可移植、引擎独立） +- PhysicalPlan 表示执行策略（引擎特定、优化） +- 分离实现： + - 跨引擎可移植性（相同的 LogicalDag，不同的 PhysicalPlan） + - 优化传递（融合、分片重新分配） + - 测试（单独验证逻辑计划） + +### 5.4 为什么基于管道的执行？ + +**替代方案**：单一全局任务图。 + +**决策**：作业划分为管道。 + +**理由**： +- 每个管道独立的检查点协调 +- 更清晰的失败边界 +- 更容易推理数据流 +- 支持复杂的 DAG（多个数据源/Sink ） + +### 5.5 为什么不使用引擎原生检查点？ + +**替代方案**：完全依赖 Flink/Spark 检查点机制。 + +**决策**：显式 SeaTunnel 检查点协议。 + +**理由**： +- 引擎独立性 - 需要跨引擎的一致语义 +- Zeta 引擎否则将没有检查点 +- 更多对精确一次语义的控制 +- 统一的监控和可观测性 + +但是，对于 Flink 转换，SeaTunnel 检查点与 Flink 检查点对齐以避免重复。 + +## 6. 未来方向 + +### 6.1 计划增强 + +- **动态扩缩容**：在作业执行期间添加/移除工作节点 +- **自适应批量大小**：根据吞吐量自动调整批量大小 +- **查询下推**：将过滤器/投影下推到数据源 +- **向量化执行**：处理批量行（列式） +- **推测执行**：缓解掉队者 + +### 6.2 研究方向 + +- **机器学习集成**：基于 ML 的优化（分片大小、并行度） +- **统一批处理和流处理**：真正的统一处理模型 +- **全局查询优化**：跨管道优化 + +## 7. 经验教训 + +### 7.1 成功之处 + +1. **引擎独立性**：通过成功添加 Zeta 引擎而无需 API 更改得到验证 +2. **基于分片的并行度**：扩展到 1000+ 并行任务 +3. **显式模式**：尽早捕获许多错误，实现模式演化 +4. **两阶段提交**：可靠的精确一次语义 + +### 7.2 可以改进之处 + +1. **API 复杂性**：枚举器/提交器增加了简单连接器的学习曲线 +2. **类加载器问题**：遮蔽依赖偶尔冲突 +3. **检查点延迟**：大状态导致检查点延迟 +4. **文档差距**：架构文档落后于代码 + +### 7.3 如果重新开始 + +1. **简化 API**：为简单的数据源/Sink 提供更高级的抽象 +2. **异步 I/O 支持**：非阻塞连接器的一等异步 API +3. **内置指标**：API 中的标准化指标收集 +4. **模式注册表集成**：与外部模式注册表更紧密的集成 + +## 8. 结论 + +SeaTunnel 的架构反映了竞争关注点之间的仔细权衡： +- 引擎独立性 vs 引擎特定优化 +- 简单性 vs 灵活性 +- 一致性 vs 延迟 +- 通用性 vs 专业化 + +V2 重新设计解决了 V1 的主要局限性，同时建立了长期演进的原则。理解这些设计理念有助于贡献者做出一致的决策，并帮助用户了解 SeaTunnel 的优势和适用场景。 + +## 9. 参考资料 + +- [架构概览](overview.md) +- [数据 Source 架构](api-design/source-architecture.md) +- [数据 Sink 架构](api-design/sink-architecture.md) +- [检查点机制](fault-tolerance/checkpoint-mechanism.md) + +### 学术论文 + +- Chandy-Lamport：["Distributed Snapshots: Determining Global States of Distributed Systems"](https://lamport.azurewebsites.net/pubs/chandy.pdf) +- Flink：["Apache Flink: Stream and Batch Processing in a Single Engine"](https://asterios.katsifodimos.com/assets/publications/flink-deb.pdf) diff --git a/docs/zh/architecture/engine/dag-execution.md b/docs/zh/architecture/engine/dag-execution.md new file mode 100644 index 000000000000..765f47d04272 --- /dev/null +++ b/docs/zh/architecture/engine/dag-execution.md @@ -0,0 +1,627 @@ +--- +sidebar_position: 2 +title: DAG 执行模型 +--- + +# DAG 执行模型 + +## 1. 概述 + +### 1.1 问题背景 + +分布式数据处理需要将用户意图转换为可执行的分布式任务: + +- **抽象层次**: 如何分离逻辑意图与物理执行? +- **优化**: 如何优化任务放置和数据混洗? +- **流水线**: 如何执行具有多个数据 Source/Sink 的复杂 DAG? +- **并行度**: 如何确定任务并行度和分布? +- **故障隔离**: 如何将故障影响限制在受影响的组件内? + +### 1.2 设计目标 + +SeaTunnel 的 DAG 执行模型旨在: + +1. **关注点分离**: 逻辑规划(用户意图) vs 物理执行(运行时细节) +2. **支持优化**: 任务融合、流水线分割、资源分配 +3. **支持复杂拓扑**: 多个数据源、目标端、分支、连接 +4. **促进容错**: 清晰的故障边界与独立检查点 +5. **最大化并行度**: 高效并行执行,最少协调开销 + +### 1.3 执行模型概览 + +``` +用户配置 (HOCON) + │ + ▼ +┌─────────────────────┐ +│ LogicalDag │ 逻辑计划 (做什么) +│ • LogicalVertex │ - 数据 Source/tranform 转换器/Sink 目标端动作 +│ • LogicalEdge │ - 数据依赖关系 +│ • Parallelism │ - 逻辑并行度 +└─────────────────────┘ + │ (计划生成) + ▼ +┌─────────────────────┐ +│ PhysicalPlan │ 物理计划 (如何执行) +│ • SubPlan[] │ - 多个流水线 +│ • Resources │ - 资源需求 +│ • Scheduling │ - 部署策略 +└─────────────────────┘ + │ (流水线分割) + ▼ +┌─────────────────────┐ +│ SubPlan (Pipeline) │ 独立执行单元 +│ • PhysicalVertex[] │ - 并行任务实例 +│ • CheckpointCoord │ - 独立检查点 +│ • PipelineLocation │ - 唯一标识符 +└─────────────────────┘ + │ (任务部署) + ▼ +┌─────────────────────┐ +│ PhysicalVertex │ 已部署任务组 +│ • TaskGroup │ - 共址任务(融合) +│ • SlotProfile │ - 分配的资源槽位 +│ • ExecutionState │ - 运行状态 +└─────────────────────┘ + │ (执行) + ▼ +┌─────────────────────┐ +│ SeaTunnelTask │ 实际执行 +│ • Source/Transform │ - 数据处理 +│ • /Sink Logic │ - 状态管理 +└─────────────────────┘ +``` + +## 2. LogicalDag: 用户意图 + +### 2.1 结构 + +LogicalDag 以引擎无关的方式表示用户的作业配置。 + +LogicalDag 的核心组成: +- **logicalVertexMap**: 顶点集合(每个顶点对应一个 Source/Transform/Sink 动作) +- **edges**: 边集合(描述数据流依赖关系) +- **jobConfig**: 作业级配置(例如并行度默认值、容错/资源/运行参数) + +### 2.2 LogicalVertex + +表示单个动作(数据 Source/转换器/Sink 目标端)及其并行度。 + +一个 LogicalVertex 通常包含: +- **vertexId**: 顶点唯一标识 +- **action**: 动作类型(SourceAction / TransformChainAction / SinkAction) +- **parallelism**: 并行实例数量(若未显式配置，可能由引擎推断) + +**动作类型**: +- **SourceAction**: 封装 `SeaTunnelSource`,生产 `CatalogTable` +- **TransformChainAction**: `SeaTunnelTransform` 链,转换模式 +- **SinkAction**: 封装 `SeaTunnelSink`,消费 `CatalogTable` + +**示例**: + +来自配置的直观映射关系: +- Vertex 1: JDBC Source，parallelism=4 +- Vertex 2: SQL Transform，parallelism=8 +- Vertex 3: Elasticsearch Sink，parallelism=2 + +### 2.3 LogicalEdge + +表示动作之间的数据流。 + +一条 LogicalEdge 通常只需要描述: +- **inputVertexId**: 上游顶点 +- **targetVertexId**: 下游顶点 + +**示例**: + +典型线性拓扑中的边: +- JDBC Source(1) → SQL Transform(2) +- SQL Transform(2) → Elasticsearch Sink(3) + +### 2.4 LogicalDag 创建 + +从用户配置构建: + +LogicalDag 在作业提交/启动阶段由作业执行环境解析配置生成（可能发生在客户端或服务端），随后作为作业不可变信息的一部分交由 JobMaster 管理执行。 + +**过程**: +1. 解析 HOCON 配置(source、transform、sink 部分) +2. 为每个配置的组件创建 `Action` 对象 +3. 从配置结构推断数据流 +4. 验证模式兼容性 +5. 构建 `LogicalDag` 对象 + +**示例配置 → LogicalDag**: +```hocon +env { + parallelism = 4 +} + +source { + JDBC { + url = "jdbc:mysql://..." + query = "SELECT * FROM orders" + } +} + +transform { + Sql { + query = "SELECT order_id, SUM(amount) FROM this GROUP BY order_id" + } +} + +sink { + Elasticsearch { + hosts = ["es-host:9200"] + index = "orders_summary" + } +} +``` + +生成的 LogicalDag: +``` +Vertex 1 (JDBC 数据源, parallelism=4) + │ + ▼ +Vertex 2 (SQL 转换器, parallelism=4) + │ + ▼ +Vertex 3 (Elasticsearch 目标端, parallelism=4) +``` + +## 3. PhysicalPlan: 执行策略 + +### 3.1 结构 + +PhysicalPlan 描述如何在分布式工作节点上执行 LogicalDag。 + +PhysicalPlan 的核心信息通常包括: +- **pipelineList(SubPlans)**: 由 LogicalDag 切分得到的多个流水线(独立执行单元) +- **jobImmutableInformation**: 作业不可变信息(例如作业 ID、提交参数、依赖等) +- **running state store**: 分布式状态存储(用于运行态状态、时间戳、元信息等) +- **jobEndFuture**: 作业完成信号(用于协调退出、回收资源、返回结果) + +### 3.2 流水线分割 + +LogicalDag 在生成 ExecutionPlan 时会被组织为一个或多个**流水线**(Pipeline/SubPlan)。以当前实现为准，主要规则是： + +1. **按连通性拆分**：DAG 中互不相连的子图会被拆成不同流水线。 +2. **遇到多输入顶点时拆分**：当存在“多输入顶点”（某个顶点有多个上游输入，例如 UNION多流汇聚）时，当前实现会沿每条 source→…→sink 的路径拆成多条线性流水线，并对共享顶点做克隆，以降低多输入拓扑在同一流水线内的协调复杂度。 + +说明： +- 如果仅存在“一个 source 分叉到多个 sink”（多输出/分支），但没有任何多输入顶点，当前实现通常不会仅因为多个 sink 就拆分流水线；该分支拓扑仍可能在同一流水线内执行。 +- 更细粒度的切分（例如按并行度/可协调能力）在代码中仍保留 TODO，后续可能演进。 + +**示例 1: 简单线性流水线**: +```hocon +source { JDBC { } } +transform { Sql { } } +sink { Elasticsearch { } } +``` + +生成: **1 个流水线** +``` +流水线 1: [JDBC 数据源] → [SQL 转换器] → [Elasticsearch 目标端] +``` + +**示例 2: 多个数据源**: +```hocon +source { + JDBC { plugin_output = "orders" } + Kafka { plugin_output = "events" } +} + +transform { + Sql { query = "SELECT * FROM orders UNION SELECT * FROM events" } +} + +sink { + Elasticsearch { } +} +``` + +生成: **2 个流水线** +``` +流水线 1: [JDBC 数据源] → [SQL 转换器] → [Elasticsearch 目标端] +流水线 2: [Kafka 数据源] → [SQL 转换器] → [Elasticsearch 目标端] +``` + +**示例 3: 多个目标端**: +```hocon +source { + MySQL-CDC { } +} + +sink { + Elasticsearch { plugin_input = "MySQL-CDC" } + JDBC { plugin_input = "MySQL-CDC" } +} +``` + +生成: **通常为 1 个流水线（包含分支）** +``` +流水线 1: [MySQL-CDC 数据源] → [Elasticsearch 目标端] + └──────→ [JDBC 目标端] +``` + +### 3.3 PhysicalPlan 生成 + +PhysicalPlan 通常由 JobMaster 在拿到 LogicalDag 后生成，并结合 ResourceManager 做资源申请与放置。 + +**步骤**: +1. **分析 LogicalDag**: 识别数据源、目标端和依赖关系 +2. **分割为流水线**: 为每个流水线创建 SubPlan +3. **生成 PhysicalVertices**: 为每个动作创建并行实例 +4. **分配资源**: 从 ResourceManager 请求槽位 +5. **分配任务**: 将 PhysicalVertices 映射到槽位 +6. **创建协调器**: 为每个流水线设置 CheckpointCoordinator + +## 4. SubPlan (流水线) + +### 4.1 结构 + +SubPlan 表示一个独立执行的流水线。 + +SubPlan(流水线)通常包含: +- **pipelineId/pipelineLocation**: 流水线的唯一标识 +- **physicalVertexList**: 此流水线中的并行任务实例列表 +- **coordinatorVertexList**: 协调器类任务(如 split enumerator、聚合提交等单实例协调任务) +- **checkpointCoordinator**: 本流水线的检查点协调器(独立协调域) +- **pipelineStatus**: 执行状态(如 CREATED/RUNNING/FAILED/FINISHED) + +### 4.2 PhysicalVertex 列表 + +每个并行度为 N 的 LogicalVertex 生成 N 个 PhysicalVertices。 + +**示例**: +``` +LogicalVertex: JDBC 数据源 (parallelism = 4) + ↓ +PhysicalVertices: + - PhysicalVertex (子任务 0, 槽位 1) + - PhysicalVertex (子任务 1, 槽位 2) + - PhysicalVertex (子任务 2, 槽位 3) + - PhysicalVertex (子任务 3, 槽位 4) +``` + +### 4.3 协调器顶点 + +用于协调任务的特殊顶点: + +- **SourceSplitEnumerator**: 通常以单实例运行,分配分片给读取器（部署位置由引擎调度决定） +- **SinkAggregatedCommitter**: 当 Sink 提供 aggregated committer 时，通常以单实例运行用于全局提交协调（部署位置由引擎调度决定） + +说明：`SinkCommitter` 的触发方式取决于引擎实现，并不一定体现为独立的协调器顶点；例如在 SeaTunnel Engine 中，committer 可能在 Sink 任务的 checkpoint 回调中被触发。 + +**示例**: +``` +JDBC → Transform → Elasticsearch 的 SubPlan: + physicalVertexList: + - JdbcSourceTask (4 个实例) + - TransformTask (4 个实例) + - ElasticsearchSinkTask (4 个实例) + + coordinatorVertexList: + - JdbcSourceSplitEnumerator (1 个实例) + - ElasticsearchSinkAggregatedCommitter (1 个实例，可选) +``` + +### 4.4 独立检查点 + +每个流水线都有自己的 `CheckpointCoordinator`: + +**优势**: +- 独立的检查点间隔 +- 隔离的故障域 +- 减少协调开销 +- 简化屏障对齐 + +**示例**: +``` +流水线 1 (JDBC → ES): + CheckpointCoordinator 按作业配置的间隔触发 + 仅管理 JDBC 和 ES 任务的检查点 + +流水线 2 (Kafka → JDBC): + CheckpointCoordinator 按作业配置的间隔触发 + 仅管理 Kafka 和 JDBC 任务的检查点 +``` + +## 5. PhysicalVertex: 已部署任务 + +### 5.1 结构 + +PhysicalVertex 表示已部署的任务实例。 + +PhysicalVertex 关注“一个并行任务实例如何被部署与运行”: +- **taskGroupLocation**: 任务实例定位信息(含并行子任务序号等) +- **taskGroup**: 任务融合后的执行单元(见下节) +- **slotProfile**: 该实例被分配到的槽位(资源容量与位置) +- **currentExecutionState**: 当前执行状态(CREATED/RUNNING/FAILED 等) +- **pluginJarsUrls**: 插件依赖(用于类加载隔离) + +### 5.2 TaskGroup: 任务融合 + +多个任务可以融合到单个 `TaskGroup` 以提高效率。 + +TaskGroup 的关键点: +- 将一段可融合的线性算子链(Source/Transform/Sink 的某些组合)放在同一执行单元内 +- 通过共享线程/队列/内存通道减少跨算子序列化与网络开销 +- 以并行度为单位生成多个 TaskGroup 实例(通常与上游并行度对齐) + +**融合条件**: +1. 相同并行度 +2. 顺序依赖(A → B) +3. 不需要数据混洗 + +**示例(带融合)**: +``` +LogicalDag: + Source (parallelism=4) → Transform (parallelism=4) → Sink (parallelism=4) + +不融合: + 12 个独立任务(4 + 4 + 4) + Source → Transform 和 Transform → Sink 有网络开销 + +融合后: + 4 个 TaskGroups,每个包含: + [SourceTask → TransformTask → SinkTask] (单线程,共享内存) +``` + +**优势**: +- 减少网络序列化/反序列化 +- 更好的 CPU 缓存局部性 +- 更低的内存占用 +- 简化部署 + +### 5.3 槽位分配 + +每个 PhysicalVertex 被分配一个 `SlotProfile`: + +SlotProfile 表达“这个任务实例运行在哪里、能用多少资源”。具体字段与语义见资源管理文档。 + +**分配过程**: +1. JobMaster 从 ResourceManager 请求槽位 +2. ResourceManager 根据分配策略选择工作节点（例如 RANDOM / SLOT_RATIO / SYSTEM_LOAD） +3. ResourceManager 分配槽位并返回 SlotProfiles +4. JobMaster 将 SlotProfiles 分配给 PhysicalVertices +5. JobMaster 通过 `DeployTaskOperation` 部署任务 + +## 6. 任务部署和执行 + +### 6.1 部署流程 + +```mermaid +sequenceDiagram + participant JM as JobMaster + participant RM as ResourceManager + participant Worker as Worker Node + participant Task as SeaTunnelTask + + JM->>JM: Generate PhysicalPlan + JM->>RM: applyResources(resourceProfiles) + RM->>RM: Allocate slots + RM-->>JM: Return SlotProfiles + + JM->>JM: Assign slots to PhysicalVertices + + loop For each PhysicalVertex + JM->>Worker: DeployTaskOperation(taskGroup) + Worker->>Task: Create SeaTunnelTask + Task->>Task: INIT → WAITING_RESTORE + Task->>JM: Report ready + end + + JM->>Worker: Start execution + Worker->>Task: READY_START → STARTING → RUNNING +``` + +### 6.2 任务执行 + +每个 `SeaTunnelTask` 执行其分配的动作: + +**SourceSeaTunnelTask**: + +执行要点: +- 持续从 SourceReader 拉取/接收数据并发出记录 +- 在检查点触发时生成并传播 barrier(屏障)，参与流水线级的一致性快照 + +**TransformSeaTunnelTask**: + +执行要点: +- 从上游通道读取记录 +- 应用 transform 逻辑并输出到下游通道 +- 若 transform 有状态，需要参与 checkpoint 的状态快照与恢复 + +**SinkSeaTunnelTask**: + +执行要点: +- 持续消费上游记录并调用 sinkWriter 写入目标端 +- 在 barrier 到达时切换到“快照边界”：准备提交信息(prepareCommit(checkpointId))、持久化 writer 状态并将提交信息交给 committer +- 在 checkpoint 成功后由 committer 进行最终提交；失败时由恢复流程回滚/重试(取决于 sink 语义) + +## 7. 优化策略 + +### 7.1 任务融合 + +**何时融合**: +- 相同并行度 +- 顺序算子(无分支) +- 无混洗边界 + +**何时不融合**: +- 不同并行度(例如 source=4, sink=8) +- 分支 DAG(一个数据源,多个目标端) +- 需要混洗(例如 GROUP BY、JOIN) + +说明：任务融合的具体策略与可配置项以当前引擎实现为准，文档不在此绑定某个固定的配置开关，避免与实际版本不一致。 + +### 7.2 并行度推断 + +并行度以配置为准： +- 若连接器显式配置了 `parallelism`，则使用连接器配置。 +- 否则使用 `env.parallelism`（默认值为 1）。 +- 某些连接器/引擎可能会根据外部系统分区数等信息做额外推断，但这是实现细节，不能在架构文档里写成固定规则。 + +**示例**: +```hocon +source { + JDBC { parallelism = 4 } # 显式 +} + +transform { + Sql { } # 推断: 4 (来自数据源) +} + +sink { + Elasticsearch { } # 推断: 4 (来自转换器) +} +``` + +### 7.3 资源分配 + +**槽位计算**: +``` +所需槽位 = 所有任务并行度之和 + +示例: + Source (parallelism=4) + Transform (parallelism=4) + Sink (parallelism=2) + = 需要 10 个槽位 + +融合后: + TaskGroup (parallelism=4, fusion[Source+Transform]) + Sink (parallelism=2) + = 需要 6 个槽位 +``` + +说明：资源画像/槽位资源的具体字段、单位与配置路径以引擎侧配置与实现为准；文档不在此给出不存在或不稳定的配置项示例。 + +## 8. 故障处理 + +### 8.1 任务故障 + +**检测**: +- 任务抛出异常 +- 心跳超时 + +**恢复**: +1. 标记任务为 FAILED +2. 使整个流水线失败(保守策略) +3. 从最新检查点恢复 +4. 重新分配资源 +5. 重新部署和重启流水线 + +### 8.2 流水线故障隔离 + +**关键见解**: 流水线故障是隔离的。 + +**示例**: +``` +有 2 个流水线的作业: + 流水线 1: JDBC → ES (RUNNING) + 流水线 2: Kafka → JDBC (FAILED) + +结果: + 流水线 2 从检查点重启 + 流水线 1 继续不受影响 +``` + +**优势**: +- 减少爆炸半径 +- 更快恢复(仅失败的流水线) +- 更好的资源利用率 + +## 9. 监控和可观测性 + +### 9.1 关键指标 + +**流水线级别**: +- `pipeline.status`: CREATED / RUNNING / FINISHED / FAILED +- `pipeline.tasks.total`: 任务总数 +- `pipeline.tasks.running`: 当前运行的任务数 +- `pipeline.checkpoint.latest_id`: 最新检查点 ID +- `pipeline.checkpoint.duration`: 检查点持续时间 + +**任务级别**: +- `task.status`: 任务执行状态 +- `task.records_in`: 接收的记录数 +- `task.records_out`: 发出的记录数 +- `task.bytes_in`: 接收的字节数 +- `task.bytes_out`: 发出的字节数 + +### 9.2 可视化 + +``` +作业: mysql-to-es +│ +├── 流水线 1 (mysql-cdc → elasticsearch) +│ ├── PhysicalVertex 0 [RUNNING] @ worker-1:slot-1 +│ ├── PhysicalVertex 1 [RUNNING] @ worker-2:slot-1 +│ ├── PhysicalVertex 2 [RUNNING] @ worker-3:slot-1 +│ └── PhysicalVertex 3 [RUNNING] @ worker-4:slot-1 +│ +└── 流水线 2 (mysql-cdc → jdbc) + ├── PhysicalVertex 0 [RUNNING] @ worker-1:slot-2 + └── PhysicalVertex 1 [RUNNING] @ worker-2:slot-2 +``` + +## 10. 最佳实践 + +### 10.1 并行度配置 + +**经验法则**: +``` +并行度 = min( + 数据分区数, + 可用槽位数, + 目标吞吐量 / 单任务吞吐量 +) +``` + +**示例**: +- **JDBC 数据源**: 设置为数据库分区数(例如 8 个分区 → parallelism=8) +- **Kafka 数据源**: 设置为分区数(例如 32 个分区 → parallelism=32) +- **文件数据源**: 设置为文件数或文件分片数 +- **CPU 密集型转换器**: 设置为 CPU 核心数 +- **I/O 密集型目标端**: 根据目标系统容量设置 + +### 10.2 流水线设计 + +**保持流水线简单**: +- 优先使用线性流水线(数据源 → 转换器 → 目标端) +- 尽可能避免复杂分支 +- 对完全独立的工作流使用多个作业 + +**何时使用多个作业**: +- 需要不同的检查点间隔 +- 需要不同的资源需求 +- 需要独立的故障域 + +### 10.3 故障排除 + +**问题**: 任务未启动 + +**检查**: +1. 是否有足够的可用槽位?(`required_slots <= available_slots`) +2. 资源配置文件是否合理?(不要请求 100 个 CPU 核心) +3. 标签过滤器是否正确?(如果使用基于标签的分配) + +**问题**: 低吞吐量 + +**检查**: +1. 并行度是否太低?(增加并行度) +2. 任务融合是否被禁用?(启用以获得更好的性能) +3. 检查点间隔是否太短?(增加间隔) + +## 11. 相关资源 + +- [引擎架构](engine-architecture.md) +- [资源管理](resource-management.md) +- [检查点机制](../fault-tolerance/checkpoint-mechanism.md) +- [架构概述](../overview.md) + +## 12. 参考资料 +### 进一步阅读 + +- [Google Borg Paper](https://research.google/pubs/pub43438/) - 任务调度灵感 +- [Apache Flink JobGraph](https://nightlies.apache.org/flink/flink-docs-stable/docs/internals/job_scheduling/) +- [Spark DAG Scheduler](https://spark.apache.org/docs/latest/job-scheduling.html) diff --git a/docs/zh/architecture/engine/engine-architecture.md b/docs/zh/architecture/engine/engine-architecture.md new file mode 100644 index 000000000000..ca1980498889 --- /dev/null +++ b/docs/zh/architecture/engine/engine-architecture.md @@ -0,0 +1,574 @@ +--- +sidebar_position: 1 +title: 引擎架构 +--- + +# SeaTunnel 引擎（Zeta）架构 + +## 1. 概述 + +### 1.1 问题背景 + +数据集成引擎必须解决基本的分布式系统挑战： + +- **分布式执行**：如何跨多台机器执行作业？ +- **资源管理**：如何高效地分配和调度任务？ +- **容错**：如何从工作节点/主节点失败中恢复？ +- **协调**：如何同步分布式任务（检查点、提交）？ +- **可扩展性**：如何处理不断增加的工作负载？ + +### 1.2 设计目标 + +SeaTunnel 引擎（Zeta）设计为原生执行引擎，具有： + +1. **轻量级**：最小依赖、快速启动、低资源开销 +2. **高性能**：针对数据同步工作负载优化 +3. **容错**：基于检查点的恢复与精确一次语义 +4. **资源效率**：基于槽位的资源管理与细粒度控制 +5. **引擎独立性**：支持与 Flink/Spark 转换相同的连接器 API + +### 1.3 架构对比 + +| 特性 | SeaTunnel Zeta | Apache Flink | Apache Spark | +|---------|---------------|--------------|--------------| +| **主要用例** | 数据同步、CDC | 流处理 | 批处理 + ML | +| **资源模型** | 基于槽位 | 基于槽位 | 基于执行器 | +| **状态后端** | 可插拔（例如 localfile/hdfs 等，取决于配置与插件） | RocksDB/堆 | 内存/磁盘 | +| **检查点** | 分布式快照 | Chandy-Lamport | RDD 血统 | +| **启动时间** | 取决于部署与依赖 | 取决于部署与依赖 | 取决于部署与依赖 | +| **依赖** | 取决于打包与插件 | 取决于打包与插件 | 取决于打包与插件 | + +## 2. 整体架构 + +### 2.1 主-工架构 + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ 主节点 │ +│ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ CoordinatorService │ │ +│ │ • 管理所有运行中的作业 │ │ +│ │ • 作业提交和生命周期管理 │ │ +│ │ • 维护作业状态（IMap） │ │ +│ │ • 资源管理器工厂 │ │ +│ └───────────────────────────────────────────────────────┘ │ +│ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ JobMaster（每个作业一个） │ │ +│ │ • 生成物理执行计划 │ │ +│ │ • 从 ResourceManager 请求资源 │ │ +│ │ • 将任务部署到工作节点 │ │ +│ │ • 协调检查点 │ │ +│ │ • 处理故障转移和恢复 │ │ +│ └───────────────────────────────────────────────────────┘ │ +│ │ │ │ +│ │ (任务部署) │ (资源请求) │ +│ ▼ ▼ │ +│ ┌─────────────────┐ ┌────────────────────────────┐ │ +│ │ CheckpointManager│ │ ResourceManager │ │ +│ │ (每个管道) │ │ • 槽位分配 │ │ +│ └─────────────────┘ │ • 工作节点注册 │ │ +│ │ • 负载均衡 │ │ +│ └────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ + │ + │ (Hazelcast 集群) + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 工作节点 │ +│ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ TaskExecutionService │ │ +│ │ • 部署和执行任务 │ │ +│ │ • 管理任务生命周期 │ │ +│ │ • 报告心跳 │ │ +│ │ • 槽位资源管理 │ │ +│ └───────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ SeaTunnelTask（每个工作节点多个） │ │ +│ │ │ │ +│ │ ┌─────────────────────────────────────────────┐ │ │ +│ │ │ SourceFlowLifeCycle │ │ │ +│ │ │ • SourceReader │ │ │ +│ │ │ • SeaTunnelSourceCollector │ │ │ +│ │ └─────────────────────────────────────────────┘ │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ ┌─────────────────────────────────────────────┐ │ │ +│ │ │ TransformFlowLifeCycle │ │ │ +│ │ │ • 转换链 │ │ │ +│ │ └─────────────────────────────────────────────┘ │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ ┌─────────────────────────────────────────────┐ │ │ +│ │ │ SinkFlowLifeCycle │ │ │ +│ │ │ • SinkWriter │ │ │ +│ │ └─────────────────────────────────────────────┘ │ │ +│ └───────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### 2.2 核心组件 + +#### CoordinatorService + +管理集群中所有作业的中心化服务。 + +**职责**： +- 接受作业提交 +- 为每个作业创建 JobMaster +- 在分布式 IMap 中维护作业状态 +- 提供作业查询和管理 API +- 处理作业生命周期事件 + +**关键数据结构**： + +- 运行中作业元信息：作业基本信息、当前状态、状态变更时间戳（分布式存储，支持多节点一致读取） +- 已完成作业历史：用于查询与审计的作业快照（通常包含最终状态与关键元数据） + +#### JobMaster + +管理单个作业执行生命周期。 + +**职责**： +- 解析配置 → 生成 LogicalDag +- 从 LogicalDag 生成 PhysicalPlan +- 从 ResourceManager 请求资源（槽位） +- 将任务部署到工作节点 +- 协调管道检查点 +- 处理任务失败并重新调度 + +**生命周期**： +``` +Created → Initialized → Scheduled → Running → Finished/Failed/Canceled +``` + +**关键操作**： +1. `init()`：生成物理计划，创建检查点协调器 +2. `run()`：请求资源，部署任务，启动执行 +3. `handleFailure()`：重启失败的任务，从检查点恢复 + +#### ResourceManager + +管理工作节点资源和槽位分配。 + +**职责**： +- 跟踪工作节点注册和心跳 +- 维护工作节点资源配置（CPU、内存） +- 基于策略分配槽位（随机、槽位比率、基于负载） +- 任务完成后释放槽位 +- 处理工作节点失败 + +**槽位分配策略**： + +- Random：在可用工作节点中随机选择 +- SlotRatio：优先选择拥有更多可用槽位的工作节点 +- SystemLoad：优先选择 CPU/内存使用率较低的工作节点 + +## 3. DAG 执行模型 + +### 3.1 执行计划转换 + +``` +用户配置（HOCON） + │ + ▼ +┌───────────────┐ +│ LogicalDag │ • 逻辑顶点（数据源/转换/数据 Sink ） +│ │ • 逻辑边（数据流） +│ │ • 并行度（每个顶点） +└───────────────┘ + │ (JobMaster.generatePhysicalPlan()) + ▼ +┌───────────────┐ +│ PhysicalPlan │ • SubPlan 列表（管道） +│ │ • JobImmutableInformation +│ │ • 资源要求 +└───────────────┘ + │ + ▼ +┌───────────────┐ +│ SubPlan │ • 管道（独立执行单元） +│ (Pipeline) │ • PhysicalVertex 列表 +│ │ • CheckpointCoordinator +└───────────────┘ + │ + ▼ +┌───────────────┐ +│PhysicalVertex │ • TaskGroup（共存任务） +│ │ • 分配的 SlotProfile +│ │ • ExecutionState +└───────────────┘ + │ + ▼ +┌───────────────┐ +│ TaskGroup │ • 多个 SeaTunnelTask 实例 +│ │ • 共享网络缓冲区 +│ │ • 线程池 +└───────────────┘ + │ + ▼ +┌───────────────┐ +│ SeaTunnelTask │ • 单个任务执行 +│ │ • 数据源/转换/数据 Sink 生命周期 +│ │ • 任务状态机 +└───────────────┘ +``` + +### 3.2 LogicalDag + +以引擎独立的方式表示用户意图。 + +**核心元素（概念级）**： +- LogicalVertex：一个逻辑算子节点（Source / TransformChain / Sink），包含并行度等执行提示 +- LogicalEdge：逻辑边，描述上游到下游的数据流向 +- JobConfig：作业级配置（并行度、容错、资源、插件等） + +**创建**： + +由 `JobConfig`/用户配置构建：解析配置 → 生成顶点/边 → 生成可执行提示（并行度、资源等）。 + +### 3.3 PhysicalPlan + +表示带资源分配的实际执行计划。 + +**核心结构（概念级）**： +- PhysicalPlan：由多个 `SubPlan`（管道）组成，并携带作业不可变元信息与终态结果句柄 +- SubPlan（Pipeline）：一个独立执行单元，包含本管道的任务顶点集合，以及本管道的 checkpoint 协调器 +- PhysicalVertex：一个可调度的并行实例，绑定到具体槽位/工作节点，并维护自身执行状态 + +**生成**： + +由 JobMaster 完成： +1. 将 LogicalDag 切分为管道 +2. 为每个顶点生成并行实例（PhysicalVertex）并计算资源需求 +3. 为每个管道创建独立的 checkpoint 协调器 + +### 3.4 管道执行 + +作业被划分为**管道**（SubPlan）以便独立执行： + +**示例**： +```hocon +# 多数据源/Sink 配置 +env { ... } + +source { + MySQL-CDC { table = "orders" } + Kafka { topic = "events" } +} + +transform { + Sql { query = "SELECT * FROM orders JOIN events ON ..." } +} + +sink { + Elasticsearch { index = "orders" } + JDBC { table = "events" } +} +``` + +**生成的管道**： +``` +管道 1: MySQL-CDC → 转换 → Elasticsearch +管道 2: Kafka → 转换 → JDBC +``` + +**好处**： +- 独立的检查点协调 +- 隔离的失败域 +- 并行管道执行 + +### 3.5 任务融合 + +多个操作可以融合到单个 TaskGroup 中以提高效率： + +``` +无融合： +[数据源任务] → 网络 → [转换任务] → 网络 → [数据 Sink 任务] + +有融合： +[TaskGroup: 数据源 → 转换 → 数据 Sink ]（单线程，无网络） +``` + +**融合条件**： +- 相同的并行度 +- 顺序依赖 +- 不需要 shuffle + +## 4. 任务生命周期 + +### 4.1 任务状态机 + +``` + [Created] + │ + ▼ + [INIT] ────────────────────────────────────┐ + │ │ + ▼ │ +[WAITING_RESTORE]（如果恢复中） │ + │ │ + ▼ │ + [READY_START] │ + │ │ + ▼ │ + [STARTING] ──────────────┐ │ + │ │ │ + ▼ ▼ ▼ + [RUNNING] ──────────> [FAILED] ─────> (重启) + │ + ▼ +[PREPARE_CLOSE] + │ + ▼ + [CLOSED] + │ + ▼ + [CANCELED]（如果作业取消） +``` + +**状态转换**： +1. **CREATED → INIT**：任务已创建，初始化资源 +2. **INIT → WAITING_RESTORE**：从检查点恢复 +3. **WAITING_RESTORE → READY_START**：状态已恢复 +4. **READY_START → STARTING**：打开数据源/转换/数据 Sink +5. **STARTING → RUNNING**：数据处理已启动 +6. **RUNNING → PREPARE_CLOSE**：正常完成 +7. **PREPARE_CLOSE → CLOSED**：资源已清理 +8. **RUNNING → FAILED**：发生异常 + +### 4.2 SeaTunnelTask 执行 + +**执行骨架（语义级）**： +1. `init`：初始化运行时资源 +2. `restoreState`：如果处于恢复路径，加载 checkpoint 状态 +3. `open`：打开 Source/Transform/Sink 生命周期 +4. 主循环：处理数据 + 处理 checkpoint 屏障/控制消息 +5. `close`：正常结束时清理资源；异常时进入失败处理与上报 + +**任务类型**： +- **SourceSeaTunnelTask**：运行 SourceReader，发送数据 +- **SinkSeaTunnelTask**：运行 SinkWriter，消费数据 +- **TransformSeaTunnelTask**：运行转换链 + +### 4.3 FlowLifeCycle 管理 + +每个任务通过 FlowLifeCycle 管理组件生命周期： + +**生命周期语义**： +- `open`：初始化 reader/transform chain/writer 等组件 +- `collect`：数据驱动的执行入口（source poll、transform 处理、sink write） +- `close`：释放资源并保证幂等（可被重复调用） + +## 5. 检查点协调 + +### 5.1 CheckpointCoordinator（每个管道） + +每个管道都有独立的检查点协调器。 + +**职责**： +- 定期触发检查点 +- 将检查点屏障注入数据流 +- 收集任务确认 +- 持久化完成的检查点 +- 清理旧检查点 + +**关键数据结构**： + +- checkpointId 生成器：单调递增生成 checkpointId +- pendingCheckpoints：进行中的 checkpoint 集合（等待 task ACK） +- completed checkpoints：最近成功的 checkpoint 列表（用于恢复与保留策略） +- checkpointStorage：外部持久化后端 + +**检查点流程**： +1. 协调器触发检查点（定期或手动） +2. 向管道中所有数据源任务发送屏障 +3. 屏障通过数据流传播 +4. 每个任务在收到屏障时快照状态 +5. 任务向协调器发送 ACK +6. 协调器等待所有 ACK +7. 创建 CompletedCheckpoint，持久化到存储 + + +### 5.2 检查点屏障 + +与数据一起流动的特殊控制消息： + +**屏障字段（概念级）**： +- checkpointId：本次 checkpoint 的唯一标识 +- timestamp：触发时间 +- type：checkpoint/savepoint 等类型标识 + +**屏障对齐**： +- 具有多个输入的任务在快照前等待来自所有输入的屏障 +- 确保分布式任务之间的一致性快照 + +## 6. 资源管理 + +### 6.1 槽位模型 + +**SlotProfile**： + +- slotId：槽位标识 +- worker：所属工作节点 +- resourceProfile：CPU/内存等资源画像 + +**WorkerProfile**： + +- address：工作节点地址 +- total/available：总资源与可用资源 +- assigned/unassigned：已分配与未分配槽位 + +### 6.2 资源分配流程 + +```mermaid +sequenceDiagram + participant JM as JobMaster + participant RM as ResourceManager + participant Worker as 工作节点 + + JM->>RM: applyResources(jobId, resourceProfiles) + RM->>RM: 选择工作节点（策略） + RM->>RM: 分配槽位 + RM->>JM: 返回槽位配置 + + JM->>Worker: 部署任务（DeployTaskOperation） + Worker->>Worker: 创建 SeaTunnelTask + Worker->>JM: ACK + + JM->>JM: 任务运行中 +``` + +### 6.3 基于标签的槽位过滤 + +将任务分配到特定工作节点组： + +```hocon +env { + # 作业级 worker 标签过滤（key/value 全量匹配） + tag_filter = { + zone = "db-zone" + } +} +``` + +**用途**： +- 数据局部性（分配到靠近数据源的工作节点） +- 资源隔离（ML 转换使用 GPU 工作节点） +- 多租户（不同团队使用不同的工作节点池） + +说明：`tag_filter` 对整个作业/流水线生效；worker 的标签来源于集群成员属性（key/value），由集群部署侧配置与维护。 + +## 7. 失败处理 + +### 7.1 任务失败 + +**检测**： +- 任务向 JobMaster 报告异常 +- JobMaster 监控任务心跳 +- 超时触发失败检测 + +**恢复**： +1. 标记任务为 FAILED +2. 释放任务的槽位 +3. 检索最新的成功检查点 +4. 使用恢复的状态重启任务 +5. 重新分配分片（对于数据源任务） + +### 7.2 工作节点失败 + +**检测**： +- ResourceManager 监控工作节点心跳 +- Hazelcast 集群检测成员移除 + +**恢复**： +1. 标记失败工作节点上的所有任务为 FAILED +2. 触发作业故障转移 +3. 从最新检查点恢复 +4. 在健康的工作节点上重新分配槽位 +5. 重新部署任务 + +### 7.3 主节点失败 + +**高可用性**： +- 多个主节点（Hazelcast 集群） +- 作业状态存储在分布式 IMap 中（已复制） +- 新主节点从 IMap 状态接管 + +**恢复**： +1. 检测主节点失败（Hazelcast） +2. 选举新主节点 +3. 新主节点从 IMap 读取作业状态 +4. 重新连接到工作节点 +5. 恢复检查点协调 + +## 8. 设计考量 + +### 8.1 为什么基于管道的执行？ + +**替代方案**：单一全局 DAG 执行 + +**决策**：划分为管道 + +**好处**： +- 独立的检查点协调（较少的协调开销） +- 清晰的失败边界（一个管道失败，其他继续） +- 更容易推理数据流 +- 支持复杂的 DAG（多数据源/Sink ） + +**缺点**： +- 无法跨管道边界融合任务 +- 管道之间潜在的数据序列化 + +### 8.2 为什么使用 Hazelcast 进行协调？ + +**替代方案**：Zookeeper、etcd、自定义 Raft 实现 + +**决策**：Hazelcast IMDG + +**好处**： +- 内存分布式数据结构（低延迟） +- 内置集群管理和失败检测 +- 易于嵌入（无外部依赖） +- 熟悉的 API（Java Collections） + +**缺点**： +- 大状态的内存开销 +- 作为协调工具，不如 Zookeeper 经过充分测试 + +### 8.3 性能优化 + +**1. 任务融合**： +- 减少网络开销 +- 改善 CPU 缓存局部性 +- 降低序列化成本 + +**2. 异步检查点**： +- 检查点上传不阻塞数据处理 +- 跨任务并行检查点 + +**3. 增量检查点**： +- 仅上传更改的状态（未来增强） + +**4. 零拷贝数据传输**： +- 共存任务之间的共享内存 +- 避免不必要的序列化 + +## 9. 相关资源 + +- [架构概览](../overview.md) +- [设计理念](../design-philosophy.md) +- [检查点机制](../fault-tolerance/checkpoint-mechanism.md) +- [资源管理](resource-management.md) +- [DAG 执行](dag-execution.md) + +## 10. 参考资料 +### 进一步阅读 + +- [Hazelcast IMDG](https://docs.hazelcast.com/imdg/latest/) +- [Google Borg 论文](https://research.google/pubs/pub43438/) - 资源管理的灵感来源 +- [Apache Flink 架构](https://flink.apache.org/flink-architecture.html) diff --git a/docs/zh/architecture/engine/resource-management.md b/docs/zh/architecture/engine/resource-management.md new file mode 100644 index 000000000000..0b1c44dbd966 --- /dev/null +++ b/docs/zh/architecture/engine/resource-management.md @@ -0,0 +1,539 @@ +--- +sidebar_position: 3 +title: 资源管理 +--- + +# 资源管理 + +## 1. 概述 + +### 1.1 问题背景 + +分布式执行引擎必须高效管理计算资源: + +- **资源分配**: 如何公平高效地将任务分配给工作节点? +- **负载均衡**: 如何在工作节点之间均匀分布工作负载? +- **资源隔离**: 如何防止作业之间的资源争用? +- **动态扩缩容**: 如何在不中断作业的情况下添加/删除工作节点? +- **异构资源**: 如何处理具有不同能力的工作节点? + +### 1.2 设计目标 + +SeaTunnel 的资源管理系统旨在: + +1. **细粒度控制**: 基于槽位的分配实现精确资源管理 +2. **灵活策略**: 针对不同场景的多种分配策略 +3. **基于标签的过滤**: 将任务分配给特定的工作节点组 +4. **高可用性**: 容忍工作节点故障并自动重新分配 +5. **可观测性**: 实时跟踪资源使用和可用性 + +### 1.3 架构概览 + +``` +┌──────────────────────────────────────────────────────────────┐ +│ JobMaster │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ 请求资源 │ │ +│ │ • 计算所需槽位 │ │ +│ │ • （可选）表达资源需求（以当前引擎实现为准） │ │ +│ │ • 应用标签过滤器(可选) │ │ +│ └────────────────────────────────────────────────────┘ │ +└──────────────────────────────┬───────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ ResourceManager │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ 工作节点注册表 │ │ +│ │ • WorkerProfile (每个工作节点) │ │ +│ │ - 总资源 │ │ +│ │ - 可用资源 │ │ +│ │ - 已分配槽位 │ │ +│ │ - 未分配槽位 │ │ +│ └────────────────────────────────────────────────────┘ │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ 分配策略 │ │ +│ │ • RandomStrategy / SlotRatioStrategy / SystemLoadStrategy│ │ +│ └────────────────────────────────────────────────────┘ │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ 槽位管理 │ │ +│ │ • 分配槽位 │ │ +│ │ • 释放槽位 │ │ +│ │ • 跟踪槽位使用 │ │ +│ └────────────────────────────────────────────────────┘ │ +└──────────────────────────────┬───────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ 工作节点 │ +│ │ +│ Worker 1 Worker 2 Worker N │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Slot 1 │ │ Slot 1 │ │ Slot 1 │ │ +│ │ Slot 2 │ │ Slot 2 │ │ Slot 2 │ │ +│ │ ... │ │ ... │ │ ... │ │ +│ └──────────┘ └──────────┘ └──────────┘ │ +└──────────────────────────────────────────────────────────────┘ +``` + +## 2. 核心概念 + +### 2.1 槽位(Slot) + +**槽位**是资源分配的基本单位。 + +一个槽位通常由以下信息描述: +- **slotID**: 槽位唯一标识 +- **worker**: 槽位所在工作节点地址 +- **resourceProfile**: 槽位可提供的资源容量(CPU/内存等) + +**关键属性**: +- **粒度化**: 每个槽位可以托管一个或多个任务(任务融合) +- **类型化**: 槽位具有资源配置文件(CPU、内存) +- **有状态**: 槽位跟踪分配状态(已分配/未分配) + +**示例**: +- slotID = 1001 +- worker = worker-1:5801 +- resourceProfile = cpu.cores / heapMemory.bytes（字段以引擎实现为准） + +### 2.2 ResourceProfile + +描述资源需求或容量。 + +一个资源配置文件(ResourceProfile)通常包括: +- **cpu.cores**: CPU 核心数（当前实现为整数 core） +- **heap-memory.bytes**: JVM 堆内存（字节） + +说明：当前资源调度在很多场景下以“slot 是否可用”为主要约束；ResourceProfile 作为扩展点存在，但是否支持按 CPU/内存精细调度取决于具体版本实现。 + +**用途**: +- **任务需求**: 引擎在申请槽位时携带资源需求（当前实现常为默认/空需求，更多能力视版本而定） +- **槽位容量**: 每个槽位公布其可用资源 +- **匹配**: ResourceManager 将任务需求与槽位容量匹配 + +### 2.3 WorkerProfile + +表示工作节点的资源和槽位清单。 + +工作节点画像(WorkerProfile)通常包含: +- **address**: 工作节点地址 +- **totalResourceProfile**: 节点总资源(常由槽位资源汇总得到) +- **availableResourceProfile**: 当前可用资源 +- **assignedSlots/unassignedSlots**: 已分配/未分配槽位清单 +- **tags**: 节点标签(用于过滤、隔离、数据局部性) + +**生命周期**: +1. **注册**: 工作节点启动时向 ResourceManager 注册 +2. **心跳**: 工作节点定期发送心跳及更新的资源信息 +3. **分配**: ResourceManager 从未分配池中分配槽位 +4. **释放**: 完成的任务释放槽位,将其移回未分配池 +5. **注销**: 工作节点离开集群(优雅或故障) + +## 3. ResourceManager + +### 3.1 接口 + +ResourceManager 对外暴露的关键能力可以概括为: +- **applyResources(jobId, resourceProfiles, tagFilters)**: 为作业申请一组满足资源需求的槽位；当资源不足时返回失败(例如抛出 NoEnoughResourceException 或以失败的 Future 表达) +- **releaseResources(jobId, slots)**: 作业完成/失败后释放槽位，回收至可分配池 +- **heartbeat(workerProfile)**: 接收工作节点心跳并更新其资源/槽位信息 +- **memberRemoved(event)**: 处理成员移除事件(故障或优雅下线)，触发资源回收与作业侧重调度 + +### 3.2 实现: AbstractResourceManager + +典型实现会维护以下状态与策略: +- **registerWorker**: 已注册工作节点到 WorkerProfile 的映射(由心跳持续刷新) +- **slotAllocationStrategy**: 选择 worker 的分配策略(随机/比例/系统负载等) +- **故障检测**: 结合 worker 心跳上报与 Hazelcast 成员事件判定节点失联（具体阈值以配置/实现为准） + +申请资源的关键流程: +1. 根据 tagFilters 过滤候选工作节点 +2. 针对每个 ResourceProfile 需求，使用策略选择一个满足容量约束的未分配槽位 +3. 将槽位从“未分配池”标记为“已分配”，并同步更新 WorkerProfile +4. 返回分配结果；如任一需求无法满足，则整体失败并由 JobMaster 决定重试/降级 + +释放资源的关键流程: +1. 将 slots 标记为未分配并回收到可分配池 +2. 更新工作节点可用资源与槽位统计 + +## 4. 槽位分配策略 + +### 4.1 RandomStrategy + +随机选择具有可用槽位的工作节点。 + +核心思路: +1. 过滤出“资源满足 requiredProfile 且存在未分配槽位”的工作节点集合 +2. 在集合中随机选择一个工作节点 +3. 从该节点的未分配槽位中挑选一个满足容量约束的槽位返回 + +**优点**: +- 简单快速 +- 无协调开销 +- 适用于同构集群 + +**缺点**: +- 无负载均衡 +- 可能造成热点 + +### 4.2 SlotRatioStrategy + +优先选择可用槽位比率更高的工作节点。 + +核心思路: +1. 过滤出资源满足 requiredProfile 的工作节点 +2. 计算并选择“可用槽位比率 = unassigned / (assigned + unassigned)”最高的节点 +3. 从该节点的未分配槽位中选择一个满足容量约束的槽位 + +**优点**: +- 更好的负载均衡 +- 均匀分布任务 +- 防止工作节点过载 + +**缺点**: +- 计算稍多 +- 可能不考虑实际 CPU/内存负载 + +### 4.3 SystemLoadStrategy + +选择系统负载(CPU/内存使用)最低的工作节点。 + +核心思路: +1. 基于心跳上报的资源使用情况计算节点负载(例如 CPU/内存利用率的加权) +2. 在满足 requiredProfile 的候选节点中选择负载最低者 +3. 从该节点挑选一个满足容量约束的未分配槽位 + +负载计算的关键在于: +- 依赖指标的时效性与稳定性(过旧会导致误判，过抖会导致分配抖动) +- 需要明确权重与采样窗口，避免频繁迁移/重分配 + +**优点**: +- 考虑实际资源使用 +- 最适合异构集群 +- 优化集群利用率 + +**缺点**: +- 需要实时指标 +- 计算成本更高 +- 如果负载快速变化可能抖动 + +## 5. 基于标签的槽位过滤 + +### 5.1 用例 + +**数据局部性**: +```hocon +env { + # 作业级 worker 标签过滤（key/value 全量匹配） + tag_filter = { + zone = "us-west-1" + } +} +``` + +**资源专业化**: +```hocon +env { + tag_filter = { + resource = "gpu" + } +} +``` + +**多租户**: +```hocon +env { + job.name = "tenant-a-job" + tag_filter = { + tenant = "a" + } +} +``` + +### 5.2 TagFilter + +TagFilter 可以视为一个简单的键值匹配条件: +- key/value 需要同时匹配工作节点的 attributes（标签由集群部署侧维护） +- 多个 TagFilter 之间通常按“与(AND)”组合：任一不匹配则该节点被过滤 + +**过滤过程**: + +过滤过程通常为: +1. 枚举所有已注册工作节点 +2. 对每个节点依次校验 filters；全部匹配则保留 +3. 得到候选节点集合，交给槽位分配策略继续挑选 + +## 6. 资源分配流程 + +### 6.1 正常分配 + +```mermaid +sequenceDiagram + participant JM as JobMaster + participant RM as ResourceManager + participant Worker as Worker Node + + JM->>JM: Generate PhysicalPlan + JM->>JM: Calculate required resources + + JM->>RM: applyResources(profiles, tags) + + RM->>RM: Filter workers by tags + RM->>RM: Select workers by strategy + RM->>RM: Allocate slots + + RM-->>JM: Return SlotProfiles + + JM->>JM: Assign slots to PhysicalVertices + + loop For each task + JM->>Worker: DeployTaskOperation(task, slot) + Worker->>Worker: Execute task in slot + Worker-->>JM: ACK + end +``` + +### 6.2 资源不足 + +```mermaid +sequenceDiagram + participant JM as JobMaster + participant RM as ResourceManager + + JM->>RM: applyResources(100 slots) + + RM->>RM: Check available slots + Note over RM: Only 50 slots available + + RM-->>JM: NoEnoughResourceException + + JM->>JM: Retry with backoff + Note over JM: Wait for resources to free up + + JM->>RM: applyResources(100 slots) + RM-->>JM: Success (after resources freed) +``` + +### 6.3 资源释放 + +```mermaid +sequenceDiagram + participant Task as SeaTunnelTask + participant JM as JobMaster + participant RM as ResourceManager + + Task->>Task: Task completes/fails + + Task->>JM: Task finished + + JM->>RM: releaseResources(slots) + + RM->>RM: Mark slots as unassigned + RM->>RM: Update WorkerProfile + + Note over RM: Slots available for
new allocations +``` + +## 7. 故障处理 + +### 7.1 工作节点故障 + +**检测**: +- worker 心跳/资源上报异常或停止（阈值以配置/实现为准） +- Hazelcast 成员移除事件 + +**恢复**: + +ResourceManager 侧的典型处理步骤: +1. 从注册表中移除失联/下线的工作节点 +2. 识别该节点上“已分配”的槽位集合(即可能承载了正在运行的任务) +3. 将槽位丢失事件通知到对应的 JobMaster(或由 Coordinator 统一转发) +4. 由作业侧触发 failover：标记任务失败、从检查点恢复、重新申请新槽位并重新部署 + +**JobMaster 响应**: +1. 标记失败槽位上的任务为 FAILED +2. 从最新检查点恢复 +3. 从 ResourceManager 请求新槽位 +4. 重新部署任务 + +### 7.2 ResourceManager 故障 + +**高可用性**: +- ResourceManager 状态是无状态的(工作节点注册表从心跳重建) +- 新的 ResourceManager 实例在主节点故障转移时启动 +- 工作节点通过心跳机制重新注册 + +**恢复**: + +恢复要点: +- ResourceManager 需要能够重新建立“工作节点注册表”：工作节点通过心跳主动上报其 address、资源、槽位与标签 +- ResourceManager 需要定期清理超时心跳的节点，避免将任务分配给已失联节点 +- 由于注册表可由心跳重建，故障转移后的新实例可以在短时间内恢复资源视图(视心跳间隔与超时参数而定) + +## 8. 配置 + +### 8.1 槽位配置 + +```hocon +seatunnel { + engine { + slot-service { + # 是否启用动态槽位 + dynamic-slot = true + + # 固定槽位数（仅在 dynamic-slot = false 时生效） + slot-num = 2 + } + } +} +``` + +### 8.2 资源策略 + +```hocon +seatunnel { + engine { + slot-service { + # worker 选择策略（取值需能映射到 AllocateStrategy 枚举） + # 选项: random / slot_ratio / system_load + slot-allocate-strategy = slot_ratio + } + } +} +``` + +### 8.3 资源配置说明 + +资源相关的可配置项以 `config/seatunnel.yaml` 与当前引擎实现为准；在没有稳定对外能力前，不建议在文档中给出“每槽位 CPU/内存”等固定配置样例，避免与实际实现不一致。 + +## 9. 监控和指标 + +### 9.1 关键指标 + +**集群级别**: +- `cluster.workers.total`: 已注册工作节点总数 +- `cluster.workers.active`: 最近有心跳的工作节点 +- `cluster.slots.total`: 所有工作节点的槽位总数 +- `cluster.slots.available`: 未分配的槽位 +- `cluster.slots.assigned`: 使用中的槽位 + +**每个工作节点**: +- `worker.cpu.available`: 可用 CPU 核心 +- `worker.memory.available`: 可用内存(MB) +- `worker.slots.total`: 工作节点上的总槽位数 +- `worker.slots.assigned`: 已分配的槽位 +- `worker.heartbeat.last`: 最后一次心跳时间戳 + +**每个作业**: +- `job.slots.requested`: 作业请求的槽位数 +- `job.slots.allocated`: 成功分配的槽位数 +- `job.resource.wait_time`: 等待资源的时间 + +### 9.2 可观测性 + +**资源仪表板示例**: +``` +集群资源: + 工作节点: 10 (全部健康) + 总槽位: 20 + 可用槽位: 8 + 利用率: 60% + +资源消费者排名: + job-123: 6 个槽位 (mysql-cdc → elasticsearch) + job-456: 4 个槽位 (kafka → jdbc) + job-789: 2 个槽位 (file → s3) + +工作节点分布: + worker-1: 2/2 槽位 (100%) + worker-2: 1/2 槽位 (50%) + worker-3: 2/2 槽位 (100%) + ... +``` + +## 10. 最佳实践 + +### 10.1 槽位大小设置 + +**一般指南**: +``` +每个工作节点的槽位数 = CPU 核心数 - 1 (为操作系统保留 1 个) + +示例: + 8 核机器 → 6-7 个槽位 + 16 核机器 → 14-15 个槽位 +``` + +**每个槽位的内存**: +``` +堆内存 = 总内存 * 0.7 / 槽位数 + +示例: + 32GB 机器, 6 个槽位 + 每个槽位的堆内存 = 32GB * 0.7 / 6 ≈ 3.7GB +``` + +### 10.2 策略选择 + +**使用 RandomStrategy 当**: +- 同构集群(所有工作节点相同) +- 简单部署 +- 快速分配比完美平衡更重要 + +**使用 SlotRatioStrategy 当**: +- 需要良好的负载均衡 +- 混合作业大小 +- 中等集群规模(< 100 个工作节点) + +**使用 SystemLoadStrategy 当**: +- 异构集群 +- 工作节点具有不同的 CPU/内存 +- 优化资源利用率至关重要 + +### 10.3 标签使用 + +**数据局部性**: +```hocon +# 按区域/可用区标记工作节点（部署侧：Hazelcast member attributes，示意） +# worker-1.attributes.zone = "us-west-1a" +# worker-2.attributes.zone = "us-east-1b" + +# 将作业分配到与数据相同的区域（作业级过滤） +env { + tag_filter = { + zone = "us-west-1a" + } +} +``` + +**资源隔离**: +```hocon +# 为关键作业分配专用工作节点（部署侧 attributes，示意） +# worker-1.attributes.priority = "high" +# worker-4.attributes.priority = "normal" + +env { + job.name = "critical-job" + tag_filter = { + priority = "high" + } +} +``` + +## 11. 相关资源 + +- [引擎架构](engine-architecture.md) +- [DAG 执行](dag-execution.md) +- [架构概述](../overview.md) + +## 12. 参考资料 +### 进一步阅读 + +- [Google Borg](https://research.google/pubs/pub43438/) - 大规模集群管理 +- [Apache YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html) - Hadoop 中的资源管理 +- [Kubernetes](https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/) - 容器编排和调度 diff --git a/docs/zh/architecture/fault-tolerance/checkpoint-mechanism.md b/docs/zh/architecture/fault-tolerance/checkpoint-mechanism.md new file mode 100644 index 000000000000..3f234e8a3d38 --- /dev/null +++ b/docs/zh/architecture/fault-tolerance/checkpoint-mechanism.md @@ -0,0 +1,550 @@ +--- +sidebar_position: 1 +title: 检查点机制 +--- + +# 检查点机制 + +## 1. 概述 + +### 1.1 问题背景 + +分布式数据处理系统面临容错的关键挑战： + +- **状态丢失**：如何在失败时保留处理状态？ +- **精确一次**：如何确保每条记录被精确处理一次？ +- **分布式一致性**：如何在分布式任务之间创建一致性快照？ +- **性能**：如何在不阻塞数据处理的情况下执行检查点？ +- **恢复**：如何在失败后高效恢复状态？ + +### 1.2 设计目标 + +SeaTunnel 的检查点机制旨在： + +1. **保证精确一次语义**：一致性状态快照 + 两阶段提交 +2. **最小化开销**：尽量降低 checkpoint 对数据处理的影响（同步/异步取决于具体实现） +3. **快速恢复**：从最新成功 checkpoint 恢复（耗时取决于状态大小与存储后端） +4. **分布式协调**：协调数百个任务的检查点 +5. **可插拔存储**：支持可插拔的 checkpoint storage（具体后端取决于引擎插件与配置） + +### 1.3 理论基础 + +SeaTunnel 的检查点基于 **Chandy-Lamport 分布式快照算法**： + +**核心思想**：在数据流中插入特殊标记（屏障）。当任务收到屏障时： +1. 快照其本地状态 +2. 向下游转发屏障 +3. 继续处理 + +结果：无需暂停整个系统即可获得全局一致性快照。 + +**参考**：["Distributed Snapshots: Determining Global States of Distributed Systems"](https://lamport.azurewebsites.net/pubs/chandy.pdf)（Chandy & Lamport，1985） + +## 2. 架构设计 + +### 2.1 检查点架构 + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ JobMaster（每个作业一个，内部按 pipeline 管理） │ +│ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ CheckpointCoordinator │ │ +│ │ │ │ +│ │ • 触发检查点（定期/手动） │ │ +│ │ • 生成检查点 ID │ │ +│ │ • 跟踪待处理的检查点 │ │ +│ │ • 收集任务确认 │ │ +│ │ • 持久化完成的检查点 │ │ +│ │ • 清理旧检查点 │ │ +│ └───────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ (触发屏障) │ +│ ▼ │ +└─────────────────────────────────────────────────────────────────┘ + │ + │ (CheckpointBarrier) + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 工作节点 │ +│ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ SourceTask 1 │ │ SourceTask 2 │ │ SourceTask N │ │ +│ │ │ │ │ │ │ │ +│ │ 1. 接收 │ │ 1. 接收 │ │ 1. 接收 │ │ +│ │ 屏障 │ │ 屏障 │ │ 屏障 │ │ +│ │ 2. 快照 │ │ 2. 快照 │ │ 2. 快照 │ │ +│ │ 状态 │ │ 状态 │ │ 状态 │ │ +│ │ 3. ACK │ │ 3. ACK │ │ 3. ACK │ │ +│ │ 4. 转发 │ │ 4. 转发 │ │ 4. 转发 │ │ +│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ +│ │ │ │ │ +│ │ (屏障传播) │ │ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ Transform 1 │ │ Transform 2 │ │ Transform N │ │ +│ │ │ │ │ │ │ │ +│ │ 1. 接收 │ │ 1. 接收 │ │ 1. 接收 │ │ +│ │ 屏障 │ │ 屏障 │ │ 屏障 │ │ +│ │ 2. 快照 │ │ 2. 快照 │ │ 2. 快照 │ │ +│ │ 状态 │ │ 状态 │ │ 状态 │ │ +│ │ 3. ACK │ │ 3. ACK │ │ 3. ACK │ │ +│ │ 4. 转发 │ │ 4. 转发 │ │ 4. 转发 │ │ +│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ SinkTask 1 │ │ SinkTask 2 │ │ SinkTask N │ │ +│ │ │ │ │ │ │ │ +│ │ 1. 接收 │ │ 1. 接收 │ │ 1. 接收 │ │ +│ │ 屏障 │ │ 屏障 │ │ 屏障 │ │ +│ │ 2. 准备 │ │ 2. 准备 │ │ 2. 准备 │ │ +│ │ 提交 │ │ 提交 │ │ 提交 │ │ +│ │ 3. 快照 │ │ 3. 快照 │ │ 3. 快照 │ │ +│ │ 状态 │ │ 状态 │ │ 状态 │ │ +│ │ 4. ACK │ │ 4. ACK │ │ 4. ACK │ │ +│ └──────────────┘ └──────────────┘ └──────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ + │ + │ (收到所有 ACK) + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ CheckpointStorage │ +│ （例如 localfile/hdfs 等，取决于插件与配置） │ +│ │ +│ CompletedCheckpoint { │ +│ checkpointId: 123 │ +│ taskStates: { │ +│ SourceTask-1: { splits: [...], offsets: [...] } │ +│ SinkTask-1: { commitInfo: XidInfo(...) } │ +│ ... │ +│ } │ +│ } │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### 2.2 关键数据结构 + +#### CheckpointCoordinator + +**职责摘要**： +- 触发 checkpoint（按 interval/并发/最小间隔约束） +- 跟踪进行中的 `PendingCheckpoint`，收集各 task 的 ACK 与状态 +- 将 `CompletedCheckpoint` 持久化到 `CheckpointStorage`，并维护“最近成功 checkpoint” + +**关键字段（概念级）**： +- `checkpointIdCounter`：生成 checkpointId +- `pendingCheckpoints`：进行中的 checkpoint 集合 +- `checkpointStorage`：状态持久化后端 +- 调度参数：`checkpointInterval` / `checkpointTimeout` / `minPauseBetweenCheckpoints` + +#### PendingCheckpoint + +表示进行中的检查点。 + +**职责摘要**： +- 持有本次 checkpoint 的中间态（已 ACK/未 ACK 的 task、收集到的 action 状态与统计） +- 在全部 task ACK 后组装 `CompletedCheckpoint`（或触发失败/超时处理） + +#### CompletedCheckpoint + +持久化的检查点数据。 + +**职责摘要**： +- 表示一次成功的 checkpoint 的“可恢复快照”，可被持久化并用于作业恢复 + +**状态组织方式（概念级）**： +- 以“算子/Action + subtask”作为索引维度收集状态 +- 每个 subtask 上报一份序列化状态（可能为空，取决于算子是否有状态） + +### 2.3 CheckpointStorage + +检查点持久化的抽象。 + +**能力要求（语义级）**： +- 持久化：将一次成功 checkpoint 的快照写入外部存储 +- 读取：支持读取“最新成功 checkpoint”以及按 checkpointId 定位读取 +- 清理：支持按保留策略删除旧 checkpoint +- 一致性：写入完成前不得对外可见“半成品”，避免恢复读到不完整快照 + +**实现**： +- `LocalFileStorage`：本地文件存储（localfile 插件） +- `HdfsStorage`：基于 Hadoop FileSystem 的存储（hdfs 插件，可通过插件配置指向不同文件系统） + +## 3. 检查点流程 + +### 3.1 触发检查点 + +```mermaid +sequenceDiagram + participant Timer as 定期计时器 + participant Coord as CheckpointCoordinator + participant Plan as CheckpointPlan + + Timer->>Coord: 触发（按配置 interval） + Coord->>Coord: 生成 checkpointId（123） + + Coord->>Coord: 检查条件 + Note over Coord: • 最小暂停已过？
• 未超过最大并发？
• 先前检查点完成？ + + Coord->>Coord: 创建 PendingCheckpoint(123) + Coord->>Plan: 获取起始任务 + + loop 对每个起始任务 + Coord->>Task: 发送 CheckpointBarrierTriggerOperation(123) + end + + Coord->>Coord: 启动超时计时器（按配置 timeout） +``` + +**触发条件**： +1. 检查点间隔已过（`checkpoint.interval` 或引擎默认值） +2. 检查点之间的最小暂停已过（`min-pause` 或引擎默认值） +3. 触发时机与并发行为以当前实现为准（文档不绑定固定“最大并发 checkpoint”配置项） + +### 3.2 屏障传播 + +```mermaid +sequenceDiagram + participant Coord as 协调器 + participant Source as SourceTask + participant Transform as TransformTask + participant Sink as SinkTask + + Coord->>Source: 触发屏障(123) + + Source->>Source: 接收屏障 + Source->>Source: snapshotState() → 分片、偏移量 + Source->>Coord: ACK(state) + Source->>Transform: 转发屏障(123) + + Transform->>Transform: 接收屏障 + Transform->>Transform: snapshotState() → 转换状态 + Transform->>Coord: ACK(state) + Transform->>Sink: 转发屏障(123) + + Sink->>Sink: 接收屏障 + Sink->>Sink: prepareCommit() → commitInfo + Sink->>Sink: snapshotState() → 写入器状态 + Sink->>Coord: ACK(commitInfo + state) + + Coord->>Coord: 收到所有 ACK + Coord->>Coord: 创建 CompletedCheckpoint +``` + +**屏障流动规则**： +1. **数据 Source 源任务**：管道起点，从协调器接收屏障 +2. **转换任务**：从上游接收，快照，向下游转发 +3. **数据 Sink 任务**：管道终点，从上游接收，快照，不转发 + +**屏障对齐**（对于具有多个输入的任务）： + +当一个任务有多个上游输入时，需要在本任务处形成一致性快照边界。典型做法是： +- 先到达屏障的输入先“对齐等待”（短暂停止向下游发出该输入的后续数据） +- 直到所有输入都收到同一 checkpointId 的屏障，才触发本地状态快照，并继续处理 + +对齐带来的直接影响是：上游数据乱序/不均衡会放大等待时间，因此需要结合并行度、分区策略与 backpressure 做调优。 + +### 3.3 状态快照 + +每种任务类型快照不同的状态： + +**SourceTask**： + +- 快照内容：reader 的“分片分配 + 分片内进度（偏移量/游标/切分点）” +- 交互行为：上报 ACK（携带状态）给协调器，并向下游转发屏障以推进全局一致性边界 + +**TransformTask**： + +- 快照内容：算子状态（无状态算子通常为空状态） +- 交互行为：上报 ACK，并转发屏障 + +**SinkTask**： + +- 快照内容：writer 的内部状态（例如未刷新的 buffer、事务句柄等） +- 提交准备：在 checkpoint 边界生成“可提交但未提交”的提交信息（2PC 的 prepare 阶段） +- 交互行为：上报 ACK（携带 writer state + commitInfo），作为管道终点不再转发屏障 + +### 3.4 检查点完成 + +```mermaid +sequenceDiagram + participant Coord as CheckpointCoordinator + participant Pending as PendingCheckpoint + participant Storage as CheckpointStorage + participant Tasks as 所有任务 + + Pending->>Pending: 所有任务已 ACK + + Pending->>Coord: notifyCheckpointComplete() + + Coord->>Coord: 创建 CompletedCheckpoint + Coord->>Storage: 持久化检查点 + Storage-->>Coord: 成功 + + Note over Coord,Tasks: 持久化成功后，框架/引擎触发提交与清理回调（触发点取决于执行引擎实现） + + Coord->>Tasks: notifyCheckpointComplete(123) + Tasks->>Tasks: 清理资源 + + Coord->>Storage: 删除旧检查点 +``` + +**完成步骤**： +1. 所有任务已确认 +2. 从 `PendingCheckpoint` 创建 `CompletedCheckpoint` +3. 将检查点持久化到存储 +4. 触发数据 Sink 提交（两阶段提交） +5. 通知所有任务完成 +6. 清理旧检查点（保留最后 N 个） + +### 3.5 检查点超时 + +协调器为每个进行中的 checkpoint 启动超时计时。 + +**超时触发后的语义**： +- 将该次 checkpoint 标记为失败并清理其进行中状态 +- 作业继续运行（仍以“最近一次成功 checkpoint”作为可恢复点） +- 是否触发 failover 取决于作业容错策略与失败类型（例如连续失败、关键任务不可用等） + +**超时处理**： +- 默认超时以引擎配置为准（作业可通过 `checkpoint.timeout` 覆盖） +- 如果超时，检查点失败 +- 作业继续使用先前的检查点 +- 下一个检查点将按计划触发 + +## 4. 恢复过程 + +### 4.1 从检查点恢复 + +```mermaid +sequenceDiagram + participant JM as JobMaster + participant Storage as CheckpointStorage + participant Source as SourceTask + participant Sink as SinkTask + + JM->>Storage: getLatestCheckpoint() + Storage-->>JM: CompletedCheckpoint(123) + + JM->>JM: 按任务提取状态 + + JM->>Source: 使用 NotifyTaskRestoreOperation 部署 + activate Source + Source->>Source: restoreState(splits, offsets) + Source->>Source: 寻找到检查点偏移量 + Source-->>JM: 就绪 + deactivate Source + + JM->>Sink: 使用 NotifyTaskRestoreOperation 部署 + activate Sink + Sink->>Sink: restoreWriter(writerState) + Sink->>Sink: 恢复未提交的事务 + Sink-->>JM: 就绪 + deactivate Sink + + JM->>Source: 开始执行 + JM->>Sink: 开始执行 +``` + +**恢复步骤**： +1. JobMaster 从存储检索最新的 `CompletedCheckpoint` +2. 为每个任务提取状态（按 ActionStateKey 和 subtaskIndex） +3. 使用包含状态的 `NotifyTaskRestoreOperation` 部署任务 +4. 任务恢复状态： + - **SourceReader**：恢复分片和偏移量，寻找到位置 + - **Transform**：恢复转换状态（通常为无） + - **SinkWriter**：恢复写入器状态，可能有未提交的事务 +5. 任务转换到 READY_START 状态 +6. 作业恢复执行 + +**示例：JDBC 数据源恢复**： + +以 JDBC 为例，恢复需要满足两点： +- 能把“分片 + 进度（offset/游标）”可靠序列化到 checkpoint +- 能在恢复时把读取位置回放到该进度（例如通过主键范围、游标、时间戳或 connector 支持的 offset 语义） + +### 4.2 精确一次恢复 + +检查点恢复 + 数据 Sink 两阶段提交的组合确保精确一次： + +``` +检查点 N（已完成）： + 数据源偏移量：[100, 200, 300] + 数据 Sink 准备的提交：[XID-1, XID-2, XID-3] + 数据 Sink 提交器提交 XID-1、XID-2、XID-3 + + ↓ [失败] + +从检查点 N 恢复： + 1. 恢复数据源偏移量：[100, 200, 300] + 2. 数据源从偏移量 100、200、300 开始读取 + 3. 数据 Sink 写入器恢复状态（可能有未提交的 XID） + 4. 数据 Sink 提交器重试提交 XID（幂等） + +结果：记录 0-99、100-199、200-299 精确提交一次 + 从 100+ 开始的记录重新处理但不重复（幂等提交） +``` + +## 5. 配置和调优 + +### 5.1 检查点配置 + +```hocon +# 作业级（env）：可覆盖 interval/timeout/min-pause +env { + checkpoint.interval = 60000 + checkpoint.timeout = 600000 + min-pause = 10000 +} +``` + +引擎侧（`config/seatunnel.yaml`）配置 checkpoint storage（示意）： + +```yaml +seatunnel: + engine: + checkpoint: + storage: + type: hdfs + max-retained: 3 + plugin-config: + namespace: /tmp/seatunnel/checkpoint_snapshot +``` + +说明： +- BATCH 模式下如果作业 env 未配置 `checkpoint.interval`，当前实现会禁用 checkpoint（以源码实现为准）。 +- checkpoint storage 主要由引擎侧配置管理；作业级配置不应假设可以随意指定 storage type/path。 + +### 5.2 调优指南 + +**检查点间隔**： +- **短间隔（10-30s）**：快速恢复，但开销更高 +- **中间隔（60-120s）**：平衡（推荐） +- **长间隔（300-600s）**：低开销，但恢复较慢 + +**权衡**： +- 更短的间隔 → 更频繁的 I/O → 更高的存储成本 +- 更长的间隔 → 更少的开销 → 更长的恢复时间 + +**经验法则**：将间隔设置为可容忍的恢复时间（数据丢失窗口）。 + +**检查点超时**： +- 应该 >> 检查点间隔 +- 取决于状态大小和存储速度 +- 默认值以引擎配置为准；建议结合状态大小与存储后端能力设置 + +**并发行为**： +- 并发 checkpoint 的能力与策略以当前实现为准；架构文档不绑定固定的“最大并发 checkpoint”配置项 + +**存储选择**： +- **localfile**：仅测试/单机场景，无 HA +- **hdfs**：生产环境常用（hdfs 插件基于 Hadoop FileSystem，可通过插件配置对接不同文件系统后端） + +## 6. 性能优化 + +### 6.1 异步检查点 + +异步 checkpoint 能降低对数据处理主路径的阻塞（是否异步、异步程度取决于具体实现）： + +核心思路是把“生成快照引用/拷贝（快）”与“序列化 + 上传（慢）”解耦： +- 任务线程快速冻结一份一致性快照（或引用）后立即继续处理 +- 后台线程异步完成序列化与外部存储写入 + +这样可以降低对数据处理主路径的阻塞，但也需要关注异步积压导致的内存压力。 + +### 6.2 增量检查点（未来） + +仅检查点更改的状态： + +- 完整 checkpoint：第一次需要上传全量状态 +- 增量 checkpoint：后续只上传变化部分，并以链式/引用方式组织快照 + +**好处**： +- 减少检查点时间 +- 降低存储 I/O +- 更快的检查点完成 + +**挑战**： +- 更复杂的状态管理 +- 需要跟踪状态变化 +- 恢复需要增量链 + +### 6.3 本地状态后端（未来） + +在本地存储热状态，仅检查点摘要： + +典型做法是把热状态存到本地（例如 RocksDB），checkpoint 时只上传“可恢复的快照引用/元数据”，从而降低远端存储压力。 + +## 7. 最佳实践 + +### 7.1 状态大小优化 + +**1. 保持状态小**： + +- 避免把“可重放的数据本身”放进状态（会放大 checkpoint 体积与时延） +- 只保存“可定位读取位置”的最小信息（offset/游标/分片进度），把数据重放交给上游存储或 connector 的读取语义 + +**2. 使用高效的序列化**： +- 优先使用 Protobuf、Kryo 而不是 Java 序列化 +- 压缩大状态（gzip、snappy） + +### 7.2 监控 + +**关键指标（示例，名称以实际 metrics 实现为准）**： +- checkpoint_duration：从触发到完成的时间 +- checkpoint_size：持久化检查点的大小 +- checkpoint_failure_rate：失败检查点的比例 +- checkpoint_alignment_duration：屏障对齐所花费的时间 + +**告警**： +- 告警阈值需结合业务可接受的恢复窗口与存储后端能力制定 +- 如果在 2x 间隔内没有完成检查点则告警 + +### 7.3 故障排除 + +**问题**：检查点超时 + +**可能原因**： +1. 任务卡住（数据处理缓慢） +2. 大状态（序列化/上传缓慢） +3. 慢速存储（网络/磁盘 I/O） +4. 屏障对齐缓慢（数据倾斜） + +**解决方案**： +- 增加检查点超时 +- 优化状态大小 +- 使用更快的存储 +- 调整并行度 + +**问题**：高检查点开销 + +**可能原因**： +1. 检查点间隔太短 +2. 大状态大小 +3. 慢速存储 + +**解决方案**： +- 增加检查点间隔 +- 优化状态大小 +- 启用增量检查点（可用时） + +## 8. 相关资源 + +- [架构概览](../overview.md) +- [设计理念](../design-philosophy.md) +- [引擎架构](../engine/engine-architecture.md) +- [数据 Sink 架构](../api-design/sink-architecture.md) +- [精确一次语义](exactly-once.md) + +## 9. 参考资料 + +### 学术论文 + +- Chandy, K. M., & Lamport, L. (1985). ["Distributed Snapshots: Determining Global States of Distributed Systems"](https://lamport.azurewebsites.net/pubs/chandy.pdf) +- Carbone, P., et al. (2017). ["State Management in Apache Flink"](http://www.vldb.org/pvldb/vol10/p1718-carbone.pdf) + +### 进一步阅读 + +- [Apache Flink 检查点](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/fault-tolerance/checkpointing/) +- [Spark 结构化流检查点](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing) diff --git a/docs/zh/architecture/fault-tolerance/exactly-once.md b/docs/zh/architecture/fault-tolerance/exactly-once.md new file mode 100644 index 000000000000..2023fe73ad98 --- /dev/null +++ b/docs/zh/architecture/fault-tolerance/exactly-once.md @@ -0,0 +1,504 @@ +--- +sidebar_position: 2 +title: 精确一次语义 +--- + +# 精确一次语义 + +## 1. 概述 + +### 1.1 问题背景 + +分布式数据处理面临基本的交付保证挑战: + +- **至多一次**: 记录可能丢失(对关键数据不可接受) +- **至少一次**: 记录可能重复(导致计数错误、重复收费) +- **精确一次**: 每条记录恰好处理一次(理想但复杂) + +**实际影响**: +``` +场景: 金融交易处理 + +至少一次: + 交易 $100 处理两次 → 用户被收费 $200 ❌ + +精确一次: + 交易 $100 处理一次 → 用户被收费 $100 ✅ +``` + +### 1.2 设计目标 + +SeaTunnel 的精确一次语义旨在: + +1. **端到端语义**: 在启用 checkpoint 且外部系统支持事务/幂等提交等前提下，尽量提供可验证的一致性语义（避免丢失或重复可见） +2. **透明实现**: 框架处理复杂性,用户最少配置 +3. **性能效率**: 在维护保证的同时最小化开销 +4. **故障弹性**: 在任务/工作节点/主节点故障时维护保证 +5. **广泛适用性**: 支持事务型和非事务型目标端 + +### 1.3 一致性级别 + +| 级别 | 保证 | 用例 | 实现 | +|------|------|------|------| +| **至多一次** | 无重复,可能丢失 | 非关键日志 | 无重试 | +| **至少一次** | 无丢失,可能重复 | 幂等处理 | 重试但无事务 | +| **精确一次** | 无丢失,无重复 | 金融、计费、审计 | 检查点 + 两阶段提交 | + +## 2. 理论基础 + +### 2.1 Chandy-Lamport 算法 + +**概念**: 无需停止整个系统的分布式快照。 + +**机制**: +1. 协调器向数据流注入**屏障**(标记) +2. 收到屏障后,每个算子: + - 快照其本地状态 + - 将屏障转发到下游 +3. 当所有算子都完成快照时,我们有一个**一致的全局快照** + +**关键属性**: 快照表示跨分布式系统状态的一致切割。 + +### 2.2 两阶段提交协议 + +**概念**: 跨分布式参与者的原子提交。 + +**阶段**: +1. **准备阶段**: 所有参与者准备(尚无副作用) +2. **提交阶段**: 协调器决定提交/中止,所有参与者执行 + +**在 SeaTunnel 中**: +- **准备**: 检查点期间的 `SinkWriter.prepareCommit(...)` +- **提交**: 检查点完成后的 `SinkCommitter.commit()` + +## 3. 精确一次架构 + +### 3.1 端到端流水线 + +``` +┌──────────────────────────────────────────────────────────────┐ +│ 数据源 │ +│ • 从外部系统读取 │ +│ • 跟踪偏移量/位置 │ +│ • 在检查点中快照偏移量 │ +└──────────────────────────────┬───────────────────────────────┘ + │ + ▼ 检查点屏障 +┌──────────────────────────────────────────────────────────────┐ +│ 转换器 │ +│ • 处理记录 │ +│ • 快照转换器状态(如果有) │ +└──────────────────────────────┬───────────────────────────────┘ + │ + ▼ 检查点屏障 +┌──────────────────────────────────────────────────────────────┐ +│ 目标端写入器 │ +│ • 缓冲写入 │ +│ • prepareCommit(checkpointId) → 生成 CommitInfo (阶段 1) │ +│ • 快照写入器状态 │ +└──────────────────────────────┬───────────────────────────────┘ + │ + │ CommitInfo + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ CheckpointCoordinator │ +│ • 收集所有 CommitInfos │ +│ • 持久化 CompletedCheckpoint │ +│ • 触发提交/回调（触发点取决于执行引擎实现） │ +└──────────────────────────────┬───────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ 目标端提交器 │ +│ • commit(CommitInfos) → 应用变更 (阶段 2) │ +│ • 必须是幂等的 │ +└──────────────────────────────┬───────────────────────────────┘ + │ + ▼ + 外部目标端 + (变更可见) +``` + +### 3.2 关键组件 + +**数据 Source 源偏移量管理**: + +Source 侧要想参与端到端精确一次，通常需要满足: +- **可追踪进度**: 读取过程持续维护“已处理到哪里”(如 Kafka offset、文件 position、CDC LSN 等) +- **可快照**: 在 checkpoint 时将进度写入状态后端(属于检查点状态的一部分) +- **可提交/可确认**: 在 checkpoint 成功后再将进度提交到外部系统(例如提交 offset) +- **幂等提交**: 由于重试、故障转移可能触发重复提交，提交动作必须可重放且结果一致 + +**目标端两阶段提交**: + +Sink 侧两阶段提交(2PC)的语义拆分: +- **Writer(阶段 1 / prepare)** + - 将写入先落到“暂不可见”的位置(事务缓冲、临时文件、暂存表/分区等) + - 在 barrier 到达时执行 prepare：封存本轮写入，并产出 CommitInfo(例如事务 ID、临时路径、批次号) + - 将 CommitInfo 上报给协调器并随 CompletedCheckpoint 一起持久化 +- **Committer(阶段 2 / commit)** + - 仅在 checkpoint 完成后运行 commit(CommitInfos)，使外部副作用“变得可见”(提交事务、原子重命名、发布 batch) + - **必须幂等**：重复提交同一 CommitInfo 不能产生重复数据；典型做法是利用外部系统的事务 ID / 唯一键 / 幂等 API + +## 4. 实现模式 + +### 4.1 事务型目标端(XA) + +**典型场景**: 支持 XA/2PC 的事务型数据库等 + +**实现**: + +实现要点: +- Writer 使用 XA/事务能力将写入暂存于事务中 +- 在 prepareCommit 阶段产出可被提交器识别的事务标识(CommitInfo) +- Committer 在 checkpoint 完成后提交事务，并对重复 commit 做幂等处理 + +**优点**: +- 强一致性保证 +- 失败时自动回滚 + +**缺点**: +- 需要数据库 XA 支持 +- 更高延迟(2PC 开销) +- 准备阶段期间锁争用 + +### 4.2 幂等目标端(Upsert) + +**典型场景**: 支持 upsert/merge 或自然幂等写入的目标端（例如按主键覆盖写入的存储） + +**实现**: + +实现要点: +- 为每条记录选择一个确定性的幂等键(通常来自主键/业务唯一键) +- 外部系统使用“按键覆盖/更新”(Upsert)语义：同一幂等键多次写入，最终只保留一个结果 +- prepareCommit 只需要保证批次边界(例如 flush 缓冲)，不一定需要单独的 commit 阶段 + +**关键**: 相同主键 → 相同文档 → 幂等更新 + +**优点**: +- 无事务开销 +- 更低延迟 + +**缺点**: +- 需要唯一键 +- 无法处理复杂事务 + +### 4.3 基于日志的目标端(Kafka) + +**实现**: + +实现要点: +- 使用 Kafka 事务能力将一个 checkpoint 边界内的写入纳入同一个事务 +- prepareCommit 阶段完成 flush 并产出事务标识(CommitInfo) +- commit 阶段提交事务，使消息对下游消费者可见 +- 对故障恢复时的重复提交，需要依赖 Kafka 事务/幂等机制保证不会产生重复可见结果 + +### 4.4 文件目标端(原子重命名) + +**实现**: + +实现要点: +- Writer 将数据写入临时路径/临时文件(对外不可见) +- prepareCommit 阶段封存临时文件并产出 CommitInfo(临时路径 + 目标路径) +- Committer 只做“原子可见化”动作(例如原子重命名/原子移动) +- 需要确认底层文件系统对 rename/move 的原子性语义；在对象存储上往往需要额外设计(否则不能直接宣称精确一次) + +**关键**: 原子重命名确保文件要么完全可见要么不可见。 + +## 5. 故障场景和恢复 + +### 5.1 检查点前任务故障 + +``` +时间线: + t0: 检查点 N 完成 + t1: 处理记录 [1000-2000] + t2: 任务失败 ❌ + t3: 从检查点 N 恢复 + t4: 重新处理记录 [1000-2000] + +结果: + ✅ 无数据丢失(记录重新处理) + ✅ 无重复(故障前未提交任何内容) +``` + +### 5.2 prepareCommit 后任务故障 + +``` +时间线: + t0: 检查点 N 进行中 + t1: SinkWriter.prepareCommit(...) → XID-123 已准备 + t2: 任务失败 ❌ (提交前) + t3: 从检查点 N-1 恢复 + t4: 重新处理记录 + t5: 新的 prepareCommit(...) → XID-124 已准备 + t6: 提交器提交 XID-124 + +结果: + ✅ XID-123 从未提交(超时后自动回滚) + ✅ XID-124 已提交(正确数据) +``` + +### 5.3 提交期间提交器故障 + +``` +时间线: + t0: 检查点 N 完成 + t1: 提交器开始提交 [XID-100, XID-101, XID-102] + t2: 提交 XID-100 ✅ + t3: 提交器失败 ❌ (XID-101, XID-102 未提交) + t4: 新提交器重试 [XID-100, XID-101, XID-102] + t5: 提交 XID-100 (已提交,幂等) ✅ + t6: 提交 XID-101 ✅ + t7: 提交 XID-102 ✅ + +结果: + ✅ 所有 XID 最终提交 + ✅ 无重复(幂等提交) +``` + +### 5.4 网络分区 + +``` +时间线: + t0: SinkWriter 准备 XID-200 + t1: 检查点完成 + t2: 提交器发送 commit(XID-200) + t3: 网络分区 ⚠️ (提交成功,但 ACK 丢失) + t4: 提交器重试 commit(XID-200) + t5: XID-200 已提交(幂等) + +结果: + ✅ 数据恰好提交一次 + ✅ 幂等性防止重复 +``` + +## 6. 幂等性要求 + +### 6.1 为什么幂等性很重要 + +**问题**: 网络故障、重试和故障转移可能导致重复的提交尝试。 + +**解决方案**: 提交器操作必须是幂等的。 + +典型对比: +- **非幂等提交**: 重试一次就会额外插入一份数据(产生重复) +- **幂等提交**: 重试多次与提交一次效果一致(例如使用唯一键约束/Upsert/事务 ID 去重) + +### 6.2 实现幂等性 + +**策略 1: 检查后执行** + +要点: +- 提交前先查询“该 CommitInfo 是否已完成提交”(通过事务表、元数据表、外部系统 API) +- 已提交则直接返回成功；未提交则提交并记录结果 + +**策略 2: 数据库级幂等性** + +要点: +- 使用唯一约束/唯一索引来承载“去重键”(事务 ID / 批次 ID / checkpointId) +- 将“写入去重标记”和“应用外部副作用”放在同一事务或同一原子语义内，避免部分成功导致的不一致 + +**策略 3: 自然幂等性(XA)** + +要点: +- 依赖 XA 协议本身对重复 commit 的处理语义 +- 对“已提交/不存在”的错误码进行兼容处理，将其视为幂等成功 + +## 7. 性能考虑 + +### 7.1 检查点间隔权衡 + +``` +短间隔(10-30s): + ✅ 快速恢复(重新处理更少) + ❌ 更高开销(频繁快照) + ❌ 更多提交操作 + +长间隔(5-10分钟): + ✅ 更低开销(快照更少) + ❌ 恢复更慢(重新处理更多) + ✅ 更少提交操作 +``` + +**建议**: 大多数工作负载 60-120 秒 + +### 7.2 批量大小优化 + +优化思路: +- 使用批量写入将外部系统交互的固定开销摊薄(例如每 1000 条 flush 一次) +- 批量过大可能增加延迟与内存占用；批量过小会增加外部 I/O 次数 + +**影响**: 1000x 批量 → ~10x 吞吐量提升 + +### 7.3 异步检查点 + +优化思路: +- 在 barrier 到达时尽快做“轻量快照”(例如复制状态引用/增量快照元数据) +- 将序列化与上传等重 I/O 工作放到异步线程执行，减少对主处理线程的阻塞 +- 需要权衡：异步快照会增加内存峰值(需要暂存 snapshot)，并要求正确处理并发可见性 + +**影响**: 快照上传时数据处理继续 + +## 8. 配置 + +### 8.1 启用精确一次 + +```hocon +env { + # 检查点配置 + checkpoint.interval = 60000 # 60 秒 + checkpoint.timeout = 600000 # 10 分钟 + + # 精确一次模式(vs 至少一次) + # 使用事务型目标端时这是隐式的 +} +``` + +### 8.2 数据源配置 + +**Kafka**: +```hocon +source { + Kafka { + bootstrap.servers = "localhost:9092" + topic = "my_topic" + + # Kafka 消费者偏移量提交 + commit_on_checkpoint = true # 检查点后提交偏移量 + } +} +``` + +**JDBC**: +```hocon +source { + JDBC { + url = "jdbc:mysql://..." + + # 基于查询的数据源(幂等重新处理) + query = "SELECT * FROM table WHERE id >= ? AND id < ?" + } +} +``` + +### 8.3 目标端配置 + +**JDBC (XA)**: +```hocon +sink { + JDBC { + url = "jdbc:mysql://..." + + # 启用 XA 事务 + xa_data_source_class_name = "com.mysql.cj.jdbc.MysqlXADataSource" + is_exactly_once = true + } +} +``` + +**Kafka (事务)**: +```hocon +sink { + Kafka { + bootstrap.servers = "localhost:9092" + topic = "output_topic" + + # Kafka 事务 + transaction.id = "seatunnel-kafka-sink" + enable.idempotence = true + } +} +``` + +## 9. 测试精确一次 + +### 9.1 功能测试 + +建议的功能测试步骤: +1. 向数据源注入固定集合的记录(可重复、可计数、最好带主键) +2. 触发/等待至少一个 checkpoint 完成 +3. 在关键窗口注入故障(例如 prepareCommit 之后、commit 之前；或 barrier 对齐期间) +4. 恢复后继续运行并结束作业 +5. 验证输出端：输入计数 = 输出计数，且基于主键/去重键无重复 + +### 9.2 混沌测试 + +建议的混沌测试维度: +- 随机杀任务/杀 worker/重启 master +- 注入网络延迟、短暂网络分区、外部存储抖动 +- 暂停/延迟 checkpoint 触发，模拟对齐与上传压力 + +验收标准: +- 输入计数与输出计数一致 +- 输出端无重复(主键/去重键唯一) +- 对关键失败窗口(prepareCommit/commit)覆盖到位 + +### 9.3 监控验证 + +``` +要跟踪的指标: + +source.records_read = 1,000,000 +sink.records_written = 1,000,000 +sink.records_committed = 1,000,000 + +✅ 所有计数匹配 → 精确一次验证 +``` + +## 10. 最佳实践 + +### 10.1 选择适当的目标端 + +**使用事务型目标端(XA)用于**: +- 金融交易 +- 计费系统 +- 审计日志 +- 关键数据 + +**使用幂等目标端用于**: +- 高吞吐量场景 +- 可接受最终一致性 +- 无事务支持 + +### 10.2 处理有毒记录 + +处理建议: +- 明确“有毒记录”的判定范围(格式错误/约束冲突/不可恢复的业务异常) +- 选择策略：写入死信队列(DLQ)并告警、跳过并计数、或触发失败(强一致场景) +- 与精确一次语义的关系：跳过会破坏端到端“无丢失”，但可能是可接受的业务权衡；需在文档/配置中显式声明 + +### 10.3 监控检查点健康 + +**关键指标**: +- `checkpoint.duration`: 应 < 间隔的 10% +- `checkpoint.failure_rate`: 应 < 1% +- `checkpoint.size`: 监控随时间增长 + +**警报**: +``` +如果 checkpoint.duration > 300s 则告警 +如果 checkpoint.failure_rate > 5% 则告警 +如果在 2x 间隔内无检查点则告警 +``` + +## 11. 相关资源 + +- [检查点机制](checkpoint-mechanism.md) +- [目标端架构](../api-design/sink-architecture.md) +- [数据源架构](../api-design/source-architecture.md) +- [引擎架构](../engine/engine-architecture.md) + +## 12. 参考资料 + +### 学术论文 + +- Chandy & Lamport (1985): ["Distributed Snapshots"](https://lamport.azurewebsites.net/pubs/chandy.pdf) +- Gray & Lamport (2006): ["Consensus on Transaction Commit"](https://lamport.azurewebsites.net/pubs/paxos-commit.pdf) +- Carbone et al. (2017): ["State Management in Apache Flink"](http://www.vldb.org/pvldb/vol10/p1718-carbone.pdf) + +### 进一步阅读 + +- [两阶段提交协议](https://en.wikipedia.org/wiki/Two-phase_commit_protocol) +- [XA 事务](https://pubs.opengroup.org/onlinepubs/009680699/toc.pdf) +- [Kafka 精确一次](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/) diff --git a/docs/zh/architecture/features/multi-table.md b/docs/zh/architecture/features/multi-table.md new file mode 100644 index 000000000000..0b40824eced4 --- /dev/null +++ b/docs/zh/architecture/features/multi-table.md @@ -0,0 +1,434 @@ +--- +sidebar_position: 3 +title: 多表同步 +--- + +# 多表同步架构 + +## 1. 概述 + +### 1.1 问题背景 + +数据库迁移和 CDC 场景通常需要同步数百张表: + +- **资源效率**: 如何避免为每张表创建一个作业? +- **一致快照**: 如何确保所有表从同一时间点开始? +- **模式路由**: 如何将数据路由到正确的目标表? +- **独立模式**: 如何处理每张表的不同模式? +- **并行写入**: 如何最大化多表的吞吐量? + +### 1.2 设计目标 + +SeaTunnel 的多表同步旨在: + +1. **单作业,多表**: 在一个作业中同步数百张表 +2. **资源效率**: 跨表共享资源 +3. **模式独立**: 每张表维护自己的模式 +4. **动态路由**: 根据表标识将记录路由到正确的目标端 +5. **水平扩展**: 支持副本写入器以实现高吞吐量 + +### 1.3 用例 + +**数据库迁移**: +```hocon +source { + MySQL-CDC { + # 捕获数据库中的所有表 + database-name = "my_db" + table-name = ".*" # 正则表达式: 所有表 + } +} + +sink { + Jdbc { + # 写入 PostgreSQL + url = "jdbc:postgresql://..." + } +} +``` + +**多表 CDC**: +```hocon +source { + MySQL-CDC { + table-name = "order_.*|user_.*|product_.*" # 多个表模式 + } +} + +sink { + Elasticsearch { + # 每张表对应不同的索引 + } +} +``` + +## 2. 核心抽象 + +### 2.1 TablePath + +用于将记录路由到表的唯一标识符。 + +TablePath 由三段信息组成: +- **databaseName**: 数据库名 +- **schemaName**: schema 名(对无 schema 的系统可为空或使用默认值) +- **tableName**: 表名 + +它需要满足两个要求: +- **可稳定序列化**: 能被序列化为唯一字符串(例如 `db.schema.table`)并在链路上传播 +- **可逆**: 能从字符串/结构化字段反解析回 TablePath + +**示例**: + +- my_db.public.orders +- my_db.public.users + +### 2.2 SeaTunnelRow 带 TableId + +记录携带表标识用于路由。 + +多表场景中，一条记录除了字段本身，还必须携带: +- **tableId**: 表标识(通常是 TablePath 的序列化形式) +- **rowKind**: 变更类型(INSERT/UPDATE/DELETE 等) + +路由侧通过 tableId 还原出 TablePath，再决定写入到哪个目标表/索引。 + +### 2.3 SinkIdentifier + +目标端写入器的唯一标识符(表 + 副本索引)。 + +SinkIdentifier 的作用是把“写入目标”精确到: +- **表标识**: TablePath/TableIdentifier +- **副本索引**: index(用于同一张表的多 writer 副本并行写入) + +示例: +- (orders, 0), (orders, 1) +- (users, 0), (users, 1) + +## 3. MultiTableSource 架构 + +多表 Source 的具体实现取决于 connector（例如 CDC connector 往往以“库/表”为维度产出变更）。 + +为了让下游能按表路由，核心要求是： +- 输出的每条 `SeaTunnelRow` 必须携带 `tableId`（通常为 `TablePath` 的序列化字符串） +- 变更流场景还需要携带 `rowKind`（INSERT/UPDATE/DELETE 等），便于下游做正确语义处理 + +至于“内部是否维护 TablePath→Reader/Enumerator 映射、如何做多表公平调度、是否共享底层连接”等，属于 connector 自身的实现选择，文档不做强绑定描述。 + +## 4. MultiTableSink 架构 + +### 4.1 结构 + +MultiTableSink 是一个“按表路由 + 可多副本并行写入”的 Sink: +- 内部维护 **TablePath → SeaTunnelSink** 的映射(每张表一个底层 sink) +- 通过 **replicaNum** 为每张表创建多个 writer 副本以提升写入吞吐 +- 依赖 catalogTables 提供各表 schema 信息(用于写入/类型转换/DDL 处理) +- 运行时要求底层 `SinkWriter` 支持多表能力（例如实现 `SupportMultiTableSinkWriter`），以提供主键路由信息与多表资源管理能力；不满足该能力的 sink 不适用于 `MultiTableSink` + +### 4.2 写入器: 带副本的多表写入 + +写入器的关键流程: +1. 从输入记录中解析 TablePath(tableId) +2. 为该表选择一个 writer 副本(replicaIndex) +3. 路由到 (TablePath, replicaIndex) 对应的底层 writer 执行写入 + +副本选择需要兼顾两类诉求: +- **顺序性/一致落点**: 对同一主键（或唯一键）相关的记录尽量路由到同一副本，降低乱序与写入冲突风险 +- **吞吐量**: 在不破坏顺序性要求的前提下，尽量分散写入压力 + +在当前 MultiTableSinkWriter 的实现中，副本选择主要依据“主键信息是否可用”： +- 有主键：对主键字段做哈希，稳定映射到某个副本 +- 无主键：使用随机策略在副本间分配 + +这意味着“是否按 rowKind（INSERT/UPDATE/DELETE）切换策略”不是该实现的默认行为；如果需要按 rowKind 细分策略，应以 connector/实现代码为准。 + +在 checkpoint 边界: +- prepareCommit: 汇总所有表/所有副本的 CommitInfo，并打包为多表级提交信息 +- snapshotState: 快照所有 writer 状态；恢复时必须能通过 SinkIdentifier 将状态路由回正确的(表,副本) + +### 4.3 提交器: 多表提交协调 + +提交器的核心责任是把多表提交信息“拆回每张表”，并委托给对应表的底层 committer: +1. 解析 commitInfos，将其按 TablePath 分组 +2. 对每个表调用对应的 SinkCommitter.commit(tableCommitInfos) +3. 汇总失败列表并按框架约定触发重试/回滚 + +注意事项: +- commit 必须幂等(可能被重试) +- 单表提交失败的处理策略需要明确：是整体失败(保守)还是允许部分表推进(取决于端到端一致性要求) + - abort/回滚相关的触发点与语义在不同执行引擎中可能不同，不能在文档层面假设一定会对每个子 sink 执行 abort；务必保证整体可重试、commit 幂等 + +## 5. 副本机制 + +### 5.1 为什么需要副本? + +**问题**: 每张表的单个写入器成为高吞吐量表的瓶颈。 + +**解决方案**: 每张表多个副本写入器用于并行写入。 + +``` +无副本: + orders 表(1000 写入/秒) → [单个写入器] → 瓶颈 + +有副本(replicaNum=4): + orders 表(1000 写入/秒) → [写入器 0] (250 写入/秒) + → [写入器 1] (250 写入/秒) + → [写入器 2] (250 写入/秒) + → [写入器 3] (250 写入/秒) +``` + +### 5.2 副本配置 + +```hocon +sink { + Jdbc { + url = "..." + + # 多表配置 + multi_table_sink_replica = 4 # 写入器副本数（对所有表生效） + } +} +``` + +### 5.3 副本选择策略 + +**基于主键哈希（稳定路由）**: + +要点: +- 以主键（或业务唯一键）做哈希，将同一键稳定映射到同一副本 +- 典型映射: $replica = hash(pk) \bmod replicaNum$ + +**随机（无主键兜底）**: + +要点: +- 当记录缺少主键字段信息时，无法提供稳定落点 +- 使用随机分配在副本间扩散压力，但不保证同一键的顺序性 + +## 6. 多表中的模式管理 + +### 6.1 独立模式 + + +每张表维护自己的 CatalogTable/Schema: +- 运行时根据 TablePath 查询对应的 schema，用于类型转换与写入 +- 不同表之间 schema 互不影响，避免“全局 schema”导致的兼容性冲突 + +### 6.2 模式演化路由 + +模式演化需要被路由到“正确的表”，并应用到该表的所有 writer 副本: +1. 从 SchemaChangeEvent 中解析出 TablePath +2. 选择该表对应的 schema/元数据更新逻辑 +3. 将变更广播到该表的所有副本 writer，保证后续写入使用一致的 schema + +## 7. 数据流示例 + +### 7.1 完整流水线 + +``` +┌──────────────────────────────────────────────────────────────┐ +│ MySQL CDC 数据源 │ +│ • 从 100 张表捕获变更 │ +│ • 用 TablePath 标记每行 │ +└──────────────────────────────┬───────────────────────────────┘ + │ + ▼ + ┌─────────────────────────────────────┐ + │ SeaTunnelRow (带 TablePath) │ + │ tableId: "my_db.public.orders" │ + │ fields: [1, "order-001", 99.99] │ + └─────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ MultiTableSinkWriter │ +│ • 从行中提取 TablePath │ +│ • 选择副本（按主键哈希或随机） │ +│ • 路由到正确的写入器 │ +└──────────────────────────────┬───────────────────────────────┘ + │ + ┌──────────────────┼──────────────────┐ + ▼ ▼ ▼ +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ orders │ │ users │ │ products │ +│ 写入器 0 │ │ 写入器 0 │ │ 写入器 0 │ +│ 写入器 1 │ │ 写入器 1 │ │ 写入器 1 │ +│ 写入器 2 │ │ │ │ │ +│ 写入器 3 │ │ │ │ │ +└──────────────┘ └──────────────┘ └──────────────┘ + │ │ │ + ▼ ▼ ▼ +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ PostgreSQL │ │ PostgreSQL │ │ PostgreSQL │ +│ orders │ │ users │ │ products │ +└──────────────┘ └──────────────┘ └──────────────┘ +``` + +### 7.2 写入流程 + +```mermaid +sequenceDiagram + participant Source as MySQL CDC + participant Writer as MultiTableSinkWriter + participant OrderWriter as Order 写入器 (副本 0) + participant UserWriter as User 写入器 (副本 0) + participant PG as PostgreSQL + + Source->>Writer: Row(tableId="orders", data=[...]) + Writer->>Writer: Extract TablePath("orders") + Writer->>Writer: Select replica (pk-hash / random) → 0 + Writer->>OrderWriter: write(row) + OrderWriter->>PG: write + + Source->>Writer: Row(tableId="users", data=[...]) + Writer->>Writer: Extract TablePath("users") + Writer->>Writer: Select replica (pk-hash / random) → 0 + Writer->>UserWriter: write(row) + UserWriter->>PG: write +``` + +### 7.3 检查点流程 + +```mermaid +sequenceDiagram + participant CP as CheckpointCoordinator + participant Writer as MultiTableSinkWriter + participant W1 as Order 写入器 0 + participant W2 as Order 写入器 1 + participant W3 as User 写入器 0 + + CP->>Writer: triggerBarrier(checkpointId) + + Writer->>W1: prepareCommit() + W1-->>Writer: CommitInfo(orders, replica=0) + + Writer->>W2: prepareCommit() + W2-->>Writer: CommitInfo(orders, replica=1) + + Writer->>W3: prepareCommit() + W3-->>Writer: CommitInfo(users, replica=0) + + Writer->>CP: ACK([CommitInfo1, CommitInfo2, CommitInfo3]) +``` + +## 8. 性能优化 + +### 8.1 副本大小设置 + +**经验法则**: +``` +replicaNum = ceil(表写入速率 / 单个写入器吞吐量) + +示例: + orders: 10,000 写入/秒 + 单个写入器: 2,500 写入/秒 + replicaNum = ceil(10,000 / 2,500) = 4 +``` + +### 8.2 表特定副本 + +优化思路: +- 不同表的写入速率差异很大时，理想情况下应允许按表配置不同的副本数 +- 但在当前实现中，`multi_table_sink_replica` 是对所有表生效的全局配置；如果需要“按表覆盖”，需要 connector/框架层提供额外能力 + +### 8.3 批量写入 + +优化思路: +- 为每个 (TablePath, replicaIndex) 维护独立缓冲区，避免不同表/不同副本相互干扰 +- 达到 batch-size 或超时阈值时触发 flush，将外部系统交互开销摊薄 +- 需要关注内存上限：多表 × 多副本 × 批次缓存会放大峰值占用 + +## 9. 监控和可观测性 + +### 9.1 关键指标 + +多表场景下建议至少具备以下维度的可观测性（具体指标命名以 connector/引擎实现为准）： + +- 按 `tableId` 维度的写入条数/字节数/延迟 +- 按（表，副本）维度的写入分布与队列堆积情况（用于判断是否存在热点） +- 全局维度的表数量、writer 数量、整体吞吐与失败重试次数 + +### 9.2 监控仪表板 + +``` +多表作业: mysql-to-postgres + +表: 100 +写入器: 250 (平均每张表 2.5 个副本) +吞吐量: 50,000 记录/秒 + +按吞吐量排名的表: + 1. orders: 15,000 记录/秒 (4 个副本) + 2. events: 10,000 记录/秒 (4 个副本) + 3. users: 5,000 记录/秒 (2 个副本) + ... + +副本分布: + orders: + 副本 0: 3,750 记录/秒 (25%) + 副本 1: 3,800 记录/秒 (25.3%) + 副本 2: 3,700 记录/秒 (24.7%) + 副本 3: 3,750 记录/秒 (25%) +``` + +## 10. 最佳实践 + +### 10.1 表选择 + +**使用正则表达式模式**: +```hocon +source { + MySQL-CDC { + # 包含特定模式 + table-name = "order_.*|user_.*" + } +} +``` + +### 10.2 副本配置 + +**保守开始**: +```hocon +sink { + Jdbc { + # 从 1 个副本开始,如果出现瓶颈则增加 + multi_table_sink_replica = 1 + } +} +``` + +**监控和调优**: + +如果单副本写入成为瓶颈（例如写入延迟持续升高、队列堆积明显），可逐步增加 `multi_table_sink_replica` 并结合目标端能力评估收益。 + +### 10.3 模式管理 + +**预创建目标表**: +```sql +-- 更好: 预创建所有目标表 +CREATE TABLE orders (...); +CREATE TABLE users (...); +CREATE TABLE products (...); +``` + +**谨慎启用自动创建**: +```hocon +sink { + Jdbc { + # 作业启动阶段：若表不存在则创建（用于首次建表） + schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST" + + # 说明：运行时 schema 变更由 CDC source 的 `schema-changes.enabled` 控制； + # 是否能自动应用新增/删除列等变更取决于 JDBC 方言与目标端能力。 + } +} +``` + +## 13. 相关资源 + +- [CatalogTable 和元数据](../api-design/catalog-table.md) +- [目标端架构](../api-design/sink-architecture.md) +- [DAG 执行](../engine/dag-execution.md) +- [模式演化](../../introduction/concepts/schema-evolution.md) + +## 14. 参考资料 + +如需进一步了解 Schema、Sink 语义与 DAG 执行，请从“相关资源”章节继续阅读。 diff --git a/docs/zh/architecture/overview.md b/docs/zh/architecture/overview.md new file mode 100644 index 000000000000..44a4bbc8cbbc --- /dev/null +++ b/docs/zh/architecture/overview.md @@ -0,0 +1,435 @@ +--- +sidebar_position: 1 +title: 架构概览 +--- + +# SeaTunnel 架构概览 + +## 1. 简介 + +### 1.1 设计目标 + +SeaTunnel 设计为分布式数据集成平台，具有以下核心目标： + +- **引擎独立性**：将连接器逻辑尽量与执行引擎解耦；连接器可通过转换层适配到不同引擎，具体可用性以连接器能力与引擎支持为准 +- **高性能**：支持高吞吐、低延迟的大规模数据同步 +- **容错性**：在启用 checkpoint 且外部系统支持事务/幂等提交等前提下，通过分布式快照与提交协议提供可验证的一致性语义 +- **易用性**：提供简单的配置方式和丰富的连接器生态系统 +- **可扩展性**：基于插件的架构，便于添加新的连接器和转换组件 + +### 1.2 目标场景 + +- **批量数据同步**：异构数据源之间的大规模批量数据迁移 +- **实时数据集成**：支持 CDC 的流式数据捕获和同步 +- **数据湖/仓入库**：高效加载数据到数据湖（Iceberg、Hudi、Delta Lake）和数据仓库 +- **多表同步**：在单个作业中同步多个表，支持模式演化 + +## 2. 整体架构 + +SeaTunnel 采用分层架构，实现关注点分离和灵活性： + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ 用户配置层 │ +│ (HOCON 配置 / SQL) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ SeaTunnel API 层 │ +│ (数据源 API / 数据 Sink API / 转换 API / 表 API) │ +│ │ +│ • SeaTunnelSource • CatalogTable │ +│ • SeaTunnelSink • TableSchema │ +│ • SeaTunnelTransform • SchemaChangeEvent │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 连接器生态系统 │ +│ │ +│ [Jdbc] [Kafka] [MySQL-CDC] [Elasticsearch] [Iceberg] ... │ +│ (连接器生态) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 转换层 │ +│ (将 SeaTunnel API 适配到引擎特定 API) │ +│ │ +│ • FlinkSource/FlinkSink • SparkSource/SparkSink │ +│ • 上下文适配器 • 序列化适配器 │ +└─────────────────────────────────────────────────────────────────┘ + │ + ┌─────────────────────┼─────────────────────┐ + ▼ ▼ ▼ +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ SeaTunnel │ │ Apache │ │ Apache │ +│ Engine (Zeta)│ │ Flink │ │ Spark │ +│ │ │ │ │ │ +│ • 主节点 │ │ • JobManager │ │ • Driver │ +│ • 工作节点 │ │ • TaskManager│ │ • Executor │ +│ • 检查点 │ │ • State │ │ • RDD/DS │ +└──────────────┘ └──────────────┘ └──────────────┘ +``` + +### 2.1 层级职责 + +| 层级 | 职责 | 核心组件 | +|-----|------|---------| +| **配置层** | 作业定义、参数配置 | HOCON 解析器、SQL 解析器、配置验证 | +| **API 层** | 连接器的统一抽象 | 数据源/数据 Sink /转换接口、CatalogTable | +| **连接器层** | 数据源/Sink 实现 | 连接器实现（JDBC、Kafka、CDC 等） | +| **转换层** | 引擎特定适配 | Flink/Spark 适配器、上下文包装器 | +| **引擎层** | 作业执行和资源管理 | 调度、容错、状态管理 | + +## 3. 核心组件 + +### 3.1 SeaTunnel API + +API 层提供引擎独立的抽象： + +#### 数据源 Source API +- **SeaTunnelSource**：创建读取器和枚举器的工厂接口 +- **SourceSplitEnumerator**：主节点侧组件，负责分片生成和分配 +- **SourceReader**：工作节点侧组件，负责从分片读取数据 +- **SourceSplit**：表示数据分区的最小可序列化单元 + +**关键设计**：协调（枚举器）与执行（读取器）分离，实现高效的并行处理和容错。 + +#### 数据 Sink API +- **SeaTunnelSink**：创建写入器和提交器的工厂接口 +- **SinkWriter**：工作节点侧组件，负责写入数据 +- **SinkCommitter**：多个写入器的提交操作协调器 +- **SinkAggregatedCommitter**：聚合提交的全局协调器 + +**关键设计**：两阶段提交协议（prepareCommit → commit）在外部系统支持事务/幂等提交且启用 checkpoint 的前提下，可提供一致性语义。 + +#### 转换 API +- **SeaTunnelTransform**：数据转换接口 +- **SeaTunnelMapTransform**：1:1 转换 +- **SeaTunnelFlatMapTransform**：1:N 转换 + +#### 表 API +- **CatalogTable**：完整的表元数据（模式、分区键、选项） +- **TableSchema**：模式定义（列、主键、约束） +- **SchemaChangeEvent**：表示模式演化的 DDL 变更 + +### 3.2 SeaTunnel Engine (Zeta) + +原生执行引擎提供： + +#### 主节点组件 +- **CoordinatorService**：管理所有运行中的 JobMaster +- **JobMaster**：管理单个作业生命周期、生成物理计划、协调检查点 +- **CheckpointCoordinator**：每个管道协调分布式快照 +- **ResourceManager**：管理工作节点资源和槽位分配 + +#### 工作节点组件 +- **TaskExecutionService**：部署和执行任务 +- **SeaTunnelTask**：执行数据源 Source/转换/数据 Sink 逻辑 +- **FlowLifeCycle**：管理数据源 Source/转换/数据 Sink 组件的生命周期 + +#### 执行模型 +``` +LogicalDag → PhysicalPlan → SubPlan (管道) → PhysicalVertex → TaskGroup → SeaTunnelTask +``` + +### 3.3 转换层 + +通过适配器模式实现引擎可移植性： + +- **FlinkSource/FlinkSink**：将 SeaTunnel API 适配到 Flink 的数据源/Sink 接口 +- **SparkSource/SparkSink**：将 SeaTunnel API 适配到 Spark 的 RDD/Dataset 接口 +- **上下文适配器**：包装引擎特定的上下文（SourceReaderContext、SinkWriterContext） +- **序列化适配器**：桥接 SeaTunnel 和引擎序列化机制 + +### 3.4 连接器生态系统 + +所有连接器遵循标准化结构： + +``` +connector-[name]/ +├── src/main/java/.../ +│ ├── [Name]Source.java # 实现 SeaTunnelSource +│ ├── [Name]SourceReader.java # 实现 SourceReader +│ ├── [Name]SourceSplitEnumerator.java +│ ├── [Name]SourceSplit.java +│ ├── [Name]Sink.java # 实现 SeaTunnelSink +│ ├── [Name]SinkWriter.java # 实现 SinkWriter +│ └── config/[Name]Config.java +└── src/main/resources/META-INF/services/ + ├── org.apache.seatunnel.api.table.factory.TableSourceFactory + └── org.apache.seatunnel.api.table.factory.TableSinkFactory +``` + +**发现机制**：Java SPI（服务提供者接口）用于动态连接器加载。 + +## 4. 数据流模型 + +### 4.1 数据读取 Source 端数据流 + +``` +数据源 Source + │ + ▼ +┌─────────────────────┐ +│ SourceSplitEnumerator│ (主节点侧) +│ • 生成分片 │ +│ • 分配给读取器 │ +└─────────────────────┘ + │ (分片分配) + ▼ +┌─────────────────────┐ +│ SourceReader │ (工作节点侧) +│ • 从分片读取 │ +│ • 发送记录 │ +└─────────────────────┘ + │ + ▼ + SeaTunnelRow + │ + ▼ + 转换链（可选） + │ + ▼ + SeaTunnelRow + │ + ▼ +┌─────────────────────┐ +│ SinkWriter │ (工作节点侧) +│ • 缓冲记录 │ +│ • 准备提交 │ +└─────────────────────┘ + │ (CommitInfo) + ▼ +┌─────────────────────┐ +│ SinkCommitter │ (协调器) +│ • 提交变更 │ +└─────────────────────┘ + │ + ▼ +数据 Sink +``` + +### 4.2 基于分片的并行度 + +- 数据源被划分为**分片**（如文件块、数据库分区、Kafka 分区） +- 每个 **SourceReader** 独立处理一个或多个分片 +- 动态分片分配实现负载均衡和故障恢复 +- 分片状态被检查点化以实现精确一次处理 + +### 4.3 管道执行 + +作业被划分为**管道**（SubPlan）： + +``` +管道 1: [数据 Source A] → [转换 1] → [数据 Sink A] + ↓ +管道 2: [数据 Source B] ───────→ [转换 2] → [数据 Sink B] +``` + +每个管道： +- 具有独立的并行度配置 +- 维护自己的检查点协调器 +- 可以并发或顺序执行 + +## 5. 作业执行流程 + +### 5.1 提交阶段 + +```mermaid +sequenceDiagram + participant Client as 客户端 + participant CoordinatorService as 协调服务 + participant JobMaster as 作业主控 + participant ResourceManager as 资源管理器 + + Client->>CoordinatorService: 提交作业配置 + CoordinatorService->>CoordinatorService: 解析配置 → LogicalDag + CoordinatorService->>JobMaster: 创建 JobMaster + JobMaster->>JobMaster: 生成物理计划 + JobMaster->>ResourceManager: 请求资源 + ResourceManager->>JobMaster: 分配槽位 + JobMaster->>TaskExecutionService: 部署任务 +``` + +### 5.2 执行阶段 + +1. **任务初始化** + - 将任务部署到分配的槽位 + - 初始化数据 Source/转换/数据 Sink 组件 + - 从检查点恢复状态（如果在恢复中） + +2. **数据处理** + - SourceReader 从分片拉取数据 + - 数据流经转换链 + - SinkWriter 缓冲和写入数据 + +3. **检查点协调** + - CheckpointCoordinator 触发检查点 + - 检查点屏障流经数据管道 + - 任务快照其状态 + - 协调器收集确认 + +4. **提交阶段** + - SinkWriter 准备提交信息 + - SinkCommitter 协调提交 + - 状态持久化到检查点存储 + +### 5.3 状态机 + +**任务状态转换**： +``` +CREATED → INIT → WAITING_RESTORE → READY_START → STARTING → RUNNING + ↓ + FAILED ← ─────────────────────── → PREPARE_CLOSE → CLOSED + ↓ + CANCELED +``` + +**作业状态转换**： +``` +CREATED → SCHEDULED → RUNNING → FINISHED + ↓ ↓ + FAILED CANCELING → CANCELED +``` + +## 6. 关键特性 + +### 6.1 容错 + +**检查点机制**： +- 受 Chandy-Lamport 算法启发的分布式快照 +- 检查点屏障在数据流中传播 +- 状态存储在可插拔的检查点存储中（HDFS、S3、本地） +- 从最新成功的检查点自动恢复 + +**故障转移策略**： +- 任务级故障转移：重启失败的任务和相关管道 +- 基于区域的故障转移：最小化对未受影响任务的影响 +- 分片重新分配：失败的分片重新分配给健康的工作节点 + +### 6.2 精确一次语义 + +**两阶段提交协议**： +1. **准备阶段**：SinkWriter 在检查点期间准备提交信息 +2. **提交阶段**：SinkCommitter 在检查点完成后提交 +3. **中止处理**：在提交前失败时回滚 + +**幂等性**：SinkCommitter 操作必须是幂等的以处理重试 + +### 6.3 动态资源管理 + +- **基于槽位的分配**：细粒度的资源管理 +- **基于标签的过滤**：将任务分配到特定的工作节点组 +- **负载均衡**：多种策略（随机、槽位比率、系统负载） +- **动态扩缩容**：无需重启作业即可添加/移除工作节点（未来特性） + +### 6.4 模式演化 + +- **DDL 传播**：从数据源捕获模式变更（ADD/DROP/MODIFY 列） +- **模式映射**：通过管道转换模式变更 +- **动态应用**：将模式变更应用到数据 Sink 表 +- **兼容性检查**：在应用前验证模式变更 + +### 6.5 多表支持 + +- **单作业多表**：在一个作业中同步数百个表 +- **表路由**：根据 TablePath 将记录路由到正确的数据 Sink +- **独立模式**：每个表维护自己的模式 +- **副本支持**：每个表多个写入器副本以获得更高吞吐量 + +## 7. 模块结构 + +``` +seatunnel/ +├── seatunnel-api/ # 核心 API 定义 +│ ├── source/ # 数据源 API +│ ├── sink/ # 数据 Sink API +│ ├── transform/ # 转换 API +│ └── table/ # 表和模式 API +│ +├── seatunnel-connectors-v2/ # 连接器实现 +│ ├── connector-jdbc/ # JDBC 连接器 +│ ├── connector-kafka/ # Kafka 连接器 +│ ├── connector-cdc/ # CDC 连接器集合 +│ │ ├── connector-cdc-mysql/ # MySQL CDC 连接器 +│ └── ... # 更多连接器 +│ +├── seatunnel-transforms-v2/ # 转换实现 +│ ├── src/ # Transform 实现源码（如：SQL、Filter 等） +│ └── ... +│ +├── seatunnel-engine/ # SeaTunnel Engine (Zeta) +│ ├── seatunnel-engine-core/ # 核心执行逻辑 +│ ├── seatunnel-engine-server/ # 服务器组件（主节点/工作节点） +│ └── seatunnel-engine-storage/ # 检查点存储 +│ +├── seatunnel-translation/ # 引擎转换层 +│ ├── seatunnel-translation-flink/ +│ └── seatunnel-translation-spark/ +│ +├── seatunnel-formats/ # 数据格式处理器 +│ ├── seatunnel-format-json/ +│ ├── seatunnel-format-avro/ +│ └── ... +│ +├── seatunnel-core/ # 作业提交和 CLI +└── seatunnel-e2e/ # 端到端测试 +``` + +## 8. 设计原则 + +### 8.1 关注点分离 + +- **API vs 实现**：清晰的 API 边界支持多种实现 +- **协调 vs 执行**：枚举器/提交器（主节点）与读取器/写入器（工作节点）分离 +- **逻辑 vs 物理**：LogicalDag（用户意图）与 PhysicalPlan（执行细节）分离 + +### 8.2 插件架构 + +- **基于 SPI 的发现**：连接器通过 Java SPI 动态加载 +- **类加载器隔离**：每个连接器使用隔离的类加载器 +- **热插拔**：无需重新构建核心即可添加连接器 + +### 8.3 引擎独立性 + +- **统一 API**：相同的连接器代码在任何引擎上运行 +- **转换层**：将 API 适配到引擎特定细节 +- **无引擎泄漏**：连接器开发人员无需了解引擎知识 + +### 8.4 可扩展性 + +- **水平扩展**：添加工作节点以提高吞吐量 +- **基于分片的并行度**：细粒度并行处理 +- **无状态工作节点**：工作节点可以动态添加/移除 + +### 8.5 可靠性 + +- **分布式检查点**：跨分布式任务的一致性快照 +- **增量状态**：优化大状态的检查点大小 +- **精确一次保证**：端到端一致性 + +## 9. 下一步 + +深入了解特定架构组件： + +- [设计理念](design-philosophy.md) - 核心设计原则和权衡 +- [数据 Source 架构](api-design/source-architecture.md) - 数据源 API 设计深入探讨 +- [数据 Sink 架构](api-design/sink-architecture.md) - 数据 Sink API 设计深入探讨 +- [引擎架构](engine/engine-architecture.md) - SeaTunnel Engine 内部机制 +- [检查点机制](fault-tolerance/checkpoint-mechanism.md) - 容错实现 + +实践指南： + +- [如何创建您的连接器](../developer/how-to-create-your-connector.md) +- [快速入门](../getting-started/locally/quick-start-seatunnel-engine.md) + +## 10. 参考资料 + +### 10.1 相关概念 + +- [Apache Flink](https://flink.apache.org/) - 检查点和状态管理的灵感来源 +- [Apache Kafka](https://kafka.apache.org/) - 消费者组模型影响了分片分配 +- [Chandy-Lamport 算法](https://en.wikipedia.org/wiki/Chandy-Lamport_algorithm) - 分布式快照算法 diff --git a/docs/zh/connectors/sink/GoogleFirestore.md b/docs/zh/connectors/sink/GoogleFirestore.md index 9e9593885494..f6cf1506e1f9 100644 --- a/docs/zh/connectors/sink/GoogleFirestore.md +++ b/docs/zh/connectors/sink/GoogleFirestore.md @@ -34,7 +34,7 @@ Google Cloud 服务账户的凭证，使用 base64 编码。如果未设置， ### 通用选项 -汇插件通用参数，请参考 [Sink Common Options](../common-options/sink-common-options.md) 了解详情。 +Sink 插件通用参数，请参考 [Sink Common Options](../common-options/sink-common-options.md) 了解详情。 ## 示例 diff --git a/docs/zh/connectors/sink/Mysql.md b/docs/zh/connectors/sink/Mysql.md index c5c58f57f28e..c3147a77995e 100644 --- a/docs/zh/connectors/sink/Mysql.md +++ b/docs/zh/connectors/sink/Mysql.md @@ -85,7 +85,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | max_commit_attempts | Int | 否 | 3 | 事务提交失败的重试次数 | | transaction_timeout_sec | Int | 否 | -1 | 事务打开后的超时，默认值为-1（永不超时）。请注意，设置超时可能会影响＜br/＞精确一次语义 | | auto_commit | Boolean | 否 | true | 默认情况下启用自动事务提交 | -| field_ide | String | 否 | - | 确定从源同步到汇时是否需要转换字段`ORIGINAL表示不需要转换`大写`表示转换为大写`LOWERCASE表示转换为小写。 | +| field_ide | String | 否 | - | 确定从源同步到 Sink 时是否需要转换字段`ORIGINAL表示不需要转换`大写`表示转换为大写`LOWERCASE表示转换为小写。 | | properties | Map | 否 | - | 其他连接配置参数，当属性和URL具有相同的参数时，优先级由驱动程序的特定实现决定。例如，在MySQL中，属性优先于URL。 | | common-options | | 否 | - | Sink插件常用参数，请参考 [Sink Common Options](../common-options/sink-common-options.md) 详见 | | schema_save_mode | Enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST | 在启动同步任务之前，对目标侧的现有表面结构选择不同的处理方案。 | diff --git a/docs/zh/connectors/sink/PostgreSql.md b/docs/zh/connectors/sink/PostgreSql.md index edcea7fa677c..f24ac91d6c6c 100644 --- a/docs/zh/connectors/sink/PostgreSql.md +++ b/docs/zh/connectors/sink/PostgreSql.md @@ -88,7 +88,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | max_commit_attempts | Int | 否 | 3 | 事务提交失败的重试次数。 | | transaction_timeout_sec | Int | 否 | -1 | 事务开启后的超时时间，默认值为 -1（永不超时）。注意设置超时可能会影响
精确一次语义。 | | auto_commit | Boolean | 否 | true | 默认启用自动事务提交。 | -| field_ide | String | 否 | - | 识别字段在从源到汇的同步时是否需要转换。`ORIGINAL` 表示无需转换；`UPPERCASE` 表示转换为大写；`LOWERCASE` 表示转换为小写。 | +| field_ide | String | 否 | - | 识别字段在从源到 Sink 的同步时是否需要转换。`ORIGINAL` 表示无需转换；`UPPERCASE` 表示转换为大写；`LOWERCASE` 表示转换为小写。 | | properties | Map | 否 | - | 附加连接配置参数，当 properties 和 URL 具有相同参数时，优先级由
驱动的具体实现决定。例如，在 MySQL 中，properties 优先于 URL。 | | common-options | | 否 | - | Sink 插件的公共参数，请参阅 [Sink 公共选项](../common-options/sink-common-options.md) 以获取详细信息。 | | schema_save_mode | Enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST | 在同步任务开启之前，根据目标端现有表结构选择不同处理方案。 | diff --git a/docs/zh/connectors/source/SqlServer.md b/docs/zh/connectors/source/SqlServer.md index 2c623d1ed3a9..bb3554132620 100644 --- a/docs/zh/connectors/source/SqlServer.md +++ b/docs/zh/connectors/source/SqlServer.md @@ -262,7 +262,7 @@ transform { sink { Console {} - # 如果你想了解更多关于如何配置 seatunnel 的信息，并查看汇插件的完整列表， + # 如果你想了解更多关于如何配置 seatunnel 的信息，并查看 Sink 插件的完整列表， # 请前往 https://seatunnel.apache.org/docs/connector-v2/sink/Jdbc } ``` diff --git a/docs/zh/developer/how-to-create-your-connector.md b/docs/zh/developer/how-to-create-your-connector.md index c8157fbb992a..f370a7d6b3b5 100644 --- a/docs/zh/developer/how-to-create-your-connector.md +++ b/docs/zh/developer/how-to-create-your-connector.md @@ -2,3 +2,15 @@ 如果你想针对SeaTunnel新的连接器API开发自己的连接器（Connector V2），请查看[这里](https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/README.zh.md) 。 +## 架构文档参考 + +如需了解 SeaTunnel 的 API 设计和引擎架构的详细信息，请参阅： + +- [架构概览](../architecture/overview.md) - 整体架构和设计原则 +- [数据源架构](../architecture/api-design/source-architecture.md) - Source API 设计深入剖析 +- [数据汇架构](../architecture/api-design/sink-architecture.md) - Sink API 设计深入剖析 +- [转换层](../architecture/api-design/translation-layer.md) - 连接器如何在不同引擎上工作 +- [检查点机制](../architecture/fault-tolerance/checkpoint-mechanism.md) - 容错和状态管理 + +这些文档将帮助你理解 SeaTunnel 连接器中使用的底层架构和设计模式。 +