Skip to content

[Docs][Architecture] Add architecture documents#10429

Open
davidzollo wants to merge 12 commits intoapache:devfrom
davidzollo:docs-architecture-design
Open

[Docs][Architecture] Add architecture documents#10429
davidzollo wants to merge 12 commits intoapache:devfrom
davidzollo:docs-architecture-design

Conversation

@davidzollo
Copy link
Contributor

This PR reviews and refines the Chinese architecture documents in docs/zh/architecture for accuracy, removing obsolete implementation details and correcting configuration keys based on the current codebase.

davidzollo and others added 7 commits January 29, 2026 12:02
This PR introduces a complete architecture documentation system for SeaTunnel, including:

**Phase 1 - Core Foundation (6 docs)**:
- Architecture Overview
- Design Philosophy
- Source Architecture
- Sink Architecture
- Engine Architecture
- Checkpoint Mechanism

**Phase 2 - Advanced Topics (6 docs)**:
- DAG Execution Model
- Resource Management
- CatalogTable & Metadata Management
- Multi-table Synchronization
- Exactly-once Semantics
- Translation Layer

**Key Features**:
- 24 professional documents (12 English + 12 Chinese)
- ~450KB total size, 15,100 lines
- Based on 2000+ lines of source code analysis
- Complete with architecture diagrams, sequence diagrams, and code examples
- Unified documentation template and terminology
- Integrated into docs/sidebars.js

This documentation system fills a critical gap in SeaTunnel's technical documentation,
providing enterprise-grade, systematic architectural design documents for:
- Architects evaluating SeaTunnel
- Core contributors understanding the codebase
- Enterprise developers customizing the system
- Technical decision makers assessing the platform

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This file is for internal tracking and should not be included in the PR.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@davidzollo davidzollo changed the title [Docs][Architecture] Refine Chinese architecture documents [Docs][Architecture] Add architecture documents Jan 31, 2026
@DanielCarter-stack
Copy link

DanielCarter-stack commented Feb 1, 2026

Issue 1: English documentation missing maintenance note

Location: docs/en/architecture/*.md (all English architecture documentation)

Problem Description:
The Chinese documentation includes a note "说明:为降低维护成本并避免'文档代码/链接随重构漂移',本文档不直接嵌入源码片段与源码链接", but the English documentation lacks a similar explanation.

Potential Risks:

  • English documentation maintainers may not understand why source code links are not embedded
  • Source code links may be added to English documentation in the future, increasing maintenance burden

Impact Scope:

  • Direct Impact: English documentation maintainers
  • Indirect Impact: Documentation and code synchronization maintenance costs
  • Affected Area: All English architecture documentation

Severity: MINOR

Improvement Suggestion:
Add a note at the beginning of the English documentation:

> Note: To minimize maintenance overhead and avoid "documentation-code drift" due to refactoring, 
> this documentation focuses on component responsibilities, interaction flows, and design motivations 
> rather than embedding source code snippets or direct source links.

Rationale:

  • Maintain consistency with Chinese documentation
  • Help future maintainers understand the design decision
  • Reduce confusion caused by inconsistent maintenance strategies between Chinese and English documentation

Issue 2: Missing documentation testing

Location: PR change root directory (recommend adding .github/workflows/docs-test.yml)

Problem Description:
This PR adds extensive documentation but does not include documentation testing CI, preventing automatic verification of:

  • Markdown link validity (especially for relative paths like ../../seatunnel-api/...)
  • Code example syntax correctness
  • Synchronization between configuration keys and source code constants

Potential Risks:

  • Documentation links become broken after code refactoring but cannot be detected in time
  • Configuration keys change in code but documentation is not updated synchronously

Impact Scope:

  • Direct Impact: Documentation reliability
  • Indirect Impact: Users encounter errors when following documentation instructions
  • Affected Area: All architecture documentation

Severity: MAJOR

Improvement Suggestion:
Add CI workflow example:

name: Docs Test

on:
  pull_request:
    paths:
      - 'docs/**'
      - 'seatunnel-api/**'
      - 'seatunnel-engine/**'

jobs:
  markdown-links:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Check Markdown links
        uses: gaurav-nelson/github-action-markdown-link-check@v1
        with:
          config-file: '.markdownlinkcheck.json'
  
  config-keys-sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Check config keys
        run: |
          # Example: Verify that configuration keys in the documentation exist in the code
          grep -r "multi_table_sink_replica" seatunnel-api || exit 1

Rationale:

  • Automated documentation quality assurance
  • Early detection of broken links and configuration key desynchronization
  • Aligns with documentation testing best practices in modern projects

Issue 3: Storage implementation naming description in checkpoint-mechanism.md is not precise enough

Location: docs/en/architecture/fault-tolerance/checkpoint-mechanism.md:260

Related Context:

**Implementations**:
- `LocalFileCheckpointStorage`: Local file system (testing)
- `HdfsCheckpointStorage`: HDFS
- `S3CheckpointStorage`: AWS S3
- `OssCheckpointStorage`: Aliyun OSS

Problem Description:
Based on source code verification, the actual class names are:

  • LocalFileStorage (not LocalFileCheckpointStorage)
  • HdfsStorage (not HdfsCheckpointStorage)
  • S3/OSS are not independent CheckpointStorage implementation classes, but are supported through HdfsStorage's S3Configuration/OssConfiguration

Potential Risks:

  • Developers searching for the S3CheckpointStorage class according to the documentation will not find it
  • Misleads users into thinking S3/OSS are independent plugins, when they actually share HdfsStorage

Impact Scope:

  • Direct Impact: Developers looking for checkpoint storage implementations
  • Indirect Impact: Users may find wrong documentation when configuring S3/OSS checkpoints
  • Affected Area: checkpoint storage configuration

Severity: MAJOR

Improvement Suggestion:
Correct to:

**Implementations**:
- `LocalFileStorage`: Local file system (testing)
- `HdfsStorage`: Hadoop FileSystem-based backend; can work with HDFS/S3A/etc depending on Hadoop configuration

Note: S3 and OSS support are provided through Hadoop FileSystem configuration (e.g., `fs.s3a.impl`) rather than separate CheckpointStorage implementations.

Rationale:

  • Consistent with actual code class names
  • Accurately reflects the implementation approach of S3/OSS
  • Avoids misleading developers into searching for non-existent classes

Issue 4: Resource management strategy class names are inaccurate

Location: docs/en/architecture/engine/resource-management.md

Related Context:
According to the review task document, strategy class names were corrected:

  • Original documentation: RandomSlotAssignStrategy / SlotRatioSlotAssignStrategy / SystemLoadSlotAssignStrategy
  • Actual code: RandomStrategy / SlotRatioStrategy / SystemLoadStrategy

Problem Description:
The English documentation (newly added) may still contain old class names (need to verify whether English documentation has applied the correction).

Potential Risks:

  • Developers searching for RandomSlotAssignStrategy cannot find the actual class
  • Misleads developers' understanding of strategy naming patterns

Impact Scope:

  • Direct Impact: Developers looking for resource management strategy implementations
  • Indirect Impact: Custom resource strategy development
  • Affected Area: Resource management module

Severity: MAJOR

Improvement Suggestion:
Verify and correct strategy class names in English documentation (if not yet corrected):

**Slot Allocation Strategies**:
```java
// 1. RandomStrategy: Random selection among available workers
// 2. SlotRatioStrategy: Prefer workers with more available slots
// 3. SystemLoadStrategy: Prefer workers with lower CPU/memory usage

** Rationale**:
- 与实际代码类名一致
- 保持中英文文档对齐
- 确保开发者能找到正确的实现类

---

# ## Issue 5: Documentation Changes Not Updated in Developer Guide

** Location**: `docs/developer/` directory

** Problem Description**:
添加了大量架构文档,但开发者指南(如 `docs/developer/how-to-create-your-connector.md`)没有相应更新,添加指向新架构文档的链接。

** Potential Risks**:
- Connector 开发者可能不知道有架构文档可以参考
- 降低新架构文档的可见性和使用率

** Scope of Impact**:
- 直接影响: Connector 开发者
- 间接影响: 新架构文档的利用
- 影响面: Connector 开发流程

** Severity**: MINOR

** Recommendation**:
在 `docs/developer/how-to-create-your-connector.md` 中添加:
```markdown
## Architecture Reference

For detailed information on SeaTunnel's API design and engine architecture, see:

- [Architecture Overview](../architecture/overview.md)
- [Source Architecture](../architecture/api-design/source-architecture.md)
- [Sink Architecture](../architecture/api-design/sink-architecture.md)
- [Checkpoint Mechanism](../architecture/fault-tolerance/checkpoint-mechanism.md)

Rationale:

  • Increase visibility of new architecture documentation
  • Help Connector developers quickly find reference materials
  • Improve overall usability of the documentation system

@misi1987107
Copy link
Contributor

misi1987107 commented Feb 4, 2026

I think there are a bit too many subdirectories. Could we reduce them to optimize navigation efficiency.the directory structure I suggest is as follows,just for your consideration:
architecture
--overview.md
----design-philosophy.md
--api-design
----catalog-table.md
----sink-architecture.md
----source-architecture.md
----translation-layer.md
--engine
----engine-architecture.md
----dag-execution.md
----resource-management.md
--taskmechanism-design
----checkpoint-mechanism.md
----exactly-once.md
----multi-table.md


```
seatunnel/
├── seatunnel-api/ # 核心 API 定义
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The structure of this module is incorrect

│ ├── connector-cdc-mysql/ # MySQL CDC 连接器
│ └── ... # 更多连接器
├── seatunnel-transforms-v2/ # 转换实现
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The structure of this module is incorrect

### 学术论文

- Chandy & Lamport (1985): ["Distributed Snapshots"](https://lamport.azurewebsites.net/pubs/chandy.pdf)
- Gray & Lamport (2006): ["Consensus on Transaction Commit"](https://lamport.azurewebsites.net/pubs/paxos-commit.pdf)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The resource has been removed

Comment on lines 48 to 54
**数据源**:
- `SourceSplitEnumerator`(协调端):生成分片、分配给读取器、处理注册
- `SourceReader`(工作节点):从分配的分片读取数据

**数据汇**:
- `SinkCommitter` / `SinkAggregatedCommitter`(协调端):协调提交
- `SinkWriter`(工作节点):写入数据、准备提交信息
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what the content under this title means, as there are no such two parts in other titles

Copy link
Collaborator

@LiJie20190102 LiJie20190102 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

davidzollo and others added 4 commits February 5, 2026 23:32
…eedback

1. Fix CheckpointStorage implementation class names in checkpoint-mechanism.md
   - Correct: LocalFileStorage, HdfsStorage (not *CheckpointStorage)
   - Clarify S3/OSS support through Hadoop FileSystem configuration

2. Update developer guide with architecture documentation links
   - Add architecture reference section to how-to-create-your-connector.md
   - Link to overview, source/sink architecture, translation layer, and checkpoint docs
   - Improve discoverability for connector developers

3. Improve design-philosophy.md to focus on principles over implementation
   - Replace specific class names with conceptual explanations
   - Explain coordination vs execution separation mechanism
   - Clarify two-phase commit protocol workflow
   - Use "Master-side" and "Worker-side" instead of "Control Plane" and "Data Plane"
   - Remove code examples, replace with principle-focused descriptions

4. Optimize directory structure (Option A)
   - Move translation/translation-layer.md -> api-design/translation-layer.md
   - Move data-flow/multi-table.md -> features/multi-table.md
   - Remove empty translation/ and data-flow/ directories
   - Reduce single-file directories for better navigation

Addresses reviewer feedback from:
- DanielCarter-stack (Issues 1, 3, 5)
- misi1987107 (directory structure)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add maintenance note explaining why source code links are not embedded
- Align with Chinese documentation maintenance strategy
- Address Issue 1 from code review (DanielCarter-stack)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- User decision: Keep overview.md without the maintenance note
- Revert previous addition

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants