Skip to content

Feature: [Connector-V2] Add HubSpot Source Connector structure (#10358)#10447

Open
Suresh-Krishna-Kusuma wants to merge 16 commits intoapache:devfrom
Suresh-Krishna-Kusuma:feature/hubspot-connector
Open

Feature: [Connector-V2] Add HubSpot Source Connector structure (#10358)#10447
Suresh-Krishna-Kusuma wants to merge 16 commits intoapache:devfrom
Suresh-Krishna-Kusuma:feature/hubspot-connector

Conversation

@Suresh-Krishna-Kusuma
Copy link

This PR introduces the initial implementation of the HubSpot Source Connector (Phase 1). It is implemented by extending the generic HttpSource to leverage existing HTTP capabilities.

Changes:

  • Added connector-http-hubspot module.
  • Implemented HubSpotSource and HubSpotSourceFactory.
  • Added dynamic configuration logic to map object_type to API URLs.
  • Added documentation in docs/en/connector-v2/source/HubSpot.md.

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

…e#10358)

This PR introduces the initial implementation of the HubSpot Source Connector (Phase 1).
It is implemented by extending the generic HttpSource to leverage existing HTTP capabilities.

Changes:
- Added connector-http-hubspot module.
- Implemented HubSpotSource and HubSpotSourceFactory.
- Added dynamic configuration logic to map object_type to API URLs.
- Added documentation in docs/en/connector-v2/source/HubSpot.md.
@DanielCarter-stack
Copy link

Issue 1: Configuration conversion has type incompatibility risk

Location: HubSpotSource.java:46
Severity: BLOCKER
Recommendation: Refactor to use custom Parameter class pattern

Issue 2: Document naming does not follow project conventions

Location: docs/en/connectors/source/HubSpot.md
Severity: MINOR
Recommendation: Rename to Hubspot.md or standardize to capitalize only the first letter

Issue 3: Missing required configuration file updates

Location: plugin-mapping.properties, seatunnel-dist/pom.xml, config/plugin_config
Severity: BLOCKER
Recommendation: Add all required configuration entries

Issue 4: Custom Parameter class not implemented

Location: HubSpotSource.java overall design
Severity: MAJOR
Recommendation: Create HubSpotSourceParameter class

Issue 6: Sensitive information may leak to logs

Location: HubSpotSource.java:59
Severity: MINOR
Explanation: This is a common issue with base classes; recommend overriding toString() in HttpParameter

Issue 7: Missing newline at end of file

Location: docs/en/connectors/source/HubSpot.md:56
Severity: MINOR
Recommendation: Add newline at end of file

Issue 8: Inconsistent changelog format

Location: docs/en/connectors/changelog/connector-http-hubspot.md
Severity: MINOR
Recommendation: Use the same format as other HTTP connectors

Issue 9: Completely lacking test coverage

Location: Entire connector-http-hubspot module
Severity: MAJOR
Recommendation: Add Factory tests and configuration conversion tests

Issue 10: Documentation missing critical usage instructions

Location: docs/en/connectors/source/HubSpot.md
Severity: MINOR
Recommendation: Add documentation for pagination, rate limiting, etc.


@LiJie20190102
Copy link
Collaborator

Thank you for your contribution. Can you add e2e test cases


## 2.3.13-SNAPSHOT

- Add HubSpot Source Connector No newline at end of file
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also contribute the sink connector

@github-actions github-actions bot added core SeaTunnel core module e2e labels Feb 5, 2026
Comment on lines 279 to 282
Hubspot {
url = "https://api.github.com/orgs/apache/repos"
access_token = "xxxx"
method = "GET"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need to update Github.md?

@davidzollo
Copy link
Contributor

Thank you for the contribution! This is a good starting point for the HubSpot connector. I found several critical issues that need to be addressed before this can be merged.

1. Invalid Default content_field causing Runtime Failure
In HubSpotSourceParameter.java, the default content_field is hardcoded as "results".
The strict JSON path implementation in HttpSourceReader expects a valid JsonPath expression (e.g., usually starting with $.). Passing "results" will likely cause a JsonPathCompile exception or fail to extract data correctly at runtime.
Action: Change the default value to $.results (to return the array) or $.results.*?

2. Mismatch between Default Schema and E2E Test Expectations

  • HubSpotSource currently reuses HttpSource without injecting a default schema or json_field configuration. This means, by default, it produces a single column named content (containing the raw JSON string).
  • However, your E2E test (hubspot_source_case.conf) asserts the presence of an id field (field_name = "id").
  • Result: The test validates a scenario that doesn't match the actual default behavior of the connector.
    Action: Either (A) Configure default json_field mappings in HubSpotSource so fields like id are extracted automatically, OR (B) Update the E2E test config to explicitly define the schema and json_field mappings required to parse the id.

3. E2E Test Network Configuration is Incorrect
The HubSpotIT.java sets up a MockServer container but fails to place it in the shared network used by the SeaTunnel engine container.

  • Missing .withNetwork(NETWORK) and .withNetworkAliases(...) in the container setup.
  • The configuration url = "http://mock-server" in the conf file cannot be resolved by the SeaTunnel container because the network alias is invalid.
    Action: Please copy the networking setup pattern from HttpIT.java (using TestSuiteBase.NETWORK and setting an alias) to ensure the test actually hits the mock server.

4. OptionRule Validation Inconsistency
In HubSpotSourceFactory, the optionRule only declares ACCESS_TOKEN and OBJECT_TYPE.
However, your test case tries to inject a url parameter (url = "http://mock-server").

  • If strict validation is enabled in the future, this will fail.
  • You are also manually reading "url" in HubSpotSourceParameter using a disjointed Options.key("url") instead of reusing HttpCommonOptions.URL.
    Action: If you intend to allow users/tests to override url, please add HttpCommonOptions.URL to the OptionRule in the factory and reuse that constant in HubSpotSourceParameter.

5. Dependency Scope in seatunnel-dist
In seatunnel-dist/pom.xml, the new connector-http-hubspot dependency is missing <scope>provided</scope>, unlike other HTTP connectors.
Action: Please add <scope>provided</scope> to align with the project's packaging strategy and avoid unnecessary artifact bloat.

6. Documentation Accuracy
In Hubspot.md, the "Key features" section marks parallelism and exactly-once as supported ([x]). Since this connector extends HttpSource (which is typically single-split/bounded for batch), please verify if these claims are strictly true or just inherited boilerplate. It is better to be accurate than to overpromise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants