Improve plugin-discovery mechanism to support dynamic runtime loading of custom connectors

[vardhan30016]

Description:

What is the feature request? What problem does it solve?
Currently, VDK supports extending data ingestion and transformation jobs via the plugin architecture, enabling new connectors and job-types.
VMware Blogs
+1

However, the mechanism requires manual packaging and deployment of plugin artifacts along with a restart of the control service (or job execution environment) to load new custom connectors. This imposes friction for organizations that want runtime flexibility: e.g., dynamically adding new data source connectors (APIs, sinks, streaming sources) without redeploying or restarting the job runner/control service.

This results in slower onboarding of new sources, reduces agility for data teams, and complicates environments where connectors need to be hot-swapped or loaded at runtime (for example in multi-tenant setups, rapid prototype phases, or “bring your own connector” scenarios).

Suggested solution
Introduce a dynamic plugin discovery and hot-load mechanism in VDK, for example by:

Scanning a designated “plugins” directory (or configured package repository) at runtime to discover new connector modules (e.g., Python packages conforming to a connector interface) without requiring a full redeploy.

Providing an API or CLI command like vdk plugin load that validates, registers and enables a connector at runtime.

Enhancing metadata/catalog updates so that newly loaded connectors appear in job templates, documentation, and discovery UIs dynamically.

Supporting versioning of connector plugins, and safe unloading/replacing of outdated plugins with minimal service interruption.

Optionally, enabling plugin isolation (sandboxing) to avoid interference between tenant-specific connectors.

Additional context

Many enterprises treat new data sources as “first-class citizens” and want the data platform to allow rapid ingestion of new APIs/data streams with minimal operational overhead.

Hot-plugin load is common in other middleware/data-engineering frameworks (for example Apache NiFi, some micro-services plugin architectures). Adding this capability to VDK would strengthen its extensibility story.

This aligns with VDK’s goal of making infrastructure tasks simpler so data engineers can focus on domain logic instead of deployment/ops work.
VMware Blogs

I would be happy to prototype an implementation and share a design/PR if this aligns with the project roadmap.