[WIP] Add custom data type support

[tigrannajaryan]

This will allow to have pipelines of data types that are not one of the
builtin type: traces and metrics.

See design.md for detailed description of how custom data types work.

[dmitryax]

I don't think this function really needed. For metrics and traces, it supposed to be a temporary function to support old->new style data model migration. What if we don't have if factory, ok := factoryBase.(component.DataExporterFactory); ok { condition? Looks like we would get a compile time error instead of this one which seems to be more useful.

[dmitryax]

the same question here

[owais]

This makes sense and would allow people to support a lot of other use-cases. +1 for adding something like this. I was thinking of doing this a bit differently. What if the core repo did not deal with building exporters, processors and receivers at all and only built pipeline objects?

1. Add a new Pipeline component interface
2. Service parses config, iterates over pipelines in config and calls pipeline factory to build each pipeline.
3. Pipeline's build method or pipeline factory would build exporters, processors and receivers. We'd move most of the code from builder package to pipeline.

Service would then just interact with pipeline objects eliminating the need for service to know and deal data types handled by each pipeline. It can basically just call Build(config, factories) Start(ctx, host) and Shutdown() methods on each pipeline. Pipeline would then internally call these methods as needed on each of it's components. Most of the builder package would become the default pipeline component.

Once we have this, anyone should be able to implement a different pipeline that deals with a completely different data-type and use it with Otel just like people build other custom components now.

Implementing a custom pipeline would mean duplicating/copying a bit of code but the flexibility this will bring to the system would be worth it IMO. Also, it should be rare for people to implement custom pipelines so copying a bit of code (if required) shouldn't make that much of a difference. We can also share a lot of code by using common/base interfaces to pass around components. Pipeline might do some type-assertion on it's components but it would happen only once as part of the startup process.

This will also allow people to customize other pipeline behavior like replacing fanout connector with something else or handling errors in a different way without a processor, etc.

In case it becomes tedious to implement a pipeline, it should be trivial to support automate it with code gen. With code gen, I'm sure we can make writing a custom pipeline as simple as the following:

// data_type.go
package myCustomPipeline

type MyCustomDataType struct {
    Name string
}

// gen.go
package myCustomPipeline

//go:generate otel-pipeline-generator -type MyCustomDataType -Name MyCustomPipeline

I'm sure this should be all that is required to build a pipeline with default behaviour that processes custom data types.

TL;DR: Add only one new type/component called Pipeline, re-work builder package into a default Pipeline component. Let people register custom pipelines. Add a code generator to trivialize building custom pipelines.

I just started a basic prototype to see how easily this can be done. Will report back once I have something more tangible.

[jrcamp]

I think this proposal originally came out of how do we send metadata from receivers. Later we also discussed having metadata be its own formal data type alongside metrics, traces, logs. I think that will be important when deployed in collector mode and users want to be able to take metadata from one collector instance inside the firewall, forward to one outside the firewall, and then send to cloud. If it's just part of the OT protocol then it can use existing receivers/exporters.

[pjanotti]

The approach LGTM @tigrannajaryan

[pjanotti]

Suggested change

	// NewFanOutConnector wraps multiple new type consumers in a single one.
	// NewFanOutConnector wraps multiple new consumers in a single one.

[pjanotti]

Suggested change

	// Consume exports the span data to all consumers wrapped by the current one.
	// Consume exports the received data to all consumers wrapped by the current one.

[pjanotti]

nit: a switch looks better in this case

[pjanotti]

Suggested change

	// Metrics data type is required. Create a trace exporter based on config.
	// Metrics data type is required. Create a metrics exporter based on config.

[codecov-io]

Codecov Report

Merging #817 into master will decrease coverage by 0.17%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master     #817      +/-   ##
==========================================
- Coverage   85.68%   85.51%   -0.18%     
==========================================
  Files         184      184              
  Lines       13061    13176     +115     
==========================================
+ Hits        11191    11267      +76     
- Misses       1419     1458      +39     
  Partials      451      451              

Impacted Files	Coverage Δ
config/configmodels/configmodels.go	100.00% <ø> (+5.00%)	⬆️
processor/cloningfanoutconnector.go	69.44% <0.00%> (-9.51%)	⬇️
processor/fanoutconnector.go	82.85% <0.00%> (-10.70%)	⬇️
service/service.go	49.09% <0.00%> (ø)
service/builder/receivers_builder.go	74.56% <57.14%> (-3.82%)	⬇️
service/builder/exporters_builder.go	72.26% <68.42%> (+4.76%)	⬆️
service/builder/pipelines_builder.go	82.89% <74.19%> (-2.48%)	⬇️
config/example_factories.go	86.54% <94.11%> (+1.70%)	⬆️
config/config.go	99.69% <100.00%> (-0.01%)	⬇️
translator/internaldata/resource_to_oc.go	76.47% <0.00%> (+2.94%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f588c89...92e86e9. Read the comment docs.

[tigrannajaryan]

Decided to go a different route: #959
Closing this.