Filter available runtimes based on configuration setting

[kevin-bates]

This pull request allows operators to NOT expose runtime processors that might otherwise be available. By default, all three out-of-the-box runtime processors ('local', 'airflow', 'kfp') are exposed because their entry points are provided as part of Elyra's packaging. With this change, operators can select which runtime processors should be exposed by setting the command-line option --PipelineProcessorRegistry.runtimes=<runtime>. If they wish to expose multiple runtimes, then additional --PipelineProcessorRegistry.runtimes=<runtime> options are required.

CLI Configuration Option

For example, a command line of jupyter elyra --PipelineProcessorRegistry.runtimes=airflow will produce the following set of Elyra tiles:

and the following log entries:

[I 2023-02-21 13:47:58.149 ElyraApp] Registering airflow processor 'elyra.pipeline.airflow.processor_airflow.AirflowPipelineProcessor'...
[I 2023-02-21 13:47:58.834 ElyraApp] Although runtime 'kfp' is installed, it is not in the set of runtimes configured via '--PipelineProcessorRegistry.runtimes' and will not be available.
[I 2023-02-21 13:47:58.918 ElyraApp] Although runtime 'local' is installed, it is not in the set of runtimes configured via '--PipelineProcessorRegistry.runtimes' and will not be available.

Similarly, a command line of jupyter elyra --PipelineProcessorRegistry.runtimes=kfp --PipelineProcessorRegistry.runtimes=local will produce tiles:

---

File-based Configuration Option

Note that these options can also be set in the jupyter_elyra_config.py configuration file:

#------------------------------------------------------------------------------
# PipelineProcessorRegistry(SingletonConfigurable) configuration
#------------------------------------------------------------------------------
## The runtimes to use during this Elyra instance.  (env
#  ELYRA_PROCESSOR_RUNTIMES) 
#  Default: []
c.PipelineProcessorRegistry.runtimes = ['local', 'airflow']

such that a command of jupyter elyra yields:

What changes were proposed in this pull request?

• See above, plus
• Add new Configuring the pipeline editor topic to the user guide
  
• Move related documentation content to this new topic and add link(s)
• Update kf-notebook Dockerfile to only enable Kubeflow Pipelines runtime

How was this pull request tested?

Implementation

• These changes were tested via a new test which sets the runtime configuration via the environment variable ELYRA_PROCESSOR_RUNTIMES and checks various combinations, as well as manual testing to produce the screenshots, etc.

Documentation

• Reviewed output of make docs

kf-notebook container image

• Build container image (make kf-notebook-image)
• Run container image (docker run -it -p 8888:8888 elyra/kf-notebook:dev)
• Open JupyterLab UI in web browser
• In terminal inspect customized content of $(jupyter --config-dir)/jupyter_elyra_config.py (should be c.PipelineProcessorRegistry.runtimes = ['kfp'])
• Inspect tiles in JupyterLab launcher window (only KFP pipeline editor tile is rendered)

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[kevin-bates]

I'm moving this to draft since there are some documentation updates and potential other changes that are needed.

[marthacryan]

• It should only be possible to create a runtime environment configuration for enabled runtimes
• Only runtime configurations for enabled runtimes should be listed in the runtime configurations list

Fixed both

[marthacryan]

• It should only be possible to create catalog connectors for enabled runtimes

  • if no kfp or airflow runtime is enabled, no connectors are supported
  • we need to filter by metadata.runtime_type to filter out connectors for disabled runtimes

• Only catalog connectors for enabled runtimes should be listed in the component catalog list (we need to filter by metadata.runtime_type)

Done - probably want to add some behavior to say that no runtimes are enabled though

[kevin-bates]

Done - probably want to add some behavior to say that no runtimes are enabled though

Are you able to configure things such that no runtimes are enabled? And if so, is that a bad thing (assuming things are graceful)?

I could see it possible to have zero runtimes enabled if a bad value was given in the config, but we should probably weigh the cost of validating that each runtime provided is legit since we may not know what runtimes exist. I suppose we could check AFTER loading runtimes, and ask "did we load everything that was asked", and, if not, warn about those "configured runtimes" that were not actually loaded. But, even then, I think we should tolerate no available runtimes if that's doable.

[ptitzler]

Checked the following user scenarios:

Open .pipeline file in VPE:

• Try to open .pipeline file in VPE for disabled runtime

Run file as pipeline:

• Try to run a notebook as pipeline when all supported runtimes are disabled: open notebook -> run as pipeline
• Try to run a Python script as pipeline when all supported runtimes are disabled: open script -> run as pipeline
• Try to run a n R script as pipeline when all supported runtimes are disabled: open script -> run as pipeline
• Try to run a notebook as pipeline when Airflow runtime is disabled: open notebook -> run as pipeline
• Try to run a Python script as pipeline when Airflow runtime is disabled: open script -> run as pipeline
• Try to run a n R script as pipeline when Airflow runtime is disabled: open script -> run as pipeline
• Try to run a notebook as pipeline when KFP runtime is disabled: open notebook -> run as pipeline
• Try to run a Python script as pipeline when KFP runtime is disabled: open script -> run as pipeline
• Try to run a n R script as pipeline when KFP runtime is disabled: open script -> run as pipeline

Manage runtimes:

• Try CRUD when KFP and Airflow runtimes are disabled
• Try CRUD when Airflow runtime is disabled
• Try CRUD when KFP runtime is disabled

Export pipeline:

• Try to export a generic pipeline when KFP and Airflow runtimes are disabled
• Try to export a generic pipeline when Airflow runtime is disabled
• Try to export a generic pipeline when KFP runtime is disabled

Run pipeline:

• Try to run a generic pipeline when KFP and Airflow runtimes are disabled
• Try to run a generic pipeline when Airflow runtime is disabled
• Try to run a generic pipeline when KFP runtime is disabled

Manage component catalogs:

• Try CRUD when KFP and Airflow runtimes are disabled
• Try CRUD when Airflow runtime is disabled
• Try CRUD when KFP runtime is disabled

[ptitzler]

Done - probably want to add some behavior to say that no runtimes are enabled though

Are you able to configure things such that no runtimes are enabled? And if so, is that a bad thing (assuming things are graceful)?

I could see it possible to have zero runtimes enabled if a bad value was given in the config, but we should probably weigh the cost of validating that each runtime provided is legit since we may not know what runtimes exist. I suppose we could check AFTER loading runtimes, and ask "did we load everything that was asked", and, if not, warn about those "configured runtimes" that were not actually loaded. But, even then, I think we should tolerate no available runtimes if that's doable.

I did test this scenario by setting PipelineProcessorRegistry.runtimes to an invalid string. In the UI all runtimes were disabled and nothing appeared to be broken in an obvious way. The log indicates that the three 'built-in' runtimes are disabled:

[I 2023-02-28 14:23:03.462 ElyraApp] Although runtime 'airflow' is installed, it is not in the set of runtimes configured via '--PipelineProcessorRegistry.runtimes' and will not be available.
[I 2023-02-28 14:23:03.935 ElyraApp] Although runtime 'kfp' is installed, it is not in the set of runtimes configured via '--PipelineProcessorRegistry.runtimes' and will not be available.
[I 2023-02-28 14:23:03.996 ElyraApp] Although runtime 'local' is installed, it is not in the set of runtimes configured via '--PipelineProcessorRegistry.runtimes' and will not be available.

What it doesn't document though in any message is what PipelineProcessorRegistry.runtimes was set to, even though that would be helpful.

[ptitzler]

A nit: it appears that the commands are still listed for all runtimes and component catalogs even when the 'associated' runtime is disabled:

(Clicking any of the commands does nothing obvious, so there doesn't seem to be any harm.)

[kevin-bates]

A nit: it appears that the commands are still listed for all runtimes and component catalogs even when the 'associated' runtime is disabled:

This may be addressed in my latest commit.

What it doesn't document though in any message is what PipelineProcessorRegistry.runtimes was set to, even though that would be helpful.

I wonder if instead of displaying ...

[I 2023-02-28 14:23:03.462 ElyraApp] Although runtime 'airflow' is installed, it is not in the set of runtimes configured via '--PipelineProcessorRegistry.runtimes' and will not be available.

we include the configured runtimes in the message? Something like this...

[I 2023-02-28 14:23:03.462 ElyraApp] Although runtime 'airflow' is installed, it is not in the set of configured runtimes {'kfp', 'local'} and will not be available.

[ptitzler]

What it doesn't document though in any message is what PipelineProcessorRegistry.runtimes was set to, even though that would be helpful.

I wonder if instead of displaying ...

[I 2023-02-28 14:23:03.462 ElyraApp] Although runtime 'airflow' is installed, it is not in the set of runtimes configured via '--PipelineProcessorRegistry.runtimes' and will not be available.

we include the configured runtimes in the message? Something like this...

[I 2023-02-28 14:23:03.462 ElyraApp] Although runtime 'airflow' is installed, it is not in the set of configured runtimes {'kfp', 'local'} and will not be available.

I like it because it combines multiple pieces of troubleshooting information! I believe in other places we use [] instead of {} to denote 'sets'. Should we do the same here for consistency?

[ptitzler]

A nit: it appears that the commands are still listed for all runtimes and component catalogs even when the 'associated' runtime is disabled:

This may be addressed in my latest commit.

Unfortunately not

[lresende]

LGTM, overall is working fine. One quick question, I built Elyra using make install-all and I can't see runtime-specific components like those described in https://elyra.readthedocs.io/en/stable/user_guide/pipeline-components.html#example-custom-components but this is the same behavior from main, so it should be unrelated.

[ptitzler]

One quick question, I built Elyra using make install-all and I can't see runtime-specific components like those described in https://elyra.readthedocs.io/en/stable/user_guide/pipeline-components.html#example-custom-components but this is the same behavior from main, so it should be unrelated.

Maybe re-raise this to question in one of our community channels?

[ptitzler]

LGTM! Thank you! Great feature.