Fix Airflow operation processing for number data types

[kiersten-stokes]

In testing #2799, I came across this issue where number-type component parameter values are being rendered by the DAG as follows because these types were being skipped during processing:

int_default_non_zero= {
  "activeControl": "StringControl",
  "NumberControl": 2,
},

After these changes, the same property appears as it should:

int_default_non_zero=2,

This is not an issue for KFP.

What changes were proposed in this pull request?

A simple 2-line change takes care of handling number-type parameter values (which was erroneously being skipped prior to this). The bulk of the changes here are test-related (see below).

How was this pull request tested?

Previously, no test existed that checked for how number-type values were being processed by Airflow. A case for that is added here to an existing test (test_create_file_custom_components). To support this test, I've added a new catalog instance (AIRFLOW_TEST_OPERATOR_CATALOG) to conftest.py that loads the TestOperator defined in tests/pipeline/resources/components/airflow_test_operator.py, which includes a component parameter for each supported data type. I've also updated the sample pipeline used during this test to include the TestOperator instead of the BashOperator.

Related, I used the same strategy to update test_same_name_operator_in_pipeline (also in test_airflow_processor.py). This test now uses the AIRFLOW_TEST_OPERATOR_CATALOG that includes the TestOperator and the test pipeline resource is similarly updated to use TestOperator parameters. While not strictly necessary, I think with only a few more similar changes, we can completely remove the BashOperator from our test resources.

I've now removed the remaining references to the BashOperator in the following tests:

• test_parse_airflow_component_url in test_component_parser_airflow.py] now fetches and parses the TestOperator file
• references to the BashOperator in the validation pipeline aa_invalid_node_op.pipeline have been replaced with TestOperator references (the values and operator-specific details are never interrogated)
• test_pipeline_aa_parent_node_missing_xcom_push now uses the new AIRFLOW_TEST_OPERATOR_CATALOG and references the TestOperator rather than the BashOperator

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[elyra-bot]

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link:

[ptitzler]

LGTM!