Airflow version 2.10.3 upgrade and code improvements to resolve sonar findings

[ramesh-maddegoda]

🗒️ Summary

This pull request upgrades the Airflow version of Nucleus to version 2.10.3. In future, we plan to upgrade Airflow version to 3.x. This version 2.10.3 upgrade is conducted to make the transition from Airflow 2.x to Airflow 3.x a smooth transition in future. Also, this pull request introduces some code improvements made to address sonar findings related with adding logs buckets for S3 buckets. In addition, this adds parameter validation for Airflow S3 backlog processing DAG parameters.

⚙️ Test Data and/or Report

• Screenshot showing the new Airflow version 2.10.3

• Screenshot showing the parameter validation for Airflow S3 backlog processing DAG parameters

♻️ Related Issues

Upgrade Nucleus Airflow version to 2.10.3 #150

[jordanpadams]

@ramesh-maddegoda it looks like the secrets.baseline is still not quite right. can you try regenerating again?

https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Git-and-Github-Guide#detect-secrets

[ramesh-maddegoda]

@ramesh-maddegoda it looks like the secrets.baseline is still not quite right. can you try regenerating again?

https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Git-and-Github-Guide#detect-secrets

@jordanpadams , I used the following command for secret detection and pushes it again.

detect-secrets scan --disable-plugin AbsolutePathDetectorExperimental \
    --exclude-files '\.secrets\..*' \
    --exclude-files '\.git.*' \
    --exclude-files '\.pre-commit-config\.yaml' \
    --exclude-files '\.mypy_cache' \
    --exclude-files '\.pytest_cache' \
    --exclude-files '\.tox' \
    --exclude-files '\.venv' \
    --exclude-files 'venv' \
    --exclude-files 'dist' \
    --exclude-files 'build' \
    --exclude-files '.*\.egg-info' \
    --exclude-files '.*\.tfstate' \
    --exclude-files '.*\.tfvars' \
    > .secrets.baseline

detect-secrets audit .secrets.baseline

[jordanpadams]

@ramesh-maddegoda I would use this in the future to ensure it matches what is in the build:

https://github.com/NASA-PDS/nucleus/blob/main/.github/workflows/secrets-detection.yaml#L47

If that is not what you expect, please update that YAML file so whatever you are running matches what the build is doing.

[jordanpadams]

@sjoshi-jpl can you please review this?

[jordanpadams]

@ramesh-maddegoda can you review the security hotspot and triage appropriately?

[ramesh-maddegoda]

@ramesh-maddegoda can you review the security hotspot and triage appropriately?

@jordanpadams all security hotspots are resolved now.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud