Update to pydantic 2

[ohrite]

Description

This PR bumps Pydantic to 2.x, and uses the v1 compatibility layer in places where the code is used today.

Note: This PR will impact both staging and production, the branch name only signifies that it's continuously deployed to staging

Resolves #4465

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

How has this been tested?

pytest for the cal-itp-data-infra package

Post-merge follow-ups

• No action required
• Actions required (specified below)

Exercise all dags

[github-actions]

Terraform plan in iac/cal-itp-data-infra-staging/composer/us

No changes. Your infrastructure matches the configuration.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1069

[github-actions]

Terraform plan in iac/cal-itp-data-infra/composer/us

Plan: 0 to add, 1 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_composer_environment.calitp-composer will be updated in-place
!~  resource "google_composer_environment" "calitp-composer" {
        id               = "projects/cal-itp-data-infra/locations/us-west2/environments/calitp-composer"
        name             = "calitp-composer"
#        (5 unchanged attributes hidden)

!~      config {
#            (8 unchanged attributes hidden)

!~          software_config {
!~              pypi_packages            = {
-                   "calitp-data-infra"      = "==2025.6.5" -> null
-                   "pydantic"               = ">=1.9,<2.0" -> null
#                    (11 unchanged elements hidden)
                }
#                (6 unchanged attributes hidden)

#                (1 unchanged block hidden)
            }

#            (8 unchanged blocks hidden)
        }

#        (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1069

[github-actions]

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 0 to add, 2 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "76FCcA==" -> (known after apply)
!~      detect_md5hash      = "0oKfay3/M7Naeex1c0UF6w==" -> "different hash"
!~      generation          = 1763690418237335 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "0oKfay3/M7Naeex1c0UF6w==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "siuP3A==" -> (known after apply)
!~      detect_md5hash      = "4ljyonNG9/incebad8A0JA==" -> "different hash"
!~      generation          = 1763690419724927 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "4ljyonNG9/incebad8A0JA==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1069

[github-actions]

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 3 to add, 17 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer["dags/airtable_loader_v2/generate_gtfs_download_configs.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "LLSRKg==" -> (known after apply)
!~      detect_md5hash      = "WEFwfVbJm4J6LdTF0abD2A==" -> "different hash"
!~      generation          = 1751416672748889 -> (known after apply)
        id                  = "calitp-composer-dags/airtable_loader_v2/generate_gtfs_download_configs.py"
!~      md5hash             = "WEFwfVbJm4J6LdTF0abD2A==" -> (known after apply)
        name                = "dags/airtable_loader_v2/generate_gtfs_download_configs.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["dags/download_gtfs_schedule_v2/download_schedule_feeds.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "LW/cjQ==" -> (known after apply)
!~      detect_md5hash      = "v4OSHRGL1Pqn5UkuoYrJ+g==" -> "different hash"
!~      generation          = 1762306517423010 -> (known after apply)
        id                  = "calitp-composer-dags/download_gtfs_schedule_v2/download_schedule_feeds.py"
!~      md5hash             = "v4OSHRGL1Pqn5UkuoYrJ+g==" -> (known after apply)
        name                = "dags/download_gtfs_schedule_v2/download_schedule_feeds.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["dags/sync_ntd_data_xlsx/scrape_ntd_xlsx_urls.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "PtTDYA==" -> (known after apply)
!~      detect_md5hash      = "MD9Xa0febR52D/2WC0YF7w==" -> "different hash"
!~      generation          = 1762556093238687 -> (known after apply)
        id                  = "calitp-composer-dags/sync_ntd_data_xlsx/scrape_ntd_xlsx_urls.py"
!~      md5hash             = "MD9Xa0febR52D/2WC0YF7w==" -> (known after apply)
        name                = "dags/sync_ntd_data_xlsx/scrape_ntd_xlsx_urls.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/calitp_data_infra/__init__.py"] will be created
+   resource "google_storage_bucket_object" "calitp-composer" {
+       bucket         = "calitp-composer"
+       content        = (sensitive value)
+       content_type   = (known after apply)
+       crc32c         = (known after apply)
+       detect_md5hash = "different hash"
+       generation     = (known after apply)
+       id             = (known after apply)
+       kms_key_name   = (known after apply)
+       md5hash        = (known after apply)
+       md5hexhash     = (known after apply)
+       media_link     = (known after apply)
+       name           = "plugins/calitp_data_infra/__init__.py"
+       output_name    = (known after apply)
+       self_link      = (known after apply)
+       source         = "../../../../airflow/plugins/calitp_data_infra/__init__.py"
+       storage_class  = (known after apply)
    }

  # google_storage_bucket_object.calitp-composer["plugins/calitp_data_infra/auth.py"] will be created
+   resource "google_storage_bucket_object" "calitp-composer" {
+       bucket         = "calitp-composer"
+       content        = (sensitive value)
+       content_type   = (known after apply)
+       crc32c         = (known after apply)
+       detect_md5hash = "different hash"
+       generation     = (known after apply)
+       id             = (known after apply)
+       kms_key_name   = (known after apply)
+       md5hash        = (known after apply)
+       md5hexhash     = (known after apply)
+       media_link     = (known after apply)
+       name           = "plugins/calitp_data_infra/auth.py"
+       output_name    = (known after apply)
+       self_link      = (known after apply)
+       source         = "../../../../airflow/plugins/calitp_data_infra/auth.py"
+       storage_class  = (known after apply)
    }

  # google_storage_bucket_object.calitp-composer["plugins/calitp_data_infra/storage.py"] will be created
+   resource "google_storage_bucket_object" "calitp-composer" {
+       bucket         = "calitp-composer"
+       content        = (sensitive value)
+       content_type   = (known after apply)
+       crc32c         = (known after apply)
+       detect_md5hash = "different hash"
+       generation     = (known after apply)
+       id             = (known after apply)
+       kms_key_name   = (known after apply)
+       md5hash        = (known after apply)
+       md5hexhash     = (known after apply)
+       media_link     = (known after apply)
+       name           = "plugins/calitp_data_infra/storage.py"
+       output_name    = (known after apply)
+       self_link      = (known after apply)
+       source         = "../../../../airflow/plugins/calitp_data_infra/storage.py"
+       storage_class  = (known after apply)
    }

  # google_storage_bucket_object.calitp-composer["plugins/hooks/kuba_hook.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "VUbcdA==" -> (known after apply)
!~      detect_md5hash      = "GV4l5STvYmCTonhpJ18EVg==" -> "different hash"
!~      generation          = 1752606164136557 -> (known after apply)
        id                  = "calitp-composer-plugins/hooks/kuba_hook.py"
!~      md5hash             = "GV4l5STvYmCTonhpJ18EVg==" -> (known after apply)
        name                = "plugins/hooks/kuba_hook.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/hooks/soda_hook.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "AEF4Cg==" -> (known after apply)
!~      detect_md5hash      = "/pG1r4pecHAscgHnGeY6MQ==" -> "different hash"
!~      generation          = 1758764038715230 -> (known after apply)
        id                  = "calitp-composer-plugins/hooks/soda_hook.py"
!~      md5hash             = "/pG1r4pecHAscgHnGeY6MQ==" -> (known after apply)
        name                = "plugins/hooks/soda_hook.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/hooks/transitland_hook.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "gChsLg==" -> (known after apply)
!~      detect_md5hash      = "xw7oV/UggwUVQ0cObCSmdg==" -> "different hash"
!~      generation          = 1752680601135618 -> (known after apply)
        id                  = "calitp-composer-plugins/hooks/transitland_hook.py"
!~      md5hash             = "xw7oV/UggwUVQ0cObCSmdg==" -> (known after apply)
        name                = "plugins/hooks/transitland_hook.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/operators/aggregator_to_gcs_operator.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "aJy3Ng==" -> (known after apply)
!~      detect_md5hash      = "Mm5EDhD5X0zxmfWG+5jJoA==" -> "different hash"
!~      generation          = 1752680601125015 -> (known after apply)
        id                  = "calitp-composer-plugins/operators/aggregator_to_gcs_operator.py"
!~      md5hash             = "Mm5EDhD5X0zxmfWG+5jJoA==" -> (known after apply)
        name                = "plugins/operators/aggregator_to_gcs_operator.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/operators/blackcat_to_gcs_operator.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "USagog==" -> (known after apply)
!~      detect_md5hash      = "O8kGhwLVle8UclxejwT6UQ==" -> "different hash"
!~      generation          = 1752009255099585 -> (known after apply)
        id                  = "calitp-composer-plugins/operators/blackcat_to_gcs_operator.py"
!~      md5hash             = "O8kGhwLVle8UclxejwT6UQ==" -> (known after apply)
        name                = "plugins/operators/blackcat_to_gcs_operator.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/operators/dbt_manifest_to_dictionary_operator.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "bQa2fw==" -> (known after apply)
!~      detect_md5hash      = "dQSSAJQ1XwwivACFbt1IQA==" -> "different hash"
!~      generation          = 1759879834012514 -> (known after apply)
        id                  = "calitp-composer-plugins/operators/dbt_manifest_to_dictionary_operator.py"
!~      md5hash             = "dQSSAJQ1XwwivACFbt1IQA==" -> (known after apply)
        name                = "plugins/operators/dbt_manifest_to_dictionary_operator.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/operators/dbt_manifest_to_metadata_operator.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "WdYbdQ==" -> (known after apply)
!~      detect_md5hash      = "6YJokkpPp3C3zbJVHFBRww==" -> "different hash"
!~      generation          = 1759879834025172 -> (known after apply)
        id                  = "calitp-composer-plugins/operators/dbt_manifest_to_metadata_operator.py"
!~      md5hash             = "6YJokkpPp3C3zbJVHFBRww==" -> (known after apply)
        name                = "plugins/operators/dbt_manifest_to_metadata_operator.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/operators/gtfs_csv_to_jsonl_hourly.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "xHIcRQ==" -> (known after apply)
!~      detect_md5hash      = "yPBX3xV1KjOxKuyScsKoSg==" -> "different hash"
!~      generation          = 1751416674229211 -> (known after apply)
        id                  = "calitp-composer-plugins/operators/gtfs_csv_to_jsonl_hourly.py"
!~      md5hash             = "yPBX3xV1KjOxKuyScsKoSg==" -> (known after apply)
        name                = "plugins/operators/gtfs_csv_to_jsonl_hourly.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/operators/littlepay_raw_sync_feed_v3.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "SUn8gQ==" -> (known after apply)
!~      detect_md5hash      = "2naYzWkGG8DDy2GEHVMuYw==" -> "different hash"
!~      generation          = 1751416671285972 -> (known after apply)
        id                  = "calitp-composer-plugins/operators/littlepay_raw_sync_feed_v3.py"
!~      md5hash             = "2naYzWkGG8DDy2GEHVMuYw==" -> (known after apply)
        name                = "plugins/operators/littlepay_raw_sync_feed_v3.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/operators/pod_operator.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "S7zO3A==" -> (known after apply)
!~      detect_md5hash      = "vjdngPhJTlpS6oHhbAfB3w==" -> "different hash"
!~      generation          = 1751572699760371 -> (known after apply)
        id                  = "calitp-composer-plugins/operators/pod_operator.py"
!~      md5hash             = "vjdngPhJTlpS6oHhbAfB3w==" -> (known after apply)
        name                = "plugins/operators/pod_operator.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/operators/scrape_ntd_xlsx.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "1vdq2w==" -> (known after apply)
!~      detect_md5hash      = "JswrPXbC0ODbp33GpR/6Fw==" -> "different hash"
!~      generation          = 1762556093245704 -> (known after apply)
        id                  = "calitp-composer-plugins/operators/scrape_ntd_xlsx.py"
!~      md5hash             = "JswrPXbC0ODbp33GpR/6Fw==" -> (known after apply)
        name                = "plugins/operators/scrape_ntd_xlsx.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/operators/scrape_state_geoportal.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "3pMECQ==" -> (known after apply)
!~      detect_md5hash      = "qwOK0bYTQ/9mzvdFyKCNGQ==" -> "different hash"
!~      generation          = 1751416675477427 -> (known after apply)
        id                  = "calitp-composer-plugins/operators/scrape_state_geoportal.py"
!~      md5hash             = "qwOK0bYTQ/9mzvdFyKCNGQ==" -> (known after apply)
        name                = "plugins/operators/scrape_state_geoportal.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/scripts/gtfs_rt_parser.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "FRbe4g==" -> (known after apply)
!~      detect_md5hash      = "91uOGxqj/8EG+cTaySO32w==" -> "different hash"
!~      generation          = 1763693339648841 -> (known after apply)
        id                  = "calitp-composer-plugins/scripts/gtfs_rt_parser.py"
!~      md5hash             = "91uOGxqj/8EG+cTaySO32w==" -> (known after apply)
        name                = "plugins/scripts/gtfs_rt_parser.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["plugins/utils.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "ZshQSQ==" -> (known after apply)
!~      detect_md5hash      = "7UdL9NZ+pHtC/CmwXFEL0g==" -> "different hash"
!~      generation          = 1751416670997258 -> (known after apply)
        id                  = "calitp-composer-plugins/utils.py"
!~      md5hash             = "7UdL9NZ+pHtC/CmwXFEL0g==" -> (known after apply)
        name                = "plugins/utils.py"
#        (17 unchanged attributes hidden)
    }

Plan: 3 to add, 17 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1069