Add Kubernetes/Kueue service backend

[cortadocodes]

Summary

This release adds preliminary support for the Kubernetes/Kueue service backend. It also adds major improvements to the octue CLI, allowing much smoother interaction with Twined services. Cloud Run support and python3.8/3.9 support is dropped.

Contents (#709)

IMPORTANT: There are 4 breaking changes.

New features

• Add new CLI commands:
  • octue question ask remote
  • octue question ask local
  • octue question events get
  • octue question events replay
• Add cancel method to Service and Child
• Support requesting compute resource amounts per question (memory, CPU, disk space)

Enhancements

• 💥 BREAKING CHANGE: Change default Octue services topic name to main.octue.services
• Allow setting of Octue services topic name via OCTUE_SERVICES_TOPIC_NAME environment variable
• Deprecate octue get-diagnostics CLI command in favour of octue question diagnostics
• Log when question event extraction is complete
• Log question acknowledgement to pub/sub
• Remove cloud run runtime warning
• Add GOOGLE_KUEUE to compute providers and remove old ones
• Add log message explaining waiting for question acceptance
• Return result from Service.answer
• Log warning if no events or result found in EventReplayer and get_events

Fixes

• Suppress google crc32c warning
• Wait for event emission futures by default

Operations

• Remove terraform config
• Remove unused dockerfile and docker build workflow

Dependencies

• Drop support for python3.8 and python3.9
• Update numpy and pandas dev dependencies

Refactoring

• 💥 BREAKING CHANGE: Remove Cloud Run support
• 💥 BREAKING CHANGE: Remove outdated octue run CLI command
• 💥 BREAKING CHANGE: Remove push subscription creation logic
• Move dockerfiles into their own directory
• Move answer_pub_sub_question module into events subpackage and rename to answer_question
• Split pub/sub event/attributes extraction into two functions
• Move attributes extraction into events subpackage
• Factor out getting local octue version
• Allow create_sruid to create default SRUIDs

Testing

• Update test for python3.13

Style

• Add missing ruff isort pre-commit check

---

Upgrade instructions

💥 Change default Octue services topic name to main.octue.services

Update the services topic in production to be named main.octue.services or set the environment variable OCTUE_SERVICES_TOPIC_NAME=octue.services

💥 Remove outdated `octue run` CLI command

Use the new octue question ask local CLI command instead.

💥 Remove Cloud Run support

• Use the Twined terraform module to set up your service network to use Kubernetes/Kueue instead of Cloud Run
• Build and push docker images for your services to the new artifact registry created by the terraform module
• Update all services in one upgrade

💥 Remove push subscription creation logic

This command is no longer needed with Kueue. Use the new service deployment reusable workflow from octue/workflows to deploy your services