feat: Add pilot logging (+2 squashed commits)

Squashed commits:
[bf5618f] feat: Add pilot logging

Squashed commits:
[490e293] feat: Logging is working, and we can search through the data
[09ddffa] fix: Lots of fixes to make it work
[b4e3d12] [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci
[f121d21] fix: remove unused code
[6f1cb7f] feat: move bulk_insert to logging.py
[ea6ecdd] feat: refactor pilot logging to include a business logic layer, rebase and fix _get_columns
[15338d5] feat: refactor pilot logging to include a business logic layer
[8fe6d31] fix: rename _get_columns to get_columns
[9ee07db] fix: remove unused PilotNotFoundError exception
[cf61acb] feat: Restructured pilot logger + test send_message and get_logs
[9800d11] feat: enable remote pilot logging system
[720f302] fix: Add security by preventing a user to access a pilot endpoint and vice-versa
[bad03be] fix: Changed the pilot token from a User to a dedicated
[f71da78] fix: Split in two pilot route
[3feaa56] fix: Add security by preventing to see other pilots
[306941f] fix: Fix, bug not caught by CI (sqlite not raising anything)
[cc8de96] fix: Change secrets from VARCHAR to BINARY
[17ce4e3] feat: Change from SecretID to SecretUUID
[5760830] fix: Generating the client
[43cfccc] feat: Migrating pilot functionalities from DIRAC, and add DiracX pilot auth

Squashed commits:
[33cfe17] test: Changing from sleep to freezegun #537
[3e5d677] refactor: Refactorized the exceptions
[d84f6fe] doc: Documenting the database
[1c0db02] test: Better testing: Tested search engine
[cd1caf4] refactor: Refactoring from the auth router to the pilot router
[9cc99f1] feat: Adding indexes to the database
[de0b580] refactor: Moved the pilot_to_secret_mapping into the pilotagents table itself
[14396f6] refactoring: Lots of refactoring of the code
[56daac5] refactor: Moving db parts to the logic, and some fixes
[79a11cf] fix: Fixed dependency and client
[8c45582] test: Tested interval deletion for the db.
[dd07bca] feat: We can clear pilots that lived more than n days
[7e93153] feat: We can search for pilots
[9da8033] feat: DIRAC can associate a pilot with a job
[77a59b0] feat: We can modify pilot fields (statuses, benchmarks, ...)
[ca0e727] fix: Added pilot management
[521164f] fix: Fixed text
[8e6e881] [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci
[adadb68] fix: Semantic
[15d9fe6] feat: We can add a number of use max for the secret in pilot registration
[4549203] fix: Fixed cli
[a9ece0a] feat: Adding a command to add a pilot to the CS, and some fixes
[42e4f0b] fix: Changed secret expiration from 600 to 3600s
[1dcb4b8] fix: Regenerated client
[02a5e31] feat: A user can create secrets, and associate them to a pilot
[8072227] feat: Ability to add only secrets, and lots of refactoring in the tests
[10205d6] feat: Autodeletion of secrets after full use or expiration.
[3f013df] feat: Restricting a secret for a certain VO
[c2181aa] fix: Chunky client
[d374ec6] feat: Changing the pilot login db, and moved from references to stamps
[aa67bda] fix: Regenerating client
[135235f] feat: Using pilot user, model for pilot creds, and documentation
[0e0a117] feat: Added last_time_used for a pilot secret, and a max value
[62977ee] fix: Few fixes with refactoring, and tests (+4 squashed commits)
Squashed commits:
[5169ee2] refactor: Moved login for pilots into token file
[12da5c0] fix: Fixed according the the new commits from main branch
[a8e06d8] fix: Fixed client
[93b83b4] fix: Fixed the tests and autorest
[a78c028] feat: A user can register pilots and get their credentials
[1e9f907] fix: Removing the pilot login cli
[3f7c632] feat: Adding duration to secrets
[8aff5d9] fix: Regenerating client, type ignore for the cli
Squashed commits:
[06a969b] fix: Type ignore for mypy...
[28e1ee1] fix: Generated code broke everything
[339ab4f] fix: Fixed cli and pilot endpoints
[31af99e] fix: Regenerated client
[a171d01] feat: Added pilot login to the cli
[54951a6] fix: Correcting client generation with the error patch
Squashed commits:
[65db1ae] fix: Regenerating clients with autorest
[5a4f370] feat: Better handling of refresh tokens for pilots
[6da7bb4] fix: Generating client with autorest
[2c4d2fb] feat: Adding pilot registrations
[a957480] feat: Adding pilot registrations, with secret management

Author: Robin-Van-de-Merghel

diracx-cli/src/diracx/cli/internal/config.py

@@ -59,7 +59,7 @@ def generate_cs(config_repo: str):
     update_config_and_commit(
         repo_path=repo_path, config=config, message="Initial commit"
     )
-    typer.echo(f"Successfully created repo in {config_repo}", err=True)
+    typer.echo(f"Successfully created repo in {config_repo}")
 
 
 @app.command()
@@ -99,7 +99,7 @@ def add_vo(
         config=config,
         message=f"Added vo {vo} registry (default group {default_group} and idp {idp_url})",
     )
-    typer.echo(f"Successfully added vo to {config_repo}", err=True)
+    typer.echo(f"Successfully added vo to {config_repo}")
 
 
 @app.command()
@@ -130,7 +130,7 @@ def add_group(
     update_config_and_commit(
         repo_path=repo_path, config=config, message=f"Added group {group} in {vo}"
     )
-    typer.echo(f"Successfully added group to {config_repo}", err=True)
+    typer.echo(f"Successfully added group to {config_repo}")
 
 
 @app.command()
@@ -177,7 +177,7 @@ def add_user(
         config=config,
         message=f"Added user {sub} ({preferred_username}) to vo {vo} and groups {groups}",
     )
-    typer.echo(f"Successfully added user to {config_repo}", err=True)
+    typer.echo(f"Successfully added user to {config_repo}")
 
 
 def update_config_and_commit(repo_path: Path, config: Config, message: str):

diracx-cli/tests/test_internal.py

@@ -222,3 +222,64 @@ def test_add_user(cs_repo, vo, user_group):
     assert sub in config.Registry[vo].Users
     for group in user_group or [TEST_USER_GROUP]:
         assert config.Registry[vo].Groups[group].Users == {sub}
+
+
+def test_add_pilot_user(cs_repo):
+
+    sub = "lhcb:rvandeme"
+
+    pilot_group = "pilot_group"
+    pilot_user = "pilot_user"
+
+    config = ConfigSource.create_from_url(backend_url=cs_repo).read_config()
+
+    # Add a second group to it
+    result = runner.invoke(
+        app,
+        [
+            "internal",
+            "add-group",
+            cs_repo,
+            f"--vo={TEST_VO}",
+            f"--group={pilot_group}",
+            "--properties",
+            "GenericPilot",
+        ],
+    )
+    assert result.exit_code == 0, result.output
+
+    # Add a user to it
+    result = runner.invoke(
+        app,
+        [
+            "internal",
+            "add-user",
+            cs_repo,
+            f"--vo={TEST_VO}",
+            f"--sub={sub}",
+            f"--preferred-username={pilot_user}",
+            f"--group={pilot_group}",
+        ],
+    )
+    assert result.exit_code == 0, result.output
+
+    # Set this user as a pilot
+    result = runner.invoke(
+        app,
+        [
+            "internal",
+            "set-user-as-pilot-user",
+            cs_repo,
+            f"--vo={TEST_VO}",
+            f"--sub={sub}",
+            f"--pilot-preferred-username={pilot_user}",
+            f"--pilot-group={pilot_group}",
+        ],
+    )
+    assert result.exit_code == 0, result.output
+
+    config = ConfigSource.create_from_url(backend_url=cs_repo).read_config()
+
+    # User pilot exists, same for pilot group
+    assert config.Operations[TEST_VO].Pilot["GenericPilotUser"] == pilot_user
+    assert config.Operations[TEST_VO].Pilot["GenericPilotGroup"] == pilot_group

diracx-client/src/diracx/client/_generated/_client.py

@@ -19,6 +19,8 @@
     AuthOperations,
     ConfigOperations,
     JobsOperations,
+    PilotOperations,
+    PilotsOperations,
     WellKnownOperations,
 )
 
@@ -34,6 +36,10 @@ class Dirac:  # pylint: disable=client-accepts-api-version-keyword
     :vartype config: _generated.operations.ConfigOperations
     :ivar jobs: JobsOperations operations
     :vartype jobs: _generated.operations.JobsOperations
+    :ivar pilot: PilotOperations operations
+    :vartype pilot: _generated.operations.PilotOperations
+    :ivar pilots: PilotsOperations operations
+    :vartype pilots: _generated.operations.PilotsOperations
     :keyword endpoint: Service URL. Required. Default value is "".
     :paramtype endpoint: str
     """
@@ -86,6 +92,12 @@ def __init__(  # pylint: disable=missing-client-constructor-parameter-credential
         self.jobs = JobsOperations(
             self._client, self._config, self._serialize, self._deserialize
         )
+        self.pilot = PilotOperations(
+            self._client, self._config, self._serialize, self._deserialize
+        )
+        self.pilots = PilotsOperations(
+            self._client, self._config, self._serialize, self._deserialize
+        )
 
     def send_request(
         self, request: HttpRequest, *, stream: bool = False, **kwargs: Any

diracx-client/src/diracx/client/_generated/aio/_client.py

@@ -19,6 +19,8 @@
     AuthOperations,
     ConfigOperations,
     JobsOperations,
+    PilotOperations,
+    PilotsOperations,
     WellKnownOperations,
 )
 
@@ -34,6 +36,10 @@ class Dirac:  # pylint: disable=client-accepts-api-version-keyword
     :vartype config: _generated.aio.operations.ConfigOperations
     :ivar jobs: JobsOperations operations
     :vartype jobs: _generated.aio.operations.JobsOperations
+    :ivar pilot: PilotOperations operations
+    :vartype pilot: _generated.aio.operations.PilotOperations
+    :ivar pilots: PilotsOperations operations
+    :vartype pilots: _generated.aio.operations.PilotsOperations
     :keyword endpoint: Service URL. Required. Default value is "".
     :paramtype endpoint: str
     """
@@ -86,6 +92,12 @@ def __init__(  # pylint: disable=missing-client-constructor-parameter-credential
         self.jobs = JobsOperations(
             self._client, self._config, self._serialize, self._deserialize
         )
+        self.pilot = PilotOperations(
+            self._client, self._config, self._serialize, self._deserialize
+        )
+        self.pilots = PilotsOperations(
+            self._client, self._config, self._serialize, self._deserialize
+        )
 
     def send_request(
         self, request: HttpRequest, *, stream: bool = False, **kwargs: Any

diracx-client/src/diracx/client/_generated/aio/operations/__init__.py

@@ -14,6 +14,8 @@
 from ._operations import AuthOperations  # type: ignore
 from ._operations import ConfigOperations  # type: ignore
 from ._operations import JobsOperations  # type: ignore
+from ._operations import PilotOperations  # type: ignore
+from ._operations import PilotsOperations  # type: ignore
 
 from ._patch import __all__ as _patch_all
 from ._patch import *
@@ -24,6 +26,8 @@
     "AuthOperations",
     "ConfigOperations",
     "JobsOperations",
+    "PilotOperations",
+    "PilotsOperations",
 ]
 __all__.extend([p for p in _patch_all if p not in __all__])  # pyright: ignore
 _patch_sdk()