-
Notifications
You must be signed in to change notification settings - Fork 14.2k
feat(caretaker): implement Cloud Run webhook ingestion service #28015
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
chadd28
wants to merge
11
commits into
google-gemini:main
Choose a base branch
from
chadd28:feature/caretaker-agent
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
8d23977
chore(ingestion): initialize project setup and environment configuration
chadd28 c9cbd86
feat(ingestion): implement Firestore IssuesStore for new documents
chadd28 2d09842
feat(ingestion): add GitHub webhook HMAC signature verification
chadd28 a6ef7a9
feat(ingestion): create Express server and webhook ingestion route
chadd28 40c59c1
test(ingestion): add unit tests for GitHub webhook signature verifica…
chadd28 a8a0fe8
feat(ingestion): skip Pub/Sub publishing for duplicate issues
chadd28 5650049
fix(ingestion): secure GitHub signature verification against DoS and …
chadd28 7c2a01b
fix(ingestion): Improve build, config, and code style
chadd28 afa05a5
fix(ingestion): resolve webhook robustness and security issues in ing…
chadd28 812b530
refactor(ingestion): split ingestion service into app and server modules
chadd28 05334cc
test(ingestion): Add unit tests for webhook endpoint
chadd28 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
12 changes: 12 additions & 0 deletions
12
tools/caretaker-agent/cloudrun/ingestion-service/.dockerignore
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| node_modules | ||
| dist | ||
| npm-debug.log | ||
| .git | ||
| .gitignore | ||
| *.py | ||
| *.pyc | ||
| __pycache__ | ||
| requirements.txt | ||
| project.toml | ||
| **/*.test.ts | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| FROM node:20-slim | ||
| WORKDIR /app | ||
| COPY package*.json ./ | ||
| RUN npm ci | ||
| COPY . . | ||
| EXPOSE 8080 | ||
| CMD ["npx", "tsx", "server.ts"] | ||
|
|
||
37 changes: 37 additions & 0 deletions
37
tools/caretaker-agent/cloudrun/ingestion-service/auth/github.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| /** | ||
| * @license | ||
| * Copyright 2026 Google LLC | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { describe, it, expect } from 'vitest'; | ||
| import { verifyGithubSignature } from './github.js'; | ||
| import * as crypto from 'node:crypto'; | ||
|
|
||
| describe('verifyGithubSignature', () => { | ||
| const secret = 'my-secret'; | ||
| const payload = '{"test":true}'; | ||
|
|
||
| it('should return true for a valid signature', () => { | ||
| const hmac = crypto.createHmac('sha256', secret); | ||
| hmac.update(payload); | ||
| const validSignature = 'sha256=' + hmac.digest('hex'); | ||
|
|
||
| const result = verifyGithubSignature(payload, validSignature, secret); | ||
| expect(result).toBe(true); | ||
| }); | ||
|
|
||
| it('should return false if signatureHeader is missing', () => { | ||
| const result = verifyGithubSignature(payload, undefined, secret); | ||
| expect(result).toBe(false); | ||
| }); | ||
|
|
||
| it('should return false for an invalid signature', () => { | ||
| const result = verifyGithubSignature( | ||
| payload, | ||
| 'sha256=invalid-signature', | ||
| secret, | ||
| ); | ||
| expect(result).toBe(false); | ||
| }); | ||
| }); |
38 changes: 38 additions & 0 deletions
38
tools/caretaker-agent/cloudrun/ingestion-service/auth/github.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| /** | ||
| * @license | ||
| * Copyright 2026 Google LLC | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import * as crypto from 'node:crypto'; | ||
|
|
||
| /** | ||
| * Verify that the payload was sent from GitHub using HMAC SHA256. | ||
| * | ||
| * @param payloadBody - The raw body of the request (Buffer or string). | ||
| * @param signatureHeader - The value of the X-Hub-Signature-256 header. | ||
| * @param secret - The GitHub Webhook secret. | ||
| * @returns True if the signature is valid, false otherwise. | ||
| */ | ||
| export function verifyGithubSignature( | ||
| payloadBody: Buffer | string, | ||
| signatureHeader: string | undefined, | ||
| secret: string, | ||
| ): boolean { | ||
| if (!signatureHeader) { | ||
| return false; | ||
| } | ||
|
|
||
| const hmac = crypto.createHmac('sha256', secret); | ||
| hmac.update(payloadBody); | ||
| const expectedSignature = 'sha256=' + hmac.digest('hex'); | ||
|
|
||
| try { | ||
| return crypto.timingSafeEqual( | ||
| Buffer.from(expectedSignature), | ||
| Buffer.from(signatureHeader), | ||
| ); | ||
| } catch { | ||
| return false; | ||
| } | ||
|
chadd28 marked this conversation as resolved.
Outdated
|
||
| } | ||
85 changes: 85 additions & 0 deletions
85
tools/caretaker-agent/cloudrun/ingestion-service/db/issuesStore.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| /** | ||
| * @license | ||
| * Copyright 2026 Google LLC | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { describe, it, expect, vi, beforeEach } from 'vitest'; | ||
| import type { Mock } from 'vitest'; | ||
| import { IssuesStore } from './issuesStore.js'; | ||
| import { Firestore, Transaction } from '@google-cloud/firestore'; | ||
|
|
||
| describe('IssuesStore', () => { | ||
| let mockTransaction: { | ||
| get: Mock; | ||
| set: Mock; | ||
| }; | ||
| let mockDb: Firestore; | ||
| let store: IssuesStore; | ||
|
|
||
| beforeEach(() => { | ||
| // Assign mock read/write methods for transaction | ||
| mockTransaction = { | ||
| get: vi.fn(), | ||
| set: vi.fn(), | ||
| }; | ||
|
|
||
| // Mock Firestore client | ||
| mockDb = { | ||
| collection: vi.fn().mockReturnThis(), | ||
| doc: vi.fn().mockReturnValue({}), | ||
| runTransaction: vi | ||
| .fn() | ||
| .mockImplementation( | ||
| (callback: (tx: Transaction) => Promise<unknown>) => { | ||
| return callback(mockTransaction as unknown as Transaction); | ||
| }, | ||
| ), | ||
| } as unknown as Firestore; | ||
|
|
||
| store = new IssuesStore(mockDb, 'issues-collection'); | ||
| }); | ||
|
|
||
| it('should initialize a new issue if it does not exist', async () => { | ||
| // The transaction should mock that the document does not exist | ||
| mockTransaction.get.mockResolvedValue({ exists: false }); | ||
|
|
||
| const result = await store.createIssue( | ||
| 'google', | ||
| 'gemini-cli', | ||
| 123, | ||
| 'Test Title', | ||
| ); | ||
|
|
||
| expect(result).toBe(true); | ||
| expect(mockTransaction.get).toHaveBeenCalled(); | ||
| expect(mockTransaction.set).toHaveBeenCalledWith( | ||
| expect.anything(), | ||
| expect.objectContaining({ | ||
| status: 'UNTRIAGED', | ||
| github_metadata: expect.objectContaining({ | ||
| owner: 'google', | ||
| repo: 'gemini-cli', | ||
| issue_number: 123, | ||
| title: 'Test Title', | ||
| }), | ||
| }), | ||
| ); | ||
| }); | ||
|
|
||
| it('should return false and skip creation if the issue already exists', async () => { | ||
| // The transaction should mock that the document already exists | ||
| mockTransaction.get.mockResolvedValue({ exists: true }); | ||
|
|
||
| const result = await store.createIssue( | ||
| 'google', | ||
| 'gemini-cli', | ||
| 123, | ||
| 'Test Title', | ||
| ); | ||
|
|
||
| expect(result).toBe(false); | ||
| expect(mockTransaction.get).toHaveBeenCalled(); | ||
| expect(mockTransaction.set).not.toHaveBeenCalled(); | ||
| }); | ||
| }); |
68 changes: 68 additions & 0 deletions
68
tools/caretaker-agent/cloudrun/ingestion-service/db/issuesStore.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,68 @@ | ||||||||
| /** | ||||||||
| * @license | ||||||||
| * Copyright 2026 Google LLC | ||||||||
| * SPDX-License-Identifier: Apache-2.0 | ||||||||
| */ | ||||||||
|
|
||||||||
| import { | ||||||||
| Firestore, | ||||||||
| FieldValue, | ||||||||
| DocumentReference, | ||||||||
| Transaction, | ||||||||
| } from '@google-cloud/firestore'; | ||||||||
|
|
||||||||
| export class IssuesStore { | ||||||||
| private db: Firestore; | ||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P3] Missing
Suggested change
|
||||||||
| private collectionName: string; | ||||||||
|
|
||||||||
| constructor(db: Firestore, collectionName: string) { | ||||||||
| this.db = db; | ||||||||
| this.collectionName = collectionName; | ||||||||
| } | ||||||||
|
|
||||||||
| // Generates the standardized Firestore document reference for an issue | ||||||||
| getIssueRef( | ||||||||
| owner: string, | ||||||||
| repo: string, | ||||||||
| issueNumber: number, | ||||||||
| ): DocumentReference { | ||||||||
| const docId = `github_${owner}_${repo}_${issueNumber}`; | ||||||||
| return this.db.collection(this.collectionName).doc(docId); | ||||||||
| } | ||||||||
|
|
||||||||
| // Initializes a new issue document in a transaction | ||||||||
| async createIssue( | ||||||||
| owner: string, | ||||||||
| repo: string, | ||||||||
| issueNumber: number, | ||||||||
| title: string, | ||||||||
| ): Promise<boolean> { | ||||||||
| const docRef = this.getIssueRef(owner, repo, issueNumber); | ||||||||
|
|
||||||||
| return this.db.runTransaction(async (transaction: Transaction) => { | ||||||||
| const snapshot = await transaction.get(docRef); | ||||||||
|
|
||||||||
| if (!snapshot.exists) { | ||||||||
| transaction.set(docRef, { | ||||||||
| status: 'UNTRIAGED', | ||||||||
| triage_attempts: 0, | ||||||||
| workable_spec: {}, | ||||||||
| lock: { | ||||||||
| holder: null, | ||||||||
| expires_at: null, | ||||||||
| }, | ||||||||
| created_at: FieldValue.serverTimestamp(), | ||||||||
| updated_at: FieldValue.serverTimestamp(), | ||||||||
| github_metadata: { | ||||||||
| owner: owner, | ||||||||
| repo: repo, | ||||||||
| issue_number: issueNumber, | ||||||||
| title: title, | ||||||||
| }, | ||||||||
| }); | ||||||||
| return true; | ||||||||
| } | ||||||||
| return false; | ||||||||
| }); | ||||||||
| } | ||||||||
| } | ||||||||
24 changes: 24 additions & 0 deletions
24
tools/caretaker-agent/cloudrun/ingestion-service/package.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| { | ||
| "name": "ingestion-service", | ||
| "version": "1.0.0", | ||
| "description": "Ingestion service for triage worker", | ||
| "main": "server.ts", | ||
| "scripts": { | ||
| "dev": "tsx watch server.ts", | ||
| "start": "tsx server.ts", | ||
| "test": "vitest run" | ||
| }, | ||
| "dependencies": { | ||
| "@google-cloud/firestore": "^7.7.0", | ||
| "@google-cloud/pubsub": "^4.4.0", | ||
| "dotenv": "^16.4.5", | ||
| "express": "^4.19.2" | ||
| }, | ||
| "devDependencies": { | ||
| "@types/express": "^4.17.21", | ||
| "@types/node": "^20.12.12", | ||
| "tsx": "^4.9.3", | ||
| "typescript": "^5.4.5", | ||
| "vitest": "^1.6.0" | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P1] Production anti-pattern.
Running
tsx(orts-node) in a production container introduces significant memory overhead and startup latency. Add a"build": "tsc"script topackage.jsonand run the compiled JavaScript here instead.