SwamphacksXI - KoalaKite

Team: Gatorworks (RJ Tabelon, Stephanie Fong, Andrew Chuang)

Description

KoalaKite is a seamless browser extension that balances the need for online parental supervision and privacy!

MVP

Overview

• Chrome MV3 extension captures the visible tab periodically and sends screenshots to a Fastify backend
• Backend classifies via Gemini Vision into SAFE / SUSPECT / HARMFUL with a strict JSON schema
• If SAFE: Image discarded immediately (never written to disk)
• If HARMFUL (and confidence >= threshold): Parent receives an email (via SMTP) with the screenshot attached
• No database - All state is in-memory.

System Design

Structure

• /shared: Zod types and shared schemas
• /server: Fastify + TypeScript backend (Gemini + SMTP email)
• /extension: Chrome MV3 (Vite + @crxjs) background worker + popup UI
• /dashboard: React 19 + Vite + Typescript webapp

Privacy & Security

• Gemini API key is only in the server, never in the extension
• All images will be discarded
• No browsing history, URLs, or page titles are stored. Minimal metadata only when necessary (timestamp + category + severity + confidence)
• Rate limit and request size limits enforced on the server
• If Gemini returns malformed JSON or fails: Treated as SUSPECT and no notification
• Image deduplication and cooldown prevents spam

Setup

Server

1. Prereqs: Node.js 18+.
2. Install deps (monorepo root):

npm install

3. Create server env file:

• Copy /server/.env.example to /server/.env and fill values:

PORT=8080
GEMINI_API_KEY=<your gemini key>
CONFIDENCE_THRESHOLD=0.7
COOLDOWN_SECONDS=120
BODY_LIMIT_BYTES=4194304
SMTP_HOST=smtp.gmail.com
SMTP_PORT=465
SMTP_SECURE=true
SMTP_USER=<your email>
SMTP_PASS=<app password>
SMTP_FROM=<from address shown in email>

4. Run the server:

cd server
npm run dev

Extension

1. Build the extension:

cd extension
npm run build

2. Load unpacked in Chrome:
   • Visit chrome://extensions
   • Enable Developer Mode
   • Load Unpacked -> select extension/dist
3. Open the popup:
   • Set Parent Email
   • Set Server Base URL (e.g., http://localhost:8080)
   • Click "Register Device" to obtain deviceId + deviceToken from the server
   • Toggle Enabled and choose interval

Backend Endpoints

• POST /api/register-device
  • Input: { parentEmail: string }
  • Output: { deviceId, deviceToken }
• POST /api/analyze
  • Auth: Authorization: Bearer <deviceToken>
  • Input: { deviceId, screenshotBase64, timestamp, screenshotHash? }
  • Output: { severity, category, reason, confidence, notified? }
  • Behavior: SAFE -> discard; HARMFUL (>= threshold) -> email with screenshot attachment; SUSPECT -> no notify.
• GET /health -> { ok: true }

Classification JSON Template

{
	"severity": "SAFE" | "SUSPECT" | "HARMFUL",
	"category": "NONE" | "NUDITY" | "SELF_HARM" | "VIOLENCE" | "HATE" | "DRUGS" | "GROOMING",
	"reason": "1-2 short sentences, non-graphic",
	"confidence": number between 0-1
}

Acceptance Tests

• Extension registers device with the parent email.
• SAFE page: no MMS; image is never stored beyond in-memory classification.
• Use an ethical test image/page that classifies HARMFUL -> email arrives containing a screenshot attachment.
• No screenshot written to disk; logs are minimal and do not include base64 or tokens.

Demo Script

1. Start server (npm run dev).
2. Load extension, register with parent email, enable at 5s interval.
3. Navigate to a test page; observe SAFE -> no email.
4. Navigate to an ethically constructed HARMFUL test; observe email with screenshot attachment.