Skip to content

TESTING_RESULTS.md

Latest commit

 

History

History
147 lines (98 loc) · 4.42 KB

TESTING_RESULTS.md

File metadata and controls

147 lines (98 loc) · 4.42 KB

Local Testing Results

Test Execution Summary

Passing Test Batches

  • Action tests (test:action) - All passing ✓
  • Sidebar tests (test:sidebar) - All passing ✓

Still Failing Test Batches

1. Content Tests (test:content) - 28 failures

Status: Content scripts are not injecting in headless mode

Failure Pattern:

  • Tests timeout waiting for #extension-root or [data-extension-root="true"] elements
  • Content scripts never appear on the page
  • Error: div with class content_script not found in Shadow DOM

Root Cause: Content scripts are not being injected at all in headless Chrome, even after:

  • Increased wait times (5 seconds after context creation)
  • Increased timeouts for Shadow DOM queries (60s in CI)
  • Added delays before page navigation

Affected Tests:

  • All content script tests that check for Shadow DOM elements
  • Tests using getShadowRootElement helper
  • Tests that directly query for extension root elements

Possible Solutions:

  1. Verify extension is actually loaded - Check extension list via CDP
  2. Wait for content script registration - Poll for chrome.runtime availability
  3. Use different navigation strategy - Navigate to a test page first, then check
  4. Check Chrome flags - May need additional flags for content script injection in headless
  5. Consider non-headless mode - Some CI systems support headed Chrome

2. New Tab Tests (test:newtab) - 27 failures

Status: Extension ID detection and navigation issues

Failure Pattern:

  • Some tests fail with ERR_INVALID_URL when navigating to chrome://newtab/
  • Extension ID may not be detected in time
  • Tests that use extension URL pattern (chrome-extension://${extensionId}/...) work better

Root Cause:

  • chrome://newtab/ navigation doesn't work reliably in headless Playwright
  • Some tests don't use extensionId fixture properly
  • Extension override may not be active when tests run

Affected Tests:

  • All new tab example tests except new/template.spec.ts (which uses extension URL)

Recommended Fix: Update all new tab tests to use extension URL pattern like new/template.spec.ts:

await page.goto(
  `chrome-extension://${extensionId}/chrome_url_overrides/newtab.html`
)

Fixes Implemented

1. Increased Timeouts

  • Shadow DOM queries: 60s timeout in CI (was 30s)
  • Extension ID detection: Increased wait times
  • Context creation: Added 3s wait + 2s verification in headless mode
  • Page fixture: Added 5s wait before page use in headless mode

2. Improved Extension Loading

  • Added delays after context creation
  • Added verification steps for extension readiness
  • Increased retry counts for extension ID detection

3. CI Script Improvements

  • Made xvfb-run optional for macOS compatibility

Remaining Issues

Critical: Content Script Injection

Content scripts are not injecting in headless Chrome. This appears to be a fundamental issue with how Chrome handles content scripts in headless mode, not just a timing problem.

Investigation Needed:

  1. Verify extension is actually loaded (check chrome://extensions via CDP)
  2. Check if content scripts are registered in manifest
  3. Test if content scripts work in non-headless mode
  4. Research Chrome flags needed for content script injection in headless

Medium: New Tab Navigation

Some new tab tests use chrome://newtab/ which doesn't work reliably. Should update to use extension URL pattern.

Next Steps

  1. Investigate content script injection

    • Check Chrome DevTools Protocol for extension loading status
    • Verify manifest content_scripts configuration
    • Test in non-headless mode to confirm scripts work

  2. Fix new tab tests

    • Update all tests to use extension URL pattern
    • Ensure extensionId fixture is used correctly

  3. Consider alternative approaches

    • Use --disable-web-security flag if needed
    • Try different Chrome launch options
    • Consider using --headless=new if available

Test Commands

# Run content tests
export CI=true
export SKIP_BUILD=true
pnpm run test:content

# Run new tab tests
pnpm run test:newtab

# Run action tests (passing)
pnpm run test:action

# Run sidebar tests (passing)
pnpm run test:sidebar

Files Modified

  1. examples/extension-fixtures.ts - Increased timeouts and waits
  2. run-ci-local.sh - macOS compatibility
  3. CI_TEST_FAILURES_REPORT.md - Initial analysis
  4. TESTING_RESULTS.md - This file