Conversation
Collect page metadata (URL, viewport, DPR, title, color scheme, scroll) from the preview iframe and send as ACP content block with every prompt. Server extracts this data and appends [Current Page Context] to user messages, plus adds system prompt guidance when page context is present.
Contributor
|
Minimum allowed coverage is Generated by 🐒 cobertura-action against 1244c62 |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Comment on lines
+783
to
+791
| let (viewportWidth, viewportHeight, dpr, scrollY) = switch previewFrame.contentWindow { | ||
| | Some(win) => ( | ||
| Some(win.innerWidth), | ||
| Some(win.innerHeight), | ||
| Some(win.devicePixelRatio), | ||
| Some(win.scrollY->Float.toInt), | ||
| ) | ||
| | None => (None, None, None, None) | ||
| } |
Contributor
There was a problem hiding this comment.
🔴 Unprotected cross-origin iframe window property access in currentPageToContentBlock can crash prompt-send
Accessing win.innerWidth, win.innerHeight, win.devicePixelRatio, and win.scrollY at lines 785-788 has no try/catch protection, unlike the getColorScheme helper (line 769-773) which wraps its win.matchMedia call in try/catch.
Root Cause: stale WindowProxy becomes cross-origin after iframe navigation
The contentWindow stored in state is a browser WindowProxy. When the preview iframe navigates from a same-origin page to a cross-origin page:
- Initial same-origin load:
setPreviewFrameatClient__WebPreview__Body.res:35-37successfully capturescontentWindow: Some(win). - Iframe navigates to a cross-origin URL: the
onLoadhandler's try/catch atClient__WebPreview__Body.res:34-41catches theSecurityErrorfromcontentDocumentaccess and silently does nothing — crucially, it does not resetcontentWindowtoNone. - The state still holds
contentWindow: Some(win), butwinis now a WindowProxy pointing at the cross-origin browsing context. - User sends a prompt →
taskToContentBlocks→currentPageToContentBlock(previewFrame)→ matchesSome(win)→ accesseswin.innerWidth→ throwsSecurityError.
This unhandled exception crashes the prompt-send flow, preventing the user from interacting with the agent.
Impact: Any user whose preview iframe navigates cross-origin (e.g., OAuth redirects, external links) will be unable to send prompts until they reload the page.
Prompt for agents
In libs/client/src/state/Client__Task__Types.res, wrap the window property access block (lines 783-791) in a try/catch, similar to how getColorScheme handles potential SecurityError. When the catch fires, fall back to (None, None, None, None). The code should look like:
let (viewportWidth, viewportHeight, dpr, scrollY) = switch previewFrame.contentWindow {
| Some(win) =>
try {
(
Some(win.innerWidth),
Some(win.innerHeight),
Some(win.devicePixelRatio),
Some(win.scrollY->Float.toInt),
)
} catch {
| _ => (None, None, None, None)
}
| None => (None, None, None, None)
}
Also wrap the contentDocument access for title (lines 794-802) in a similar try/catch, and consider resetting contentWindow/contentDocument to None in the WebPreview onLoad catch handler at Client__WebPreview__Body.res:38-41 so stale cross-origin references don't persist in state.
Was this helpful? React with 👍 or 👎 to provide feedback.
The current_page field was stored in the DB but never deserialized back into UserMessage when loading interactions from the database. This caused append_current_page_context to receive nil instead of the page data, so the [Current Page Context] section was never appended to LLM messages.
The raw JS function returns null when no chip element is found, but
the ReScript type was option<{..}> which expects undefined for None.
This caused null to be wrapped as Some(null), leading to a TypeError
when calling getAttribute on null. Changed type to Nullable.t and
convert with Nullable.toOption.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #339
Adds implicit "current page" context that collects browser/page metadata from the preview iframe and passes it to the AI agent's system prompt. Unlike selected component (which requires user action), this is always present — giving the agent awareness of the user's browsing context.
What's included
_meta: { current_page: true }marker.current_pagefrom content blocks, appends a[Current Page Context]section to LLM user messages, and addscurrent_page_guidanceto the system prompt explaining how to use the context.Key changes
libs/client/src/state/Client__Task__Types.rescurrentPageToContentBlockbuilder, included intaskToContentBlocksfor all task statesapps/frontman_server/lib/frontman_server/tasks/interaction.exextract_current_page/1,append_current_page_context/2,has_current_page?/1apps/frontman_server/lib/frontman_server/agents/prompts.excurrent_page_guidance/0system prompt sectionapps/frontman_server/lib/frontman_server/agents/root_agent.exhas_current_pagefieldapps/frontman_server/lib/frontman_server/agents.exhas_current_pagetoRootAgent.newapps/frontman_server/lib/frontman_server/tasks.exhas_current_page?/2delegationHow it works
previewFrame.contentWindow/contentDocumentat prompt time{ type: "resource", resource: { _meta: { current_page: true, url, viewport_width, ... } } }UserMessage.current_pagestruct[Current Page Context]appended to user message text before LLM call