Skip to content

[Feature Request] Scrape timeouts and other timeout defaults configuration via environment variable #103

@huzhekun

Description

@huzhekun

I have encountered very often that when using this MCP to scrape on my self hosted firecrawl, there would be instances where when using search, if a website fails to scrape due to anti bot, regardless of the search timeout, scrape will continue to run and try to scrape the website until the scrape timeout is reached. This leads to later search queries failing as often multiple pages would fail and my two firecrawl workers would not have the ability to take on more tasks, leading to later searches to timeout as well. i would like to be able to set these timeout parameters for search, scrape in search, and scrape separately via env var (assuming there is a good reason why in firecrawl self host scrape timeout doesnt inherit from search timeout) so the model doesnt have to worry about it

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions