Skip to content

Commit d5c56d3

Browse files
authored
refactor: General improvements (#33)
* chore: change license to MIT, and add ws dependency Closes #29 * refactor: standardize Firecrawl naming, update README and CHANGELOG, and adjust license year - Updated instances of "FireCrawl" to "Firecrawl" for consistency across documentation and code - Enhanced README with additional configuration instructions and acknowledgments - Revised CHANGELOG to reflect recent changes and optimizations - Updated license year from 2023 to 2025 Closes #30
1 parent 51532e7 commit d5c56d3

File tree

9 files changed

+2283
-849
lines changed

9 files changed

+2283
-849
lines changed

CHANGELOG.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@
3939

4040
### Changed
4141

42-
- Removed redundant batch configuration to rely on FireCrawl library's built-in functionality
42+
- Removed redundant batch configuration to rely on Firecrawl library's built-in functionality
4343
- Simplified batch processing logic by leveraging library's native implementation
4444
- Optimized parallel processing and rate limiting handling
4545
- Reduced code complexity and potential configuration conflicts
@@ -74,7 +74,7 @@
7474
- Enhanced content validation with configurable criteria
7575
- Added comprehensive logging system for operations and errors
7676
- New search tool (`firecrawl_search`) for web search with content extraction
77-
- Support for self-hosted FireCrawl instances via optional API URL configuration
77+
- Support for self-hosted Firecrawl instances via optional API URL configuration
7878
- New `FIRECRAWL_API_URL` environment variable
7979
- Automatic fallback to cloud API
8080
- Improved error messages for self-hosted instances

LICENSE

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
MIT License
22

3-
Copyright (c) 2023 vrknetha
3+
Copyright (c) 2025 vrknetha
44

55
Permission is hereby granted, free of charge, to any person obtaining a copy
66
of this software and associated documentation files (the "Software"), to deal

README.md

Lines changed: 41 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,9 @@
22

33
A Model Context Protocol (MCP) server implementation that integrates with [Firecrawl](https://github.com/mendableai/firecrawl) for web scraping capabilities.
44

5-
Big thanks to [@vrknetha](https://github.com/vrknetha), [@cawstudios](https://caw.tech) for the initial implementation!
5+
> Big thanks to [@vrknetha](https://github.com/vrknetha), [@cawstudios](https://caw.tech) for the initial implementation!
6+
>
7+
> You can also play around with [our MCP Server on MCP.so's playground](https://mcp.so/playground?server=firecrawl-mcp-server). Thanks to MCP.so for hosting and [@gstarwd](https://github.com/gstarwd) for integrating our server.
68
79
## Features
810

@@ -11,10 +13,10 @@ Big thanks to [@vrknetha](https://github.com/vrknetha), [@cawstudios](https://ca
1113
- URL discovery and crawling
1214
- Web search with content extraction
1315
- Automatic retries with exponential backoff
14-
- - Efficient batch processing with built-in rate limiting
16+
- Efficient batch processing with built-in rate limiting
1517
- Credit usage monitoring for cloud API
1618
- Comprehensive logging system
17-
- Support for cloud and self-hosted FireCrawl instances
19+
- Support for cloud and self-hosted Firecrawl instances
1820
- Mobile/Desktop viewport support
1921
- Smart content filtering with tag inclusion/exclusion
2022

@@ -36,22 +38,44 @@ npm install -g firecrawl-mcp
3638

3739
Configuring Cursor 🖥️
3840
Note: Requires Cursor version 0.45.6+
41+
For the most up-to-date configuration instructions, please refer to the official Cursor documentation on configuring MCP servers:
42+
[Cursor MCP Server Configuration Guide](https://docs.cursor.com/context/model-context-protocol#configuring-mcp-servers)
3943

40-
To configure FireCrawl MCP in Cursor:
44+
To configure Firecrawl MCP in Cursor **v0.45.6**
4145

4246
1. Open Cursor Settings
43-
2. Go to Features > MCP Servers
47+
2. Go to Features > MCP Servers
4448
3. Click "+ Add New MCP Server"
4549
4. Enter the following:
4650
- Name: "firecrawl-mcp" (or your preferred name)
4751
- Type: "command"
4852
- Command: `env FIRECRAWL_API_KEY=your-api-key npx -y firecrawl-mcp`
4953

54+
To configure Firecrawl MCP in Cursor **v0.48.6**
55+
56+
1. Open Cursor Settings
57+
2. Go to Features > MCP Servers
58+
3. Click "+ Add new global MCP server"
59+
4. Enter the following code:
60+
```json
61+
{
62+
"mcpServers": {
63+
"firecrawl-mcp": {
64+
"command": "npx",
65+
"args": ["-y", "firecrawl-mcp"],
66+
"env": {
67+
"FIRECRAWL_API_KEY": "YOUR-API-KEY"
68+
}
69+
}
70+
}
71+
}
72+
```
73+
5074
> If you are using Windows and are running into issues, try `cmd /c "set FIRECRAWL_API_KEY=your-api-key && npx -y firecrawl-mcp"`
5175
52-
Replace `your-api-key` with your FireCrawl API key.
76+
Replace `your-api-key` with your Firecrawl API key. If you don't have one yet, you can create an account and get it from https://www.firecrawl.dev/app/api-keys
5377

54-
After adding, refresh the MCP server list to see the new tools. The Composer Agent will automatically use FireCrawl MCP when appropriate, but you can explicitly request it by describing your web scraping needs. Access the Composer via Command+L (Mac), select "Agent" next to the submit button, and enter your query.
78+
After adding, refresh the MCP server list to see the new tools. The Composer Agent will automatically use Firecrawl MCP when appropriate, but you can explicitly request it by describing your web scraping needs. Access the Composer via Command+L (Mac), select "Agent" next to the submit button, and enter your query.
5579

5680
### Running on Windsurf
5781

@@ -64,17 +88,16 @@ Add this to your `./codeium/windsurf/model_config.json`:
6488
"command": "npx",
6589
"args": ["-y", "firecrawl-mcp"],
6690
"env": {
67-
"FIRECRAWL_API_KEY": "YOUR_API_KEY_HERE"
91+
"FIRECRAWL_API_KEY": "YOUR_API_KEY"
6892
}
6993
}
7094
}
7195
}
7296
```
7397

74-
7598
### Installing via Smithery (Legacy)
7699

77-
To install FireCrawl for Claude Desktop automatically via [Smithery](https://smithery.ai/server/@mendableai/mcp-server-firecrawl):
100+
To install Firecrawl for Claude Desktop automatically via [Smithery](https://smithery.ai/server/@mendableai/mcp-server-firecrawl):
78101

79102
```bash
80103
npx -y @smithery/cli install @mendableai/mcp-server-firecrawl --client claude
@@ -86,7 +109,7 @@ npx -y @smithery/cli install @mendableai/mcp-server-firecrawl --client claude
86109

87110
#### Required for Cloud API
88111

89-
- `FIRECRAWL_API_KEY`: Your FireCrawl API key
112+
- `FIRECRAWL_API_KEY`: Your Firecrawl API key
90113
- Required when using cloud API (default)
91114
- Optional when using self-hosted instance with `FIRECRAWL_API_URL`
92115
- `FIRECRAWL_API_URL` (Optional): Custom API endpoint for self-hosted instances
@@ -206,7 +229,7 @@ These configurations control:
206229

207230
### Rate Limiting and Batch Processing
208231

209-
The server utilizes FireCrawl's built-in rate limiting and batch processing capabilities:
232+
The server utilizes Firecrawl's built-in rate limiting and batch processing capabilities:
210233

211234
- Automatic rate limit handling with exponential backoff
212235
- Efficient parallel processing for batch operations
@@ -372,10 +395,10 @@ Example response:
372395
- `enableWebSearch`: Enable web search for additional context
373396
- `includeSubdomains`: Include subdomains in extraction
374397

375-
When using a self-hosted instance, the extraction will use your configured LLM. For cloud API, it uses FireCrawl's managed LLM service.
376-
398+
When using a self-hosted instance, the extraction will use your configured LLM. For cloud API, it uses Firecrawl's managed LLM service.
377399

378400
### 7. Deep Research Tool (firecrawl_deep_research)
401+
379402
Conduct deep web research on a query using intelligent crawling, search, and LLM analysis.
380403

381404
```json
@@ -391,6 +414,7 @@ Conduct deep web research on a query using intelligent crawling, search, and LLM
391414
```
392415

393416
Arguments:
417+
394418
- query (string, required): The research question or topic to explore.
395419
- maxDepth (number, optional): Maximum recursive depth for crawling/search (default: 3).
396420
- timeLimit (number, optional): Time limit in seconds for the research session (default: 120).
@@ -402,6 +426,7 @@ Returns:
402426
- May also include structured activities and sources used in the research process.
403427

404428
### 8. Generate LLMs.txt Tool (firecrawl_generate_llmstxt)
429+
405430
Generate a standardized llms.txt (and optionally llms-full.txt) file for a given domain. This file defines how large language models should interact with the site.
406431

407432
```json
@@ -422,10 +447,8 @@ Arguments:
422447
- showFullText (boolean, optional): Whether to include llms-full.txt contents in the response.
423448

424449
Returns:
425-
- Generated llms.txt file contents and optionally the llms-full.txt (data.llmstxt and/or data.llmsfulltxt)
426-
427-
428450

451+
- Generated llms.txt file contents and optionally the llms-full.txt (data.llmstxt and/or data.llmsfulltxt)
429452

430453
## Logging System
431454

@@ -440,7 +463,7 @@ The server includes comprehensive logging:
440463
Example log messages:
441464

442465
```
443-
[INFO] FireCrawl MCP Server initialized successfully
466+
[INFO] Firecrawl MCP Server initialized successfully
444467
[INFO] Starting scrape for URL: https://example.com
445468
[INFO] Batch operation queued with ID: batch_1
446469
[WARNING] Credit usage has reached warning threshold

package-lock.json

Lines changed: 9 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

package.json

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"name": "firecrawl-mcp",
33
"version": "1.7.2",
4-
"description": "MCP server for FireCrawl web scraping integration. Supports both cloud and self-hosted instances. Features include web scraping, batch processing, structured data extraction, and LLM-powered content analysis.",
4+
"description": "MCP server for Firecrawl web scraping integration. Supports both cloud and self-hosted instances. Features include web scraping, batch processing, structured data extraction, and LLM-powered content analysis.",
55
"type": "module",
66
"bin": {
77
"firecrawl-mcp": "dist/index.js"
@@ -22,13 +22,14 @@
2222
"prepare": "npm run build",
2323
"publish": "npm run build && npm publish"
2424
},
25-
"license": "ISC",
25+
"license": "MIT",
2626
"dependencies": {
2727
"@mendable/firecrawl-js": "^1.19.0",
2828
"@modelcontextprotocol/sdk": "^1.4.1",
2929
"dotenv": "^16.4.7",
3030
"p-queue": "^8.0.1",
31-
"shx": "^0.3.4"
31+
"shx": "^0.3.4",
32+
"ws": "^8.18.1"
3233
},
3334
"devDependencies": {
3435
"@jest/globals": "^29.7.0",

0 commit comments

Comments
 (0)