feat: Add ScrapyWebReader Integration (#20212)

Author: hammadbinsajjad

docs/api_reference/api_reference/readers/web.md

@@ -1,3 +1,3 @@
 ::: llama_index.readers.web
 options:
-members: - AgentQLWebReader - AsyncWebPageReader - BeautifulSoupWebReader - BrowserbaseWebReader - FireCrawlWebReader - HyperbrowserWebReader - KnowledgeBaseWebReader - MainContentExtractorReader - NewsArticleReader - OlostepWebReader - OxylabsWebReader - ReadabilityWebPageReader - RssNewsReader - RssReader - ScrapflyReader - SimpleWebPageReader - SitemapReader - SpiderReader - TrafilaturaWebReader - UnstructuredURLLoader - WholeSiteReader - ZenRowsWebReader
+members: - AgentQLWebReader - AsyncWebPageReader - BeautifulSoupWebReader - BrowserbaseWebReader - FireCrawlWebReader - HyperbrowserWebReader - KnowledgeBaseWebReader - MainContentExtractorReader - NewsArticleReader - OlostepWebReader - OxylabsWebReader - ReadabilityWebPageReader - RssNewsReader - RssReader - ScrapflyReader - ScrapyWebReader - SimpleWebPageReader - SitemapReader - SpiderReader - TrafilaturaWebReader - UnstructuredURLLoader - WholeSiteReader - ZenRowsWebReader

docs/examples/data_connectors/WebPageDemo.ipynb

@@ -1328,6 +1328,134 @@
     "\n",
     "print(response)"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "07117c04",
+   "metadata": {},
+   "source": [
+    "# Using Scrapy Web Reader 🕸️"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "22fd0310",
+   "metadata": {},
+   "source": [
+    "Scrapy is a popular web crawling framework for Python. The ScrapyWebReader allows you to leverage Scrapy's powerful crawling capabilities to extract data from websites. It can be used in 2 ways\n",
+    "\n",
+    "1. By providing an Scrapy spider class.\n",
+    "2. By providing the path to a Scrapy project."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0462b632",
+   "metadata": {},
+   "source": [
+    "### 1. Using with Scrapy Spider Class"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "25da4f69",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from scrapy.spiders import Spider\n",
+    "from llama_index.readers.web import ScrapyWebReader\n",
+    "\n",
+    "\n",
+    "class SampleSpider(Spider):\n",
+    "    name = \"sample_spider\"\n",
+    "    start_urls = [\"http://quotes.toscrape.com\"]\n",
+    "\n",
+    "    def parse(self, response):\n",
+    "        ...\n",
+    "\n",
+    "\n",
+    "reader = ScrapyWebReader()\n",
+    "docs = reader.load_data(SampleSpider)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e99c6e02",
+   "metadata": {},
+   "source": [
+    "### 2. Using with Scrapy Project Path"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1110e52e",
+   "metadata": {},
+   "source": [
+    "Downloading a Sample Scrapy Project"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "40060d02",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%git clone https://github.com/scrapy/quotesbot.git"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "91d304d4",
+   "metadata": {},
+   "source": [
+    "Using the scrapy project with spider named \"toscrape-css\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8cf448df",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from llama_index.readers.web import ScrapyWebReader\n",
+    "\n",
+    "reader = ScrapyWebReader(project_path=\"./quotesbot\")\n",
+    "docs = reader.load_data(\"toscrape-css\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12c85cd4",
+   "metadata": {},
+   "source": [
+    "### Metadata"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ce6769ec",
+   "metadata": {},
+   "source": [
+    "Some keys from the scraped items can be stored as metadata in the Document object. You can specify which keys to include as metadata using the `metadata_keys` parameter. If you want to keep the keys in both the content and as metadata, you can set the `keep_keys` parameter to `True`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1c3f6112",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "reader = ScrapyWebReader(\n",
+    "    project_path=\"./quotesbot\",\n",
+    "    metadata_keys=[\"author\", \"tags\"],\n",
+    "    keep_keys=True,\n",
+    ")\n",
+    "docs = reader.load_data(\"toscrape-css\")"
+   ]
   }
  ],
  "metadata": {

llama-index-integrations/readers/llama-index-readers-web/llama_index/readers/web/__init__.py

@@ -33,6 +33,9 @@
 from llama_index.readers.web.scrapfly_web.base import (
     ScrapflyReader,
 )
+from llama_index.readers.web.scrapy_web.base import (
+    ScrapyWebReader,
+)
 from llama_index.readers.web.simple_web.base import (
     SimpleWebPageReader,
 )
@@ -73,6 +76,7 @@
     "RssReader",
     "RssNewsReader",
     "ScrapflyReader",
+    "ScrapyWebReader",
     "SimpleWebPageReader",
     "SitemapReader",
     "SpiderWebReader",

llama-index-integrations/readers/llama-index-readers-web/llama_index/readers/web/scrapy_web/README.md

@@ -0,0 +1,43 @@
+# LlamaIndex Scrapy Web Reader Integration
+
+This integration provides the `ScrapyWebReader` class that allows you to use Scrapy to scrape data and load it into LlamaIndex.
+
+## Installation
+
+```bash
+pip install llama-index llama-index-readers-web
+```
+
+## Usage
+
+The `ScrapyWebReader` can be used in 2 ways
+
+1. By providing an Scrapy spider class.
+2. By providing the path to a Scrapy project.
+
+### 1. Using with Scrapy Spider Class
+
+```python
+from llama_index.readers.web import ScrapyWebReader
+
+
+class SampleSpider(Spider):
+    name = "sample_spider"
+    start_urls = ["http://quotes.toscrape.com"]
+
+    def parse(self, response):
+        ...
+
+
+reader = ScrapyWebReader()
+docs = reader.load_data(SampleSpider)
+```
+
+### 2. Using with Scrapy Project Path
+
+```python
+from llama_index.readers.web import ScrapyWebReader
+
+reader = ScrapyWebReader(project_path="/path/to/scrapy/project")
+docs = reader.load_data("spider_name")
+```

llama-index-integrations/readers/llama-index-readers-web/llama_index/readers/web/scrapy_web/__init__.py

@@ -0,0 +1,93 @@
+from typing import List, Optional, Union
+from multiprocessing import Process, Queue
+
+from scrapy.spiders import Spider
+
+from llama_index.core.readers.base import BasePydanticReader
+from llama_index.core.schema import Document
+
+from .utils import run_spider_process, load_scrapy_settings
+
+
+class ScrapyWebReader(BasePydanticReader):
+    """
+    Scrapy web page reader.
+
+    Reads pages from the web.
+
+    Args:
+        project_path (Optional[str]): The path to the Scrapy project for
+            loading the project settings (with middlewares and pipelines).
+            The project path should contain the `scrapy.cfg` file.
+            Settings will be set to empty if path not specified or not found.
+            Defaults to "".
+
+        metadata_keys (Optional[List[str]]): List of keys to use
+            as document metadata from the scraped item. Defaults to [].
+
+        keep_keys (bool): Whether to keep metadata keys in items.
+            Defaults to False.
+
+    """
+
+    project_path: Optional[str] = ""
+    metadata_keys: Optional[List[str]] = []
+    keep_keys: bool = False
+
+    def __init__(
+        self,
+        project_path: Optional[str] = "",
+        metadata_keys: Optional[List[str]] = [],
+        keep_keys: bool = False,
+    ):
+        super().__init__(
+            project_path=project_path,
+            metadata_keys=metadata_keys,
+            keep_keys=keep_keys,
+        )
+
+    @classmethod
+    def class_name(cls) -> str:
+        return "ScrapyWebReader"
+
+    def load_data(self, spider: Union[Spider, str]) -> List[Document]:
+        """
+        Load data from the input spider.
+
+        Args:
+            spider (Union[Spider, str]): The Scrapy spider class or
+                the spider name from the project to use for scraping.
+
+        Returns:
+            List[Document]: List of documents extracted from the web pages.
+
+        """
+        if not self._is_spider_correct_type(spider):
+            raise ValueError(
+                "Invalid spider type. Provide a Spider class or spider name with project path."
+            )
+
+        documents_queue = Queue()
+
+        config = {
+            "keep_keys": self.keep_keys,
+            "metadata_keys": self.metadata_keys,
+            "settings": load_scrapy_settings(self.project_path),
+        }
+
+        # Running each spider in a separate process as Scrapy uses
+        # twisted reactor which can only be run once in a process
+        process = Process(
+            target=run_spider_process, args=(spider, documents_queue, config)
+        )
+
+        process.start()
+        process.join()
+
+        if documents_queue.empty():
+            return []
+
+        return documents_queue.get()
+
+    def _is_spider_correct_type(self, spider: Union[Spider, str]) -> bool:
+        return not (isinstance(spider, str) and not self.project_path)

llama-index-integrations/readers/llama-index-readers-web/llama_index/readers/web/scrapy_web/base.py

@@ -0,0 +1 @@
+Scrapy

llama-index-integrations/readers/llama-index-readers-web/llama_index/readers/web/scrapy_web/requirements.txt

@@ -0,0 +1,87 @@
+import json
+import os
+from multiprocessing import Queue
+from typing import Dict
+
+from scrapy.spiders import signals, Spider
+from scrapy.crawler import CrawlerProcess
+from scrapy.utils.project import get_project_settings
+
+from llama_index.core.schema import Document
+
+
+def load_scrapy_settings(project_path: str) -> Dict:
+    """
+    Load Scrapy settings from the given project path.
+    """
+    if not project_path:
+        return {}
+
+    if not os.path.exists(project_path):
+        return {}
+
+    cwd = os.getcwd()
+
+    try:
+        os.chdir(project_path)
+
+        try:
+            settings = get_project_settings() or {}
+        except Exception:
+            settings = {}
+    finally:
+        os.chdir(cwd)
+
+    return settings
+
+
+def run_spider_process(spider: Spider, documents_queue: Queue, config: Dict):
+    """
+    Run the Scrapy spider process and collect documents in the queue.
+    """
+    documents = []
+
+    def item_scraped(item, response, spider):
+        documents.append(item_to_document(dict(item), config))
+
+    process = CrawlerProcess(settings=config["settings"])
+    crawler = process.create_crawler(spider)
+    crawler.signals.connect(item_scraped, signal=signals.item_scraped)
+    process.crawl(crawler)
+    process.start()
+
+    documents_queue.put(documents)
+
+
+def item_to_document(item: Dict, config: Dict) -> Dict:
+    """
+    Convert a scraped item to a Document with metadata.
+    """
+    metadata = setup_metadata(item, config)
+    item = remove_metadata_keys(item, config)
+
+    return Document(text=json.dumps(item), metadata=metadata)
+
+
+def setup_metadata(item: Dict, config: Dict) -> Dict:
+    """
+    Set up metadata for the document from the scraped item.
+    """
+    metadata = {}
+
+    for key in config["metadata_keys"]:
+        if key in item:
+            metadata[key] = item[key]
+
+    return metadata
+
+
+def remove_metadata_keys(item: Dict, config: Dict) -> Dict:
+    """
+    Remove metadata keys from the scraped item if keep_keys is False.
+    """
+    if not config["keep_keys"]:
+        for key in config["metadata_keys"]:
+            item.pop(key, None)
+
+    return item