This project incrementally builds a retrieval-augmented question answering system over scientific PDFs.
Phase 1: Scientific PDF Text Extraction
This phase focuses on reliable extraction of raw text from text-based academic PDFs.The pdf used is a research paper "Attention is all you need"
Phase 2: Text Cleaning and Normalization
This phase introduces a dedicated text cleaning step to prepare extracted PDF content for downstream retrieval tasks. The raw text is normalized by fixing broken line breaks, reducing excessive whitespace, and preserving paragraph and section boundaries. No semantic content is removed or altered at this stage, ensuring the cleaned text remains faithful to the original document while being suitable for chunking and embedding.
Phase 3: Text Chunking Strategy
This phase introduces a controlled text chunking step that splits the cleaned document into overlapping, fixed-size segments suitable for retrieval. A character-based chunking strategy with overlap is used to preserve contextual continuity across chunk boundaries while maintaining deterministic and reproducible behavior. The chunked output serves as the foundational unit for embedding generation and similarity-based retrieval in later phases.
Phase 4: Chunk Validation and Quality Assurance
This phase focuses on validating the quality of generated text chunks before embedding. Basic sanity checks are applied to eliminate empty or low-information chunks, and representative samples are manually inspected to ensure semantic coherence, proper boundaries, and reasonable overlap behavior. This step ensures that only meaningful and structurally sound text units are passed to the embedding stage.
Phase 5: Embedding Generation
This phase introduces the transformation of validated text chunks into fixed-size numerical embeddings using a pre-trained sentence embedding model. Each chunk is mapped to a dense vector representation that captures its semantic meaning while maintaining consistent dimensionality across the dataset. These embeddings form the basis for similarity-based retrieval in subsequent phases of the system.
Phase 6: Chunk Boundary Refinement
This phase refines the chunking process to ensure that text chunks do not start or end in the middle of words. While preserving the original chunk size and overlap parameters, chunk boundaries are adjusted to the nearest word boundaries using lightweight, deterministic rules. This refinement improves semantic coherence within chunks without introducing additional dependencies or altering the overall chunking strategy.