AC logoAndrii Chekh
← All key projects

Scientific Paper Scraping & AI Pipeline

A high-throughput data pipeline for collecting, processing, and preparing scientific papers for downstream LLM and AI workflows. The project processed more than 5,000 scientific PDF documents published between 2017 and 2025. The main challenge was automating the entire ingestion process while maintaining high throughput and preserving the structure and mathematical content of the original documents.

  • Python
  • Mathpix API
  • LLM / AI processing
  • Parallel processing
  • Proxy rotation
  • Web scraping
  • PDF processing

The Challenge

Processing thousands of scientific papers manually or sequentially would be extremely time-consuming. Standard PDF parsers can also struggle with scientific documents containing mathematical formulas, images, tables, and complex formatting.

The pipeline needed to:

  • Collect a large number of documents automatically

  • Handle requests in parallel

  • Avoid rate limits and blocking through proxy rotation

  • Process thousands of files reliably

  • Convert PDFs into structured Markdown

  • Preserve important document elements such as formulas and images

  • Produce clean data suitable for further LLM processing

The Solution

I built the processing pipeline primarily with Python, using parallel processing and proxy rotation to handle multiple requests simultaneously. The ingestion process was organized as a multi-stage pipeline:

Scraping → Proxy Rotation → Parallel Processing → PDF Download → PDF-to-Markdown Conversion → Data Preparation

This structure allowed each stage to focus on a specific part of the ingestion and processing workflow.

PDF-to-Markdown Conversion

For PDF conversion, I integrated the Mathpix API, which is designed to convert complex documents into machine-readable formats while preserving scientific notation, mathematical formulas, images, and document structure. The resulting Markdown files could then be used as structured input for downstream AI and LLM workflows.

Performance

The pipeline processed the complete dataset of more than 5,000 documents in approximately 8 hours. Parallel processing significantly reduced the total processing time compared with sequential execution, while proxy rotation allowed multiple requests to be processed concurrently without relying on a single network connection.

Tech Stack

  • Python

  • Mathpix API

  • LLM / AI processing

  • Parallel processing

  • Proxy rotation

  • Web scraping

  • PDF processing

  • Markdown

  • Data pipelines

Result

The final result was an automated, high-throughput pipeline capable of turning thousands of scientific PDF documents into structured, machine-readable Markdown. Instead of manually downloading, parsing, and preparing each document, the entire workflow could be executed automatically and produce a consistent dataset ready for further AI and LLM processing.