> ## Documentation Index
> Fetch the complete documentation index at: https://acem-52171079.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Intelligent document processing and normalizer

The Document Collector is a high-performance **FastAPI** service responsible for ingesting, normalizing, and converting diverse document formats into a standardized JSON structure optimized for RAG pipelines.

## Capabilities

The service supports a wide range of input formats:

<CardGroup cols={3}>
  <Card title="PDF Documents" icon="file-pdf">
    High-fidelity extraction using `pymupdf` (fitz) with `PyPDF2` fallback.
  </Card>

  <Card title="Microsoft Office" icon="file-word">
    Native support for Word (`.docx`), Excel (`.xlsx`), and PowerPoint (`.pptx`).
  </Card>

  <Card title="Text & Code" icon="file-code">
    Parses `.txt`, `.md`, `.json`, `.csv`, `.html` and other plain text formats.
  </Card>

  <Card title="Multimedia" icon="file-audio">
    Integrates with OpenAI Whisper API for transcribing Audio and Video files.
  </Card>

  <Card title="Web Scraping" icon="globe">
    Extracts cleaner content from URLs using `playwright` and `beautifulsoup4`.
  </Card>

  <Card title="E-Books & Email" icon="book">
    Processes `.epub` books and `.mbox` email archives.
  </Card>
</CardGroup>

## Standardized Output

Regardless of the input format, the collector outputs a consistent JSON structure. This normalization is crucial for downstream embedding and indexing.

```json theme={null}
{
  "id": "uuid-v4",
  "url": "file:///path/to/source.pdf",
  "title": "Clean Filename.pdf",
  "docAuthor": "Extracted Author",
  "wordCount": 1500,
  "pageContent": "The full extracted text content...",
  "token_count_estimate": 1950,
  "location": "storage/custom-documents/uuid.json"
}
```

## Getting Started

To learn more about the internals and how to integrate with the collector:

<CardGroup cols={2}>
  <Card title="Architecture" icon="sitemap" href="/docs/collector/architecture">
    Review the internal processing pipeline and tech stack.
  </Card>

  <Card title="Extensions" icon="puzzle-piece" href="/docs/collector/extensions">
    Learn about external integrations like YouTube and GitHub.
  </Card>
</CardGroup>