haive.core.engine.document.processors¶

Document Processing Components.

This module provides document processing capabilities including chunking and content transformation that integrate with the DocumentEngine.

The processors handle: - Content normalization - Document chunking strategies - Metadata extraction - Format conversion

Classes¶

`ChunkingProcessor`	Processor for chunking documents into smaller pieces.
`ContentNormalizer`	Processor for normalizing document content.
`DocumentProcessor`	Base class for document processing operations.
`FormatDetector`	Processor for detecting document formats.
`MetadataExtractor`	Processor for extracting metadata from documents.

Module Contents¶

class haive.core.engine.document.processors.ChunkingProcessor(chunking_strategy=ChunkingStrategy.RECURSIVE, chunk_size=1000, chunk_overlap=200, **kwargs)¶

Bases: DocumentProcessor

Processor for chunking documents into smaller pieces.

Initialize the chunking processor.

Parameters:

chunking_strategy (haive.core.engine.document.config.ChunkingStrategy) – Strategy for chunking
chunk_size (int) – Size of chunks in characters
chunk_overlap (int) – Overlap between chunks
**kwargs – Additional configuration

chunk_text(text, strategy, chunk_size, chunk_overlap, metadata)¶

Chunk text according to the specified strategy.

Parameters:

text (str) – Text to chunk
strategy (haive.core.engine.document.config.ChunkingStrategy) – Chunking strategy
chunk_size (int) – Size of chunks
chunk_overlap (int) – Overlap between chunks
metadata (dict[str, Any]) – Base metadata for chunks

Returns:

List of document chunks

Return type:

list[haive.core.engine.document.config.DocumentChunk]

class haive.core.engine.document.processors.ContentNormalizer(normalize_whitespace=True, remove_extra_newlines=True, strip_content=True, **kwargs)¶

Bases: DocumentProcessor

Processor for normalizing document content.

Initialize the content normalizer.

Parameters:

normalize_whitespace (bool) – Whether to normalize whitespace
remove_extra_newlines (bool) – Whether to remove extra newlines
strip_content (bool) – Whether to strip leading/trailing whitespace
**kwargs – Additional configuration

normalize_content(content)¶

Normalize document content.

Parameters:: content (str) – Content to normalize
Returns:: Normalized content
Return type:: str

class haive.core.engine.document.processors.DocumentProcessor(**kwargs)¶

Base class for document processing operations.

Initialize the processor.

abstractmethod process(document)¶

Process a document.

Parameters:: document (langchain_core.documents.Document) – Document to process
Returns:: Processed document
Return type:: haive.core.engine.document.config.ProcessedDocument

class haive.core.engine.document.processors.FormatDetector(**kwargs)¶

Bases: DocumentProcessor

Processor for detecting document formats.

Initialize the processor.

detect_format(content, metadata)¶

Detect document format from content and metadata.

Parameters:

content (str) – Document content
metadata (dict[str, Any]) – Document metadata

Returns:

Detected document format

Return type:

haive.core.engine.document.config.DocumentFormat

class haive.core.engine.document.processors.MetadataExtractor(**kwargs)¶

Bases: DocumentProcessor

Processor for extracting metadata from documents.

Initialize the processor.

extract_metadata(content, existing_metadata)¶

Extract additional metadata from document content.

Parameters:

content (str) – Document content
existing_metadata (dict[str, Any]) – Existing metadata

Returns:

Enhanced metadata dictionary

Return type:

dict[str, Any]