Mib# SrSSKJr SSKJr SSKJrJrJr SSK J r J r J r J r JrJrJr SSKJr SSSSSSSSSSSSS. S S jjrSS jr"S S \ 5rg)zdImplementation of chunking by title. Main entry point is the `@add_chunking_strategy()` decorator. ) annotations)cached_property)IterableIteratorOptional)CHUNK_MULTI_PAGE_DEFAULTBoundaryPredicateChunkingOptionsPreChunkCombiner PreChunkeris_on_next_pageis_title)ElementN combine_text_under_n_charsinclude_orig_elementsmax_characters max_tokensmultipage_sectionsnew_after_n_charsnew_after_n_tokensoverlap overlap_all tokenizerrepeat_table_headersskip_table_chunkingc T[RUUUUUUUUU U U U S9 n [X 5$)a Uses title elements to identify sections within the document for chunking. Splits off into a new CompositeElement when a title is detected or if metadata changes, which happens when page numbers or sections change. Cuts off sections once they have exceeded a character length of max_characters (or token count of max_tokens). Parameters ---------- elements A list of unstructured elements. Usually the output of a partition function. combine_text_under_n_chars Combines elements (for example a series of titles) until a section reaches a length of n characters. Defaults to `max_characters` which combines chunks whenever space allows. Specifying 0 for this argument suppresses combining of small chunks. Note this value is "capped" at the `new_after_n_chars` value since a value higher than that would not change this parameter's effect. include_orig_elements When `True` (default), add elements from pre-chunk to the `.metadata.orig_elements` field of the chunk(s) formed from that pre-chunk. Among other things, this allows access to original-element metadata that cannot be consolidated and is dropped in the course of chunking. max_characters Chunks elements text and text_as_html (if present) into chunks of length n characters (hard max). Mutually exclusive with `max_tokens`. max_tokens Chunks elements into chunks of n tokens (hard max). Requires `tokenizer` to be specified. Mutually exclusive with `max_characters`. multipage_sections If True, sections can span multiple pages. Defaults to True. new_after_n_chars Cuts off new sections once they reach a length of n characters (soft max). Defaults to `max_characters` when not specified, which effectively disables any soft window. Specifying 0 for this argument causes each element to appear in a chunk by itself (although an element with text longer than `max_characters` will be still be split into two or more chunks). new_after_n_tokens Token-based equivalent of `new_after_n_chars`. Cuts off new sections once they reach n tokens (soft max). Requires `max_tokens` and `tokenizer` to be specified. overlap Specifies the length of a string ("tail") to be drawn from each chunk and prefixed to the next chunk as a context-preserving mechanism. By default, this only applies to split-chunks where an oversized element is divided into multiple chunks by text-splitting. overlap_all Default: `False`. When `True`, apply overlap between "normal" chunks formed from whole elements and not subject to text-splitting. Use this with caution as it entails a certain level of "pollution" of otherwise clean semantic chunk boundaries. tokenizer The tokenizer to use for token-based chunking. Can be either an encoding name (e.g., "cl100k_base") or a model name (e.g., "gpt-4"). Required when using `max_tokens`. repeat_table_headers Default: `True`. When `True`, repeated table-header behavior is enabled for chunked table continuations. Specify `False` to opt out and preserve legacy table-chunk behavior. skip_table_chunking Default: `False`. When `True`, `Table` elements are passed through unchanged without being split into `TableChunk` elements, regardless of their size. r)_ByTitleChunkingOptionsnew_chunk_by_title)elementsrrrrrrrrrrrroptss u/var/www/eduai.edurigo.com/storigo/production/storigo_env/lib/python3.13/site-packages/unstructured/chunking/title.pychunk_by_titler$sKP # & &#=3%-+-1/ ' D 8 **rc[[R"X5US9R5nUVVs/sHo3R 5HoDPM M snn$s snnf)z-Implementation of actual "by-title" chunking.)r")r r iter_pre_chunksiter_combined_pre_chunks iter_chunks)r!r" pre_chunks pre_chunkchunks r#r r psT"""82 $. S:i;P;P;R%E;RE: SS Ss"Acl^\rSrSrSr\SSj5r\S Sj5r\S Sj5rS U4Sjjr Sr U=r $) r{aAdds the by-title-specific chunking options to the base case. `by_title`-specific options: combine_text_under_n_chars A remedy to over-chunking caused by elements mis-identified as Title elements. Every Title element would start a new chunk and this setting mitigates that, at the expense of sometimes violating legitimate semantic boundaries. multipage_sections Indicates that page-boundaries should not be respected while chunking, i.e. elements appearing on two different pages can appear in the same chunk. c4^SU4Sjjn[U"55$)aThe semantic-boundary detectors to be applied to break pre-chunks. For the `by_title` strategy these are sections indicated by a title (section-heading), an explicit section metadata item (only present for certain document types), and optionally page boundaries. c3Z># [v TR(d [5v gg7f)N)rrr )selfsr#iter_boundary_predicatesM_ByTitleChunkingOptions.boundary_predicates..iter_boundary_predicatess"N**%''+s(+)returnzIterator[BoundaryPredicate])tuple)r1r2s` r#boundary_predicates+_ByTitleChunkingOptions.boundary_predicatess ( -/00r%cZURRS5nUc UR$U$)aCombine consecutive text pre-chunks if former is smaller than this and both will fit. - Does not combine text chunks if together they would exceed the chunking window. - Defaults to `max_characters` when not specified. - Is reduced to `new_after_n_chars` when it exceeds that value. r)_kwargsgethard_maxr1 arg_values r#r2_ByTitleChunkingOptions.combine_text_under_n_charss-LL$$%AB ) 1t}}@y@r%c`URRS5nUc[$[U5$)z0When False, break pre-chunks on page-boundaries.r)r9r:rboolr<s r#r*_ByTitleChunkingOptions.multipage_sectionss.LL$$%9: +4+<'Q$y/Qr%c>[TU]5 URS:a[SUR35eURUR:a%[SURSUR35eg)z2Raise ValueError if request option-set is invalid.rz8'combine_text_under_n_chars' argument must be >= 0, got zR'combine_text_under_n_chars' argument must not exceed `max_characters` value, got z > N)super _validater ValueErrorr;)r1 __class__s r#rD!_ByTitleChunkingOptions._validates   * *Q .778:   * *T]] :#>>?s4==/S  ;r%)r4ztuple[BoundaryPredicate, ...])r4int)r4r@)r4None) __name__ __module__ __qualname____firstlineno____doc__rr6rrrD__static_attributes__ __classcell__)rFs@r#rr{sR  1 1 A ARR r%)r!Iterable[Element]r Optional[int]rOptional[bool]rrSrrSrrTrrSrrSrrSrrTrz Optional[str]rrTrrTr4 list[Element])r!rRr"rr4rU)rO __future__r functoolsrtypingrrrunstructured.chunking.baserr r r r r runstructured.documents.elementsrr$r rrHr%r#r[s #%//4 15,0$( $)-'+(,!"&#+/*.V+V+!.V+* V+ " V+  V+'V+%V+&V+V+ V+V+)V+(V+V+rTEoEr%