MiH SrSSKJr SSKJrJr SSKJrJr SSK J r SSSSSSSSSSS. S Sjjr S S jr "S S \5r g)aImplementation of baseline chunking. This is the "plain-vanilla" chunking strategy. All the fundamental chunking behaviors are present in this strategy and also in all other strategies. Those are: - Maximally fill each chunk with sequential elements. - Isolate oversized elements and divide (only) those chunks by text-splitting. - Overlap when requested. "Fancier" strategies add higher-level semantic-unit boundaries to be respected. For example, in the by-title strategy, section boundaries are respected, meaning a chunk never contains text from two different sections. When a new section is detected the current chunk is closed and a new one started. ) annotations)IterableOptional)ChunkingOptions PreChunker)ElementN include_orig_elementsmax_characters max_tokensnew_after_n_charsnew_after_n_tokensoverlap overlap_all tokenizerrepeat_table_headersskip_table_chunkingc P[RUUUUUUUUU U S9 n [X 5$)a Combine sequential `elements` into chunks, respecting specified text-length limits. Produces a sequence of `CompositeElement`, `Table`, and `TableChunk` elements (chunks). Parameters ---------- elements A list of unstructured elements. Usually the output of a partition function. include_orig_elements When `True` (default), add elements from pre-chunk to the `.metadata.orig_elements` field of the chunk(s) formed from that pre-chunk. Among other things, this allows access to original-element metadata that cannot be consolidated and is dropped in the course of chunking. max_characters Hard maximum chunk length. No chunk will exceed this length. A single element that exceeds this length will be divided into two or more chunks using text-splitting. Mutually exclusive with `max_tokens`. max_tokens Hard maximum chunk token count. No chunk will exceed this token count. Requires `tokenizer` to be specified. Mutually exclusive with `max_characters`. new_after_n_chars A chunk that of this length or greater is not extended to include the next element, even if that element would fit without exceeding `max_characters`. A "soft max" length that can be used in conjunction with `max_characters` to limit most chunks to a preferred length while still allowing larger elements to be included in a single chunk without resorting to text-splitting. Defaults to `max_characters` when not specified, which effectively disables any soft window. Specifying 0 for this argument causes each element to appear in a chunk by itself (although an element with text longer than `max_characters` will be still be split into two or more chunks). new_after_n_tokens Token-based equivalent of `new_after_n_chars`. A chunk with this token count or greater is not extended. Requires `max_tokens` and `tokenizer` to be specified. overlap Specifies the length of a string ("tail") to be drawn from each chunk and prefixed to the next chunk as a context-preserving mechanism. By default, this only applies to split-chunks where an oversized element is divided into multiple chunks by text-splitting. overlap_all Default: `False`. When `True`, apply overlap between "normal" chunks formed from whole elements and not subject to text-splitting. Use this with caution as it produces a certain level of "pollution" of otherwise clean semantic chunk boundaries. tokenizer The tokenizer to use for token-based chunking. Can be either an encoding name (e.g., "cl100k_base") or a model name (e.g., "gpt-4"). Required when using `max_tokens`. repeat_table_headers Default: `True`. When `True`, repeated table-header behavior is enabled for chunked table continuations. Specify `False` to opt out and preserve legacy table-chunk behavior. skip_table_chunking Default: `False`. When `True`, `Table` elements are passed through unchanged without being split into `TableChunk` elements, regardless of their size. r )_BasicChunkingOptionsnew_chunk_elements) elementsr r r r rrrrrroptss u/var/www/eduai.edurigo.com/storigo/production/storigo_env/lib/python3.13/site-packages/unstructured/chunking/basic.pychunk_elementsrsEB ! $ $3%+-1/ % D 8 **rc[R"X5VVs/sHnUR5HnUPM M snn$s snnf)z(Implementation of actual basic chunking.)riter_pre_chunks iter_chunks)rr pre_chunkchunks rrrisJ $33HC CI**,E , C  s$Ac\rSrSrSrSrg)rtzOptions for `basic` chunking.N)__name__ __module__ __qualname____firstlineno____doc____static_attributes__r$rrrrts'r)rIterable[Element]r Optional[bool]r Optional[int]r r-r r-rr-rr-rr,rz Optional[str]rr,rr,return list[Element])rr+rrr.r/)r) __future__rtypingrrunstructured.chunking.baserrunstructured.documents.elementsrrrrr$rrr4s #%B3 -1$( $'+(,!"&#+/*.N+N+*N+" N+  N+ % N+&N+N+ N+N+)N+(N+N+b(O(r