Mi:l%SrSSKJr SSKrSSKrSSKrSSKJr SSKJ r J r J r J r J r Jr SSKrSSKJrJr SSKJr SSKJrJr SS KJrJrJr SS KJrJrJrJ r J!r!J"r"J#r#J$r$ SS K%J&r& S r'S \(S'Sr)S\(S'\ \ /\*4r+S\(S'\,\-\-4r.S\(S'"SS5r/"SS5r0S4Sjr1S5Sjr2S6Sjr3"SS5r4"SS5r5"S S!5r6"S"S#5r7"S$S%5r8"S&S'5r9"S(S)5r:"S*S+5r;"S,S-5r<"S.S/5r="S0S15r>S7S2jr?S4S3jr@g)8z@Chunking objects not specific to a particular chunking strategy.) annotationsN)cached_property)AnyCallable DefaultDictIterableIteratorcast) ParserErrortostring)fragment_fromstring)Self TypeAlias)HtmlCellHtmlRow HtmlTable) CodeSnippetCompositeElementConsolidationStrategyElementElementMetadataTable TableChunkTitle)loggeriintCHUNK_MAX_CHARS_DEFAULTTboolCHUNK_MULTI_PAGE_DEFAULTrBoundaryPredicate TextAndHtmlc<\rSrSrSrSSjr\S5rS SjrSr g) TokenCounter6zyToken counting using tiktoken for token-based chunking. Lazily imports tiktoken only when token counting is first used. cXlgN_tokenizer_nameself tokenizers t/var/www/eduai.edurigo.com/storigo/production/storigo_env/lib/python3.13/site-packages/unstructured/chunking/base.py__init__TokenCounter.__init__<s(cSSKnURUR5$![a UR UR5s$f=f)z'Lazily initialize the tiktoken encoder.rN)tiktokenencoding_for_modelr(KeyError get_encoding)r*r1s r,_encoderTokenCounter._encoder?sI  ?..t/C/CD D ?(()=)=> > ?s!%A A cJ[URRU55$)z&Return the number of tokens in `text`.)lenr5encoder*texts r,countTokenCounter.countKs4==''-..r/r'N)r+strr;r>returnr) __name__ __module__ __qualname____firstlineno____doc__r-rr5r<__static_attributes__r/r,r#r#6s% ) ? ?/r/r#cb\rSrSrSrSSjr\SSj5r\SSj5r \SSj5r \SSj5r \SSj5r \SS j5r \SS j5r\SS j5r\SS j5r\SS j5r\SSj5r\SSj5r\SSj5r\SSj5r\SSj5rS SjrS!SjrSrg)"ChunkingOptionsUaSpecifies parameters of optional chunking behaviors. Parameters ---------- max_characters Hard-maximum text-length of chunk. A chunk longer than this will be split mid-text and be emitted as two or more chunks. Mutually exclusive with `max_tokens`. max_tokens Hard-maximum token count of chunk. A chunk with more tokens than this will be split mid-text and be emitted as two or more chunks. Requires `tokenizer` to be specified. Mutually exclusive with `max_characters`. new_after_n_chars Preferred approximate chunk size. A chunk composed of elements totalling this size or greater is considered "full" and will not be enlarged by adding another element, even if it will fit within the remaining `max_characters` for that chunk. Defaults to `max_characters` when not specified, which effectively disables this behavior. Specifying 0 for this argument causes each element to appear in a chunk by itself (although an element with text longer than `max_characters` will be still be split into two or more chunks). new_after_n_tokens Token-based equivalent of `new_after_n_chars`. Preferred approximate chunk size in tokens. Requires `tokenizer` and `max_tokens` to be specified. combine_text_under_n_chars Provides a way to "recombine" small chunks formed by breaking on a semantic boundary. Only relevant for a chunking strategy that specifies higher-level semantic boundaries to be respected, like "section" or "page". Recursively combines two adjacent pre-chunks when the first pre-chunk is smaller than this threshold. "Recursively" here means the resulting pre-chunk can be combined with the next pre-chunk if it is still under the length threshold. Defaults to `max_characters` which combines chunks whenever space allows. Specifying 0 for this argument suppresses combining of small chunks. Note this value is "capped" at the `new_after_n_chars` value since a value higher than that would not change this parameter's effect. overlap Specifies the length of a string ("tail") to be drawn from each chunk and prefixed to the next chunk as a context-preserving mechanism. By default, this only applies to split-chunks where an oversized element is divided into multiple chunks by text-splitting. overlap_all Default: `False`. When `True`, apply overlap between "normal" chunks formed from whole elements and not subject to text-splitting. Use this with caution as it entails a certain level of "pollution" of otherwise clean semantic chunk boundaries. repeat_table_headers Default: `True`. When `True`, repeated table-header behavior is enabled for chunked table continuations. Specify `False` to opt out and preserve legacy table-chunk behavior. text_splitting_separators A sequence of strings like `(" ", " ")` to be used as target separators during text-splitting. Text-splitting only applies to splitting an oversized element into two or more chunks. These separators are tried in the specified order until one is found in the string to be split. The default separator is `""` which matches between any two characters. This separator should not be specified in this sequence because it is always the separator of last-resort. Note that because the separator is removed during text-splitting, only whitespace character sequences are suitable. tokenizer The tokenizer to use for token-based chunking. Can be either an encoding name (e.g., "cl100k_base") or a model name (e.g., "gpt-4"). Required when using `max_tokens`. c Xlgr&_kwargs)r*kwargss r,r-ChunkingOptions.__init__s r/c 6U"S0UD6nUR5 U$)zUReturn instance or raises `ValueError` on invalid arguments like overlap > max_chars.rG) _validate)clsrNr*s r,newChunkingOptions.news}V}  r/cg)zThe semantic-boundary detectors to be applied to break pre-chunks. Overridden by sub-typs to provide semantic-boundary isolation behaviors. rGrGr*s r,boundary_predicates#ChunkingOptions.boundary_predicatess r/cFURRS5nUbU$S$)zCombine two consecutive text pre-chunks if first is smaller than this and both will fit. Default applied here is `0` which essentially disables chunk combining. Must be overridden by subclass where combining behavior is supported. combine_text_under_n_charsrrMgetr* arg_values r,rZ*ChunkingOptions.combine_text_under_n_charss)LL$$%AB %1y8q8r/cUR(aURS$URRS5nUbU$[$)a The maximum size for a chunk (in characters or tokens depending on mode). A pre-chunk will only exceed this size when it contains exactly one element which by itself exceeds this size. Such a pre-chunk is subject to mid-text splitting later in the chunking process. max_tokensmax_characters)use_token_countingrMr\rr]s r,hard_maxChunkingOptions.hard_maxsB  " "<< - -LL$$%56 %1yN7NNr/cXURRS5nUcS$[U5$)zqWhen True, add original elements from pre-chunk to `.metadata.orig_elements` of chunk. Default value is `True`. include_orig_elementsTrMr\rr]s r,rg%ChunkingOptions.include_orig_elementss- LL$$%<=  (t=d9o=r/cXURRS5nUcS$[U5$)zaWhen True, repeat detected table headers in continuation table chunks. Default value is `True`. repeat_table_headersTrhr]s r,rk$ChunkingOptions.repeat_table_headerss- LL$$%;<  (t=d9o=r/cXURRS5nUcS$[U5$)zZWhen True, Table elements are passed through without chunking. Default value is `False`. skip_table_chunkingFrhr]s r,rn#ChunkingOptions.skip_table_chunkings- LL$$%:; !)u>tI>r/cbURRS5nU(a UR$S$)zCharacters of overlap to add between chunks. This applies only to boundaries between chunks formed from whole elements and not to text-splitting boundaries that arise from splitting an oversized element. overlap_allr)rMr\overlap)r*overlap_all_args r,inter_chunk_overlap#ChunkingOptions.inter_chunk_overlaps),,**=9.t||5A5r/cNURRS5nU=(d S$)zThe number of characters to overlap text when splitting chunks mid-text. The actual overlap will not exceed this number of characters but may be less as required to respect splitting-character boundaries. rrrr[)r* overlap_args r,rrChunkingOptions.overlaps#ll&&y1 ar/cURnUR(a)URRS5nUcU$X!:aU$U$URRS5nUcU$X1:aU$U$)zA pre-chunk of this size or greater is considered full. Note that while a value of `0` is valid, it essentially disables chunking by putting each element into its own chunk. new_after_n_tokensnew_after_n_chars)rdrcrMr\)r*rdnew_after_n_tokens_argnew_after_n_chars_args r,soft_maxChunkingOptions.soft_maxs==  " "%)\\%5%56J%K "%-%0) ) $ 0 01D E ! (O ! +O%$r/c[U5$)zA text-splitting function suitable for splitting the text of an oversized pre-chunk. The function is pre-configured with the chosen chunking window size and any other applicable options specified by the caller as part of this chunking-options instance. ) _TextSplitterrVs r,splitChunkingOptions.splitsT""r/cg)a;The string to insert between elements when concatenating their text for a chunk. Right now this is just " " (a blank line in plain text), but having this here rather than as a module-level constant provides a way for us to easily make it user-configurable in future if we want to. z rGrVs r,text_separatorChunkingOptions.text_separatorsr/cXURRS5nUcS$[U5$)zLSequence of text-splitting target strings to be used in order of preference.text_splitting_separators)  )rMr\tuple)r*text_splitting_separators_args r,r)ChunkingOptions.text_splitting_separatorss<)- (8(89T(U%-4  45 r/c`URRS5nU(a [U5$S$)zQThe token counter for token-based chunking, or None for character-based chunking.r+N)rMr\r#r)s r, token_counterChunkingOptions.token_counter$s*LL$$[1 *3|I&==r/c<URRS5SL$)zHTrue when token-based chunking is configured, False for character-based.raNr[rVs r,rc"ChunkingOptions.use_token_counting*s|| -T99r/cUR(a,UR(aURRU5$[U5$)zIReturn the size of `text` in the configured units (characters or tokens).)rcrr<r8r:s r,measureChunkingOptions.measure/s4  " "t'9'9%%++D1 14yr/cURRS5nURRS5nURRS5nUbUb [S5eUbUc [S5eUbUS::a[SU35eURRS 5nUbUc [S 5eUbUS:a[S U35eURnUS::a[S U35eURRS 5nUbUS:a[SU35eURU:a[SURSU35eg)z5Raise ValueError if requestion option-set is invalid.rarbr+Nz\'max_tokens' and 'max_characters' are mutually exclusive; specify one or the other, not bothz/'tokenizer' is required when using 'max_tokens'rz''max_tokens' argument must be > 0, got rzz:'new_after_n_tokens' requires 'max_tokens' to be specifiedz0'new_after_n_tokens' argument must be >= 0, got z+'max_characters' argument must be > 0, got r{z/'new_after_n_chars' argument must be >= 0, got z;'overlap' argument must be less than `max_characters`, got z >= )rMr\ ValueErrorrdrr)r*rarbr+rzrdr{s r,rQChunkingOptions._validate5s\\%%l3 ))*:;LL$$[1   !n&@6   !i&7NO O  !jAoFzlST T"\\--.BC  )j.@YZ Z  ).@1.DBCUBVW  == q=J8*UV V!LL,,-@A  (->-BNO`Nabc c <<8 # ~T(5  $r/rLN)rNr)rNrr@rr@ztuple[BoundaryPredicate, ...]r@rr@r)r@z Callable[[str], tuple[str, str]]r@r>)r@ztuple[str, ...])r@zTokenCounter | Noner?)r@None)rArBrCrDrEr- classmethodrSrrWrZrdrgrkrnrtrrr~rrrrrcrrQrFrGr/r,rIrIUsE5n 99 O O>>>>??66  %%>##  >> :: 0r/rIc"[U[5$)aTrue when ``element`` is a `Table` or a concrete subtype such as `TableChunk`. Subclasses share the same isolation contract: they must not share a pre-chunk with arbitrary text elements, and two table-bearing sequences must not be merged by `PreChunkCombiner`. ) isinstancerelements r,_element_is_table_familyrps gu %%r/c&[SU55$)zJTrue when ``elements`` already includes at least one table-family element.c38# UHn[U5v M g7fr&)r).0es r, 1_elements_contain_table_family..{s=Hq'**H)any)elementss r,_elements_contain_table_familyrys =H= ==r/c<[U5=(d [U5$)a/True when a proposed merge of two element streams must be rejected for table isolation. If either side already contains a table, the combiner must flush before accepting the other side. This keeps `combine_text_under_n_chars` from concatenating a table pre-chunk with neighboring narrative pre-chunks. )r)leftrights r,+_table_isolation_forbids_side_by_side_merger~s *$ / X3QRW3XXr/cj\rSrSrSrS Sjr\S Sj5rS Sjr\ S Sj5r SSjr Sr g ) PreChunkeria9Gathers sequential elements into pre-chunks as length constraints allow. The pre-chunker's responsibilities are: - **Segregate semantic units.** Identify semantic unit boundaries and segregate elements on either side of those boundaries into different sections. In this case, the primary indicator of a semantic boundary is a `Title` element. A page-break (change in page-number) is also a semantic boundary when `multipage_sections` is `False`. - **Minimize chunk count for each semantic unit.** Group the elements within a semantic unit into sections as big as possible without exceeding the chunk window size. - **Minimize chunks that must be split mid-text.** Precompute the text length of each section and only produce a section that exceeds the chunk window size when there is a single element with text longer than that window. A Table element is placed into a section by itself. CheckBox elements are dropped. The "by-title" strategy specifies breaking on section boundaries; a `Title` element indicates a new "section", hence the "by-title" designation. cXlX lgr& _elements_opts)r*roptss r,r-PreChunker.__init__s ! r/c.U"X5R5$)zEGenerate pre-chunks from the element-stream provided on construction.)_iter_pre_chunks)rRrrs r,iter_pre_chunksPreChunker.iter_pre_chunkss 8"3355r/c#># [UR5nURHXnURU5(dUR U5(dUR 5ShvN UR U5 MZ UR 5ShvN gN2N7f)aGenerate pre-chunks from the element-stream provided on construction. A *pre-chunk* is the largest sub-sequence of elements that will both fit within the chunking window and respects the semantic boundary rules of the chunking strategy. When a single element exceeds the chunking window size it is placed in a pre-chunk by itself and is subject to mid-text splitting in the second phase of the chunking process. N)PreChunkBuilderrr_is_in_new_semantic_unitwill_fitflush add_element)r*pre_chunk_builderrs r,rPreChunker._iter_pre_chunkss,DJJ7~~G--g66(11'::,22444  ) )' 2&%**,,,5 -s$A$B&B',BBBBc.URR$)zBThe semantic-boundary detectors to be applied to break pre-chunks.)rrWrVs r,_boundary_predicatesPreChunker._boundary_predicatesszz---r/cdURVs/sH o""U5PM nn[U5$s snf)zITrue when `element` begins a new semantic unit such as a section or page.)rr)r*rpredsemantic_boundariess r,r#PreChunker._is_in_new_semantic_units7 :>9R9RS9RtG}9RS&''Ts-rN)rIterable[Element]rrI)rrrrIr@Iterator[PreChunk]r@rrrrr@r) rArBrCrDrEr-rrrrrrrFrGr/r,rrsV,6(60?6 66 -6..(r/rcr\rSrSrSrS SjrS SjrSSjrSSjr\ SSj5r SSjr \ SS j5r S r g )riaNAn element accumulator suitable for incrementally forming a pre-chunk. Provides the trial method `.will_fit()` a pre-chunker can use to determine whether it should add the next element in the element stream. `.flush()` is used to build a PreChunk object from the accumulated elements. This method returns an iterator that generates zero-or-one `PreChunk` object and is used like so: yield from builder.flush() If no elements have been accumulated, no `PreChunk` instance is generated. Flushing the builder clears the elements it contains so it is ready to build the next pre-chunk. c|Xl[UR5Ul/UlSUl/UlSUlg)Nr)rr8r_separator_lenr_overlap_prefix_text_segments _text_lenr*rs r,r-PreChunkBuilder.__init__s: !$"5"56(*%')+r/ct[UR5S:Xa%[U5(aSUl/UlSUlURR U5 UR(aNURR UR5 U=R [UR5- slgg)zAdd `element` to this section.rrN)r8rrrrrappendr;)r*rs r,rPreChunkBuilder.add_elements t~~ ! #(@(I(I#%D "$D DN g& <<    & &w|| 4 NNc',,/ /N r/c## URnU(dg[XRUR5nURn[ U5S:Xa[ US5(aSnURU5 Uv g7f)zGenerate zero-or-one `PreChunk` object and clear the accumulator. Suitable for use to emit a PreChunk when the maximum size has been reached or a semantic boundary has been reached. Also to clear out a terminal pre-chunk at the end of an element stream. Nrr)rPreChunkrr overlap_tailr8r _reset_state)r*r pre_chunkoverlap_for_nexts r,rPreChunkBuilder.flushsq>> X';';TZZH %11 x=A ":8A;"G"G!  *+sA;A=c[U5(a[UR5S:H$[UR5(ag[UR5S:XagURUR R :agUR R(aUR RRURUR(a UR/O/-5nUR RU5UR R:*$UR[UR=(d S5:+$)a+True when `element` can be added to this prechunk without violating its limits. There are several limits: - A `Table` element will never fit with any other element. It will only fit in an empty pre-chunk. - No element will fit in a pre-chunk that already contains a `Table` element. - A text-element will not fit in a pre-chunk that already exceeds the soft-max (aka. new_after_n_chars/new_after_n_tokens). - A text-element will not fit when together with the elements already present it would exceed the hard-max (aka. max_characters/max_tokens). rFTr)rr8rr _text_lengthrr~rcrjoinrr;rrd_remaining_space)r*rnew_texts r,rPreChunkBuilder.will_fits $G , ,t~~&!+ + *$.. 9 9 t~~ ! #   tzz22 2 :: ( (zz0055## ~2NH::%%h/4::3F3FF F((3w||/Ar+BBBr/cUR[UR5-nURRUR - U- $)zMMaximum text-length of an element that can be added without exceeding maxlen.)rr8rrrdrr*separators_lens r,r PreChunkBuilder._remaining_space?s=,,s43F3F/GGzz""T^^3nDDr/cXlURR5 U(aU/O/Ul[ U5Ulg)zMSet working-state values back to "empty", ready to accumulate next pre-chunk.N)rrclearrr8r)r*overlap_prefixs r,rPreChunkBuilder._reset_stateFs3- 2@~.b^,r/clURR(a\UR(dgURRR UR5nURR U5$[ UR5nU(aUS- OSnURX0R--$)aSize of the text in this pre-chunk (in characters or tokens depending on mode). This value represents the chunk-size that would result if this pre-chunk was flushed in its current state. In particular, it does not include the length of a trailing separator (since that would only appear if an additional element was added). Not suitable for judging remaining space, use `.remaining_space` for that value. rr) rrcrrrrr8rr)r*r;nseparator_counts r,rPreChunkBuilder._text_lengthMs :: ( (&&::,,11$2E2EFD::%%d+ + ## $#$!a%!~~3F3F!FGGr/)rrrrrrNrrIr@r)rrr@rrrr)rr>r@r)rArBrCrDrEr-rrrpropertyrrrrFrGr/r,rrsN    0."CHEE -HHr/rc\rSrSrSrS SjrSSjrSSjrSSjrSSjr \ SSj5r SS jr \ SS j5r S rg )rilzYSequence of elements staged to form a single chunk. This object is purposely immutable. c<[U5UlX lX0lgr&)listrrr)r*rrrs r,r-PreChunk.__init__rsh- r/c[U[5(dgURUR:H=(a URUR:H$)NF)rrrr)r*others r,__eq__PreChunk.__eq__ys>%**##u'<'<<bSXSbSbAbbr/c"[URUR5(ag[UR5URR :ag[UR U5R5nX RR:*$)zRTrue when `pre_chunk` can be combined with this one without exceeding size limits.F)rrr8_textrrZcombinerd)r*r combined_lens r, can_combinePreChunk.can_combine~sg 6t~~yGZGZ [ [ tzz?djjCC C 4<< 2889 zz2222r/cn[URUR-URURS9$)z?Return new `PreChunk` that combines this and `other_pre_chunk`.)rr)rrrr)r*other_pre_chunks r,rPreChunk.combines3 NN_66 6//  r/c## [UR5S:Xa[URS[5(anURR (aURSv g[ RURSURUR5ShvN g[RURURUR5ShvN gNCN7f)a4Form this pre-chunk into one or more chunk elements maxlen or smaller. When the total size of the pre-chunk will fit in the chunking window, a single chunk it emitted. When this prechunk contains an oversized element (always isolated), it is split into two or more chunks that each fit the chunking window. rrN) r8rrrrrn _TableChunker iter_chunksr_ChunkerrrVs r,r PreChunk.iter_chunkss t~~ ! # 4>>!3De(L(Lzz--nnQ''(44NN1%t';';TZZ ++DNNDJJ S S S  Ts$B$C.&C*'=C.$C,%C.,C.c|URRnU(aURU*SR5$S$)aThe portion of this chunk's text to be repeated as a prefix in the next chunk. This value is the empty-string ("") when either the `.overlap` length option is `0` or `.overlap_all` is `False`. When there is a text value, it is stripped of both leading and trailing whitespace. Nr)rrtrstrip)r*rrs r,rPreChunk.overlap_tails6**0007tzz7()$**,?R?r/c## UR(aURv URHnUR(dM[UR5(dM2[ U[ 5(aURv MWSR URR5R55nU(dMUv M g7f)zGenerate overlap text and each element text segment in order. Empty text segments are not included. CodeSnippet elements preserve their original whitespace (including newlines) to maintain code formatting. rN) rrr;r8rrrrr)r*rr;s r,_iter_text_segmentsPreChunk._iter_text_segmentss   && &Avvv#aff++a--&&L88AFFLLN$8$8$:;Dt"  s>CC A!C CchURRRUR55$)zThe concatenated text of all elements in this pre-chunk, including any overlap. Whitespace is normalized to a single space. The text of each element is separated from that of the next by a blank line (" "). )rrrrrVs r,rPreChunk._texts(zz((--d.F.F.HIIr/)rrrN)rrrr>rrIr@r)rrr@rrrr@r)rrr@r)r@z/Iterator[CompositeElement | Table | TableChunk]rr@z Iterator[str])rArBrCrDrEr-rrrr rrrrrFrGr/r,rrlss );>FU c 3 T*@@#$JJr/rc\rSrSrSrS Sjr\SSj5rSSjr\ SSj5r \ SSj5r \ SSj5r \ SS j5r \ SS j5rS rg )r iz{Forms chunks from a pre-chunk other than one containing only a `Table`. Produces zero-or-more `CompositeElement` objects. c<[U5UlX lX0lgr&)rrrr)r*rr;rs r,r-_Chunker.__init__sh  r/c0U"XU5R5$)zForm zero or more chunks from `elements`. One `CompositeElement` is produced when all `elements` will fit. Otherwise there is a single `Text`-subtype element and chunks are formed by splitting.  _iter_chunks)rRrr;rs r,r _Chunker.iter_chunkss84(5577r/c## UR(dgURRnU"UR5up#[X RS9v U(a)U"U5up#[X R S9v U(aM(gg7f)z)Form zero or more chunks from `elements`.Nr;metadata)rrrr_consolidated_metadata_continuation_metadata)r*rs remainders r,r_Chunker._iter_chunkssnzz    TZZ( A0K0KLL +LA"4O4OP Pis A?BBcSSjn[R"[5nURH1nU"UR5HupEX$R U5 M M3 [ U5$)aLCollection of all populated metadata values across elements. The resulting dict has one key for each `ElementMetadata` field that had a non-None value in at least one of the elements in this pre-chunk. The value of that key is a list of all those populated values, in element order, for example: { "filename": ["sample.docx", "sample.docx"], "languages": [["lat"], ["lat", "eng"]] ... } This preprocessing step provides the input for a specified consolidation strategy that will resolve the list of values for each field to a single consolidated value. cDSURR55$)zM(field_name, value) pair for each non-None field in single `ElementMetadata`.c36# UHupUcM X4v M g7fr&rG)r field_namevalues r,rO_Chunker._all_metadata_values..iter_populated_fields..s")F%J$#)Fs  ) known_fieldsitems)r"s r,iter_populated_fields<_Chunker._all_metadata_values..iter_populated_fieldss#)1)>)>)D)D)F r/)r"rr@Iterator[tuple[str, Any]]) collections defaultdictrrr"rdict)r*r0 field_valuesrr+r,s r,_all_metadata_values_Chunker._all_metadata_valuess_$ 5@4K4KD4Q A%:1::%F! (//6&G L!!r/c[S0URD6nURR(aURUlU$)acMetadata applicable to this pre-chunk as a single chunk. Formed by applying consolidation rules to all metadata fields across the elements of this pre-chunk. For the sake of consistency, the same rules are applied (for example, for dropping values) to a single-element pre-chunk too, even though metadata for such a pre-chunk is already "consolidated". rG)r _meta_kwargsrrg_orig_elements orig_elements)r*consolidated_metadatas r,r#_Chunker._consolidated_metadata$s:!0 D$2C2C D :: + +262E2E ! /$$r/cT[R"UR5nSUlU$)zMetadata applicable to the second and later text-split chunks of the pre-chunk. The same metadata as the first text-split chunk but includes `.is_continuation = True`. Unused for non-oversized pre-chunks since those are not subject to text-splitting. T)copyr#is_continuation)r*continuation_metadatas r,r$_Chunker._continuation_metadata4s(!% $*E*E F04-$$r/cr^^^[m[R"5mSUUU4Sjjn[U"55$)a The consolidated metadata values as a dict suitable for constructing ElementMetadata. This is where consolidation strategies are actually applied. The output is suitable for use in constructing an `ElementMetadata` object like `ElementMetadata(**self._meta_kwargs)`. c 3H># TRR5HupTRU5nUTRLa XS4v M/UTRLaU[ U[ S/554v MZUTRLa<UVVs0sHo3HoDS_M M nnnU[UR554v MUTRLaUSRSU554v MUTRLaM[S[U5S35e gs snnf7f)zKGenerate (field-name, value) pairs for each field in consolidated metadata.rz list[Any]Nrc3@# UHoR5v M g7fr&)r)rvals r,rB_Chunker._meta_kwargs..iter_kwarg_pairs..[s.Mfsyy{{fszmetadata field z& has no defined consolidation strategy)r7r/r\FIRSTLIST_CONCATENATEsumr LIST_UNIQUErkeysSTRING_CONCATENATErDROPNotImplementedErrorrepr) r+valuesstrategyval_listkeyordered_unique_keysCSfield_consolidation_strategiesr*s r,iter_kwarg_pairs/_Chunker._meta_kwargs..iter_kwarg_pairsLs&*&?&?&E&E&G" 9==jIrxx'$Qi//!4!44$c&${B2G&HHH/EK*`VW_PS9W_3V'*`$d+>+C+C+E&FFF!6!66$chh.Mf.M&MMM(.)$z*:);;ab''H+asB D" D!BD")r@r2)rrXr5)r*rYrWrXs` @@r,r:_Chunker._meta_kwargsBs4#)>)])])_&  2$&''r/c0^U4Sjn[U"55$)zJThe `.metadata.orig_elements` value for chunks formed from this pre-chunk.c3># TRHKnURRcUv M [R"U5nSURlUv MM g7fr&)rr"r<r@)r orig_elementr*s r,iter_orig_elements3_Chunker._orig_elements..iter_orig_elementsksQ^^::++3G $yy| 7; %%3""$sAA )r)r*r_s` r,r;_Chunker._orig_elementsgs #&())r/)rrrN)rrr;r>rrIr@r)rrr;r>rrIr@Iterator[CompositeElement])r@rb)r@zdict[str, list[Any]]r@r)r@zdict[str, Any]r@z list[Element])rArBrCrDrEr-rr rrr7r#r$r:r;rFrGr/r,r r s  8(8038;J8 #88Q( " "D % % % %"("(H**r/r c \rSrSrSrSSjr\SSj5rSSjr\ SSj5r \ SSj5r \ SSj5r SS jr SS jrSSS jjr\SS j5r\ SS j5r\ SSj5r\ SSj5rSrg)r i{zResponsible for forming chunks, especially splits, from a single-table pre-chunk. Table splitting is specialized because we recursively split on an even row, cell, text boundary. This object encapsulate those details. c(XlX lX0lgr&)_tablerr)r*tablerrs r,r-_TableChunker.__init__s - r/c0U"XU5R5$)LSplit this pre-chunk into `Table` or `TableChunk` objects maxlen or smaller.r)rRrhrrs r,r _TableChunker.iter_chunkss 5$/<<>>r/c## UR(dgURRnURRnU"UR5nUR (aU"UR 5OSnX1::aBXA::a=UR nUR =(d SUl[URUS9v gURR(aSOSnUR (aURRU:aUR5ShvN gUR5ShvN gNN7f)rkNrr!2) _table_textrrdr_text_with_overlap_html _metadata text_as_htmlrrc_iter_text_only_table_chunks _iter_text_and_html_table_chunks)r*maxlenr text_size html_sizer"min_html_thresholds r,r_TableChunker._iter_chunkss $$**$$D334 +/::GDJJ'1  9#6~~H$(JJ$6$H !T44xH H  $(::#@#@RbzzTZZ003EE88: : : 88::: ; ;s$DD?D;D?5D=6D?=D?cBUR=n(dgUR$)zThe compactified HTML for this table when it has text-as-HTML. The empty string when table-structure has not been captured, perhaps because `infer_table_structure` was set `False` in the partitioning call. r) _html_tablehtml)r* html_tables r,rr_TableChunker._htmls"#... .r/c0URRR=ncgUR5nU(dg[R "U5$![ [4a1 [R"S[U5S:aUSSS-OU5 gf=f)zqThe `lxml` HTML element object for this table. `None` when the `Table` element has no `.metadata.text_as_html`. Nz^Could not parse text_as_html for table element; skipping HTML-based chunking. text_as_html: %sdz...) rgr"rtrrfrom_html_textr rrwarningr8)r*rts r,r}_TableChunker._html_tables !KK00== =L F#))+  ++L9 9Z(  NN$.1,.?#.E Tc"U*<    sA>BBcURnUcgSnUR5HnUR(d U$US- nM U$)zJNumber of contiguous leading rows that should be treated as table headers.rr)r} iter_rows is_header)r*rr<rows r,_leading_header_row_count'_TableChunker._leading_header_row_countsQ%%  '')C==  QJE* r/c#># UR=nc [S5eURR(a UROSn[ UURUS9nUR [ RUURUS9URS9ShvN gN7f)zSplit table into chunks where HTML corresponds exactly to text. `.metadata.text_as_html` for each chunk is a parsable `` HTML fragment. Nz# TRnTRRnU(aU"U5up US4v U(aMgg7fr&)rqrr)text_remainderr chunk_textr*s r,_iter_text_splitsE_TableChunker._iter_text_only_table_chunks.._iter_text_splitssB!44NJJ$$E -2>-B*  $&&!.s AAAN)r@zIterator[tuple[str, None]])r)r*rs` r,ru*_TableChunker._iter_text_only_table_chunkss% '**+<+>???s "-+-c#f# [[R"55n[SU5n[ U5HtunupgUR nUbXxlOSUlUS:=(d SUlUS:aUOSUl[XhS9n X9Rl XYRl U v Mv g7f)aForm `TableChunk` objects from (text, html) pairs. Handles `is_continuation` and chunk sequencing metadata (`table_id`, `chunk_index`) so the original table can be reconstructed from its chunks. Carries `num_carried_over_header_rows` so synthetic repeated header rows can be removed. rNr!) r>uuiduuid4max enumeratersrtrArrr"table_id chunk_index) r*text_html_pairsrrcarried_header_row_countrr;r~r"chunks r,r _TableChunker._make_table_chunksstzz|$#&q*F#G )2?)C %K$~~H(,%(,%(3a'@DH $,7!O(  1DrrIr@r)rhrrr>rrIr@Iterator[Table | TableChunk])r@rr)r@zHtmlTable | Noner)r@Iterator[TableChunk]r)rz Iterator[tuple[str, str | None]]rrr@rrcrd)rArBrCrDrEr-rr rrrrr}rrvrurrrsr;rprqrFrGr/r,r r {s  ??+.?6E? %?? ;@  ,   .@(-.9'*  >6  22 TTr/r c4\rSrSrSrSSSjjr\SSSjj5rSSjrSSjr SSjr \ SSj5r \ SS j5r \ SS j5r\ SS j5r\ SS j5r\ SS j5r\ SSj5rSSjrS Sjr\S!Sj5r\S"Sj5rSrg)#rifa\Produces (text, html) pairs for a `
` HtmlElement. Each chunk contains a whole number of rows whenever possible. An oversized row is split on an even cell boundary and a single cell that is by itself too big to fit in the chunking window is divided by text-splitting. The returned `html` value is always a parseable HTML `
` subtree. c>XlX l[SU5UlgNr)_table_elementrr_header_row_count)r* table_elementrrs r,r-_HtmlTableSplitter.__init__ps+ !$Q(8!9r/c,U"XUS9R5$)zGenerate (text, html) pair for each split of this table pre-chunk. Each split is on an even row boundary whenever possible, falling back to even cell and even word boundaries when a row or cell is by itself oversized, respectively. r)_iter_subtables)rRrrrs r,r!_HtmlTableSplitter.iter_subtablesus=9IJZZ\\r/c## Sn[URU5URRS9nURR 5GH nUR U5(d\UR5HupEURXEU5v SnM [URU5URRS9nUR U5(aURU5 MURX0RU5S9HupEURXEU5v SnM [URU5URRS9nGM UR5HupEURXEU5v SnM g7f)zGenerate (text, html) pairs containing as many whole rows as will fit in window. Falls back to splitting rows into whole cells when a single row is by itself too big to fit in the chunking window. T)rwrFrwN) _RowAccumulator_maxlenrrrrrr_prepend_repeated_headersadd_row_iter_row_splits)r*is_first_chunkaccumrr;r~s r,r"_HtmlTableSplitter._iter_subtablessK t||N'CTZZM_M_`&&002C>>#&&"'++-JD88^TT%*N#0(<<7ASAS~~c"" c""&"7"7LLQ_D`"7"aJD88^TT%*N#b(<<7ASAS!3( ++-JD00^L L"N(sFF c#d# [US9nUR5HqnURU5(dUR5ShvN URU5(aUR U5 MZUR XBS9ShvN Ms UR5ShvN gNaN#N 7f)zQSplit oversized row into (text, html) pairs containing as many cells as will fit.rN)_CellAccumulator iter_cellsrradd_cell_iter_cell_splits)r*rrwrcells r,r#_HtmlTableSplitter._iter_row_splitss /NN$D>>$'' ;;=((~~d##t$11$1FFF%;;=  ) G s6AB0B* ?B0B, B0$B.%B0,B0.B0c## URR(a;[[SUS- 5URRR S5S9nO[[SUS- 5S9n[ U5nU"UR5upVUSUS34v U(aU"U5upVUSUS34v U(aMg g 7f) zDSplit a single oversized cell into sub-sub-sub-table HTML fragments.r r+)rar+!)rbz
z
N)rrcrIrrMr\rr;)r*rrwrrr;r&s r,r$_HtmlTableSplitter._iter_cell_splitss :: ( (#q&2+.**,,00=D ##a"2EFDd# *odV+=>>>#I.OD/$/ABB Bis B8B><B>cFSRSUR55$)z?Concatenated text for leading header rows identified by caller.rc3R# UHoR5Ho"v M M g7fr&)iter_cell_texts)rrr;s r,r2_HtmlTableSplitter._header_text..s"\(9FYFYF[dF[(9s%')r _header_rowsrVs r, _header_text_HtmlTableSplitter._header_textsxx\(9(9\\\r/cURS::ag/n[URR55H'up#X R:a OUR U5 M) [ U5$)zDLeading rows that should be repeated on continuation chunks, if any.rrG)rrrrrr)r*rowsidxrs r,r_HtmlTableSplitter._header_rowssb  ! !Q & !$"5"5"?"?"ABHC,,, KK CT{r/c~^TR(dgSRU4SjTR55nSUS3$)z;HTML for repeated header rows, preserving header semantics.rc3F># UHnTRU5v M g7fr&)_as_header_row_htmlrrr*s r,r7_HtmlTableSplitter._header_rows_html..s!WEVcD44S99EVs!zz)rr)r* rows_htmls` r,_header_rows_html$_HtmlTableSplitter._header_rows_htmls:  GGWTEVEVWW  8,,r/cRUR(a[UR5$S$)zJHeader-row count prepended to each continuation chunk, or 0 when disabled.r)_should_repeat_headersr8rrVs r,r0_HtmlTableSplitter.carried_over_header_row_counts"*.)D)Ds4$$%K!Kr/cxUR(dgURURRS-S-:*$)zHTrue when header repetition is enabled and not pathologically expensive.Fr)r_max_header_row_lenrrdrVs r,r)_HtmlTableSplitter._should_repeat_headerss7  ''DJJ,?,?!,C+IIIr/cf^TR(dg[U4SjTR55$)z'Largest leading-header row text length.rc3># UH=nTRRSRUR555v M? g7f)rN)rrrrrs r,r9_HtmlTableSplitter._max_header_row_len..s7dRc34::%%chhs/B/B/D&EFFRcsAA)rrrVs`r,r&_HtmlTableSplitter._max_header_row_lens)  dRVRcRcdddr/cLURRUR5$)z/Size of repeated header text in chunking units.)rrrrVs r,_header_text_len#_HtmlTableSplitter._header_text_lenszz!!$"3"344r/cU(dUR(dURR$[SURRUR- S- 5$)z)rr removeprefix removesuffixr)r*r;r~r header_textr html_inner chunk_htmls r,r,_HtmlTableSplitter._prepend_repeated_headerssy !>zJ t556zl(K %%r/cPUR=(d URn[RU5nUc0UR(a[RUR5nUc UR$UHn[ USS5S:XdMSUlM [ U[S9$)zVSerialize `row` preserving source HTML while converting direct-child `` to ``.Ntagtdth)encoding) source_htmlr~r_parse_row_fragmentgetattrrr r>)rrow_htmltrrs r,r&_HtmlTableSplitter._as_header_row_htmls??.chh  3 3H = :#//#77AB :88ODtUD)T1S))r/c[U5nURS:XaU$UR S5nU(aUS$S$![[4a gf=f)z>Parse `row_html` and return a `` element when recoverable.Nr z.//trr)r r rrxpath)r parsedrs r,r &_HtmlTableSplitter._parse_row_fragment#s\ (2F :: M||G$tAw(D(Z(  s >AA)rrrNr)rrrrIrr)rrrrIrrr@Iterator[TextAndHtml]r@r)rrrwrr@r)rrrwrr@rr)r@ztuple[HtmlRow, ...]rr)rrr@r)r;r>r~r>rrr@r!)rrr@r>)r r>)rArBrCrDrEr-rrrrrrrrrrrrrrr staticmethodrr rFrGr/r,rrfs!: VW]%]-<]PS] ]]#B! C0]]  --LLJJee 55G & * * ) )r/rcn\rSrSrSrS SjrS SjrS SjrS Sjr\ SSj5r SSjr S r g )ri2aProvides a text-splitting function configured on construction. Text is split on the best-available separator, falling-back from the preferred separator through a sequence of alternate separators. - The separator is removed by splitting so only whitespace strings are suitable separators. - A "blank-line" (" ") is unlikely to occur in an element as it would have been used as an element boundary during partitioning. This is a *callable* object. Constructing it essentially produces a function: split = _TextSplitter(opts) fragment, remainder = split(s) This allows it to be configured with length-options etc. on construction and used throughout a chunking operation on a given element-stream. cXlgr&rrs r,r-_TextSplitter.__init__Es r/c>URRnURR(a4URRU5U::aUS4$UR U5$[ U5U::aUS4$UR HZup4URX4U5upVU(a[ U5[ U5:aM:UR5UR54s $ USUR5XURR- SR54$)aReturn pair of strings split from `s` on the best match of configured patterns. The first string is the split, the second is the remainder of the string. The split string will never be longer than `maxlen` (in characters or tokens depending on mode). The separators are tried in order until a match is found. The last separator is "" which matches between any two characters so there will always be a split. The separator is removed and does not appear in the split or remainder. An `s` that is already less than the maximum length is returned unchanged with no remainder. This allows this function to be called repeatedly with the remainder until it is consumed and returns a remainder of "". rN) rrdrcr_split_by_tokensr8 _patterns_split_from_maxlenrstriplstriprr)r*r%rwpsep_lenfragmentr&s r,__call___TextSplitter.__call__Hs$$ :: ( (zz!!!$."u ((+ + q6V b5L..JA#'"9"9!a"H Hy>SV+??$i&6&6&88 8) &z  "Atzz/A/A&A&C$D$K$K$MMMr/c URRnURRnURRnURHupV[ UR U55nUHnUR5upUSU R5n U"U 5U::dM6U"U 5S:XaMDXSR5n US:a URX5n U S-U -nX4s s $X4s s $ M S[U5nn[US-S5nUU::a,UU-S-nU"USU5U::aUnUS-nOUS- nUU::aM,Un[U[US5S5H0nU[U5:dMUUR5(dM.Un O USUR5n U"U 5U:aUUS-:aUSUR5n UnUUSR5n US:a#U (aURX5n U S-U -nX4$X4$)aSplit text `s` on a separator boundary while respecting token limits. Tries each separator in order of preference, looking for the rightmost split position that keeps the fragment under the token limit. Falls back to splitting on whitespace boundaries if no separator works. Nrrrr)rrdrrrrrfinditerspanrr_get_token_overlap_tailr8rrangeisspace)r*r%rwrrrpattern_matchesmatch match_start match_endr! raw_remaindertailoverlapped_remainderlowhighbest_posmid split_posis r,r_TextSplitter._split_by_tokensts*$$**$$**$$..JG7++A./G ).& \k?1138$.x(A- $%jM$8$8$:M{#;;HN/3czM/I,'==#22! )0s1vTw{A&Tk:!#Cq#w6)AgQw Tk xWa"5A3q6zadllnn 6 Zi='') 8 v %)gk*A(|**,H I)* ,,. Q;8//BD#'#: #= 1 1&&r/cpURRnU"U5U::aUR5$S[U5pTXE:a'XE-S-nXSnU"U5nX:aUS-nOUnXE:aM'Un U [U5:aDXR 5(d-U S- n U [U5:aXR 5(dM-U [U5:aDXR 5(a-U S- n U [U5:aXR 5(aM-U [U5:aXSR 5$XS$)zExtract tail of text containing approximately `target_tokens` tokens. Uses binary search to find the character position from which the tail contains approximately the specified number of tokens. Adjusts to word boundaries to avoid splitting words. rrNr)rrrr8r*r) r*r; target_tokensrr4r5r7r2 token_countposs r,r(%_TextSplitter._get_token_overlap_tails,**$$ 4=M )::< s4yTj:!#C:D!$-K*AgjCIodi&7&7&9&9 1HCCIodi&7&7&9&9CIo$)"3"3"5"5 1HCCIo$)"3"3"5"5 #d) :$$& &Dzr/cRURRn[SU55$)aSequence of (pattern, len) pairs to match against. Patterns appear in order of preference, those following are "fall-back" patterns to be used if no match of a prior pattern is found. NOTE these regexes search *from the end of the string*, which is what the "(?r)" bit specifies. This is much more efficient than starting at the beginning of the string which could result in hundreds of matches before the desired one. c3j# UH)n[R"SU35[U54v M+ g7f)z(?r)N)regexcompiler8)rseps r,r*_TextSplitter._patterns..s(S emmd3%L13s8< s13)rrr)r* separatorss r,r_TextSplitter._patternss$ZZ99 S SSSr/c|URRURRpTURX5S-XB-S9nUcSU4$UR 5upxSn USUR 5n X8SR 5n U[U 5::aX4$U[U 5- n X*SR 5n X-U -nX4$)aReturn (split, remainder) pair split from `s` on the right-most match before `maxlen`. Returns `"", s` if no suitable match was found. Also returns `"", s` if splitting on this separator produces a split shorter than the required overlap (which would produce an infinite loop). `split` will never be longer than `maxlen` and there is no longer split available using `pattern`. The separator is removed and does not appear in either the split or remainder. r)r>endposNrr)rrdrrsearchr'rrr8)r*r+r r%rwrrr.r/r0 separatorr!r1tail_lenr2r3s r,r _TextSplitter._split_from_maxlens**--tzz/A/Aqk&:JK =q5L"'  \k?))+* ,,. c)n $* *S^+ #**,#/-?--r/rN)rrI)r%r>r@tuple[str, str])r;r>r<rr@r>)r@z*tuple[tuple[regex.Pattern[str], int], ...])r+zregex.Pattern[str]r rr%r>r@rN) rArBrCrDrEr-r"rr(rrrrFrGr/r,rr2s[$*NXD'L$L T T(.)(.47(.` fragment cell-by-cell to maximally fill chunking window. Accumulate cells until chunking window is filled, then generate the text and HTML for the subtable composed of all those rows that fit in the window. cXl/Ulgr&)r_cells)r*rws r,r-_CellAccumulator.__init__ s &( r/c:URRU5 g)zPAdd `cell` to this accumulation. Caller is responsible for ensuring it will fit.N)rQrr*rs r,r_CellAccumulator.add_cell$s 4 r/c## UR(dgSRUR55nSRSUR55nSUS3nURR5 X4v g7f)zFGenerate zero-or-one (text, html) pairs for accumulated sub-sub-table.Nrrc38# UHoRv M g7fr&r~rcs r,r)_CellAccumulator.flush..-s6+Q&&+rz z
)rQr_iter_cell_textsr)r*r;tds_strr~s r,r_CellAccumulator.flush(sd{{ xx--/0''6$++66WI]3 jsA:A<cFUR[UR5:$)zLTrue when `cell` will fit within remaining space left by accummulated cells.)rr8r;rTs r,r_CellAccumulator.will_fit2s$$DII66r/c#d# URHnUR=n(dMUv M g7f)zGenerate contents of each accumulated cell as a separate string. A cell that is empty or contains only whitespace does not generate a string. N)rQr;)r*rr;s r,r\!_CellAccumulator._iter_cell_texts6s, KKD II%D%J s.0c[UR5nURU- [SUR55- $)zHNumber of characters remaining when text of accumulated cells is joined.c3L# UHn[UR5v M g7fr&)r8r;rYs r,r4_CellAccumulator._remaining_space..Fs2T 13qvv;; s"$)r8rQrrKrs r,r!_CellAccumulator._remaining_space@s5 T[[)||n,s2T 2T/TTTr/)rQrN)rwr)rrr@rr)rrr@rrr) rArBrCrDrEr-rrrr\rrrFrGr/r,rrs6 )!7UUr/rcn\rSrSrSr\4S SjjrS SjrSSjrSSjr SSjr \ SSj5r SS jr S rg )riIzMaybe `SubtableAccumulator`. Accumulate rows until chunking window is filled, then generate the text and HTML for the subtable composed of all those rows that fit in the window. c8XlX l/UlSUlgr)r_measure_rows _row_text_len)r*rwrs r,r-_RowAccumulator.__init__Ps  $& r/cURRU5 U=RURU5- slg)zOAdd `row` to this accumulation. Caller is responsible for ensuring it will fit.N)rjrrk_measured_row_text_lenr*rs r,r_RowAccumulator.add_rowVs0 # d99#>>r/c# # UR(dgSRUR55nSRSUR55nSUS3nURR5 SUlX4v g7f)zBGenerate zero-or-one (text, html) pairs for accumulated sub-table.Nrrc38# UHoRv M g7fr&rX)rrs r,r(_RowAccumulator.flush..`s5*Q&&*rrrr)rjrr\rrk)r*r;trs_strr~s r,r_RowAccumulator.flush[slzz xx--/0''5$**55 * jsBBc>URURU5:$)zJTrue when `row` will fit within remaining space left by accummulated rows.)rrnros r,r_RowAccumulator.will_fitfs$$(C(CC(HHHr/c#f# URHnUR5ShvN M gN 7f)zGenerate contents of each row cell as a separate string. A cell that is empty or contains only whitespace does not generate a string. N)rjr)r*rss r,r\ _RowAccumulator._iter_cell_textsjs* A((* * * *s #1/ 1cd[UR5nURU- UR- $)z>Number of chunk-size units remaining for accumulated row text.)r8rjrrkrs r,r _RowAccumulator._remaining_spacers+ TZZ||n,t/A/AAAr/c^URSRUR555$)z4Length of `row` text in configured chunk-size units.r)rirrros r,rn&_RowAccumulator._measured_row_text_lenzs#}}SXXc&9&9&;<==r/)rrirkrjN)rwrrzCallable[[str], int])rrr@rr)rrr@rrr)rrr@r)rArBrCrDrEr8r-rrrr\rrrnrFrGr/r,rrIsC EH ? I+BB>r/rc,\rSrSrSrSSjrSSjrSrg) PreChunkCombinerizDFilters pre-chunk stream to combine small pre-chunks where possible.cXlX lgr&) _pre_chunksr)r* pre_chunksrs r,r-PreChunkCombiner.__init__s % r/c## [UR5nURHBnURU5(dUR 5ShvN UR U5 MD UR 5ShvN gN2N7f)zTGenerate pre-chunk objects, combining `PreChunk` objects when they'll fit in window.N)_PreChunkAccumulatorrrrr add_pre_chunk)r*rrs r,iter_combined_pre_chunks)PreChunkCombiner.iter_combined_pre_chunkssj$TZZ0))I>>),, ;;=((    * *;;=  ) !s$ABB,B=B>BB)rrN)rzIterable[PreChunk]rrIr)rArBrCrDrEr-rrFrGr/r,rrsN !r/rc@\rSrSrSrS SjrS SjrS SjrS SjrSr g) ria+Accumulates, measures, and combines pre-chunks. Used for combining pre-chunks for chunking strategies like "by-title" that can potentially produce undersized chunks and offer the `combine_text_under_n_chars` option. Provides `.add_pre_chunk()` allowing a pre-chunk to be added to the chunk and provides monitoring properties `.remaining_space` and `.text_length` suitable for deciding whether to add another pre-chunk. `.flush()` is used to combine the accumulated pre-chunks into a single `PreChunk` object. This method returns an interator that generates zero-or-one `PreChunk` objects and is used like so: yield from accum.flush() If no pre-chunks have been accumulated, no `PreChunk` is generated. Flushing the builder clears the pre-chunks it contains so it is ready to accept the next pre-chunk. cXlSUlgr&r _pre_chunkrs r,r-_PreChunkAccumulator.__init__s +/r/cnURcUUlgURRU5Ulg)zPAdd a pre-chunk to the accumulator for possible combination with next pre-chunk.N)rrr*rs r,r"_PreChunkAccumulator.add_pre_chunks00I 6:oo6M6Mi6X r/c#Z# UR(dgURv SUlg7f)z~Generate accumulated pre-chunk as a single combined pre-chunk. Does not generate a pre-chunk when none has been accumulated. N)rrVs r,r_PreChunkAccumulator.flushs"  oos)+cTURcgURRU5$)aTrue when there is room for `pre_chunk` in accumulator. An empty accumulator always has room. Otherwise there is only room when `pre_chunk` can be combined with any other pre-chunks in the accumulator without exceeding the combination limits specified for the chunking run. T)rrrs r,r_PreChunkAccumulator.will_fits& ?? "**955r/rNr)rrr@rrr) rArBrCrDrEr-rrrrFrGr/r,rrs&0  6r/rc$^^SmSmSUU4SjjnU$)atNot a predicate itself, calling this returns a predicate that triggers on each new page. The lifetime of the returned callable cannot extend beyond a single element-stream because it stores current state (current page-number) that is particular to that element stream. The returned predicate tracks the "current" page-number, starting at 1. An element with a greater page number returns True, indicating the element starts a new page boundary, and updates the enclosed page-number ready for the next transition. An element with `page_number == None` or a page-number lower than the stored value is ignored and returns False. rTcv>URRnT(aU=(d SmSmgUcgUT:XagUmg)NrFT)r" page_number)rrcurrent_page_numberis_firsts r,page_number_incremented0is_on_next_page..page_number_incrementedsP&&22 "-"2 H   - -*r/rrG)rrrs @@r,is_on_next_pagers" !H6 #"r/c"[U[5$)z:True when `element` is a `Title` element, False otherwise.)rrrs r,is_titlers gu %%r/r)rrr@r)rrrrr@r)r@r )ArE __future__rr3r@r functoolsrtypingrrrrr r rB lxml.etreer r lxml.htmlr typing_extensionsrrunstructured.common.html_tablerrrunstructured.documents.elementsrrrrrrrrunstructured.loggerrr__annotations__rrr rr>r!r#rIrrrrrrr r rrrrrrrrrGr/r,rsqF" %GG ,)-GG   ' #""&$% ( 4898XsCx Y(//>PPv&> Y  Y  Y  Y$H(H(VIHIHbcJcJVa*a*HcTcTVI)I)Xd.d.N-U-U`3>3>v!!*6666X+#\&r/