Mi^ %SSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK r SSK r SSK J r Jr SSKJr SSKJrJrJrJrJrJrJrJrJrJr SSKrSSKJrJr SSK J!r! \(aSSK"J#r#J$r$ \\%\%\%\%4r&S \'S '\\%\%4r(S \'S '\\(S 4r)S \'S 'Sr*\"S5r+\"S5r,S+Sjr-S,Sjr.S-Sjr/S.Sjr0S/S0Sjjr1\ "SS9S1Sj5r2S2Sjr3S3Sjr4S4Sjr5S5Sjr6S5Sjr7Sr8S6Sjr9S7S jr:S8S!jr;S9S:S"jjr<S;SS$jjr>S?S%jr?S@SAS&jjr@S@SBS'jjrASCSDS(jjrB"S)S*5rCg)E) annotationsN) lru_cachewraps) combinations) TYPE_CHECKINGAnyCallableIterableIteratorListOptionalTupleTypeVarcast) ParamSpec TypeAlias) __version__)ElementTextrBoxPoint.Points)z%Y-%m-%dz%Y-%m-%dT%H:%M:%Sz%Y-%m-%d+%H:%M:%Sz%Y-%m-%dT%H:%M:%S%z_T_Pc R[R"U5n[S0[[URU55DUD6nURR 5HHnUR U;dMURURLdM0URXER 'MJ U$)zMMap both explicit and default arguments of decorated func call by param name.) inspect signaturedictzip parametersvaluesnamedefaultempty)funcargskwargssig call_argsargs l/var/www/eduai.edurigo.com/storigo/production/storigo_env/lib/python3.13/site-packages/unstructured/utils.pyget_call_args_applying_defaultsr-/s   D !C $ QtC,E'F Q& QI~~$$& 889 $CII)E"%++Ihh ' cJUR[R"55$)zTrue when file_path is in the Python-defined tempdir. The Python-defined temp directory is platform dependent (macOS != Linux != Windows) and can also be determined by an environment variable (TMPDIR, TEMP, or TMP). ) startswithtempfile gettempdir) file_paths r,is_temp_file_pathr4=s    3 3 5 66r.c[US5nURSU55 SSS5 g!,(df  g=f)Nzw+c3T# UHn[R"U5S-v M g7f) N)jsondumps).0datums r, save_as_jsonl..HsJTEtzz%047Ts&()open writelines)datafilename output_files r, save_as_jsonlrCFs. h JTJJ   s/ =c[U5nUVs/sHn[R"U5PM snsSSS5 $s snf!,(df  g=fN)r>r8loads)rA input_filelines r,read_from_jsonlrIKs8 h:-78ZT 4 Z8 8 sA =AA AcL^^[T[5(aT/mSUU4SjjnU$)Nc>^^UU4Sjm[T5SUU4Sjj5n[T5SUU4Sjj5n[R"T5(aU$U$)Nc>/nTH&n[U5(aMURU5 M( [U5S:aG[SSR U5S3T(a STS3-5eSSR U5S 3-5eg) Nrz$Following dependencies are missing: z, . z5Please install them using `pip install "unstructured[z]"`.z'Please install them using `pip install  z`.)dependency_existsappendlen ImportErrorjoin) missing_depsdep dependenciesextrass r, run_check;requires_dependencies..decorator..run_checkXs&(L#(-- '',$< 1$!:499\;R:SSUV"RRXQYY]` GsxxP\G]F^^`a %r.c">T"5 T"U0UD6$rErr'r(r&rXs r,wrapper9requires_dependencies..decorator..wrappergs K(( (r.c>># T"5 T"U0UD6IShvN $N7frErr[s r, wrapper_async?requires_dependencies..decorator..wrapper_asyncls! Kt.v.. ..s )r'_P.argsr( _P.kwargs)rasyncioiscoroutinefunction)r&r\r_rXrVrWs` @r, decorator(requires_dependencies..decoratorWsV  t )  ) t /  /  & &t , , r.)r&Callable[_P, _T]returnrg) isinstancestr)rVrWres`` r,requires_dependenciesrkPs+,$$$~ < r.)maxsizec[R"U5 g![anU[U5;aSnAgSnAgSnAff=f)NFT) importlib import_modulerRrepr) dependencyes r,rOrOxsC +   a  !  s A<Ach[U5n[U5nX!4$![a [S5ef=f)NzYExpected at least 1 element in iterable from which to retrieve first, got empty iterable.)iternext StopIteration ValueError)ititeratorouts r,_first_and_remaining_iteratorr|sFBxH 8n =      s 1c [U5upU$)zRReturns the first item from an iterable. Raises an error if the iterable is empty.)r|)ryr{_s r,firstrs *2 .FC Jr.cd[U5up[SU55(a [S5eU$)ziReturns the only element from a singleton iterable. Raises an error if the iterable is not a singleton. c3&# UHnSv M g7f)TNr)r:r~s r,r<only..s "A4szRExpected only 1 element in passed argument, instead there are at least 2 elements.)r|anyrx)ryr{rzs r,onlyrs8 2"5MC " """ `   Jr.c[[R"S5=(d SR55=(d6 [[R"S5=(d SR55$)zTrue if telemetry should be disabled via env. DO_NOT_TRACK and SCARF_NO_ANALYTICS both follow the same rule: any non-empty value (after strip) opts out. See README/CHANGELOG for the public contract. DO_NOT_TRACKSCARF_NO_ANALYTICS)boolosgetenvstriprr.r,_telemetry_opt_outrsO >*0b779 : d ' ( .B557?r.c~[R"S5=(d SR5R5S;$)zLTrue if telemetry is explicitly enabled via env. Only 'true' and '1' opt in.UNSTRUCTURED_TELEMETRY_ENABLEDr)true1)rrrlowerrr.r,_telemetry_opt_inrs6 II6 7 =2 D D F L L NS r.c [5(d[5(dg[R"S/[RS9 SnSR[R"5RS5SS5n[R"[5 [R"S[ [R""5U[R$"5['U5['S [ ;5R)5S .S S 9 SSS5 g![ [R 4a SnNf=f!,(df  g=f) zSend a lightweight analytics ping. Off by default. Set UNSTRUCTURED_TELEMETRY_ENABLED=true to opt in. Opt-out env vars (DO_NOT_TRACK, SCARF_NO_ANALYTICS): any non-empty value opts out. Nz nvidia-smi)stderrTF.z1https://packages.unstructured.io/python-telemetrydev)versionplatformpythonarchgpur )paramstimeout)rr subprocess check_outputDEVNULLOSErrorCalledProcessErrorrSrpython_versionsplit contextlibsuppress Exceptionrequestsgetrsystemmachinerjr) gpu_presentrs r,scarf_analyticsrs #4#6#6z7I7IJ XXh557==cB2AFGN   Y ' ?&$OO-( ((*;'5K/0668   ( ' Z22 3  ( 's&DA/D1D.-D.1 D?c US::a[SU35e[[U5U- S-5Vs/sHn[XX!-5PM sn$s snf)zSGenerate n-grams from a list of strings where `n` (int) is the size of each n-gram.rz!n must be positive, received n = )rxrangerQtuple)snis r,ngramsrsT Av ?)>AE!, )> ?? ?sA cU(d S[54$[UR5U5n[UR5U5nU(d S[54$[U5[U5-n[U5[U5- S-nXe4$)zTCalculate the percentage of common_ngrams between string A and B with reference to Ard)setrrrQ) first_string second_stringrfirst_string_ngramssecond_string_ngrams common_ngrams percentages r,!calculate_shared_ngram_percentagers #%x !3!3!5q9!-"5"5"7; #%x+,s3G/HHMm$s+>'??3FJ  $$r.c[5n[UR55[UR55:a[UR55S- nO[UR55S- nXpSnU(d$[UUU5upBUS:XaOUS-nU(dM$[ US5U[ US-54$)a3From two strings, calculate the shared ngram percentage. Returns a tuple containing... - The largest n-gram percentage shared between the two strings. - A set containing the shared n-grams found during the calculation. - A string representation of the size of the largest shared n-grams found. rrr)rrQrrroundrj)rr shared_ngramsrngram_percentages r,"calculate_largest_ngram_percentagers+.%M <    3}':':'<#==  ""$ % )  ##% & *&3m*K   + ' 6  FA !1 %}c!a%j @@r.c[U5S:wag/SQnU(aN[U5S:Xa?[U5nUS==U-ss'US==U-ss'US==U- ss'US==U- ss'[U5S:Xa1USUS:a%USUS:aUSUS::a USUS::ag[U5S:H=(aC USUSs=:*=(a US:*Os =(a USUSs=:*=(a US:*$s $) zTrue if the child_target bounding box is nested in the parent_target. Box format: [x_bottom_left, y_bottom_left, x_top_right, y_top_right]. The parameter 'add' is the pixel error tolerance for extra pixels outside the parent region FrrrrrrrT)rQlist) parent_target child_targetaddparent_targetss r, is_parent_boxrs)  =Q!N s=!Q&m,qS qS qS qS  LQ !_q 1 1l1oXYIZ6Z !_q 1 1l1oXYIZ6Z |  ! q\!_AAq0AA F 1 a E EN14E E Er.cUSup4USupVUSupxUSupXS- Xd- -n X- X- -n [X75n [XH5n[XY5n[Xj5n[SX- 5[SUU- 5-n[X5n[X5nX-nUS:XaUS:XagUU- S-nO,US:XaUS:XagUU- S-nOX-S:XagUX-U- - S-n[US5UUU4$)a#Calculate the percentage of overlapped region. Calculate the percentage with reference to the biggest element-region (intersection_ratio_method="parent"), the smallest element-region (intersection_ratio_method="partial"), or the disjunctive union region (intersection_ratio_method="total") rrparentrrpartial)maxminr)box1box2intersection_ratio_methodx1y1x2y2x3y3x4y4 area_box1 area_box2x_intersection1y_intersection1x_intersection2y_intersection2intersection_areamax_areamin_area total_areaoverlap_percentages r,calculate_overlap_percentager+s;!WFB !WFB !WFB !WFBRW%IRW%I"kO"kO"kO"kOA@AC /)E9(H9(H&J H, q=/(:cA "i / q=/(:cA  !a '/93HK\3\]add #Q '8Z GGr.c`SupVpxUupUupUupUunn[U U SS9unnnnXt:aU SUS3U SUS3/nSnOU (dU SUS3U SUS3/nSU 3nOU(dU SUS3U SUS3/nSU 3nOX;dX;aU SUS3U SUS3/nSnO[X5unnn[US 5nU(dU SUS3U SUS3/nS nO\U SUS3U SUS3/n[U R 55[UR 55:aU OU nS U-S US 3-nSUSU3nUUUUUUU4$)auClassifies the overlapping case for an element_pair input. There are 5 cases of overlapping: 'Small partial overlap' 'Partial overlap with empty content' 'Partial overlap with duplicate text (sharing 100% of the text)' 'Partial overlap without sharing text' 'Partial overlap sharing {calculate_largest_ngram_percentage(...)}% of the text' Returns: overlapping_elements: List[str] - List of element types with their `ix` value. Ex: ['Title(ix=0)'] overlapping_case: str - See list of cases above overlap_percentage: float largest_ngram_percentage: float max_area: float min_area: float total_area: float )NNNNrr(ix=)zSmall partial overlapz&partial overlap with empty content in z#partial overlap with duplicate textrz$partial overlap without sharing textzof the text from(z-gram)zpartial overlap sharing z% )rrrrQr)box_pair label_pair text_pairix_pairsm_overlap_thresholdoverlapping_elementsoverlapping_caserlargest_ngram_percentagerrtype1type2text1text2 ix_element1 ix_element2rrrr~ largest_nref_types r,identify_overlapping_caser\s54\X,> JDLELE&K9U  "+:6(J 0gT+a (gT+a ( 3'k]!,'k]!,$ "HwO 'k]!,'k]!,$ "HwO  ^u~'k]!,'k]!,$  E 6X6 2 $a(--Eq'I $+gT+a0gT+a0($$J gT+a0gT+a0($%( $6U[[]9K$K5QV-8Qyk;PP%=>V=WWYZbYc#d   r.c&USupUSup4XX44$)zAccepts a set of Points and returns the lower-left and upper-right coordinates. Expects four coordinates representing the corners of a rectangle, listed in this order: bottom-left, top-left, top-right, bottom-right. rrr) coordinatesx_bottom_left_1y_bottom_left_1 x_top_right_1 y_top_right_1s r,_convert_coordinates_to_boxr s' (31~$O#.q> M ] IIr.c UupVUupxSRUV s/sHoR5(dMU PM sn 5n SRUV s/sHoR5(dMU PM sn 5n USSR5nUSSR5n[U5n [U5n U upnnU unnnnUU:=(a UU:nUU:=(a UU:nSunnnnnnSunnn U(aU(a[ UUSS9un n![ UUSS9unnnn [ XUS 9(a#US U S 3US U S 3/nS US U3nSnUS U S 3nOH[ XUS 9(a#US U S 3US U S 3/nS US U3nSnUS U S 3nO[ UUUX4US9unnnnnnn UUUU=(d SU=(d SU=(d SU=(d SU=(d SU =(d S4 $s sn fs sn f)aIdentify if overlapping or nesting elements exist and, if so, the type of overlapping case. Returns: overlapping_elements: List[str] - List of element types & their `ix` value. Ex: ['Title(ix=0)'] overlapping_case: str - See list of cases above overlap_percentage: float overlap_percentage_total: float largest_ngram_percentage: float max_area: float min_area: float total_area: float rrN)NNNNNN)NNNtotalrr)rrrznested z in r)rr)rS isnumericrr rrr)"rrrnested_error_tolerance_pxrrrrrchrr box1_corners box2_cornersrrrr x_bottom_left_2y_bottom_left_2 x_top_right_2 y_top_right_2horizontal_overlapvertical_overlaprparent_elementrroverlap_percentage_totalrrrrr~s" r,$identify_overlapping_or_nesting_casers&JDLE''@"2@AK''@"2@AK !"IOO E !"IOO E.t4L.t4LEQBOm]EQBO_m](=8\]_=\&6Z=?;Z   &6"Hh .,H  &-- ) !Q >Z  &.> :Hh 9R S'k]!,'k]!,$ ")tE7; !$  %wd;-q9N <;T U'k]!,'k]!,$ ")tE7; !$  %wd;-q9N**%9  $ "( a %A %A A Aa  UA@sGGG!,G!cUSRR=(d Sn[U5Vs/sHn/PM nn[U5Vs/sHn/PM nn[U5Vs/sHn/PM nn[U5HupU RR=(d Sn U S- n U RR(aH[ [ U RRR5S5n X[RU 5 XkRUSU R35 X{RU R5 M Sn /n[[XVU5SS9Hun unnn[[US55n[[US55n[[US55n[UUU5Huunnn[UUUUU5u nnnnnnnnn U(dM)URUUUUS 3UUS 3[US5S 3[US5S 3[U S5S 3S .S .5 S n Mw M X4$s snfs snfs snf)zLCatch overlapping and nested bounding boxes cases across a list of elements.rrpointsrMF)startr%upxˆ2)rrrrr)rrrrmetadataT)r  page_numberr enumeraterrrto_dictrPcategorytextr rrrr)!elementsrr num_pagesr~pages_of_bboxes text_labels text_contentixelementr! n_page_to_ixboxdocument_with_overlapping_flagoverlapping_cases page_bboxes page_labels page_textpage_bboxes_combinationspage_labels_combinationstext_content_combinationsrrrrrrrrrrrrs! r,#catch_overlapping_and_nested_bboxesr79s~ %%116QI7G O,7?: :k; $( [!(D#E #' [!(D#E $(i)C$D!/2 $ $ %0 +Hj)5)$  $ "(( !((0D*8,<1C0DA.F8P;S:TTU8V+01+=*>e(D+01+=*>e(D-2:q-A,B%*H % 26.O0 ?b * <>> elements = partition("example.pdf") >>> grouped = group_elements_by_parent_id(elements) >>> for parent_id, children in grouped.items(): ... print(f"Parent {parent_id}: {len(children)} children") >>> # Assign orphan elements to previous element's group >>> grouped = group_elements_by_parent_id(elements, assign_orphans=True) r) defaultdictN parent_id) collectionsr9rgetattrr rPr)r&assign_orphansr9groupslast_parent_idr,r:s r,group_elements_by_parent_idr@si4(3>t3DF$(NG,,k4@  &I  "&N  ) <r.c4\rSrSrSSjrSrS SjrSrSrg) FileHandlericDXl[R"5UlgrE)r3 threadingLocklock)selfr3s r,__init__FileHandler.__init__s"NN$ r.cUR [UR5nUR5nSSS5 WsSSS5 $!,(df  N=f!,(df  g=frE)rFr>r3read)rGfiler@s r, read_fileFileHandler.read_files@ YYdnn%yy{&Y%%Ys!AA A A A A&cUR [URS5nURU5 SSS5 SSS5 g!,(df  N=f!,(df  g=f)Nw)rFr>r3write)rGr@rLs r, write_fileFileHandler.write_files@ YYdnnc*d 4 +Y**Ys!AAA A A A&cUR [RRUR5(a [R "UR5 SSS5 g!,(df  g=frE)rFrpathexistsr3remove)rGs r, cleanup_fileFileHandler.cleanup_files: YYww~~dnn-- $..)YYs AA%% A3)r3rFN)r3rj)r@rjrhNone) __name__ __module__ __qualname____firstlineno__rHrMrRrX__static_attributes__rr.r,rBrBs% ! *r.rB)r&zCallable[_P, List[Element]]r'rar(rbrhzdict[str, Any])r3rjrhr)r@list[dict[str, Any]]rArjrhrZ)rArjrhr`rE)rVzstr | list[str]rWz Optional[str]rhz.Callable[[Callable[_P, _T]], Callable[_P, _T]])rrrj)ry Iterable[_T]rhzTuple[_T, Iterator[_T]])ryrarhr)ryz Iterable[Any]rhr)rhr)rz list[str]rintrhzlist[tuple[str, ...]])rrjrrjrrbrhz"tuple[float, set[tuple[str, ...]]])rrjrrjrhz'tuple[float, set[tuple[str, ...]], str])g)rrrrrfloatrhr)r )rrrrrrjrhz!tuple[float, float, float, float])$@) r$list[Points] | tuple[Points, Points]rlist[str] | tuple[str, str]rrfrrfrrc)rr)rd) rrerrfrrfrrbrrc)r&z list['Text']rrbrrcrhz!tuple[bool, list[dict[str, Any]]])F)r&zIterable['Element']r=rrhz$dict[Optional[str], list['Element']])D __future__rrcrrorr8rrrr1rD functoolsrr itertoolsrtypingrrr r r r r rrrrtyping_extensionsrrunstructured.__version__runstructured.documents.elementsrrrcr__annotations__rr DATE_FORMATSrrr-r4rCrIrkrOr|rrrrrrrrrrrr rr7r@rBrr.r,rqs" &"   20=ueUE12Y2&y&%*% %\  T]t_  %     7K 9!%!% %4%P 3   @@%%% %( %&AA&)A,A>>&-.H .H .H #.H' .Hl#' b2b+b+b) b  bJJ &'"& i2i+i+i # i  i\&'"&I=I="I= I=' I=\!)!))*)X**r.