Mi# SSKJrJr SSKrSSKrSSKJrJrJrJ r J r J r J r SSK JrJr SSKrSSKrSSKJrJr SSKJrJrJrJrJr SSKJrJrJrJrJ r J!r! SS K"7 SS K"J#r#J$r$J%r%J&r&J'r'J(r(J)r) SS K*7 SS K*J+r+ SS K,7 SS K,J-r-J.r.J/r/J0r0 SS K1J2r2J3r3 SSK4J5r5 SSK6r7SSK8r8SSK9J:r: SSK;JS\>4Sjr?SJS\>S\@S\>4SjjrA"SS\5rB"SS\B5rC"SS\B5rD"SS\B5rE0S\>_S \@_S!\F_S"\G_S#\H_S$\I_S%\J_S&\K_S'\L_S(\M_S)\N_S*\O_S+\P_S,\Q_S-\R_S.\S_S/\T_\U\V\W\X\Y\ZS0S1S\[\\S2. Er]S3\>S4\QS\4S5jr^"S6S7\B5r_"S8S9\_5r`"S:S;\_5ra"S<S=\_5rb"S>S?\_5rc\V"\d\Z"S@55Vs0sHoSA;dM USB\e"U5SC3_M snrf\8R"SD5rh\8R"SE5riSF\\>\>4S\\>\>44SGjrj"SHSI\B5rkgs snf)K)ABCabstractmethodN)AnyListDictOptionalTuplePatternUnion)ThreadPoolExecutor as_completed)IntFlagauto)PROMPT_EXTRACT_BLOCKS&PROMPT_EXTRACT_BLOCKS_WITH_INSTRUCTION&PROMPT_EXTRACT_SCHEMA_WITH_INSTRUCTIONJSON_SCHEMA_BUILDER_XPATHPROMPT_EXTRACT_INFERRED_SCHEMA)DEFAULT_PROVIDERDEFAULT_PROVIDER_API_KEYCHUNK_TOKEN_THRESHOLD OVERLAP_RATEWORD_TOKEN_RATEHTML_EXAMPLE_DELIMITER)*) sanitize_htmlescape_json_stringperform_completion_with_backoffextract_xml_datasplit_and_parse_json_objectssanitize_input_encode merge_chunks) TokenUsage) get_deviceload_HF_embedding_modelload_text_multilabel_classifiercalculate_batch_size) LLMConfigcreate_llm_config)partial BeautifulSoup)htmletreetextreturncUR5n[R"SSU[RS9R5$)zEStrip markdown code fences (e.g. ```json ... ```) from LLM responses.z^```(?:[a-zA-Z0-9_-]+)?\s*|```$)flags)stripresub MULTILINE)r0s v/var/www/eduai.edurigo.com/storigo/production/storigo_env/lib/python3.13/site-packages/crawl4ai/extraction_strategy.py_strip_markdown_fencesr:3s2 ::UT:d[UR[5(dgURSS5R 5nU(aSSR U5-OSnURS5(aSURS53OSnTR SU-SURUUS3-5 UHnT"XQS -5 M g) Nclassr3.id#z <>r) isinstancetagstrgetsplitjoinappend) eldepthclassescls_strid_strchild_walklinesr=s r9rS'_get_top_level_structure.._walkIs 9 Jrvvs$;$; &&"%++--4#))"')vvd||1RVVD\N# TE\axxy$BBCE % #r; N<)r. fromstring ExceptionxpathrK)r<r=treebodyrootrSrTs ` @@r9_get_top_level_structurer^;sq |, ::h D47D E$ $N 99U3BZ  # sA!! A.-A.c \rSrSrSrSS\4Sjjr\S\S\S\\ \\ 44Sj5r S\S \\S\\ \\ 44S jr S\S \\S\\ \\ 44S jr S rg )ExtractionStrategyWz4 Abstract base class for all extraction strategies. input_formatc XlSUlURRUlUR SS5Ulg)z Initialize the extraction strategy. Args: input_format: Content format to use for extraction. Options: "markdown" (default), "html", "fit_markdown" **kwargs: Additional keyword arguments z<|DEL|>verboseFN)rbDEL __class____name__namerIrd)selfrbkwargss r9__init__ExtractionStrategy.__init__\s4)NN++ zz)U3 r;urlr.r1cg)z Extract meaningful blocks or chunks from the given HTML. :param url: The URL of the webpage. :param html: The HTML content of the webpage. :return: A list of extracted blocks or chunks. Nrirmr.qrjs r9extractExtractionStrategy.extractjs r;sectionsc /n[5nUVs/sH!nUR"URX40UD6PM# nn[U5H"n UR U R 55 M$ SSS5 U$s snf!,(df  U$=f)z Process sections of text in parallel by default. :param url: The URL of the webpage. :param sections: List of sections (strings) to process. :return: A list of processed JSON blocks. N)r submitrrr extendresult) rirmrtrqrjextracted_contentexecutorsectionfuturesfutures r9runExtractionStrategy.runus  !X ('G cEfE' 'w/!((90 "! " !! sA<(A73A<7A<< B ch# SSKnUR"URX/UQ70UD6IShvN $N7f)aC Async version: Process sections of text in parallel using asyncio. Default implementation runs the sync version in a thread pool. Subclasses can override this for true async processing. :param url: The URL of the webpage. :param sections: List of sections (strings) to process. :return: A list of processed JSON blocks. rN)asyncio to_threadr~)rirmrtrqrjrs r9arunExtractionStrategy.aruns1 &&txxMMfMMMMs )202)rerbrhrdN)markdown)rg __module__ __qualname____firstlineno____doc__rHrkrrrrrrr~r__static_attributes__ror;r9r`r`Ws 4S 4 3 c Dc3h[T U]"S 0UD6 SSKn XlX lX0lX@lXPlXpl[R"5Ul URSS5Ul U R"/5UlSUl[!5Ul[%UR"5UlUR(a#[)SUR"R*S35 [-U5uUlUlUR0R3UR"5 UR0R55 SUlU R"/5UlUR(a#[)S UR"R*S35 [75uUln UR(a?[)S US 3[;[R"5UR- 5-S -5 gg)a Initialize the strategy with clustering parameters. Args: semantic_filter (str): A keyword filter for document filtering. word_count_threshold (int): Minimum number of words per cluster. max_dist (float): The maximum cophenetic distance on the dendrogram to form clusters. linkage_method (str): The linkage method for hierarchical clustering. top_k (int): Number of top categories to extract. rNrdFdirectz#[LOG] Loading Extraction Model for z device.batchz([LOG] Loading Multilabel Classifier for z[LOG] Model loaded z, models/reuters, took  secondsro)superrknumpysemantic_filterword_count_thresholdmax_distlinkage_methodtop_k sim_thresholdtimetimerrIrdarraybuffer_embeddingsget_embedding_methodr%devicer(default_batch_sizeprinttyper& tokenizermodeltoevalr'nlprH) rirrrrr model_namerrjnp_rfs r9rkCosineStrategy.__init__s{* "6".$8! , *YY[ zz)U3 !#"$,! l #7t{{"C << 7 8H8H7IR S&=Z%H"  dkk" $+!!#" << .Hsadr;T)keyreverseN) lenmaxsklearn.metrics.pairwiserget_embeddingsflattenziprsortrw) rirrrrquery_embeddingdocument_embeddings similaritiesdocsim filtered_docsremaining_docsrs r9filter_documents_embeddings*CosineStrategy.filter_documents_embeddingssg  y>J &QI! 34J>--.?@C#11)<)  2 ')  8 8((( SJ8   }  *!$I <  sentencesc URRS;GaSSKnUc URn/n[ S[ U5U5HnXXb-nUR USSSS9nUR5V V s0sH upXRUR5_M" nn n UR5 UR"S 0UD6n SSS5 W RRSS9R5R5n URU 5 M [ R""U5UlUR$$URRS :XaqUc URn/n[ S[ U5U5H+nXXb-nURU5n URU 5 M- [ R""U5UlUR$$s sn n f!,(df  GN=f) z Get BERT embeddings for a list of sentences. Args: sentences (List[str]): A list of text chunks (sentences). Returns: NumPy array of embeddings. )cpugpucudampsrNTpt)padding truncationreturn_tensorsr)dimrro)rrtorchrrangerritemsrno_gradrlast_hidden_statemeanrrrLrvstackr) rir batch_size bypass_bufferrall_embeddingsrbatch_sentences encoded_inputrtensor model_output embeddingss r9rCosineStrategy.get_embeddingsPs ;;  < < !!44 N1c)nj9"+"? $#TdSW!/! DQCVCVCX!CXKCC4;;//CX! ]]_#'::#> #>L%*;;@@Q@GKKMSSU %%j1:"&(YY~%>D "%%%[[   &!!44 N1c)nj9"+"?!ZZ8 %%j1: &(YY~%>D "%%%3! %_s5'G/G G' cSSKJnJn SSKJn [ R "5UlURUSS9nU"US5nU"X`RS9nU"XpRSS 9nU$) z Perform hierarchical clustering on sentences and return cluster labels. Args: sentences (List[str]): A list of text chunks (sentences). Returns: NumPy array of cluster labels. r)linkagefcluster)pdistT)rcosine)methoddistance) criterion) scipy.cluster.hierarchyrrscipy.spatial.distancerrrrrr) rirrrrrdistance_matrixlinkedlabelss r9hierarchical_clustering&CosineStrategy.hierarchical_clusterings_ >0YY[ (($(G  H51D1DE&--:F r;clustersc0nUR5HDup4SRU5n[UR55nX`R:dM@XBU'MF U$)z Filter clusters to remove those with a word count below the threshold. Args: clusters (Dict[int, List[str]]): Dictionary of clusters. Returns: Dict[int, List[str]]: Filtered dictionary of clusters.  )rrKrrJr)rirfiltered_clusters cluster_idtexts full_text word_counts r9filter_clusters_by_word_count,CosineStrategy.filter_clusters_by_word_countsZ!)!1 JIY__./J66605*-"2! r;rmr.c 6[R"5nUR"UR5n[U5S:XaFUR"S5Vs/sH)owR 5(dMUR 5PM+ nnUR X`R 5nU(d/$URU5n[R"5n0n [U5H(upU RU /5RXj5 M* URU 5n [U 5V s/sH#n [U 5/SRX5S.PM% nn UR(a[!SUR"35 UR"R$S;a<UR'UVs/sHoSPM sn5n[)X5H upXS'M UR(a&[!S [R"5U- S S 35 U$s snfs sn fs snf) z Extract clusters from HTML content using hierarchical clustering. Args: url (str): The URL of the webpage. html (str): The HTML content of the webpage. Returns: List[Dict[str, Any]]: A list of processed JSON blocks. r rru[LOG] 🚀 Assign tags using )rrrrrru"[LOG] 🚀 Categorization done in z.2fr)rrJrerr5rrrr setdefaultrLrsortedintrKrdrrrrr)rirmr.rqrjt text_chunkschunkrrrlabelridx cluster_listclusters r9rrCosineStrategy.extracts IIKjj* { q 6:jj6HZ6HUKKM=5;;=6HKZ66 -- I--k: IIK%f-LE   r * 1 1+2D E.!>>xH /0 0#hsxx@Q@V7W X0  << 1$++? @ ;;  < <XXNgy1NOF"%l";"'#<& << 6tyy{Qs6K8T Us[0 OsH 1H (*H&HrtcZUR"XRRU540UD6$)z Process sections using hierarchical clustering. Args: url (str): The URL of the webpage. sections (List[str]): List of sections (strings) to process. Returns: )rrrerK)rirmrtrqrjs r9r~CosineStrategy.runs%||Cx!8CFCCr;)rrrrrrrrrrrrrrdr)N g?wardz&sentence-transformers/all-MiniLM-L6-v2g333333?))NFN)rgrrrrrkrrHr rrrrrrrrr~r __classcell__rfs@r9rrs,;TnMO6*c6*586*FI6* c6*rDI5&c5&nc2!S$s)^,! c49n !2I3IcIDc3h[TU]"SSU 0UD6 XlUR(d4[[[ R R[5S9UlX l X@l X0l U(aSUl Xl U=(d [UlX`lXplXlURS05UlUR"(dSUlXl/Ul[+5UlXlXlXlXlg)ae Initialize the strategy with clustering parameters. Args: llm_config: The LLM configuration object. instruction: The instruction to use for the LLM model. schema: Pydantic model schema for structured data. extraction_type: "block" or "schema". chunk_token_threshold: Maximum tokens per chunk. overlap_rate: Overlap between chunks. word_token_rate: Word to token conversion rate. apply_chunking: Whether to apply chunking. input_format: Content format to use for extraction. Options: "markdown" (default), "html", "fit_markdown" force_json_response: Whether to force a JSON response from the LLM. verbose: Whether to print verbose output. # Deprecated arguments, will be removed very soon provider: The provider to use for extraction. It follows the format /, e.g., "ollama/llama3.3". api_token: The API token for the provider. base_url: The base URL for the API request. api_base: The base URL for the API request. extra_args: Additional arguments for the API request, such as temperature, max_tokens, etc. rbr"r#r) extra_argsgeANro)rrkr'r*rosenvironrIrr( extract_typer)force_json_responserchunk_token_threshold overlap_rateword_token_rateapply_chunkingr,rdusagesr$ total_usager"r#r$r%)rir'r(r)extraction_typer1r2r3r4rbr0rdr"r#r$r%rjrfs r9rkLLMExtractionStrategy.__init__-sX >|>v>$/)**..)ABDO'+  (D #6 %:%S>S"(., **\26""),D &  %< "  r;c>[R"UR5nURnXR;a/X$UR La[ SUSURU35e[TU]!X5 g)zHandle attribute setting. Setting '' is deprecated. N) inspect signaturerk parameters_UNWANTED_PROPSdefaultAttributeErrorr __setattr__)rirhvaluesig all_paramsrfs r9rB!LLMExtractionStrategy.__setattr__wsq .^^ '' 'ED9I9Q9Q,Q 9TF2CDDXDXY]D^C_!`a a D(r;rmixr.r1c  UR(a[SUSU35 U[[U55S.n[nUR (aUR US'[ nURS:Xa9UR(a([R"URSS9US'[nURS:XaUR(d[nUHnURS U-S -XF5nM [URR UURR"URR$UR&UR(URR*URR,URR.S 9 n[1UR2R4UR2R6UR2R8UR2R:(a UR2R:R<O0UR2R>(a UR2R>R<O0S 9nUR@RCU5 URD=R4UR4- slURD=R6UR6- slURD=R8UR8- slURFS RHRJn Sn U (d([MURFS SS5n S SS/SU S3S./n OUR&(a[RN"[QU 55n [SU [T5(a_[WU 5S:XaL[S[YU R[55S [X5(a[YU R[55S n OBU /n O>[SU [X5(aU n O&[]S/U 5Sn [RN"U 5n U Hn SU S'M UR(a[S[WU 5SUSU5 U $![^a\ URFS RHRJ=(d Sn [aU 5upUn U(aU RCS SS/US.5 Nf=f![^a;nUR(a[SU35 USS/[cU5S./sSnA$SnAff=f)a Extract meaningful blocks or chunks from the given HTML using an LLM. How it works: 1. Construct a prompt with variables. 2. Make a request to the LLM using the prompt. 3. Parse the response and extract blocks or chunks. Args: url: The URL of the webpage. ix: Index of the block. html: The HTML content of the webpage. Returns: A list of extracted blocks or chunks. [LOG] Call LLM for  - block index: URLHTMLREQUESTr)rindentSCHEMA{}r$ json_responser, base_delay max_attemptsexponential_factorcompletion_tokens prompt_tokens total_tokenscompletion_tokens_detailsprompt_tokens_detailsrN finish_reasonunknownTerror(LLM returned no content (finish_reason: )rrarrrblocksFr3[LOG] Extractedblocks from URL: block index:[LOG] Error in LLM extraction: )2rdrrrrr(rr/r)jsondumpsrrreplacerr'r"r#r$r0r,backoff_base_delaybackoff_max_attemptsbackoff_exponential_factorr$usagerZr[r\r]__dict__r^r5rLr6choicesmessagergetattrloadsr:rFdictrlistvaluesr rYr!rH)rirmrGr.variable_valuesprompt_with_variablesvariableresponserprrer_r& raw_contentparsedunparsedes r9rrLLMExtractionStrategy.extracts6" << 'u,>;;+3..*R*R*[*[>>77'/nn&J&J&S&S E KK  u %    . .%2I2I I .    * *e.A.A A *    ) )U-?-? ? )" "**1-55==$+H,<,w(GHF!&$//v;!+ 4 ;PQR;SUY0Z0Z%)&--/%:1%=F'-XF#FD11!'.xj'B8LF!ZZ/F#E%*E'N$||%K&" M% &..q199AAGR #? #L MM"#dWIRZ[  & ||7s;< !$I"1v   sE:G2S-EQ1+SA#SSSS T 0TT T c2[UUUURS9nU$)zK Merge documents into sections based on chunk_token_threshold and overlap. )docs target_sizeoverlapword_token_ratio)r#r3)rirr1rrts r9_mergeLLMExtractionStrategy._merges'!.!11   r;rtc URUUR[URUR-5S9n/nURR R S5(ac[U5HRupV[URU5nURU"U[U555 [R"S5 MT U$[SS9n[URU5n[U5VVs/sH upVURXu[U55PM" n nn[!U 5H#n URU R#55 M% S S S 5 U$s snnf![$aJn UR&(a[)SU 35 UR+SSS /[-U 5S .5 S n A MS n A ff=f!,(df  U$=f) z Process sections sequentially with a delay for rate limiting issues, specifically for LLMExtractionStrategy. Args: url: The URL of the webpage. sections: List of sections (strings) to process. Returns: A list of extracted blocks or chunks. rzgroq/g? max_workerszError in thread execution: rTrardN)rr1r r2r'r" startswithrr+rrrwr"rsleepr rvr rxrYrdrrLrH) rirmrtmerged_sectionsryrGr{ extract_funcrzr|r}rs r9r~LLMExtractionStrategy.runs++   & &22T5F5FFG&   ?? # # . .w 7 7(9 &t||S9 !(( %:7%CD 3 :F! /$2h&t||S9 (1'A'A OOL6KG6TU'A +73F )00A43.! +% <<!$?s"CD)00)*)-)0 +.q6  32.! sH%F<9'E F<2E%F<F<% F9/?F4.F<4F99F<< G c B # SSKJn UR(a[SUSU35 U[ [ U55S.n[ nUR(aURUS'[nURS:Xa9UR(a([R"URSS 9US '[nURS:XaUR(d[nUHnURS U-S -XW5nM U"UR R"UUR R$UR R&UR(UR*UR R,UR R.UR R0S 9 IShvN n[3UR4R6UR4R8UR4R:UR4R<(a UR4R<R>O0UR4R@(a UR4R@R>O0S9n URBREU 5 URF=R6U R6- slURF=R8U R8- slURF=R:U R:- slURHSRJRLn Sn U (d([OURHSSS5n SSS/SU S3S./n OUR((a[RP"[SU 55n [UU [V5(a_[YU 5S:XaL[U[[U R]55S[Z5(a[[U R]55Sn OBU /n O>[UU [Z5(aU n O&[_S/U 5Sn [RP"U 5n U Hn SU S'M UR(a[SU 5SUSU5 U $GN![`a] URHSRJRL=(d Sn[cU5unnUn U(aU RESSS/US.5 Nf=f![`a;nUR(a[SU35 USS/[eU5S./sSnA$SnAff=f7f)a Async version: Extract meaningful blocks or chunks from the given HTML using an LLM. How it works: 1. Construct a prompt with variables. 2. Make an async request to the LLM using the prompt. 3. Parse the response and extract blocks or chunks. Args: url: The URL of the webpage. ix: Index of the block. html: The HTML content of the webpage. Returns: A list of extracted blocks or chunks. r aperform_completion_with_backoffrIrJrKrNr)rrOrQrRrSrTNrYrr_r`TrarbrcrdreFr3rfrgrhri)3utilsrrdrrrrr(rr/r)rjrkrrrlr'r"r#r$r0r,rmrnror$rprZr[r\r]rqr^r5rLr6rrrsrrtrur:rFrvrrwrxr rYr!rH)rirmrGr.rryrzr{r|rprrer_r&r}r~rrs r9aextractLLMExtractionStrategy.aextractLsE" < << 'u,>;;+3..*R*R*[*[>>77'/nn&J&J&S&S E KK  u %    . .%2I2I I .    * *e.A.A A *    ) )U-?-? ? ) "**1-55==$+H,<,w(GHF!&$//v;!+ 4 ;PQR;SUY0Z0Z%)&--/%:1%=F&,XF#FD11!'-xj'B8LF!ZZ/F#E%*E'N$||%K&" MO j &..q199AAGR #? #L MM"#dWIRZ[  & ||7s;< !$I"1v   spC?TBSQ* ES:EQ->+S)T*S-A$SSSS T!0TTTTTc $# SSKnURUUR[URUR-5S9n/n[ U5VVs/sH upgUR X[U55PM" nnnUR"USS06IShvN n U Hjn [U [5(aAUR(a[SU 35 URSSS/[U 5S.5 MYURU 5 Ml U$s snnfN|7f) z Async version: Process sections with true parallelism using asyncio.gather. Args: url: The URL of the webpage. sections: List of sections (strings) to process. Returns: A list of extracted blocks or chunks. rNrreturn_exceptionsTzError in async extraction: rard)rrr1r r2rrr"gatherrFrYrdrrLrHrw) rirmrtrrryrGr{tasksresultsrxs r9rLLMExtractionStrategy.aruns ++   & &22T5F5FFG&   )9 9  MM##8#A B9   FFFF&),,<<7x@A!((!"!%!( #&v; "((0! 1 GsAD'D9DDA=Dc F[S5 [SSSSS35 [S5 [SSSURRS 35 [S SSURRS 35 [S SSURRS 35 [S 5 [S SSSSSS SSS S35 [S5 [ UR S5H=up[USSURS SURS SURS 35 M? g)zHPrint a detailed token usage report showing total and per-request usage.z === Token Usage Summary ===Typez<15rCountz>12z------------------------------ Completionz>12,PromptTotalz === Usage History ===z Request #z<10z0------------------------------------------------rN)rr6rZr[r\rr5)rirrps r9 show_usage LLMExtractionStrategy.show_usages( -.  Agc]+, h c"!D$4$4$F$Ft#LMN #a 0 0 > >tDEF  Qt//<>RB.!c.!T#Y.!4S#X;O.!`r;rrHr floatboolrroundabsminrsumr reversedrwrvtuplesetrTF) rmapfilteranyallrTrueFalseNonerFr expression local_varsc[R"USS9n[R"U5GHvn[ U[R [R45(a [S5e[ U[R5(a9URRS5(a[SURS35e[ U[R5(dMURn[ U[R5(a9URRS5(a[S URS 35e[ U[R5(dGM=URRS5(dGM`[S URS 35e S [0n[![#US S5Xa5$![an[SU35eSnAff=f) a Evaluate a computed field expression safely using AST validation. Allows simple transforms (math, string methods, attribute access on data) while blocking dangerous operations (__import__, dunder access, etc.). Args: expression: The Python expression string to evaluate. local_vars: The local variables (extracted item fields) available to the expression. Returns: The result of evaluating the expression. Raises: ValueError: If the expression contains disallowed constructs. r)modezInvalid expression syntax: Nz0Import statements are not allowed in expressionsrz$Access to private/dunder attribute 'z' is not allowedz Calling 'z' is not allowed in expressions __builtins__z )astparse SyntaxError ValueErrorwalkrFImport ImportFrom AttributeattrrCallfuncNamerB_SAFE_EVAL_BUILTINSrcompile)rrr[rnoder safe_globalss r9_safe_eval_expressionr s"<yy&1 dSZZ8 9 9OP P dCMM * *tyy/C/CC/H/H6tyykAQR  dCHH % %99D$))dgg.@.@.E.E y(GH$ ..4993G3G3L3L  {*IJ'.#$78L nf5| PP7 <6qc:;;S%\S&\S\S'S(S\\4 S)jj5r"\S*\S\ \4S+j5r#S,S-S..r$\S>S/\S\S%\S0\S\4 S1jj5r%\SSSS\&"5SSSS2S3S4 S/\S\S%\S0\S4S5S6\S7\S\'\\ \4S8\ S9\S'S(S\4S:jj5r(\S?S/\S\S%\S0\S4S5S6\S7\S\'\\ \4S8\ S9\S'S(S\4S;jj5r)S[TU]"S0UD6 XlURSS5Ulg)z Initialize the JSON element extraction strategy with a schema. Args: schema (Dict[str, Any]): The schema defining the extraction rules. rdFNro)rrkr)rIrdrir)rjrfs r9rk&JsonElementExtractionStrategy.__init__ss, "6" zz)U3 r;rmr<r1cURU5nURXPRS5n/nUHn0n SUR;a3URSH n URX5n U cMXU S'M" UR XRS5n U R U 5 U (dMUR U 5 M U$)a? Extract structured data from HTML content. How it works: 1. Parses the HTML content using the `_parse_html` method. 2. Identifies base elements using the schema's base selector. 3. Extracts fields from each base element using `_extract_item`. Args: url (str): The URL of the page being processed. html_content (str): The raw HTML content to parse and extract. *q: Additional positional arguments. **kwargs: Additional keyword arguments for custom extraction. Returns: List[Dict[str, Any]]: A list of extracted items, each represented as a dictionary. baseSelector baseFieldsrhfields) _parse_html_get_base_elementsr)_extract_single_field _extract_itemupdaterL) rirmr<rqrj parsed_html base_elementsrelementitemfieldrC field_datas r9rr%JsonElementExtractionStrategy.extract~s*&&|4 // ^4 $GDt{{*![[6E 66wFE(.3U6]+7 ++G[[5JKJ KK #tt$% r;cg)z*Parse HTML content into appropriate formatNrorir<s r9r)JsonElementExtractionStrategy._parse_html r;selectorcg)(Get all base elements using the selectorNrorirrs r9r0JsonElementExtractionStrategy._get_base_elementsrr;cg)z%Get child elements using the selectorNrorirrs r9 _get_elements+JsonElementExtractionStrategy._get_elementsrr;c SU;a(URXS5nUcURS5$USS:Xa?URXS5nU(aUSOSnU(aURXBS5$0$USS:Xa8URXS5nUVs/sHo`R XbS5PM sn$USS :Xa8URXS5nUVs/sHo`RXbS5PM sn$UR X5$s snfs snf![ aInUR(a[S US S [U535 URS5sSnA$SnAff=f) Nsourcer@rnestedrrrrw nested_listzError extracting field rh: ) _resolve_sourcerIrr_extract_list_itemrrYrdrrH)rirrnested_elementsnested_elementelementsrMrs r9_extract_field,JsonElementExtractionStrategy._extract_fieldsy (5 ..whH? 99Y//V}("&"4"4WJ>O"P7F!3D&&&~XG V}&--gZ7HIOWXx//(ODxXXV} ---gZ7HIJRS(B**2X?(SS--g= = YT (||/f bQIJ99Y' ' (sY-DAD6D8!DD 6D9!DD7D:D D E(>E#E(#E(cSU;a2URXS5nU(dURS5$USnOUnUSn[U[5(dU/nUnUHnUS:XaUR U5nOUS:XaUR XRS5nOUS:XaUR U5nOUS:XaURS 5nU(a~[U[5(dUR U5n[U[5(aA[R"Xu5nU(a!URURS S 55OS nOS nUbM O S U;aURXRS 5nUbU$URS5$)a Extract a single field based on its type. How it works: 1. Selects the target element using the field's selector. 2. Extracts the field value based on its type (e.g., text, attribute, regex). 3. Applies transformations if defined in the schema. Args: element: The base element to extract the field from. field (Dict[str, Any]): The field definition in the schema. Returns: Any: The extracted field value. rr@rrr0 attributer.regexpatterngrouprN transform) rrIrFrw_get_element_text_get_element_attribute_get_element_htmlrHr6searchr_apply_transform) rirrselected type_pipelinerCsteprmatchs r9r3JsonElementExtractionStrategy._extract_single_fieldsk"  ))'3DEHyy++{HHf -..*OM!Dv~..u5$33E;MN..u5))I.%eS11 $ 6 6u =!%-- " ' 9FK EIIgq,A BQU $}'"* % ))%{1CDE)uCuyy/CCr;cV0nUH nURX5nUcMXSUS'M" U$)Nrh)rrirrrrrCs r9r0JsonElementExtractionStrategy._extract_list_items:E..w>E &+U6]# r;c0nUH;nUSS:XaURX45nOURX5nUcM4XSUS'M= U$)a Extracts fields from a given element. How it works: 1. Iterates through the fields defined in the schema. 2. Handles computed, single, and nested field types. 3. Updates the item dictionary with extracted field values. Args: element: The base element to extract fields from. fields (List[Dict[str, Any]]): The list of fields to extract. Returns: Dict[str, Any]: A dictionary representing the extracted item. rcomputedrh)_compute_fieldrr s r9r+JsonElementExtractionStrategy._extract_itemsX"EV} *++D8++G; &+U6]#  r;cUS:XaUR5$US:XaUR5$US:XaUR5$U$)aU Apply a transformation to a value. How it works: 1. Checks the transformation type (e.g., `lowercase`, `strip`). 2. Applies the transformation to the value. 3. Returns the transformed value. Args: value (str): The value to transform. transform (str): The type of transformation to apply. Returns: str: The transformed value. lowercase uppercaser5)lowerupperr5)rirCrs r9r.JsonElementExtractionStrategy._apply_transform2sE"  #;;= + %;;= ' !;;=  r;c SU;a[USU5$SU;a US"U5$g![aInUR(a[SUSS[ U535 UR S5sSnA$SnAff=f)NrfunctionzError computing field rhrr@)rrYrdrrHrI)rirrrs r9r,JsonElementExtractionStrategy._compute_fieldKs (u$,U<-@$GGu$Z(..% (||.uV}oRAxHI99Y' ' (s)) A<>A71A<7A<rtc^URRU5nUR"X40UD6$)a Run the extraction strategy on a combined HTML content. How it works: 1. Combines multiple HTML sections using the `DEL` delimiter. 2. Calls the `extract` method with the combined HTML. Args: url (str): The URL of the page being processed. sections (List[str]): A list of HTML sections. *q: Additional positional arguments. **kwargs: Additional keyword arguments for custom extraction. Returns: List[Dict[str, Any]]: A list of extracted items. )rerKrr)rirmrtrqrj combined_htmls r9r~!JsonElementExtractionStrategy.runVs*$ h/ ||C9&99r;cg)zGet text content from elementNrorirs r9r/JsonElementExtractionStrategy._get_element_textkrr;cg)zGet HTML content from elementNrors r9r/JsonElementExtractionStrategy._get_element_htmlprr;rcg)z Get attribute value from elementNrorirrs r9r4JsonElementExtractionStrategy._get_element_attributeurr;rcg)aNavigate to a sibling element relative to the base element. Used when a field's data lives in a sibling of the base element rather than a descendant. For example, Hacker News splits each submission across two sibling rows. Args: element: The current base element. source: A sibling selector string. Currently supports the ``"+ "`` syntax which navigates to the next sibling matching ````. Returns: The resolved sibling element, or ``None`` if not found. Nro)rirrs r9r-JsonElementExtractionStrategy._resolve_sourcezs" r;CSSN schema_typeexpected_fieldsc 2^SSSSS//SSS. nUR5S:Xa[O[nU"US9nURSUS9nUR U5n URXS 5n [U 5US 'U (aURU S5n U S SUS'US S:Xa8US R SURS S5S35 [U5US'U$URS/5n U V s/sHoSPM nn [U5US'UH|nUVs/sHnURU5PM nn[SU55n[SU5S 5nUb[U5S SnUSR UU[U5US.5 M~ [SUS55US'USS:aUSUS- US'USS:XaUS R S5 OPUSVs/sHnUSS:XdMUSPM nnU(a&US R S S!RU535 U(ajURS/5V s1sHoSiM nn UV s/sH oU;dM U PM nn U(a&US R S"S!RU535 U(aqURS/5V s1sHoSiM nn USVs1sHnUSS:dMUSiM snmUS S:=(a [!U4S#jU55US$'U$US S:=(a USS:US$'U$![a#nUS R S U35 UsS nA$S nAff=f![a GNf=fs sn fs snfs snfs sn fs sn fs sn fs snf)%u7Run the generated schema against HTML and return a diagnostic result. Args: schema: The extraction schema to validate. html_content: The HTML to validate against. schema_type: "CSS" or "XPATH". expected_fields: When provided, enables strict mode — success requires ALL expected fields to be present and populated. When None, uses fuzzy mode (populated_fields > 0). Returns a dict with keys: success, base_elements_found, total_fields, populated_fields, field_coverage, field_details, issues, sample_base_html, top_level_structure. Frgr3) successbase_elements_found total_fieldspopulated_fieldsfield_coverage field_detailsissuessample_base_htmltop_level_structurer()r))rmr<r2zExtraction crashed: Nrr-r3zbaseSelector 'z' matched 0 elementsr4rrhr.c3># UHocMUS:wdMSv M g7f)Nr3rro.0vs r9 AJsonElementExtractionStrategy._validate_schema..s!QV!b!!V  c3># UHocMUS:wdMUv M g7f)Nr3ror7s r9r:r;sN&Qqa2gqq&r<xr1)rhpopulated_count total_count sample_valuec3:# UHoSS:dM Sv M g7f)r?rrNro)r8fds r9r:r;s!) /"6G3H13LAA/s  r/r0u9All fields returned None/empty — selectors likely wrongr?zFields always empty: z, z%Expected fields missing from schema: c3,># UH oT;v M g7frro)r8fpopulated_namess r9r:r;sFo_,or,)rJsonCssExtractionStrategyJsonXPathExtractionStrategyrrrYrLrrrrrIr^rnextrHrKr)r)r<r)r*rx StrategyClassstrategyrrr~rsample all_fieldsrE field_namesfnamerrxr? sample_valrC empty_fieldsschema_field_namesmissingrFs @r9_validate_schema.JsonElementExtractionStrategy._validate_schemasW,#$ !! "#%   $$&%/*0  %F3H$$,$GE )),7F$77~@VWM,/ ,>F( )!33M!4DE-3ET])* ' (A - 8  # # NB!? @@TU -E\,RF( )MZZ"- *45*Qy* 5!$[!1~ E278%$dhhuo%F8!!QV!QQON&NPTUJ% _Tc2 ? # * *#2"5z * ,  !&)) /) & !" . !A %'-.@'AF>DZ'ZF# $ $ % * 8  # #K  !11B'(A-6 1  x ''+DIIl,C+DE 5;ZZ"5M!N5MF)5M !N"1Q/Q>P5Pq/GQx '';DIIg ?M    690"OQ"Osl6L5AM%3M6M;' N8 N N N ,N <N N$ N5 M"?MM"M"% M32M3validation_resultattempt is_repeatedc bUn/nURSU35 USS:XaJURSURSS5S35 US(aURS US-S -5 OURS USS 35 US (aURSUSSUSS35 URS5 URS5 US H8nUS=(d SnURSUSSUSSUSSUS3 5 M: US(a1URS5 USHnURSU35 M USS:Xa$US (aURS!US -S -5 U(aURS"5 URS#5 S$RU5$)%z=Build a structured feedback message from a validation result.u!## Schema Validation — Attempt r-rz**CRITICAL:** baseSelector `rr3z=` matched **0 elements**. The schema cannot extract anything.r4zKHere is the top-level HTML structure so you can pick a valid selector: ```  ```zbaseSelector matched **z** element(s).r1z **Field coverage:** r//r.z fields have data z| Field | Populated | Sample |z|-------|-----------|--------|rAz *(empty)*z| rhz | r?r@z |r2z **Issues:**z- r3ue Here is the innerHTML of the first base element — use it to find correct child selectors: ```html zr **WARNING:** You returned the exact same schema as before. You MUST change the selectors to fix the issues above.z@ Please fix the schema and return ONLY valid JSON, nothing else.rV)rLrIrK) rWr)rXrYvrpartsrCrMissues r9_build_feedback_message5JsonElementExtractionStrategy._build_feedback_messages 8 BC # $ ) LL.vzz."/M.NONO '( c./0 LL)"-B*C)DNS  o  LL(,>)?(@"^BTAUUhi  LL9 : LL9 :)N+:{ F C+<(='>a=@Q?RRUV\U]]_`* h< LL )H r%\*&  !Q &2.@+A LLE'()   LLI   O yyr;query html_snippetrpr$c# SSKJn U(aSUS3OSnSUSUSUS S S 3nU"URUS URURS 9IS hvN nUb{U=R UR R - slU=RUR R- slU=RUR R- slURSRRn U (aU R5(dg [R"[U 55$N![ a g f=f7f)a Infer a target JSON example from a query and HTML snippet via a quick LLM call. Args: usage: Optional TokenUsage accumulator. If provided, token counts from this LLM call are added to it in-place. Returns the parsed dict, or None if inference fails. rrzURL: rVr3uYou are given a data extraction request and a snippet of HTML from a webpage. Your job is to produce a single example JSON object representing ONE item that the user wants to extract. Rules: - Return ONLY a valid JSON object — one flat object, NOT wrapped in an array or outer key. - The object represents a single repeated item (e.g., one product, one article, one row). - Use clean snake_case field names matching the user's description. - If the item has nested repeated sub-items, represent those as an array with one example inside. - Fill values with realistic examples from the HTML so the meaning is clear. Example — if the request is "extract product name, price, and reviews": {"name": "Widget Pro", "price": "$29.99", "reviews": [{"author": "Jane", "text": "Great product"}]} zExtraction request: z HTML snippet: ```html Nr5z/ ``` Return ONLY the JSON object for ONE item:T)r"rzrUr#r$r)rrr"r#r$rZrpr[r\rrrsrr5rjrur:rY) rbrcr'rmrprurl_linepromptr|raws r9_infer_target_json0JsonElementExtractionStrategy._infer_target_jsonWs7 <&)U3%r?r vj"5'*''3ET':&;<8 8 $ =#,,&,"$..#,, H ''8>>+K+KK'##x~~'C'CC#""hnn&A&AA"""1%--55Cciikk::4S9: :  sG%E+D7D5CD7ED74E5D77 EEEE target_jsonc4[UR55$)z9Extract top-level field names from a target JSON example.)rwkeys)rjs r9_extract_expected_fields6JsonElementExtractionStrategy._extract_expected_fieldssK$$&''r;r r!r+r.target_json_examplec"SSKJn US:XaUO[nSU3nSUS3nU(aUSU3- nU(a USUS 3- nU(a U(dUS - nO'U(d U(aUS - nOU(d U(dUS - nUS - nSRXg/5$)z~ Build the prompt for schema generation. Shared by sync and async methods. Returns: str: Combined system and user prompt r)JSON_SCHEMA_BUILDERr(a# You specialize in generating special JSON schemas for web scraping. This schema uses CSS or XPATH selectors to present a repetitive pattern in crawled HTML, such as a product in a product list or a search result item in a list of search results. We use this JSON schema to pass to a language model along with the HTML content to extract structured data from the HTML. The language model uses the JSON schema to extract data from the HTML and retrieve values for fields in the JSON schema, following the schema. Generating this HTML manually is not feasible, so you need to generate the JSON schema using the HTML content. The HTML copied from the crawled website is provided below, which we believe contains the repetitive pattern. # Schema main keys: - name: This is the name of the schema. - baseSelector: This is the CSS or XPATH selector that identifies the base element that contains all the repetitive patterns. - baseFields: This is a list of fields that you extract from the base element itself. - fields: This is a list of fields that you extract from the children of the base element. {name, selector, type} based on the type, you may have extra keys such as "attribute" when the type is "attribute". # Extra Context: In this context, the following items may or may not be present: - Example of target JSON object: This is a sample of the final JSON object that we hope to extract from the HTML using the schema you are generating. - Extra Instructions: This is optional instructions to consider when generating the schema provided by the user. - Query or explanation of target/goal data item: This is a description of what data we are trying to extract from the HTML. This explanation means we're not sure about the rigid schema of the structures we want, so we leave it to you to use your expertise to create the best and most comprehensive structures aimed at maximizing data extraction from this page. You must ensure that you do not pick up nuances that may exist on a particular page. The focus should be on the data we are extracting, and it must be valid, safe, and robust based on the given HTML. # What if there is no example of target JSON object and also no extra instructions or even no explanation of target/goal data item? In this scenario, use your best judgment to generate the schema. You need to examine the content of the page and understand the data it provides. If the page contains repetitive data, such as lists of items, products, jobs, places, books, or movies, focus on one single item that repeats. If the page is a detailed page about one product or item, create a schema to extract the entire structured data. At this stage, you must think and decide for yourself. Try to maximize the number of fields that you can extract from the HTML. # What are the instructions and details for this schema generation? zJ HTML to analyze: ```html z% ``` z4 ## Query or explanation of target/goal data item: z, ## Example of target JSON object: ```json r[aIMPORTANT: To remind you, in this process, we are not providing a rigid example of the adjacent objects we seek. We rely on your understanding of the explanation provided in the above section. Make sure to grasp what we are looking for and, based on that, create the best schema..aIMPORTANT: Please remember that in this process, we provided a proper example of a target JSON object. Make sure to adhere to the structure and create a schema that exactly fits this example. If you find that some elements on the page do not match completely, vote for the majority.zIMPORTANT: Since we neither have a query nor an example, it is crucial to rely solely on the HTML content provided. Leverage your expertise to determine the schema based on the repetitive patterns observed in the content.aIMPORTANT: 0/ Ensure your schema remains reliable by avoiding selectors that appear to generate dynamically and are not dependable. You want a reliable schema, as it consistently returns the same data even after many page reloads. 1/ DO NOT USE use base64 kind of classes, they are temporary and not reliable. 2/ Every selector must refer to only one unique element. You should ensure your selector points to a single element and is unique to the place that contains the information. You have to use available techniques based on CSS or XPATH requested schema to make sure your selector is unique and also not fragile, meaning if we reload the page now or in the future, the selector should remain reliable. 3/ Do not use Regex as much as possible. Analyze the HTML and generate a JSON schema that follows the specified format. Only output valid JSON schema, nothing else. r )promptsrqrrK)r.r)rbrorqprompt_templatesystem_content user_contents r9_build_schema_prompt2JsonElementExtractionStrategy._build_schema_prompts 11<1E-Kd(),   UV[U\] ]L  NObNcchi iL , { {L. } }L2 @ @L  {{N9::r;Trr'r)r"r#validatemax_refinementsc  SSKn U R5n [R"SUUUUUUUUUU U S. U D6nU cU R U5$SSKnURRSS9nURU R U5nUR5sSSS5 $![a Sn Nf=f!,(df  g=f)ae Generate extraction schema from HTML content or URL(s) (sync version). Args: html (str, optional): The HTML content to analyze. If not provided, url must be set. schema_type (str): "CSS" or "XPATH". Defaults to "CSS". query (str, optional): Natural language description of what data to extract. target_json_example (str, optional): Example of desired JSON output. llm_config (LLMConfig): LLM configuration object. provider (str): Legacy Parameter. LLM provider to use. api_token (str): Legacy Parameter. API token for LLM provider. url (str or List[str], optional): URL(s) to fetch HTML from. If provided, html parameter is ignored. When multiple URLs are provided, HTMLs are fetched in parallel and concatenated. validate (bool): If True, validate the schema against the HTML and refine via LLM feedback loop. Defaults to False (zero overhead). max_refinements (int): Max refinement rounds when validate=True. Defaults to 3. usage (TokenUsage, optional): Token usage accumulator. If provided, token counts from all LLM calls (including inference and validation retries) are added to it in-place. **kwargs: Additional args passed to LLM processor. Returns: dict: Generated schema following the JsonElementExtractionStrategy format. Raises: ValueError: If neither html nor url is provided. rN) r.r)rbror'r"r#rmrxryrprrro) rget_running_loop RuntimeErrorragenerate_schemar~concurrent.futuresr|r rvrx)r.r)rbror'r"r#rmrxryrprjrloopcoro concurrentrzr}s r9generate_schema-JsonElementExtractionStrategy.generate_schemasT  ++-D-==  # 3!+     <;;t$ $ %##6616E!d;}}FE- D ,FEsB),B1 B.-B.1 B?c j # SSKJn Jn Uc2Ub$[U[5(a[ U5S:Xa [ S5e[RR5H$up[5UcM[SUSU35e Uc [5n/nUGb SSK Jn SS KJnJnJn U"S S S S 9nU"UR&S 9n[U[(5(aU/OUnU"US 9IShvN n[ U5S:XaUR+USUS9IShvN nUR,(d[/SUSSUR035eUR2S:a[/SUR2SUSS35eUR4nUR4/nGOUR7UUS9IShvN n/nS5HunnUR,(d%[/SUR:SUR035eUR2S:a&[/SUR2SUR:S35eUR=UR45 U "UR4SSSS9n[>R@"US9nUR=USU35 M SRCU5nSSS5IShvN OU/nUb[U[(5(a U "USSSS9nSnU(aU(aE[U[(5(a[DRF"U5n OUn [RIU 5nO~U(awSn!Ub[U[(5(aUOUSn![ROX UU!U S9IShvN n"U"(a1[RIU"5nU(d[DRP"U"SS9n[RSXX#5n#S U#S!./n$Sn%Sn&SU(aU OS-n'[UU'5GHn(U "URVU#S URXURZU$U S"9IShvN n)U b{U =R\U)R^R\- sl.U =R`U)R^R`- sl0U =RbU)R^Rb- sl1U)RdSRfRhn*U*(aU*Rk5(d [ S#5e[DRF"[mU*55n+U+n&U(dU+s $Sn-UH6n.[RoU+U.UUS)9n/U-b U/S*U-S*:aU/n-U/S+(dM6 O U-S+(aU+s $U(U'S- :aU+s $[DRP"U+S S,9n0U0U%:Hn1U0n%[RqU-U+U(S-U15n2U$R=S%U*S!.5 U$R=S U2S!.5 GM U&bU&$[/S-5eGNGNGN1GN7!,IShvN (df  GNI=f![DRJ[L4a GNuf=fGNGN![DRJa_n,U(a U(U'S- :a[/S$[)U,535eU$R=S%W*S!.5 U$R=S S&U,S'3S!.5 Sn,A,GMSn,A,f[.an,[/S([)U,535eSn,A,ff=f7f).a  Generate extraction schema from HTML content or URL(s) (async version). Use this method when calling from async contexts (e.g., FastAPI) to avoid issues with certain LLM providers (e.g., Gemini/Vertex AI) that require async execution. Args: html (str, optional): The HTML content to analyze. If not provided, url must be set. schema_type (str): "CSS" or "XPATH". Defaults to "CSS". query (str, optional): Natural language description of what data to extract. target_json_example (str, optional): Example of desired JSON output. llm_config (LLMConfig): LLM configuration object. provider (str): Legacy Parameter. LLM provider to use. api_token (str): Legacy Parameter. API token for LLM provider. url (str or List[str], optional): URL(s) to fetch HTML from. If provided, html parameter is ignored. When multiple URLs are provided, HTMLs are fetched in parallel and concatenated. validate (bool): If True, validate the schema against the HTML and refine via LLM feedback loop. Defaults to False (zero overhead). max_refinements (int): Max refinement rounds when validate=True. Defaults to 3. usage (TokenUsage, optional): Token usage accumulator. If provided, token counts from all LLM calls (including inference and validation retries) are added to it in-place. **kwargs: Additional args passed to LLM processor. Returns: dict: Generated schema following the JsonElementExtractionStrategy format. Raises: ValueError: If neither html nor url is provided. r)rpreprocess_html_for_schemaNrz'Either 'html' or 'url' must be providedr:r;)AsyncWebCrawler) BrowserConfigCrawlerRunConfig CacheModeT)headless text_mode light_mode) cache_mode)config)rmrzFailed to fetch URL '': izHTTP z error for URL '')urlsrr5ii )r<text_thresholdattr_value_thresholdmax_size)rrVr )rbrcr'rmrprrOuser)roler)r"rzrUr#r$messagesr,zLLM returned an empty responsezFailed to parse schema JSON: assistantz/Your response was not valid JSON. Parse error: z- Please return ONLY valid JSON, nothing else.zFailed to generate schema: )r*r/r,) sort_keysz0Failed to generate schema: no attempts succeeded)9rrrrFrwrrr_GENERATE_SCHEMA_UNWANTED_PROPSrlocalsrAr*async_webcrawlerr async_configsrrrBYPASSrHrr,rY error_message status_coder. arun_manyrrmrLrformatrKrjrurmJSONDecodeError TypeErrorrhrkrvrr"r#r$rZrpr[r\rrrsrr5r:rUr`)3r.r)rbror'r"r#rmrxryrprjrrrhrsoriginal_htmlsrrrrbrowser_configcrawler_configrcrawlerrxr html_partsrcleanedheaderr* target_obj first_urlinferredrfrprev_schema_json last_schemarWrXr|rgr)r best_result orig_htmlr] current_jsonrYfeedbacks3 r9r}.JsonElementExtractionStrategy.agenerate_schemas\ X #*<>'*?QyFL`L`Ka(bcc))S0'%0B0B/CCSTXYZT[S\\](^__!;;D&,kk]N$+$5$54$5$WWG!#J%.w%: 6%~~"+.CFJJ!>Q!G"))VHBwi*@A&;";;z2D5>==8#VN ;*S#..-!#%( D "!"5s;;%)ZZ0C%D %8 &C&\&\]g&hO ?'1#s';';QI!>!Q!Qzy`e"R"&C&\&\]e&fO..2jj!.L+.CCDW\r#78 xOQ? \*G H!A'00*0"&(22'00%%"$++x~~/O/OO+''8>>+G+GG'&&(..*E*EE&&&q)1199#))++$%EFF$:3$?@$  K+ 2CCI{$3D&"-?*@;OaCb*b"$Ki==,9% ,**  ::f=L&*::K+ 5LLVWq[+H OO[SA B OOVA BI+N  " JKKe>SX>===d,,i8$&'' 7lQ.>#>#&CCF8$LMM EFEaSICC! H"=c!fX FGG HsA-Z33A2Z3%W &Z3)&WWBW W!C1W Z3W9Z3AW2AZ3XA9Z3-XXC-X3AZ38BZ3WWZ3W/W  W/*Z32X Z3XZ3XZ0-AZZ3 Z0Z++Z00Z3)r)rd)r(N)NN) Nr(NNNNNNTrN),rgrrrrrerrHrrkrrrrrrrrrrrrrr~rrrr staticmethodrvrrUr rr`rhrmrrvr*r rr}rrrs@r9rrQsT> C 4tCH~ 4++&)+ d38n +Z       s  (<5Dn62 (:s:d3i:$tCQTH~BV:* C   C      s  $!/3 AAAA"$s), A  AAFC C C C  C C C J//3/QT/dp/}EFJ}K//b(d(tCy(( HI'# =;3=;S=;=;be=;qt=;=;~ #'"3"5%) "E'E'E'E'! E'  E'  E'E'3S > "E'E'E'E' E'E'N #'"&%) "_L_L_L_L! _L  _L  _L_L3S > "_L_L_L_L _L_Lr;rc^\rSrSrSrS\\\44U4SjjrS\4Sjr S\4Sjr S\4S jr S \4S jr S \4S jr S \4SjrS\4SjrSrU=r$)rHia Concrete implementation of `JsonElementExtractionStrategy` using CSS selectors. How it works: 1. Parses HTML content with BeautifulSoup. 2. Selects elements using CSS selectors defined in the schema. 3. Extracts field data and applies transformations as defined. Attributes: schema (Dict[str, Any]): The schema defining the extraction rules. verbose (bool): Enables verbose logging for debugging purposes. Methods: _parse_html(html_content): Parses HTML content into a BeautifulSoup object. _get_base_elements(parsed_html, selector): Selects base elements using a CSS selector. _get_elements(element, selector): Selects child elements using a CSS selector. _get_element_text(element): Extracts text content from a BeautifulSoup element. _get_element_html(element): Extracts the raw HTML content of a BeautifulSoup element. _get_element_attribute(element, attribute): Retrieves an attribute value from a BeautifulSoup element. r)c 2>SUS'[TU]"U40UD6 gNr.rbrrkrs r9rk"JsonCssExtractionStrategy.__init__!'~ *6*r;r<c[US5$)Nlxmlr,rs r9r%JsonCssExtractionStrategy._parse_htmls\622r;rc$URU5$rselectrs r9r,JsonCssExtractionStrategy._get_base_elementss!!(++r;c$URU5$rrrs r9r'JsonCssExtractionStrategy._get_elementss~~h''r;r1c URSS9$)NTr5)get_textrs r9r+JsonCssExtractionStrategy._get_element_text#sd++r;c[U5$r)rHrs r9r+JsonCssExtractionStrategy._get_element_html&s 7|r;rc$URU5$rrIr$s r9r0JsonCssExtractionStrategy._get_element_attribute){{9%%r;rcUR5nURS5(dgUSSR5nURS5nUSR5=(d SnUSSVs/sH)ofR5(dMUR5PM+ nn0nU(a U4SjUS'UR"U40UD6$s snf)N+rrArc@^T=(a [U4SjU55$)Nc3,># UH oT;v M g7frro)r8clcs r9r:NJsonCssExtractionStrategy._resolve_source....6sA"&Ba$rG)r)r_clss` r9r;JsonCssExtractionStrategy._resolve_source..6s%q8SA"&A>8r;class_)r5rrJfind_next_sibling) rirrselr^rGprOrjs r9r)JsonCssExtractionStrategy._resolve_source,s  %%QRj  #Ahnn&$&+ABi=i77991779i= .5 F8 ((777 >s /C Cro)rgrrrrrrHrrkrrrrrrrrrrs@r9rHrHsu*+tCH~+33,,(s( ,C,C&& 8s 8 8r;rHc^\rSrSrS\\\44U4SjjrS\4SjrSr Sr Sr S r S r S rS \4S jrS \4SjrS\4SjrS\4SjrS\4SjrS\4SjrSrSrU=r$)JsonLxmlExtractionStrategyi;r)c >SUS'[TU]"U40UD6 0Ul0Ul0UlUR SS5UlUR SS5UlSSKJ nJ n SSK J n X0l X@l XPl g) Nr.rb use_cachingToptimize_common_patternsr)r/r. CSSSelector)rrk_selector_cache _xpath_cache _result_cacherIrrrr/r.lxml.cssselectr html_parser)rir)rjr/r.rrfs r9rk#JsonLxmlExtractionStrategy.__init__<sy!'~ *6*!"::mT:(. 3Mt(T% %. &r;r<cURRSSS9nURRX5$![anUR(a[ SU35 UR RU5sSnA$![aInUR(a[ SU35 URRS5sSnAsSnA$SnAff=fSnAff=f)z&Parse HTML content with error recoveryT)recoverremove_blank_textz8Error parsing HTML, falling back to alternative method: NzCritical error parsing HTML: r.)r/ HTMLParserrXrYrdrrElement)rir<parserre2s r9r&JsonLxmlExtractionStrategy._parse_htmlNs 2ZZ**44*PF::((> > 2||PQRPSTU 2''22<@@ 2<<9">?zz))&111  2 2sK47 CC!B;C C 9CCC CCCCcUR(dU$SSKnUR"SU5(aU$UR5n[ U5S::aU$[ U5S:ab[ SU55(aKUVs/sH2oDR S5(dUR S5(dM0UPM4 nnU(aUS $U$s snf) z8Optimize common selector patterns for better performancerNztd:nth-child\(\d+\)rrc3r# UH-oRS5=(d URS5v M/ g7f)rArCN)r)r8rs r9r:@JsonLxmlExtractionStrategy._optimize_selector..ns*!XRWQ,,s"3"Hq||C7H"HRWs57rArC)rr6rrJrrr)ri selector_strr6r^rspecific_partss r9_optimize_selector-JsonLxmlExtractionStrategy._optimize_selector^s,,   99+\ : : ""$ u:?  u:>c!XRW!XXX).YA,,s2C2Cq||TWGXaNY%b)) Zs =/C0CcF^^^^^TmTR(aTRT5mTRT5mTRmTTRT'SUUUUU4SjjnU$![ a1nTR (a[STSU35 SSjsSnA$SnAff=f)z6Create a selector function that handles all edge casescX>SnTR(aSURSS5=(d [[U55nUST 3nUTR;aTRU$/nT "U5nU(dU(aTR TU5nU(aUR U5nU(dST ;aTRUT 5nU(dTRUT 5nU(d_T R5nU(aHUSn[R"SU5nU(a%URS5n UR SU 35nTR(aU(aUTRU'U$![a-n TR(a[S T S U 35 Sn A U$Sn A ff=f) NrBr3z:: nth-childrz^(\w+)r.//Error applying selector 'r)rrIrHhashr_make_context_sensitive_xpathrZ_handle_nth_child_selector_fallback_class_id_searchrJr6r rrYrdr)rcontext_sensitive cache_key element_idr context_xpathr^ last_part tag_matchtag_namercompiledoriginal_selectorrrirZs r9 selector_funcKJsonLxmlExtractionStrategy._create_selector_function..selector_funcs ##!(T2!6!L#d7m:LJ#-,b ?I D$6$66#11)<<$P&w/G#'8(,(J(J5RY(Z (&-mmM&BG ';:K+K&*&E&EgO`&aG '&*&D&DWN_&`G '$5$;$;$=E$,1"I ,.HHY ,J #,/8q/AH.5mmc(.s2r;N)T)rrrpathrrYrdr)rirr rr r rZs`` @@@r9_create_selector_function4JsonLxmlExtractionStrategy._create_selector_functionus(  ( (22<@LE >'' 5HMME/4D  l +3 3 j!  >||2<.A3GH> =  >s9A%% B /&BB B cURS5(aU$URS5(aSU3nOSU3nURU5 U$! SURS5S3s$=f! g=f)z1Convert absolute XPath to context-sensitive XPathdescendant-or-self::r\rArrN)rrZrJ)rirZrrs r9r8JsonLxmlExtractionStrategy._make_context_sensitive_xpaths  677 $$"#E7 "%eW  4 m,$$ 4U[[-b1233 s'A-!A-AA*(A-*A--A1cSSKn/nUR"SU5nU(aURS5nURSUS3S5SR 5nU(aFUR"SU5nU(aURS5OS n UR S US U 35nU$UR S US 35nU$![ a*n UR(a[S U 35 Sn A U$Sn A ff=f)z2Special handling for nth-child selectors in tablesrNtd:nth-child\((\d+)\)rz td:nth-child(rcrz(\w+)r.//td[]//]z#Error handling nth-child selector: ) r6rrrJr5rZrYrdr) rirrr6rr col_numremaining_selectorr r rs r9r5JsonLxmlExtractionStrategy._handle_nth_child_selectors AII6 EE++a.&2%7%7-yPQ8RTU%VWY%Z%`%`%b"%!# (4F GI5>yq1CH%mmfWIS ,KLG &mmfWIQ,?@G   A||;A3?@  AsBB>'B>> C2C--C2c/nSSKnUR"SU5nUR"SU5nUH)nURSUS35nURU5 M+ UH)n URSU S35n URU 5 M+ U$![a*n UR (a[ S U 35 Sn A U$Sn A ff=f) z!Fallback to search by class or IDrNz\.([a-zA-Z0-9_-]+)z#([a-zA-Z0-9_-]+)z.//*[contains(@class, 'z')]z .//*[@id='z']z#Error in fallback class/id search: )r6findallrZrwrYrdr) rirrrr6 class_matches id_matches class_name class_resultsid_name id_resultsrs r9r4JsonLxmlExtractionStrategy._fallback_class_id_searchs A JJ'Get child elements using the selector with context sensitivityTr-r.rirrr s r9r(JsonLxmlExtractionStrategy._get_elements s**84 W==r;r1cSRSURS555nU$![aSnUR(a[ SU35 UR 5R 5sSnA$! SnAg=fSnAff=f)z$Extract normalized text from elementrc3n# UH+oR5(dMUR5v M- g7frr)r8r s r9r:?JsonLxmlExtractionStrategy._get_element_text..( s"W/I!WWYIAGGII/Is55 .//text()zError extracting text: Nr3)rKrZrYrdr text_contentr5)rirr0rs r9r,JsonLxmlExtractionStrategy._get_element_text$ sv 88Ww}}[/IWWDK ||/s34 ++-3355  s2(+ BBA82B8B:BBBcURRUSSS9$![a)nUR(a[ SU35 SnAgSnAff=f)z)Get HTML string representation of elementunicoder.)encodingrzError serializing HTML: Nr3)r/tostringrYrdr)rirrs r9r,JsonLxmlExtractionStrategy._get_element_html3 sM ::&&w6&R R ||045 s AA  ArcURU5$![a,nUR(a[SUSU35 SnAgSnAff=f)zGet attribute value safelyzError getting attribute 'rN)rIrYrdr)rirrrs r9r1JsonLxmlExtractionStrategy._get_element_attribute< sF ;;y) ) ||1)CsCD s A "AA rcUR5nURS5(dgUSSR5nURS5nUSR5=(d SnUSSVs/sH)ofR5(dMUR5PM+ nnSU3nUH n USU S3- nM US - nURU5n U (aU S$S$s snf NrrrArrz./following-sibling::z4[contains(concat(' ',normalize-space(@class),' '),' z ')]z[1]r5rrJrZ rirrrr^rGrrOrZclsrs r9r*JsonLxmlExtractionStrategy._resolve_sourceE   %%QRj  #Ahnn%#&+ABi=i77991779i='u-C KC5PTU UE --&$wqz.$. > /C Cc\UR(aURR5 gg)zClear caches to free memoryN)rrclear)ris r9 _clear_caches(JsonLxmlExtractionStrategy._clear_cachesT s"       $ $ & r;)rrrrr/rrr)rgrrrrrHrrkrrrrrrr*rrrrrrrKrrrs@r9rr;s'tCH~'$22 .M>^.:62 CC >s> C C /s /''r;rc^\rSrSrS\\\44U4SjjrS\4SjrSr S\4Sjr S\4S jr S \4S jr S \4S jr S \4SjrS\4SjrSrU=r$) JsonLxmlExtractionStrategy_naiveiY r)c @>SUS'[TU]"U40UD6 0Ulgr)rrkrrs r9rk)JsonLxmlExtractionStrategy_naive.__init__Z s'!'~ *6*!r;r<cRSSKJn UR"SS9nUR"X5$)Nrr/T)r)rr/rrX)rir<r/rs r9r,JsonLxmlExtractionStrategy_naive._parse_html_ s&!!$/ 55r;c(^^^TTR;a&SSKJn U"T5mUUU4SjnUTRT'TRT$![a>nTR(a[ STSU35 SnUTRT'SnANRSnAff=f)zCGet a selector function that works within the context of an elementrrc>T "U5nU(aU$T RnURS5(aUnOSURS53nURU5nU(aU$ST ;aSSKnUR "ST 5nU(agUR S5nT RSS5S R5nU(aURS US U35$URS US 35$T R5n[U5S:a!US (aURS US 35$/$![a.n T R(a[ST SU 35 /sSn A $Sn A ff=f)Nrz./r\rrrrrcrrrrrrr) rrlstriprZr6rrrJr5rrYrdr) rrrZrr6r r sub_selectorr^rr rris r9 select_funcCJsonLxmlExtractionStrategy_naive._get_selector..select_funcm sc+""*7"3"#*N!) !++,BCC,1M/1c1B0C,DM")-- ">"#*N',6%$&II.F $UE$*/++a./;/A/A#q/I"/M/S/S/U #/+2==6'#l^9\+]$]+2==6'!9L+M$M!- 2 2 4u:>eBi#*==3uRyk1B#CC! $"<<!$=l^3qc"RS! "s<D4AD4'A5D4D42?D42D44 E,>#E'!E,'E,rrc/$rro)rs r9 fallback_funcEJsonLxmlExtractionStrategy_naive._get_selector..fallback_func sIr;N)rrrrYrdr)rirrrXrr[r s`` @r9r*.JsonLxmlExtractionStrategy_naive._get_selectord s t33 3 2< C&|4,"\6A$$\2##L11 C<<6|nCsKL6C$$\2 CsA B4B  Brc4URU5nU"U5$rr.r/s r9r3JsonLxmlExtractionStrategy_naive._get_base_elements s**84 [))r;c4URU5nU"U5$rr.r2s r9r.JsonLxmlExtractionStrategy_naive._get_elements s**84 W%%r;r1c^SRURS55R5$Nr3r7rKrZr5rs r9r2JsonLxmlExtractionStrategy_naive._get_element_text #www}}[1288::r;c0SSKJn UR"USS9$)NrrRr;r<)rr/r=)rirr/s r9r2JsonLxmlExtractionStrategy_naive._get_element_html s~~g ::r;rc$URU5$rrr$s r9r7JsonLxmlExtractionStrategy_naive._get_element_attribute rr;rcUR5nURS5(dgUSSR5nURS5nUSR5=(d SnUSSVs/sH)ofR5(dMUR5PM+ nnSU3nUH n USU S3- nM US - nURU5n U (aU S$S$s snfrBrCrDs r9r0JsonLxmlExtractionStrategy_naive._resolve_source rGrH)r)rgrrrrrHrrkrr*rrrrrrrrrs@r9rNrNY sw"tCH~" 66 B2H**&s&;C;;C;&& /s / /r;rNc^\rSrSrSrS\\\44U4SjjrS\4Sjr S\4Sjr S \S \4S jr S \S \4S jr S\4S jr S \4SjrS \4SjrS\4SjrS\4SjrSrU=r$)rIi a Concrete implementation of `JsonElementExtractionStrategy` using XPath selectors. How it works: 1. Parses HTML content into an lxml tree. 2. Selects elements using XPath expressions. 3. Converts CSS selectors to XPath when needed. Attributes: schema (Dict[str, Any]): The schema defining the extraction rules. verbose (bool): Enables verbose logging for debugging purposes. Methods: _parse_html(html_content): Parses HTML content into an lxml tree. _get_base_elements(parsed_html, selector): Selects base elements using an XPath selector. _css_to_xpath(css_selector): Converts a CSS selector to an XPath expression. _get_elements(element, selector): Selects child elements using an XPath selector. _get_element_text(element): Extracts text content from an lxml element. _get_element_html(element): Extracts the raw HTML content of an lxml element. _get_element_attribute(element, attribute): Retrieves an attribute value from an lxml element. r)c 2>SUS'[TU]"U40UD6 grrrs r9rk$JsonXPathExtractionStrategy.__init__ rr;r<c.[R"U5$r)r.rXrs r9r'JsonXPathExtractionStrategy._parse_html s|,,r;rc$URU5$r)rZrs r9r.JsonXPathExtractionStrategy._get_base_elements s  **r; css_selectorr1c4SU;aU$URU5$)z'Convert CSS selector to XPath if neededr\)_basic_css_to_xpath)rirus r9 _css_to_xpath)JsonXPathExtractionStrategy._css_to_xpath s! ,  '' 55r;cSU;a%URS5nSSRU5-$SU;a%URS5nSSRU5-$SU-$)z.Basic CSS to XPath conversion for common casesz > z//r\r)rJrK)rirur^s r9rw/JsonXPathExtractionStrategy._basic_css_to_xpath sd L  &&u-E#((5/) ) ,  &&s+E$))E** *l""r;c|URU5nURS5(dSU-nURU5$)NrA)rxrrZ)rirrrZs r9r)JsonXPathExtractionStrategy._get_elements s;""8,$$%KE}}U##r;c^SRURS55R5$rcrdrs r9r-JsonXPathExtractionStrategy._get_element_text rfr;c,[R"USS9$)Nr;rh)r/r=rs r9r-JsonXPathExtractionStrategy._get_element_html s~~g ::r;rc$URU5$rrr$s r9r2JsonXPathExtractionStrategy._get_element_attribute rr;rcUR5nURS5(dgUSSR5nURS5nUSR5=(d SnUSSVs/sH)ofR5(dMUR5PM+ nnSU3nUH n USU S3- nM US - nURU5n U (aU S$S$s snfrBrCrDs r9r+JsonXPathExtractionStrategy._resolve_source rGrHro)rgrrrrrrHrrkrrrxrwrrrrrrrrs@r9rIrI s,+tCH~+--++6#6#6 ###$s$ ;C;;C;&& /s / /r;rI z z\x02xz\x08z(?Fix common JSON-escape goofs coming from LLMs or manual edits.z\\b\\\\z Regex for 'u' won’t compile after fix: N) r_WB_FIXr7 translate_CTRL _NEEDS_ESCAPEr6rrar)r)saferpatrs r9_sanitize_schemar" s Dlln kk&#&007- ^ JJsOU % K xx ^{5'1NqcRSY] ] ^sA88B# BB#c ^\rSrSr%Sr"SS\5r\Rr\Rr \Rr \Rr\Rr\R"r\R&r\R*r\R.r\R2r\R6r\R:r\R>r \RBr"\RFr$\RJr&\RNr(\RRr*\RVr,\RZr.\R^r0\Rbr2\"S5r30SS_SS _S S _S S _SS_SS_SS_SS_SS_SS_SS_SS_SS_S S!_S"S#_S$S%_S&S'_S(S)S*S+S,.Er4\5\6\64\7S-'\8Rr\8Rt-r;S.S.S/.r<\Rz4S0S1S2.S3SS4\>\?\5\6\64\@\A\6\644S5\6S6S04U4S7jjjjrBS \6S8\6S6\@\5\6\C44S9jrDS8\6S6\64S:jrE\FS0S0S0S;.S<\6S=\6S>\>\6S?\>\@\6S@\>\GS6\5\6\644 SAjj5rHSBrIU=rJ$)CRegexExtractionStrategyi6 ap A lean strategy that finds e-mails, phones, URLs, dates, money, etc., using nothing but pre-compiled regular expressions. Extraction returns:: { "url": "", "label": "", "value": "", "span": [start, end] } Only `generate_schema()` touches an LLM, extraction itself is pure Python. c\rSrSr\"5r\"5r\"5r\"5r\"5r \"5r \"5r \"5r \"5r \"5r\"5r\"5r\"5r\"5r\"5r\"5r\"5r\"5r\"5r\"5r\"5r\"5r\\-\-\-\ -\ -\ -\ -\ -\-\-\-\-\-\-\-\-\-\-\-\-rSrg)RegexExtractionStrategy._BiJ roN)rgrrrrEMAIL PHONE_INTLPHONE_USrLIPV4IPV6UUIDCURRENCY PERCENTAGENUMBERDATE_ISODATE_USTIME_24H POSTAL_US POSTAL_UKHTML_COLOR_HEXTWITTER_HANDLEHASHTAGMAC_ADDRIBAN CREDIT_CARDNOTHINGALLrror;r9_BrJ s=&& &&&&&&& &&&&& & &&&&&& & J  )C /$ 6 = D # $&, -/7 8:A BDL M # $&4 57E F ! !$( (+6 6 r;rremailz[\w.+-]+@[\w-]+\.[\w.-]+ phone_intlz\+?\d[\d .()-]{7,}\dphone_usz#\(?\d{3}\)?[ -. ]?\d{3}[ -. ]?\d{4}rmzhttps?://[^\s\"'<>]+ipv4z(?:\d{1,3}\.){3}\d{1,3}ipv6z"[A-F0-9]{1,4}(?::[A-F0-9]{1,4}){7}uuidzG[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}currencyu,(?:USD|EUR|RM|\$|€|£)\s?\d+(?:[.,]\d{2})? percentagez\d+(?:\.\d+)?%numberz%\b\d{1,3}(?:[,.\s]\d{3})*(?:\.\d+)?\bdate_isoz\d{4}-\d{2}-\d{2}date_usz\d{1,2}/\d{1,2}/\d{2,4}time_24hz.\b(?:[01]?\d|2[0-3]):[0-5]\d(?:[:.][0-5]\d)?\b postal_usz\b\d{5}(?:-\d{4})?\b postal_ukz$\b[A-Z]{1,2}\d[A-Z\d]? ?\d[A-Z]{2}\bhtml_color_hexz#[0-9A-Fa-f]{6}\btwitter_handlez @[\w]{1,15}z#[\w-]+z$(?:[0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}z[A-Z]{2}\d{2}[A-Z0-9]{11,30}zH\b(?:4\d{12}(?:\d{3})?|5[1-5]\d{14}|3[47]\d{13}|6(?:011|5\d{2})\d{12})\b)hashtagmac_addriban credit_cardDEFAULT_PATTERNSzUse llm_config insteadr+Nfit_html)customrbrrrbr1c T>[T U]"SSU0UD6 URR5VVs0sH?upV[ UR UR 55RU-(dM=XV_MA nnnU(aM[U[5(aURU5 O&URUVVs0sHupX_M snn5 UR5VVs0sH&upU[R"X`R5_M( snnUlgs snnfs snnfs snnf)z Args: patterns: Custom patterns overriding or extending defaults. Dict[label, regex] or list[tuple(label, regex)]. input_format: "html", "markdown" or "text". **kwargs: Forwarded to ExtractionStrategy. rbNro)rrkrrrtrrrCrFrvrr6r_FLAGS _compiled) rirrrbrjrrxmergedlblrfs r9rk RegexExtractionStrategy.__init__ s =l=f=  00668" 8tww ,22W< CG8 "  &$'' f% f=f73swf=>=CLLN. . s ?    o (3B-0HH I;;z*/(("(++z.D"E **((  ll1o%%--kk&%(ff+Wc: O::c?L %**,HC N 3-  O>seDE3 N O88 N #6se3qc!BCM Ns0 F9F6 F3F..F36G! GG!)r)KrgrrrrrrrEmailr PhoneIntlrPhoneUSrLUrlrIPv4rIPv6rUuidrCurrencyr PercentagerNumberrDateIsorDateUSrTime24hrPostalUSrPostalUKrHexColorr TwitterHandlerHashtagrMacAddrrIbanr CreditCardrAllNothingrrrH__annotations__r6 IGNORECASEr8rr?rrr rr rkrrrrrr)rrrrs@r9rr6 s1& W >XXE]]I[[GVVCWWDWWDWWD[[H]]JYYF[[GZZF[[G\\H\\H&&H&&MZZG[[GWWD^^JVVCUG (6( 2( A ( 2 ( 5( @( e( J( ,( C( /!(" 5#($ L%(( 2)(* B+(, /-(. >/(0&B:f7(d38n<]]R\\ )F,-O ! JN& ! ! tCH~tE#s(O/DDEF !  !  ! ! L34S#X;O&H3H3H $(,*. SSS} S 49% S Y' S c3hSSr;r)r)labcrrrr<typingrrrrr r r r~r r rjrenumrrrrrrrrrrrrrrrrrrrrr r!r"r#modelsr$ model_loaderr%r&r'r(typesr)r* functoolsr+rrr6bs4r-rr.r/rHr:r r^r`rrrrrrrrrrrr rrwrvrrrrrrrrrrFrrrrrHrrNrIchrordrrrrrr)rs0r9rs# CCC? vv0 !3!3!s!8<N<N~ - .cD'cDR v.v|  3 s #U ,2D  3   %s ,13 8=s  3 & #-h  D $ !( 16s    $'sf sU 5$d .Qc.Qt.Q.QbjL$6jLX<8 =<8|\'!>\'|n/'Dn/`M/"?M/^ ),Cr(;Q(;1?P r#a&  (;Q **W  01 T#s(^S#X(M0M3 Rs G-G-