Mi7@SrSSKrSSKrSSKJrJrJrJr SSKJ r SSK J r SSK J r Jr SSKJrJr SS KJr "S S 5rg) z Link Extractor for Crawl4AI Extracts head content from links discovered during crawling using URLSeeder's efficient parallel processing and caching infrastructure. N)DictListOptionalAny) AsyncLogger)AsyncUrlSeeder) SeedingConfigCrawlerRunConfig)LinksLink)calculate_total_scorec 8\rSrSrSrSS\\4SjjrSrSr Sr S r SS \ S \ S \ 4S jjr S\S\S\4SjrS\S\\ \4S\\ 4SjrS\\ S\\ \4S\\\ \44SjrS\\ S\S\\ \4S\\\ \44SjrS\S\\\ \4S\S\4SjrSrg) LinkPreviewal Extracts head content from links using URLSeeder's parallel processing infrastructure. This class provides intelligent link filtering and head content extraction with: - Pattern-based inclusion/exclusion filtering - Parallel processing with configurable concurrency - Caching for performance - BM25 relevance scoring - Memory-safe processing for large link sets Nloggerc,XlSUlSUlg)z^ Initialize the LinkPreview. Args: logger: Optional logger instance for recording events NF)rseeder _owns_seeder)selfrs o/var/www/eduai.edurigo.com/storigo/production/storigo_env/lib/python3.13/site-packages/crawl4ai/link_preview.py__init__LinkPreview.__init__s 04 !cB# UR5IShvN U$N7f)zAsync context manager entry.N)startrs r __aenter__LinkPreview.__aenter__)sjjl  s c@# UR5IShvN gN7f)zAsync context manager exit.N)close)rexc_typeexc_valexc_tbs r __aexit__LinkPreview.__aexit__.sjjls c# UR(dB[URS9UlURR5IShvN SUlggN 7f)z"Initialize the URLSeeder instance.)rNT)rr rrrrs rrLinkPreview.start2sB{{(  ;; eT:J7FJJxQS  IIfC DL &N!3}#56  8"99-UU --e6J  &U#S\)`\UU8_X_E_!\)`%ab  d+ V *as: C(CA C(C!-C(1C# C# C(!C(#C(rEc^/nUR(anURURVs/sH!oDR(dMURPM# sn5 UR SSS[ UR50S9 UR (anURURVs/sH!oDR(dMURPM# sn5 UR SSS[ UR50S9 URnU(aKUV^s/sH!m[U4SjU55(dMTPM# nnUR SSS[ U50S9 URnU(aKUV^s/sH!m[U4SjU55(aMTPM# nnUR SS S[ U50S9 URnUS :a'[ U5U:aUS UnUR SS S U0S9 [5n /n UH-mTU ;dM U RT5 U RT5 M/ UR SSS[ U 50S9 U $s snfs snfs snfs snf)z Filter links based on configuration parameters. Args: links: Links object containing internal and external links link_config: Configuration dictionary for link extraction Returns: List of filtered URL strings debugzAdded {count} internal linksr;r<zAdded {count} external linksc3R># UHn[R"TU5v M g7fNfnmatch.0patternurls r ,LinkPreview._filter_links..s!UDTwsG44DT$'z,After include patterns: {count} links remainc3R># UHn[R"TU5v M g7frOrPrRs rrVrWs!YHXW7??388HXrXz,After exclude patterns: {count} links remainrNzLimited to {max_links} links max_linksz)Final filtered URLs: {count} unique links)include_internalextendinternalhrefr4rBinclude_externalexternalinclude_patternsanyexclude_patternsrZsetaddappend) rr6rErFlinkrarUrcrZseen unique_urlss ` rrALinkPreview._filter_linkspsR   ' '  !T)))$))!T U IIg=%s5>>':;  =  ' '  !T)))$))!T U IIg=%s5>>':;  ='77 ,,UDTUU}  IIgM%s='9:  <'77 ,,YHXYY}  IIgM%s='9:  < )) q=S/);)*95M IIg=)95  7u  C$ ""3'! 'F!3{#34  6a"U "Us/II-III<I8I#I#urlsc # URnURnU(aURSS[U5US.S9 [ SU[ USS5UR URUR (aSOSUS 9nU(aURXU5IShvN nU$URRUUUURS 9IShvN nU$N6N7f) a Extract head content for URLs using URLSeeder's parallel processing. Args: urls: List of URLs to process link_config: Configuration dictionary for link extraction Returns: List of dictionaries with url, status, head_data, and optional relevance_score r:zNStarting batch processing: {total} links with {concurrency} concurrent workers)total concurrencyr<T hits_per_secNbm25) extract_headrnroqueryscore_thresholdscoring_methodverboserkr7rntimeout) rurnr4rBr r0rrrs_extract_with_progressrextract_head_for_urlsrw)rrkrErurnseeding_configresultss rrC#LinkPreview._extract_heads_parallels%%!--  IIfn'*4yM  O'# ndC##'77%0%6%66D   77kZZG!KK==%'#++ >G[s$BCC/C C CCrzc t# [U5nURn[SUS-5nSnSnSn URSS5 URR UUUUR S9IShvN n U H)n US- nU RS5S :XaUS- nM$U S- n M+ URSS UUUU S .S 9 U $NK7f) z-Extract head content with progress reporting.r rr:zProcessing links in batches...rvNr>r?zcBatch processing completed: {completed}/{total} processed, {successful} successful, {failed} failed) completedrm successfulfailedr<)rBrnmaxr4rryrwr1) rrkrzrE total_urlsrn batch_sizerrrr{results rrx"LinkPreview._extract_with_progresssY !-- J",-    &:; 99!#'' :  F NIzz(#w.a !   &#,)$. &   1 sA(B8*B6+A B8original_linksrGc0nUHnURS5nURS05URSS5URS5URS5S.nU(aXtU'URS5nU(dM~X:wdMXtU'M /n URGHnn U RU;GaXJRnURS5n [U RU RU R U R USUSURS5[U S S 5U S 9 n U b(U R=(d 0U lXRS'[U RU R[US S 5[URR5S9U lU R!U 5 GM[[U S S 5S [US S 5[URR5S9U lU R!U 5 GMq /n UR"GHnn U RU;GaXJRnURS5n [U RU RU R U R USUSURS5[U S S 5U S 9 n U b(U R=(d 0U lXRS'[U RU R[US S 5[URR5S9U lU R!U 5 GM[[U S S 5S [US S 5[URR5S9U lU R!U 5 GMq [%SX-55(a#SnU R'USS9 U R'USS9 [)U U S9$)z Merge head extraction results back into Link objects. Args: original_links: Original Links object head_results: Results from head extraction Returns: Links object with head_data attached to matching links rU head_datar>unknownerrorrelevance_score)rr>rr original_urlintrinsic_scoreN) r^texttitle base_domainrhead_extraction_statushead_extraction_errorrcontextual_score score_linksF)rrscore_links_enabledquery_providedc3# UH:n[US5=(a" UR=(a SUR;v M< g7f)rrNhasattrr)rSrgs rrV/LinkPreview._merge_head_data..s=@>4t[)ddnndARVZVdVdAdd>sAAc[US5(a0UR(aSUR;aURS$g)Nrrgr)rgs rget_relevance_score9LinkPreview._merge_head_data..get_relevance_scores74--$..EVZ^ZhZhEh>>*;<<rT)keyreverse)r]r`)r1r]r^r rrrr0rrrrboolr@rr total_scorerfr`rbsortr )rrrGr7url_to_head_datarrU head_inforupdated_internalrgr updated_linkupdated_externalrs rrDLinkPreview._merge_head_datas""F**U#C#ZZ R8 **Xy9G,#)::.?#@ I (1%!::n5L| 31:.#"++Dyy,,,YY7 #,==1B#C #** $ 0 0' 4+4X+>*3--*@$+D2CT$J%5  $/-9-C-C-IrL*@P**+<=,A$0$@$@%1%B%B(/ u(M#'(B(B(H(H#I , (!'' 5$9$+D2CT$J%)(/ u(M#'(B(B(H(H#I $ !''-Q,V"++Dyy,,,YY7 #,==1B#C #** $ 0 0' 4+4X+>*3--*@$+D2CT$J%5  $/-9-C-C-IrL*@P**+<=,A$0$@$@%1%B%B(/ u(M#'(B(B(H(H#I , (!'' 5$9$+D2CT$J%)(/ u(M#'(B(B(H(H#I $ !''-Q,V @+>@ @ @   ! !&94 ! H  ! !&94 ! H%%  r)rrrrO) LINK_EXTRACT)__name__ __module__ __qualname____firstlineno____doc__rrrrr%rr!strr4r r rJrrrrArCr rxrD__static_attributes__rrrrsP  "x 4 " %&V#VV#V''!'  'R?5?tCH~?$s)?B-3i-#s(^- d38n  -^23i2&2#s(^ 2 d38n  2hG G 4S>*G ! G  G rr)rasynciorQtypingrrrr async_loggerrasync_url_seederr async_configsr r modelsr r utilsrrrrrrs3,,%,:(I I r