Miu ~SSKrSSKrSSKrSSKrSSKrSSKrSSKJrJrJ r J r J r J r J r JrJrJr SSKJrJr SSKJrJr SSKJrJr SSKJrJr SSKJrJr SS K J!r! SS K"J#r#J$r$ SS K%J&r& SS K'J(r( SSK)r)SS K*J+r+J,r,J-r-J.r.J/r/J0r0J1r1 SSK2J3r3 \1\4\-\4/\54\/\1\4\-\4/\5444r6Sr7"SS\35r81Skr9S-S\+S\54Sjjr:S\+S\+4Sjr;S\+S\54Sjr<"SS5r="SS5r>\7"SS 55r?"S!S"5r@"S#S$5rA"S%S&5rB\7"S'S(55rC"S)S*5rD"S+S,5rEg).N) DEFAULT_PROVIDERDEFAULT_PROVIDER_API_KEYMIN_WORD_THRESHOLD$IMAGE_DESCRIPTION_MIN_WORD_THRESHOLDPROVIDER_MODELSPROVIDER_MODELS_PREFIXESSCREENSHOT_HEIGHT_TRESHOLD PAGE_TIMEOUTIMAGE_SCORE_THRESHOLDSOCIAL_MEDIA_DOMAINS)UAGenValidUAGenerator)ExtractionStrategyLLMExtractionStrategy)ChunkingStrategy RegexChunking)MarkdownGenerationStrategyDefaultMarkdownGenerator)ContentScrapingStrategyLXMLWebScrapingStrategy)DeepCrawlStrategy)TableExtractionStrategyDefaultTableExtraction) CacheMode)ProxyRotationStrategy)Any AwaitableCallableDictListOptionalUnion)Enumc^^^ URm[R"T5nURVs/sH o"S:wdM UPM snm[ T5m [ R "T5UU4Sj5nX0l0Ul[U 4Sj5n[S5n[S5nX@l XPl X`l U$s snf)uClass decorator: adds set_defaults/get_defaults/reset_defaults classmethods. After decorating, every new instance resolves parameters as: explicit arg > class-level user defaults > hardcoded default Usage:: BrowserConfig.set_defaults(headless=False, viewport_width=1920) cfg = BrowserConfig() # headless=False, viewport_width=1920 cfg = BrowserConfig(headless=True) # explicit wins → headless=True selfcx>[U5RnU(a[UR55n[ [ U55H(nU[ T 5:dMUR T U5 M* UR5H$upgXd;dM [R"U5X&'M& T"U/UQ70UD6 gN) type_user_defaultssetkeysrangelenadditemscopydeepcopy) r&argskwargs user_defaultsexplicitikeyvalue original_init param_namess p/var/www/eduai.edurigo.com/storigo/production/storigo_env/lib/python3.13/site-packages/crawl4ai/async_configs.py wrapped_init$_with_defaults..wrapped_init9sT 11 6;;=)H3t9%s;''LLQ0&,113 &"&--"6FK4 d,T,V,c>[U5T- nU(a[SURSU35eUR5H(up4[R "U5UR U'M* g)zSet class-level default overrides for new instances. Args: **kwargs: Parameter names and their default values. Raises: ValueError: If any key is not a valid ``__init__`` parameter. zInvalid parameter(s) for : N)r+ ValueError__name__r0r1r2r*)klassr4invalidkv valid_paramss r< set_defaults$_with_defaults..set_defaultsKsef+ , +ENN+;2gYG LLNDA&*mmA&6E  ##r?cB[R"UR5$)z7Return a deep copy of the current class-level defaults.)r1r2r*)rDs r< get_defaults$_with_defaults..get_defaults]s}}U1122r?cU(a&UHnURRUS5 M! gURR5 g)zyClear class-level defaults. With no arguments, removes all overrides. With arguments, removes only the named overrides. N)r*popclear)rDnamesns r<reset_defaults&_with_defaults..reset_defaultsbs; $$((D1  & & (r?) __init__inspect signature parameters frozenset functoolswrapsr* classmethodrIrLrS) clssigpr=rIrLrSr:r;rHs @@@r<_with_defaultsr`(sLLM   M *C!nn.r LLMConfigrb SEOFilter URLFilter URLScorer DisplayMode FilterChain ProxyConfig DomainFilter BrowserConfigr SeedingConfigCosineStrategyCompositeScorerFreshnessScorerPathDepthScorerCrawlerRunConfigLLMContentFilterURLPatternFilterBM25ContentFilterContentTypeFilterGeolocationConfigHTTPCrawlerConfigLinkPreviewConfigNoTableExtractionSemaphoreDispatcherVirtualScrollConfigBFSDeepCrawlStrategyDFSDeepCrawlStrategyPruningContentFilterDomainAuthorityScorerrContentRelevanceFilterrKeywordRelevanceScorerrRegexExtractionStrategyRoundRobinProxyStrategyrMemoryAdaptiveDispatcherBestFirstCrawlingStrategyJsonCssExtractionStrategyJsonLxmlExtractionStrategyPDFContentScrapingStrategyJsonXPathExtractionStrategyobjignore_default_valuec Ucg[U[[[[45(aU$[U[ 5(a#UR RURS.$[US5(aUR5$[U[[[45(d&[US5(a1[U[5(dUVs/sHn[U5PM sn$[U[ 5(a%[U5Vs/sHn[U5PM sn$[U[5(a;SUR#5VVs0sHup4[U5[U5_M snnS.$UR Rn[US5(a[$R&"UR R(5nUR*n0nUR#5HvupU S:XaM [-X U R.5n [1U 5(a[1U R.5(aMOXR.:wdM`U(aMi[U 5X'Mx UR RUS.$[U5$s snfs snfs snnf) zp Recursively convert an object to a serializable dictionary using {type, params} structure for complex objects. N)r)params isoformat__iter__dict)r)r9 __class__r&) isinstancestrintfloatboolr$rrCr9hasattrrlisttupler+rto_serializable_dictrYr0rVrWrUrXgetattrdefaultis_empty_value) rritemrFrG_typer^rcurrent_valuesnameparamr9s r<rrs  {#S%.// #t ..#))DDsK  }}#eS)**gc:.F.FzZ]_cOdOd7:;st$T*s;;#y!!7;CyAyt$T*yAA#tBE))+N+$!c!f2155+N  MM " "EsK   6 67!<<>KDv~Cu}}5E#5))nU]].K.KMM)2F2F+?+FN(*.MM**$  s8Oe<B OsI> J !JdatareturncUcg[U[[[[45(aU$[U[ 5(Ga!SU;GaSU;dUSS:XGa SU;GaUSS:Xa8SU;a2USR 5VVs0sHupU[U5_M snn$USnU[;a[SUS35eSnS/nUH7n[R"U5n[Xs5(a [Xs5n OM9 Ub_[!U["5(a U"US5$SU;a9USR 5VVs0sHupU[U5_M nnnU"S 0UD6$[U[$5(aUV s/sHn [U 5PM sn $[U[ 5(a/UR 5VVs0sHupU[U5_M snn$U$s snnf![[4a GM.f=fs snnfs sn fs snnf) zK Recursively convert a serializable dictionary back to an object instance. Nr)rrr9zDeserialization of type 'zX' is not allowed. Only allowlisted configuration and strategy types can be deserialized.crawl4airf)rrrrrrr0from_serializable_dictALLOWED_DESERIALIZE_TYPESrB importlib import_modulerr ImportErrorAttributeError issubclassr$r) rrFrG type_namer] module_paths module_pathmodconstructor_argsrs r<rrs& |$c5$/00  4 dN  $v,&"8W_ <6 !go=A']=P=P=RS=RTQA-a00=RS SL 5 5+I;7YZ  "| 'K --k:3**!#1C+( ?#t$$4>**4>B(^=Q=Q=S$=STQA-a00=S!$.-..$9=>&t,>>$9=F)!,,FF KST$ 0  $?Gs*G 1G G8<G><H G54G5r9cUcg[U[[[[[ 45(a[ U5S:Xagg)z+Check if a value is effectively empty/null.TrF)rrrr+rrr.)r9s r<rr/s2 }%$sD#677CJ!O r?cj\rSrSrS S\S\S\\4Sjjr\S\SS4Sj5r S\4S jr SS jr S r g )ri7latitude longitudeaccuracyc(XlX lX0lg)zConfiguration class for geolocation settings. Args: latitude: Latitude coordinate (e.g., 37.7749) longitude: Longitude coordinate (e.g., -122.4194) accuracy: Accuracy in meters. Default: 0.0 Nrrr)r&rrrs r<rUGeolocationConfig.__init__8s! " r?geo_dictrct[URS5URS5URSS5S9$)z-Create a GeolocationConfig from a dictionary.rrrr)rget)rs r< from_dictGeolocationConfig.from_dictIs7!\\*-ll;/\\*c2  r?cJURURURS.$)%Convert to dictionary representation.rrr&s r<to_dictGeolocationConfig.to_dictRs#    r?c nUR5nURU5 [RU5$)zCreate a copy of this configuration with updated values. Args: **kwargs: Key-value pairs of configuration options to update Returns: GeolocationConfig: A new instance with the specified updates )rupdaterrr&r4 config_dicts r<cloneGeolocationConfig.cloneZs.lln 6" **;77r?)rrrN)r)rr) rCrgrhrirr"rU staticmethodr rrrrlrfr?r<rr7s` %( !!!5/ !" D %8     8r?rc \rSrSrSrSS\S\\S\\S\\4SjjrS \\4S jr\ S \S S4S j5r \ S \ S S4Sj5r \ SS\S \ S4Sjj5rS \ 4SjrSSjrSrg)rsigdirectNserverusernamepasswordipcdXlX lX0lU=(d UR5Ulg)aConfiguration class for a single proxy. Args: server: Proxy server URL (e.g., "http://127.0.0.1:8080") username: Optional username for proxy authentication password: Optional password for proxy authentication ip: Optional IP address for verification purposes N)rrr_extract_ip_from_serverr)r&rrrrs r<rUProxyConfig.__init__js*   6446r?rcSUR;a2URRS5SRS5nUS$URRS5nUS$![a gf=f)z#Extract IP address from server URL.://r:rN)rsplit Exception)r&partss r<r#ProxyConfig._extract_ip_from_serverso  # ))%0399#>Qx ))#.Qx  sAA$A$$ A10A1 proxy_strcU=(d SR5nSU;aXSU;aRURSS5up#URSS5upESU;a$URSS5upg[USU3UUS9$SU;aSU;a [US9$URS5n[U5S:XaUuppg[S U SU 3XgS9$[U5S :XaUup[S U SU 3S9$[ S U35e) zCreate a ProxyConfig from a string. Supported formats: - 'http://username:password@ip:port' - 'http://ip:port' - 'socks5://ip:port' - 'ip:port:username:password' - 'ip:port' @rrrrrr)rzhttp://zInvalid proxy string format: )striprrsr.rB) rs auth_part server_partprotocol credentialsrrrrports r< from_stringProxyConfig.from_strings_" # # % !8 %&WWS!_ "I$-OOE1$= !Hk!%0%6%6sA%>""&Zs;-8%% A:#Q,a( (  u:?+0 (Bht1TF&;hb b u:?HBt1TF&;< <8 DEEr? proxy_dictc[URS5URS5URS5URS5S9$)z'Create a ProxyConfig from a dictionary.rrrrrrrr)rsr)rs r<rProxyConfig.from_dictsA>>(+^^J/^^J/~~d#   r?env_varc/n[R"US5RS5nUH0nU(dM UR[R U55 M2 U$![ an[SU35 SnAU$SnAff=f)zLoad proxies from environment variable. Args: env_var: Name of environment variable containing comma-separated proxy strings Returns: List of ProxyConfig objects r,z(Error loading proxies from environment: N)osgetenvrappendrsrrprint)rproxies proxy_listproxyes r<from_envProxyConfig.from_envs B7B/55c:J#{66u=>$  B S4S?j5rS@rSArS>\ 4SBjr\SC\ S>S4SDj5rSNSE\SF\SG\SH\SI\SJ\SK\ 4SLjjrSMrg)Oruia" Configuration class for setting up a browser instance and its context in AsyncPlaywrightCrawlerStrategy. This class centralizes all parameters that affect browser and context creation. Instead of passing scattered keyword arguments, users can instantiate and modify this configuration object. The crawler code will then reference these settings to initialize the browser in a consistent, documented manner. Attributes: browser_type (str): The type of browser to launch. Supported values: "chromium", "firefox", "webkit". Default: "chromium". headless (bool): Whether to run the browser in headless mode (no visible GUI). Default: True. browser_mode (str): Determines how the browser should be initialized: "builtin" - use the builtin CDP browser running in background "dedicated" - create a new dedicated browser instance each time "cdp" - use explicit CDP settings provided in cdp_url "docker" - run browser in Docker container with isolation Default: "dedicated" use_managed_browser (bool): Launch the browser using a managed approach (e.g., via CDP), allowing advanced manipulation. Default: False. cdp_url (str): URL for the Chrome DevTools Protocol (CDP) endpoint. Default: "ws://localhost:9222/devtools/browser/". browser_context_id (str or None): Pre-existing CDP browser context ID to use. When provided along with cdp_url, the crawler will reuse this context instead of creating a new one. Useful for cloud browser services that pre-create isolated contexts. Default: None. target_id (str or None): Pre-existing CDP target ID (page) to use. When provided along with browser_context_id, the crawler will reuse this target instead of creating a new page. Default: None. cdp_cleanup_on_close (bool): When True and using cdp_url, the close() method will still clean up the local Playwright client resources. Useful for cloud/server scenarios where you don't own the remote browser but need to prevent memory leaks from accumulated Playwright instances. Default: False. cdp_close_delay (float): Seconds to wait after disconnecting a CDP WebSocket before stopping the Playwright subprocess. Gives the connection time to fully release. Set to 0 to skip the delay entirely. Only applies when cdp_cleanup_on_close=True. Default: 1.0. cache_cdp_connection (bool): When True and using cdp_url, the Playwright subprocess and CDP WebSocket are cached at the class level and shared across multiple BrowserManager instances connecting to the same cdp_url. Reference-counted; the connection is only closed when the last user releases it. Eliminates the overhead of repeated Playwright/CDP setup and teardown. Default: False. create_isolated_context (bool): When True and using cdp_url, forces creation of a new browser context instead of reusing the default context. Essential for concurrent crawls on the same browser to prevent navigation conflicts. Default: False. debugging_port (int): Port for the browser debugging protocol. Default: 9222. use_persistent_context (bool): Use a persistent browser context (like a persistent profile). Automatically sets use_managed_browser=True. Default: False. user_data_dir (str or None): Path to a user data directory for persistent sessions. If None, a temporary directory may be used. Default: None. chrome_channel (str): The Chrome channel to launch (e.g., "chrome", "msedge"). Only applies if browser_type is "chromium". Default: "chromium". channel (str): The channel to launch (e.g., "chromium", "chrome", "msedge"). Only applies if browser_type is "chromium". Default: "chromium". proxy (Optional[str]): Proxy server URL (e.g., "http://username:password@proxy:port"). If None, no proxy is used. Default: None. proxy_config (ProxyConfig or dict or None): Detailed proxy configuration, e.g. {"server": "...", "username": "..."}. If None, no additional proxy config. Default: None. viewport_width (int): Default viewport width for pages. Default: 1080. viewport_height (int): Default viewport height for pages. Default: 600. viewport (dict): Default viewport dimensions for pages. If set, overrides viewport_width and viewport_height. Default: None. device_scale_factor (float): The device pixel ratio used for rendering pages. Controls how many physical pixels map to one CSS pixel, allowing simulation of HiDPI or Retina displays. For example, a viewport of 1920x1080 with a device_scale_factor of 2.0 produces screenshots at 3840x2160 resolution. Increasing this value improves screenshot quality but may increase memory usage and rendering time. Default: 1.0. verbose (bool): Enable verbose logging. Default: True. accept_downloads (bool): Whether to allow file downloads. If True, requires a downloads_path. Default: False. downloads_path (str or None): Directory to store downloaded files. If None and accept_downloads is True, a default path will be created. Default: None. storage_state (str or dict or None): An in-memory storage state (cookies, localStorage). Default: None. ignore_https_errors (bool): Ignore HTTPS certificate errors. Default: True. java_script_enabled (bool): Enable JavaScript execution in pages. Default: True. cookies (list): List of cookies to add to the browser context. Each cookie is a dict with fields like {"name": "...", "value": "...", "url": "..."}. Default: []. headers (dict): Extra HTTP headers to apply to all requests in this context. Default: {}. user_agent (str): Custom User-Agent string to use. Default: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) " "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36". user_agent_mode (str or None): Mode for generating the user agent (e.g., "random"). If None, use the provided user_agent as-is. Default: None. user_agent_generator_config (dict or None): Configuration for user agent generation if user_agent_mode is set. Default: None. text_mode (bool): If True, disables images and other rich content for potentially faster load times. Default: False. light_mode (bool): Disables certain background features for performance gains. Default: False. extra_args (list): Additional command-line arguments passed to the browser. Default: []. enable_stealth (bool): If True, applies playwright-stealth to bypass basic bot detection. Cannot be used with use_undetected browser mode. Default: False. memory_saving_mode (bool): If True, adds aggressive cache discard and V8 heap cap flags to reduce Chromium memory growth. Recommended for high-volume crawling (1000+ pages). May slightly reduce performance due to cache eviction. Default: False. max_pages_before_recycle (int): Number of pages to crawl before recycling the browser process to reclaim leaked memory. 0 = disabled. Recommended: 500-1000 for long-running crawlers. Default: 0. avoid_ads (bool): If True, blocks ad-related and tracker network requests at the browser context level using a curated blocklist of top ad/tracker domains. Default: False. avoid_css (bool): If True, blocks loading of CSS files (css, less, scss, sass) to reduce resource usage and speed up crawling. Default: False. chromiumT dedicatedFN?8XzQMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36ri$ localhostr browser_typeheadless browser_modeuse_managed_browsercdp_urlbrowser_context_id target_idcdp_cleanup_on_closecdp_close_delaycache_cdp_connectioncreate_isolated_contextuse_persistent_context user_data_dirchrome_channelchannelr proxy_configviewport_widthviewport_heightviewportdevice_scale_factoraccept_downloadsdownloads_path storage_stateignore_https_errorsjava_script_enabledsleep_on_closeverbosecookiesheaders user_agentuser_agent_modeuser_agent_generator_config text_mode light_mode extra_argsdebugging_porthostenable_stealth avoid_ads avoid_css init_scriptsmemory_saving_modemax_pages_before_recyclec-XlX lX0lX@lXPlX`lXplXlXlXl Xl Xl Xl U=(d UR=(d SUl U=(d UR=(d SUlURS;aSUlSUl U(a[R "S["5 UUlUUl[)UR&[*5(a$[,R/UR&5Ul[)UR&[05(a$[,R3UR&5UlUR$(a4UR&(a#[R "S["5 SUlOr?rgeneratergenerate_client_hints browser_hint setdefaultrB).r&rrrrrrrrrrrr r!r"r#rr$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?fa_user_agenr_generators. r<rUBrowserConfig.__init__Wsj) (#6  "4"$8!.$8!'>$&<#*,O0A0AOZA$"3"3Az    5 5DL"$D   MMEGR S ( d'' . . + 5 5d6G6G HD  d'' - - + 7 78I8I JD  ::$++ MMnp{ |DJ ZZ + 7 7 CD DJ,.  == $"&--"3"3GT"BD #'==#4#4Xs#CD #6 0,*#6 #6 ")"5w2 ")"5w2 $.+F("$(2(>*B, , ,"",8,DL""4(@%"2"4   8 +5>>339rDO !77H  T->->?    )'+D $   ( *'+D $   ( *t||'+D $   + -   & &'+D $   4#;#;@Q@QU^@^F A_#; r?r4rc UR5VVs0sH.upU[U[5(aSU;a [U5OU_M0 nnn[R "[ R5RR5S1- n[ S0UR5VVs0sHupX;dM X_M snnD6$s snnfs snnfNr)r&rf) r0rrrrVrWrurUrXr,r4rFrGvalids r< from_kwargsBrowserConfig.from_kwargss   & Jq$,?,?FaK%a(UV V&  !!-"8"89DDIIKvhVON1:NOO  O5C& C5Cc 0SUR_SUR_SUR_SUR_SUR_SUR _SUR _SUR_S UR_S UR_S UR_S UR_S UR_SUR_S[URS5(aURR!5O UR_SUR"_SUR$_0SUR&_SUR(_SUR*_SUR,_SUR._SUR0_SUR2_SUR4_SUR6_SUR8_SUR:_SUR<_SUR>_S UR@_S!URB_S"URD_S#URF_EURHURJURLURNURPURRURTS$.EnU$)%Nrrrrrrrrrr r!r"r#rr$rr%r&r(r)r*r+r,r-r0r1r2r3r4r5r6r7r.r/r8)r9r:r;r<r=r>r?)+rrrrrrrrrr r!r"r#rrr$rr%r&r(r)r*r+r,r-r0r1r2r3r4r5r6r7r.r/r8r9r:r;r<r=r>r?)r&results r<rBrowserConfig.to_dicts* D--*  *  D--*  "4#;#; * t|| * !$"9"9 *  *  #D$=$=*  &t'C'C*  %d&A&A*  T//*  d11*  t||*  TZZ*  74CTCTV_;`;`D--557fjfwfw* d11!* " t33#* $ "4#;#;%* &  5 5'* ( d11)* * T//+* , "4#;#;-* . "4#;#;/* 0 t||1* 2 t||3* 4 $//5* 6 t337* 8 *4+K+K9* : ;* < $//=* > $//?* @ d11A* B t||C* D d11E* FII"11 --"&"9"9(,(E(ES* Z r?c nUR5nURU5 [RU5$)zCreate a copy of this configuration with updated values. Args: **kwargs: Key-value pairs of configuration options to update Returns: BrowserConfig: A new instance with the specified updates )rrrurVrs r<rBrowserConfig.clone1s.lln 6"((55r?c[U5$r(rrs r<dumpBrowserConfig.dump? #D))r?rcp[U5n[U[5(aU$[RU5$r()rrrurVrconfigs r<loadBrowserConfig.loadCs1(- fm , ,M((00r?token channel_idcountrystatecityrsession_durationcU(aU(d [S5eUS;a[SU35eSSUUUUUS.nU(aXHS'U(aXXS'S n [R"XS S 9n U R5 U R 5n [ U [ 5(a4U RS 5(a[S U RSS535e[ U [5(aU (d [S5eU Sn [USU SSU S3U SU SS9Ul g![an [SU 35 eSn A ff=f)a Fetch a proxy from NSTProxy API and automatically assign it to proxy_config. Get your NSTProxy token from: https://app.nstproxy.com/profile Args: token (str): NSTProxy API token. channel_id (str): NSTProxy channel ID. country (str, optional): Country code (default: "ANY"). state (str, optional): State code (default: ""). city (str, optional): City name (default: ""). protocol (str, optional): Proxy protocol ("http" or "socks5"). Defaults to "http". session_duration (int, optional): Session duration in minutes (0 = rotate each request). Defaults to 10. Raises: ValueError: If the API response format is invalid. PermissionError: If the API returns an error message. z,[NSTProxy] token and channel_id are required)httpsocks5z[NSTProxy] Invalid protocol: rr)fTypecount channelIdrjrsessionDurationrhrkrlz3https://api.nstproxy.com/api/v1/generate/apiproxies )rtimeouterrz[NSTProxy] API Error: msgz Unknown erroru=[NSTProxy] Invalid API response — expected a non-empty listrrrrrrrru$[NSTProxy] ❌ Failed to set proxy: N) rBrequestsrraise_for_statusjsonrrPermissionErrorrrsr$rr)r&rhrirjrkrlrrmrurlresponser proxy_infors r< set_nstproxyBrowserConfig.set_nstproxyKsX<JKL L - -txx?_>`&abbdD)) !`aaaJ!,"3z$'7&8*V:L9MN#J/#J/!D    8< =  sCD D>*D99D>)-r)r;r<rrNrrrrrrr#r"r0rr8r(r*r:r7r1rr9r,r=r-r6r?r>rr$r.r+rr5rr r2r4r3r!r/r'r&r%)ANYrrroru)rCrgrhri__doc__rrrr#rsrrrr!rUrrVrrr`rfrrlrfr?r<ruruszmb''$)"&%*!$%*(-',!(!7;""%(!&"04$($($ `!,. "$"&#(()e[[[ [ " [  [ [[#[[#["&[!%[[[ !["#[$Kt34%[&'[()[*+[,#-[./[01[2S$_-3[4"5[6"7[89[:;[<=[>?[@A[LM[N&*O[PQ[RS[TU[VW[XY[Z[[\][^_[`3ia[b!c[d#&e[z PD P_ P P.` 6*d*141O11 "MMM M  M  MMMMr?ruc p\rSrSrSrSS\S\S\\\4S\4Sjjr S\ 4S jr \ S \ SS4S j5r S rg )rizConfiguration for virtual scroll handling. This config enables capturing content from pages with virtualized scrolling (like Twitter, Instagram feeds) where DOM elements are recycled as user scrolls. container_selector scroll_count scroll_bywait_after_scrollc4XlX lX0lX@lg)a Initialize virtual scroll configuration. Args: container_selector: CSS selector for the scrollable container scroll_count: Maximum number of scrolls to perform scroll_by: Amount to scroll - can be: - "container_height": scroll by container's height - "page_height": scroll by viewport height - int: fixed pixel amount wait_after_scroll: Seconds to wait after each scroll for content to load Nrrrr)r&rrrrs r<rUVirtualScrollConfig.__init__s&#5("!2r?rc`URURURURS.$)z(Convert to dictionary for serialization.rrrs r<rVirtualScrollConfig.to_dicts0#'"9"9 --!%!7!7   r?rcU"S0UD6$)z Create instance from dictionary.rfrf)r]rs r<rVirtualScrollConfig.from_dicts{T{r?)rrrrN)rucontainer_heightg?)rCrgrhrirrrr#rrUrrr\rrlrfr?r<rrst%7#& 333c? 3 ! 30  T&;r?rc\rSrSrSrSS\S\S\\\S\\\S\ S \ S \ S \\S \\ S \4Sjjr \ S\ \\4SS4Sj5rS\ \\44SjrSSjrSrg)riz3Configuration for link head extraction and scoring.Ninclude_internalinclude_externalinclude_patternsexclude_patterns concurrencyrv max_linksqueryscore_thresholdr/c PXlX lX0lX@lXPlX`lXplXlXlXl US::a [S5eUS::a [S5eUS::a [S5eU bSU s=::aS::d O [S5eU(dU(d [S 5egg) a Initialize link extraction configuration. Args: include_internal: Whether to include same-domain links include_external: Whether to include different-domain links include_patterns: List of glob patterns to include (e.g., ["*/docs/*", "*/api/*"]) exclude_patterns: List of glob patterns to exclude (e.g., ["*/login*", "*/admin*"]) concurrency: Number of links to process simultaneously timeout: Timeout in seconds for each link's head extraction max_links: Maximum number of links to process (prevents overload) query: Query string for BM25 contextual scoring (optional) score_threshold: Minimum relevance score to include links (0.0-1.0, optional) verbose: Show detailed progress during extraction rzconcurrency must be positiveztimeout must be positivezmax_links must be positiveNrrz+score_threshold must be between 0.0 and 1.0zAAt least one of include_internal or include_external must be True) rrrrrrvrrrr/rB) r&rrrrrrvrrrr/s r<rULinkPreviewConfig.__init__s8!1 0 0 0& " .  ! ;< < a<78 8 >9: :  &0M#0MJK K(8`a a)9r?rrcnU(dg[URSS5URSS5URS5URS5URSS 5URS S 5URS S 5URS5URS5URSS5S9 $)zFCreate LinkPreviewConfig from dictionary (for backward compatibility).NrTrFrrrrurvrdrrr/ rrrrrrvrrrr/)rr)rs r<rLinkPreviewConfig.from_dicts (__-?F(__-?G(__-?@(__-?@# r:OOIq1!ook37//'*'OO,=>OOIu5  r?c URURURURURUR UR URURURS. $)zConvert to dictionary format.rrrs r<rLinkPreviewConfig.to_dicts`!% 5 5 $ 5 5 $ 5 5 $ 5 5++||ZZ#33||  r?c nUR5nURU5 [RU5$)z"Create a copy with updated values.)rrrrrs r<rLinkPreviewConfig.clones,lln 6" **;77r?) rrrrrrrrrvr/) TFNNrurrNNF)rr)rCrgrhrirrr"r!rrrrUrr rrrrrlrfr?r<rrs="&!&0404#+/1b1b1b#49- 1b #49- 1b  1b1b1b}1b"%1b1bf tCH~ 2E  $  c3h  8r?rc R\rSrSr%SrSr\\S'Sr\ \ \\4\S'Sr \ \ \\ 4\S'Sr \ \ \\ 4\S'S r\\S 'S r\\S 'SS\S\ \ \\4S\ \ \\ 4S\ \ \\ 4S \S \4 S jjr\S \SS4Sj5rSrSrS\4Sjr\S\SS4Sj5rSrg)ri&z#HTTP-specific crawler configurationGETmethodNr1rr{Tfollow_redirects verify_sslcLXlX lX0lX@lXPlX`lgr(rr1rr{rr)r&rr1rr{rrs r<rUHTTPCrawlerConfig.__init__0s#    0$r?r4rc [URSS5URS5URS5URS5URSS5URSS5S 9$) Nrrr1rr{rTrr)rrr4s r<rVHTTPCrawlerConfig.from_kwargs@s^ ::h.JJy)F#F##ZZ(:DAzz,5   r?cURURURURURUR S.$)Nrrrs r<rHTTPCrawlerConfig.to_dictKs8kk||IIII $ 5 5//   r?c nUR5nURU5 [RU5$)zCreate a copy of this configuration with updated values. Args: **kwargs: Key-value pairs of configuration options to update Returns: HTTPCrawlerConfig: A new instance with the specified updates )rrrrVrs r<rHTTPCrawlerConfig.cloneUs.lln 6" ,,[99r?c[U5$r(r_rs r<r`HTTPCrawlerConfig.dumpbs #D))r?cp[U5n[U[5(aU$[RU5$r()rrrrVrds r<rfHTTPCrawlerConfig.loades0'- f/ 0 0M ,,V44r?)rrr1r{rr)rNNNTT)rCrgrhrirrr__annotations__r1r"r rrr{rrrrUrrrVrrr`rfrlrfr?r<rr&s?-FC(,GXd38n %,%)D(4S> ")%)D(4S> ")!d!J,0)-)-!%%%$sCx.)%tCH~& % tCH~& %  %%  D %8    :*d*545/55r?rcd^\rSrSrSrSSSSS.r\S\"5\"5S SSSSS SS S S SSSSSS SSSS \ RSS S S S SS S S \ SSS S S SSSSSS SS SSS S S S S S S S S S\ S S S \ \SSS S SS S SS S S SS S S SS S S SSS SS0SSSS\R SSS4cS\S\S\S\S\S\S\\S\S\S\S\S \S!\S"\S#\S$\S%\S%\\S4S&\\S'\\S(\\S)\S*\\S+\\S,\\S-\S.\ S/\S0\S1\S2\S3\S4\S5\S6\S7\S8\S9\S:\S;\S<\S=\S>\S?\S@\\\\4SA\\\\4SB\\\\4SC\SD\SE\SF\SG\\SH\SI\SJ\SK\SL\SM\SN\SO\SP\SQ\SR\SS\ST\SU\SV\SW\SX\SY\ SZ\S[\S\\S]\S^\S_\S`\Sa\Sb\Sc\Sd\Se\Sf\Sg\Sh\Si\Sj\Sk\Sl\Sm\Sn\So\Sp\Sq\\!Sr\\"\#\\$44Ss\\%\#\\$44St\\&Su\Sv\#\\$4Sw\Sx\\'\/\(\44Syjjr)\*Sz5r+\,S{5r-\-R\S|5r-S}\4S~jr/Sr0Sk\S}\4Sjr1Sr2U4Sjr3\*S\S}S4Sj5r4S}\4Sjr5\*S\S}S4Sj5r6Sr7Sr8Sr9U=r:$)r{ilax@ Configuration class for controlling how the crawler runs each crawl operation. This includes parameters for content extraction, page manipulation, waiting conditions, caching, and other runtime behaviors. This centralizes parameters that were previously scattered as kwargs to `arun()` and related methods. By using this class, you have a single place to understand and adjust the crawling options. Attributes: # Deep Crawl Parameters deep_crawl_strategy (DeepCrawlStrategy or None): Strategy to use for deep crawling. # Content Processing Parameters word_count_threshold (int): Minimum word count threshold before processing content. Default: MIN_WORD_THRESHOLD (typically 200). extraction_strategy (ExtractionStrategy or None): Strategy to extract structured data from crawled pages. Default: None (NoExtractionStrategy is used if None). chunking_strategy (ChunkingStrategy): Strategy to chunk content before extraction. Default: RegexChunking(). markdown_generator (MarkdownGenerationStrategy): Strategy for generating markdown. Default: None. only_text (bool): If True, attempt to extract text-only content where applicable. Default: False. css_selector (str or None): CSS selector to extract a specific portion of the page. Default: None. target_elements (list of str or None): List of CSS selectors for specific elements for Markdown generation and structured data extraction. When you set this, only the contents of these elements are processed for extraction and Markdown generation. If you do not set any value, the entire page is processed. The difference between this and css_selector is that this will shrink the initial raw HTML to the selected element, while this will only affect the extraction and Markdown generation. Default: None excluded_tags (list of str or None): List of HTML tags to exclude from processing. Default: None. excluded_selector (str or None): CSS selector to exclude from processing. Default: None. keep_data_attributes (bool): If True, retain `data-*` attributes while removing unwanted attributes. Default: False. keep_attrs (list of str): List of HTML attributes to keep during processing. Default: []. remove_forms (bool): If True, remove all `
` elements from the HTML. Default: False. prettiify (bool): If True, apply `fast_format_html` to produce prettified HTML output. Default: False. parser_type (str): Type of parser to use for HTML parsing. Default: "lxml". scraping_strategy (ContentScrapingStrategy): Scraping strategy to use. Default: LXMLWebScrapingStrategy. proxy_config (ProxyConfig or dict or None): Detailed proxy configuration, e.g. {"server": "...", "username": "..."}. If None, no additional proxy config. Default: None. # Sticky Proxy Session Parameters proxy_session_id (str or None): When set, maintains the same proxy for all requests sharing this session ID. The proxy is acquired on first request and reused for subsequent requests. Session expires when explicitly released or crawler context is closed. Default: None. proxy_session_ttl (int or None): Time-to-live for sticky session in seconds. After TTL expires, a new proxy is acquired on next request. Default: None (session lasts until explicitly released or crawler closes). proxy_session_auto_release (bool): If True, automatically release the proxy session after a batch operation. Useful for arun_many() to clean up sessions automatically. Default: False. # Browser Location and Identity Parameters locale (str or None): Locale to use for the browser context (e.g., "en-US"). Default: None. timezone_id (str or None): Timezone identifier to use for the browser context (e.g., "America/New_York"). Default: None. geolocation (GeolocationConfig or None): Geolocation configuration for the browser. Default: None. # SSL Parameters fetch_ssl_certificate: bool = False, # Caching Parameters cache_mode (CacheMode or None): Defines how caching is handled. If None, defaults to CacheMode.ENABLED internally. Default: CacheMode.BYPASS. session_id (str or None): Optional session ID to persist the browser context and the created page instance. If the ID already exists, the crawler does not create a new page and uses the current page to preserve the state. bypass_cache (bool): Legacy parameter, if True acts like CacheMode.BYPASS. Default: False. disable_cache (bool): Legacy parameter, if True acts like CacheMode.DISABLED. Default: False. no_cache_read (bool): Legacy parameter, if True acts like CacheMode.WRITE_ONLY. Default: False. no_cache_write (bool): Legacy parameter, if True acts like CacheMode.READ_ONLY. Default: False. shared_data (dict or None): Shared data to be passed between hooks. Default: None. # Cache Validation Parameters (Smart Cache) check_cache_freshness (bool): If True, validates cached content freshness using HTTP conditional requests (ETag/Last-Modified) and head fingerprinting before returning cached results. Avoids full browser crawls when content hasn't changed. Only applies when cache_mode allows reads. Default: False. cache_validation_timeout (float): Timeout in seconds for cache validation HTTP requests. Default: 10.0. # Page Navigation and Timing Parameters wait_until (str): The condition to wait for when navigating, e.g. "domcontentloaded". Default: "domcontentloaded". page_timeout (int): Timeout in ms for page operations like navigation. Default: 60000 (60 seconds). wait_for (str or None): A CSS selector or JS condition to wait for before extracting content. Default: None. wait_for_timeout (int or None): Specific timeout in ms for the wait_for condition. If None, uses page_timeout instead. Default: None. wait_for_images (bool): If True, wait for images to load before extracting content. Default: False. delay_before_return_html (float): Delay in seconds before retrieving final HTML. Default: 0.1. mean_delay (float): Mean base delay between requests when calling arun_many. Default: 0.1. max_range (float): Max random additional delay range for requests in arun_many. Default: 0.3. semaphore_count (int): Number of concurrent operations allowed. Default: 5. # Page Interaction Parameters js_code (str or list of str or None): JavaScript code/snippets to run on the page after wait_for and delay_before_return_html. Default: None. js_code_before_wait (str or list of str or None): JavaScript to run BEFORE wait_for. Use for triggering loading that wait_for then checks. Default: None. js_only (bool): If True, indicates subsequent calls are JS-driven updates, not full page loads. Default: False. ignore_body_visibility (bool): If True, ignore whether the body is visible before proceeding. Default: True. scan_full_page (bool): If True, scroll through the entire page to load all content. Default: False. scroll_delay (float): Delay in seconds between scroll steps if scan_full_page is True. Default: 0.2. max_scroll_steps (Optional[int]): Maximum number of scroll steps to perform during full page scan. If None, scrolls until the entire page is loaded. Default: None. process_iframes (bool): If True, attempts to process and inline iframe content. Default: False. flatten_shadow_dom (bool): If True, flatten shadow DOM content into the light DOM before HTML capture so page.content() includes it. Also injects an init script to force-open closed shadow roots. Default: False. remove_overlay_elements (bool): If True, remove overlays/popups before extracting HTML. Default: False. remove_consent_popups (bool): If True, remove GDPR/cookie consent popups (IAB TCF/CMP) before extracting HTML. Targets known CMP providers like OneTrust, Cookiebot, TrustArc, Quantcast, Didomi, etc. Default: False. simulate_user (bool): If True, simulate user interactions (mouse moves, clicks) for anti-bot measures. Default: False. override_navigator (bool): If True, overrides navigator properties for more human-like behavior. Default: False. magic (bool): If True, attempts automatic handling of overlays/popups. Default: False. adjust_viewport_to_content (bool): If True, adjust viewport according to the page content dimensions. Default: False. # Media Handling Parameters screenshot (bool): Whether to take a screenshot after crawling. Default: False. screenshot_wait_for (float or None): Additional wait time before taking a screenshot. Default: None. screenshot_height_threshold (int): Threshold for page height to decide screenshot strategy. Default: SCREENSHOT_HEIGHT_TRESHOLD (from config, e.g. 20000). force_viewport_screenshot (bool): If True, always take viewport-only screenshots regardless of page height. When False, uses automatic decision (viewport for short pages, full-page for long pages). Default: False. pdf (bool): Whether to generate a PDF of the page. Default: False. image_description_min_word_threshold (int): Minimum words for image description extraction. Default: IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD (e.g., 50). image_score_threshold (int): Minimum score threshold for processing an image. Default: IMAGE_SCORE_THRESHOLD (e.g., 3). exclude_external_images (bool): If True, exclude all external images from processing. Default: False. table_score_threshold (int): Minimum score threshold for processing a table. Default: 7. table_extraction (TableExtractionStrategy): Strategy to use for table extraction. Default: DefaultTableExtraction with table_score_threshold. # Virtual Scroll Parameters virtual_scroll_config (VirtualScrollConfig or dict or None): Configuration for handling virtual scroll containers. Used for capturing content from pages with virtualized scrolling (e.g., Twitter, Instagram feeds). Default: None. # Link and Domain Handling Parameters exclude_social_media_domains (list of str): List of domains to exclude for social media links. Default: SOCIAL_MEDIA_DOMAINS (from config). exclude_external_links (bool): If True, exclude all external links from the results. Default: False. exclude_internal_links (bool): If True, exclude internal links from the results. Default: False. exclude_social_media_links (bool): If True, exclude links pointing to social media domains. Default: False. exclude_domains (list of str): List of specific domains to exclude from results. Default: []. exclude_internal_links (bool): If True, exclude internal links from the results. Default: False. score_links (bool): If True, calculate intrinsic quality scores for all links using URL structure, text quality, and contextual relevance metrics. Separate from link_preview_config. Default: False. # Debugging and Logging Parameters verbose (bool): Enable verbose logging. Default: True. log_console (bool): If True, log console messages from the page. Default: False. # HTTP Crwler Strategy Parameters method (str): HTTP method to use for the request, when using AsyncHTTPCrwalerStrategy. Default: "GET". data (dict): Data to send in the request body, when using AsyncHTTPCrwalerStrategy. Default: None. json (dict): JSON data to send in the request body, when using AsyncHTTPCrwalerStrategy. # Connection Parameters stream (bool): If True, enables streaming of crawled URLs as they are processed when used with arun_many. Default: False. process_in_browser (bool): If True, forces raw:/file:// URLs to be processed through the browser pipeline (enabling js_code, wait_for, scrolling, etc.). When False (default), raw:/file:// URLs use a fast path that returns HTML directly without browser interaction. This is automatically enabled when browser-requiring parameters are detected (js_code, wait_for, screenshot, pdf, etc.). Default: False. check_robots_txt (bool): Whether to check robots.txt rules before crawling. Default: False Default: False. user_agent (str): Custom User-Agent string to use. Default: None. user_agent_mode (str or None): Mode for generating the user agent (e.g., "random"). If None, use the provided user_agent as-is. Default: None. user_agent_generator_config (dict or None): Configuration for user agent generation if user_agent_mode is set. Default: None. # Experimental Parameters experimental (dict): Dictionary containing experimental parameters that are in beta phase. This allows passing temporary features that are not yet fully integrated into the main parameter set. Default: None. url: str = None # This is not a compulsory parameter z*Instead, use cache_mode=CacheMode.DISABLEDz(Instead, use cache_mode=CacheMode.BYPASSz,Instead, use cache_mode=CacheMode.WRITE_ONLYz+Instead, use cache_mode=CacheMode.READ_ONLY) disable_cache bypass_cache no_cache_readno_cache_writeNFlxmlg$@domcontentloadedg?g333333?rTg?rrword_count_thresholdextraction_strategychunking_strategymarkdown_generator only_text css_selectortarget_elements excluded_tagsexcluded_selectorkeep_data_attributes keep_attrs remove_forms prettiify parser_typescraping_strategyr$rsproxy_rotation_strategyproxy_session_idproxy_session_ttlproxy_session_auto_releaselocale timezone_id geolocationfetch_ssl_certificate cache_mode session_idrrrr shared_datacheck_cache_freshnesscache_validation_timeout wait_until page_timeoutwait_forwait_for_timeoutwait_for_imagesdelay_before_return_html mean_delay max_rangesemaphore_countjs_codejs_code_before_wait c4a_scriptjs_onlyignore_body_visibilityscan_full_page scroll_delaymax_scroll_stepsprocess_iframesflatten_shadow_domremove_overlay_elementsremove_consent_popups simulate_useroverride_navigatormagicadjust_viewport_to_content screenshotscreenshot_wait_forscreenshot_height_thresholdforce_viewport_screenshotpdf capture_mhtml$image_description_min_word_thresholdimage_score_thresholdtable_score_thresholdtable_extractionexclude_external_imagesexclude_all_imagesexclude_social_media_domainsexclude_external_linksexclude_social_media_linksexclude_domainsexclude_internal_links score_links!preserve_https_for_internal_linksr/ log_consolecapture_network_requestscapture_console_messagesrstreamprefetchprocess_in_browserr}base_urlcheck_robots_txtr2r3r4deep_crawl_strategylink_preview_configvirtual_scroll_config url_matcher match_mode experimental max_retriesfallback_fetch_functioncd WVUlWWUlXlX lX0lX@lXPlX`lU=(d /UlU=(d /Ul U =(d SUl Xl U =(d /Ul Xl XlXlU=(d [!5UlUUlUUlUUlUUlUUlUUlUUlUUlUUlUUlUUlUUlUUlUUlUUl UUl!U Ul"U!Ul#U"Ul$U#Ul%U$Ul&U%Ul'U&Ul(U'Ul)U(Ul*U)Ul+U*Ul,U+Ul-U,Ul.U-Ul/U.Ul0U/Ul1U0Ul2U1Ul3U2Ul4U3Ul5U4Ul6U5Ul7U6Ul8U7Ul9U8Ul:U9Ul;U:Ul<U;Ul=UU=Ul?U>Ul@U?UlAW@UlBWAUlCWBUlDWEUlEWFUlFWCUlGWDc[WCS9UlIOWDUlIWG=(d [UlKWHUlLWIUlMWJ=(d /UlNWKUlOWLUlPWMUlQWNUlRWOUlSWPUlTWQUlUWSUlVWTUlWWUUlXWRUlYWXUlZWYUl[WZUl\W[Ul]URb*[UR[5(d [S5eURb*[UR[5(d [S5eURc[5UlW\UlcW]cSUldOX[W][5(aW]UldO;[W][5(aW]5UldO [S5eW^cSUlhOX[W^[5(aW^UlhO;[W^[5(aW^5UlhO [S5eW_UljW`UlkWa=(d 0UllWbUlmWcUlnUR^(a#URZ(dUR5 ggg)Nr)rz=extraction_strategy must be an instance of ExtractionStrategyz9chunking_strategy must be an instance of ChunkingStrategyz$!1!2*D' &&&;"%$(**,&%:"(@%%(  0.(@%$". #6 $ &<#,( 0."4'>$%:"*"4 *D'%#6 +F()B&*4X1%:"'>$"4%:"  #$:Qf$gD !$4D ! ) @,@ )'=#*D'.4"&<#&1R. &)A%(@%   "4 !1%.+F(  # # /  $ $&89 9 O   ! ! -j  " "$47 7 K   ! ! )%2_D "$7   &'+D $ +-> ? ?':D $ +T 2 2'8'B'BCV'WD $[\ \ ! ()-D & -/B C C)>D & -t 4 4)<)F)FG\)]D &_` `'$).B''>$ ??4<<  $ $ &$0?r?cF[U[5(a/nUHnUbUS:XaURS5 M[U[5(a&UR[R U55 MZ[U[ 5(a&UR[RU55 MURU5 M U$[U[5(a[R U5$[U[ 5(aUS:Xag[RU5$U$)zINormalize proxy_config to ProxyConfig, list of ProxyConfig/None, or None.Nr)rrrrrsrrr)r9 normalizedr_s r<_normalize_proxy_config(CrawlerRunConfig._normalize_proxy_configs eT " "J9X %%d+4((%%k&;&;A&>?3''%%k&=&=a&@A%%a(  t $ $((/ / s # # **51 1 r?cUR$r() _proxy_configrs r<r$CrawlerRunConfig.proxy_configs!!!r?c8[RU5Ulgr()r{rr!)r&r9s r<r$r"s-EEeLr?rcURcS/$[UR[5(a UR(a UR$S/$UR/$)z4Normalize proxy_config to a list for the retry loop.N)r$rrrs r<_get_proxy_list CrawlerRunConfig._get_proxy_listsR    $6M d'' . .(,(9(94$$ Ev E!!""r?c SSKJn [ UR [ 5(aUR /nO UR n/n[U5HupEU"U5nUR(aURUR5 M;URnSUS-SURSURSURSUR3 nUR (aUS UR SR3- n[#U5e X0l g ![a SSKJn GN f=f![a [#S 5e[$a:n S [ U 5R'5;a[#S [ U 535eeS n A ff=f)z Compile C4A script to JavaScriptr)compilerz%C4A Script compilation error (script z ): Line z , Column rAz Code: z Suggestion: z^C4A script compiler not available. Please ensure crawl4ai.script module is properly installed.zcompilation errorzFailed to compile C4A script: N)scriptr(rcrawl4ai.scriptrrr enumeratesuccessextendr first_errorlinecolumnmessage source_line suggestionsrBrlower) r&r(scripts compiled_jsr7r)rZerror error_msgrs r<r$CrawlerRunConfig._compile_c4a_scriptsz,  4+ $//3//??+//K&w/  >>&&v~~6#..E?!uE""'**Yu||nBu}}oV##(#4#4"57 ((!'78I8I!8L8T8T7U%VV $Y//!0$'L; 433 4> N  "#a&,,.8 #A#a&!JKK   s5DD D-D*&D-)D**D--F 5FFcURcg[UR5(aURU5$[UR[5(aSSKJn U"XR5$[UR[ 5(aUR(dg/nURH`n[U5(aUR U"U55 M,[U[5(aSSKJn UR U"X55 M`Mb UR[R:XaU(a [U5$S$U(a [U5$S$g)zCheck if this config matches the given URL. Args: url: The URL to check against this config's matcher Returns: bool: True if this config should be used for the URL or if no matcher is set. Tr)fnmatchF) rcallablerrr;rrrrbrjanyall)r&r}r;resultsmatchers r<is_matchCrawlerRunConfig.is_matchs    # D$$ % %##C( ( ((# . . '3 0 01 1 (($ / /##G++G$$NN73<0--/NN73#89,),,.'.s7|9E9'.s7|9E9r?cXR;a[SUSURU35e[SURRSUS35e)zHandle attribute access.z Getting '' is deprecated. 'z' has no attribute ')_UNWANTED_PROPSrrrC)r&rs r< __getattr__CrawlerRunConfig.__getattr__(s[ '' ' 9TF2CDDXDXY]D^C_!`a aq!8!8 99MdVSTUVVr?c>[R"UR5nURnXR;a/X$UR La[ SUSURU35e[TU]!X5 g)zHandle attribute setting.z Setting 'rDN) rVrWrUrXrFrrsuper __setattr__)r&rr9r^ all_paramsrs r<rKCrawlerRunConfig.__setattr__.sq .^^ '' 'ED9I9Q9Q,Q 9TF2CDDXDXY]D^C_!`a a D(r?r4c UR5VVs0sH.upU[U[5(aSU;a [U5OU_M0 nnn[R "[ R5RR5S1- n[ S0UR5VVs0sHupX;dM X_M snnD6$s snnfs snnfrS) r0rrrrVrWr{rUrXr,rTs r<rVCrawlerRunConfig.from_kwargs9s  & Jq$,?,?FaK%a(UV V&  !!"2";";<GGLLNRXQYYRFLLN"QNDAaj414N"QRR #RrXc[U5$r(r_rs r<r`CrawlerRunConfig.dumpHrbr?rcp[U5n[U[5(aU$[RU5$r()rrr{rVrds r<rfCrawlerRunConfig.loadLs2(- f. / /M++F33r?c  0SUR_SUR_SUR_SUR_SUR_SUR _SUR _SUR_S UR_S UR_S UR_S UR_S UR_SUR_SUR_S[UR ["5(a>UR Vs/sH&n[%US5(aUR'5OUPM( snO@[%UR S5(aUR R'5O UR _SUR(_0SUR*_SUR,_SUR._SUR0_SUR2_SUR4_SUR6_SUR8_SUR:_SUR<_SUR>_SUR@_SURB_S URD_S!URF_S"URH_S#URJ_E0S$URL_S%URN_S&URP_S'URR_S(URT_S)URV_S*URX_S+URZ_S,UR\_S-UR^_S.UR`_S/URb_S0URd_S1URf_S2URh_S3URj_S4URl_E0S5URn_S6URp_S7URr_S8URt_S9URv_S:URx_S;URz_SUR_S?UR_S@UR_SAUR_SBUR_SCUR_SDUR_SEUR_E0SFUR_SGUR_SHUR_SIUR_SJUR_SKUR_SLUR_SMUR_SNUR_SOUR_SPUR_SQUR_SRUR_SSUR_STUR_SUUR_SVUR_EURUR(aURR'5OSURURURURURSW.E$s snf)XNrrrrrrrrrrrrrrrr$rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr/r r r rr r rrr2r3r4)rrr}rrrr)`rrrrrrrrrrrrrrrrr$rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr/r r r rr r rrr2r3r4rrr}rrrr)r&r_s r<rCrawlerRunConfig.to_dictTsra "D$=$=a !4#;#;a  !7!7a  !$"9"9 a  a D-- a  t33a  T//a  !7!7a  #D$=$=a  $//a  D--a  a  4++a  !7!7a d//66GKFWFWXFW9 5 51 D--?a @ T//Aa B T//Ca D d11Ea F 4++Ga H $//Ia J D--Ka L  Ma N  5 5Oa P t33Qa R '(E(ESa T $//Ua V Wa X t33Ya Z t||[a \ "4#;#;]a ^ t||_a ` %d&A&Aaa b d11ca d D--ea f  5 5ga h t33ia j !$"9"9ka l &t'C'Cma n $T%?%?oa p T//qa r !$"9"9sa t TZZua v )$*I*Iwa x $//ya z "4#;#;{a | *4+K+K}a ~ 488a @ T//Aa B 3D4]4]Ca D $T%?%?Ea F $T%?%?Ga H  5 5Ia J !$"9"9Ka L &t'C'CMa N +D,M,MOa P %d&A&AQa R )$*I*ISa T t33Ua V %d&A&AWa X 4++Ya Z 01W1W[a \ t||]a ^ 4++_a ` '(E(Eaa b '(E(Eca d dkkea f dkkga h  ia j !$"9"9ka l  5 5ma n $//oa p t33qa r *4+K+Ksa t$(#;#;IMIaIa4#;#;#C#C#Egk88++// --++Aa a "Ys3-Uc nUR5nURU5 [RU5$)aCreate a copy of this configuration with updated values. Args: **kwargs: Key-value pairs of configuration options to update Returns: CrawlerRunConfig: A new instance with the specified updates Example: ```python # Create a new config with streaming enabled stream_config = config.clone(stream=True) # Create a new config with multiple updates new_config = config.clone( stream=True, cache_mode=CacheMode.BYPASS, verbose=True ) ``` )rrr{rVrs r<rCrawlerRunConfig.clones.,lln 6"++K88r?)dr!rrrrrrr rr rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr rrrrrrrrrrrrrrrr rrrrr$rrrrrrrrrrrrrrrrrrr rrrrr}rr2r4r3r/rrrrrr);rCrgrhrirrFrrrrBYPASSr r rr rbrjrrrrrrr!rrr#rr"rrrrrrr rr UrlMatcherrrrUrrpropertyr$setterr%rrArGrKrVr`rfrrrl __classcell__)rs@r<r{r{lsvpGCHH O%726.;o9Q9S %)"!%%*"!59SWCG*.+/+0 $%)37&+ ) 0 0"##$ &+*.,( $ %*- )-59,0'+$!*. %#((-&+##(+0 %)+E*/#4X%:%&48(-#(-1',+0 $',!27!).).#(!&#,.;?HLLP,0 ) '+MQos'"s'0 s' , s' 7 s's's'cs's's'#s's's's' !s'"3#s'$M4 +>c4OP%s'&"**?!@'s'*#3-+s',$C=-s'.%)/s'2 3s'4c]5s'6/07s': $;s'>?s'@As'BCs'DEs'FGs'HIs'JKs'N $Os'P#(Qs'TUs'VWs'XYs'Z[s'\]s'^#(_s'`as'bcs'des'hsDI~&is'j#3S >2ks'l#tCy.)ms'nos'p!%qs'rss'tus'v#3-ws'xys'z!{s'|"&}s'~ $s'@As'B!Cs'DEs'F%)Gs'JKs'L#Ms'N&)Os'P$(Qs'RSs'TUs'V/2Ws'X #Ys'Z #[s'\2]s'^"&_s'`!as'd'+es'f!%gs'h%)is'jks'l!%ms'nos'p,0qs'tus'vws'z#'{s'|#'}s'@As'BCs'DEs'F!Gs'HIs'JKs'LMs'NOs'PQs'R&*Ss'V&&78Ws'Z##4d38n#DE[s'^ %%8$sCx.%HI_s'bj)cs'des'h38nis'lms'n"*(C5)C.3H*I!Jos'l ,""MM##.`+C+D+\W ) SD S%7 S S*d*444.44b H99r?r{c\rSrSr\SSSSSSSSSSSS4 S\S\\S\\S\\S\\S\\S \\S \\S \\ \S \\S \\S\\S\\4Sjjr \ S\ SS4Sj5r SrSrSrg)rmiNprovider api_tokenr temperature max_tokenstop_pfrequency_penaltypresence_penaltystoprRbackoff_base_delaybackoff_max_attemptsbackoff_exponential_factorc^TUlU(aURS5(dX lOU(a5URS5(a[R"USS5UlO[ R "5n[U4SjU55(a2[U4SjU5S5n[ R"U5UlO*[Ul[R"[5UlX0l X@l XPlX`lXplXlXlXlU bU OSUlU bU OSUlU bXlgSUlg)z4Configuaration class for LLM provider and API token.zenv:rNc3F># UHnTRU5v M g7fr( startswith.0prefixr^s r< %LLMConfig.__init__..sFX68&&v..Xs!c3X># UHnTRU5(dMUv M! g7fr(rkrms r<rprqs R(h6I6I&6QVV(s* *r)r^rlr_rrr r,r=nextrrrrr`rarbrcrdrerRrfrgrh)r&r^r_rr`rarbrcrdrerRrfrgrhprefixesselected_prefixs ` r<rULLMConfig.__init__s"! Y11&99&N 9//77YYy}5DN0446HFXFFF"&R(R#":!=!=o!N 0 !#+C!D &$ !2 0 8J8V"4\]?!',B!C'-zz2N'O  r?c&URURURURURUR UR URURURURURURS. $)Nryryrs r<rLLMConfig.to_dictss  ++//ZZ!%!7!7 $ 5 5II"&"9"9$($=$=*.*I*I  r?c nUR5nURU5 [RU5$)zCreate a copy of this configuration with updated values. Args: **kwargs: Key-value pairs of configuration options to update Returns: llm_config: A new instance with the specified updates )rrrmrVrs r<rLLMConfig.clone%s.lln 6"$$[11r?) r_rfrhrgrrcrarRrdr^rer`rb)rCrgrhrirrr"rrr!rUrrrVrrrlrfr?r<rmrms )#'"&'+$(!%-1,0$(,0.248-v-vC=-v3- -v e_ -v SM -v-v$E?-v#5/-vtCy!-v C=-v%SM-v'sm-v%-SM-v^ D [  " " 2r?rmc#\rSrSrSrSS\S\\S\S\S\S \S \S \S \\S \\ S\\S\\S\\ S\S\S\S\4"Sjjr S\ \\ 44Sjr\S\ \\ 4SS4Sj5rS\ SS4SjrSrg)rvi2zN Configuration class for URL discovery and pre-validation via AsyncUrlSeeder. Nsourcepattern live_check extract_headmax_urlsr hits_per_secforcebase_directory llm_configr/rrscoring_methodfilter_nonsense_urlscache_ttl_hoursvalidate_sitemap_lastmodcXlX lX0lX@lXPlX`lXplXlXlXl Xl Xl Xl Xl XlUUlUUlg)a Initialize URL seeding configuration. Args: source: Discovery source(s) to use. Options: "sitemap", "cc" (Common Crawl), or "sitemap+cc" (both). Default: "sitemap+cc" pattern: URL pattern to filter discovered URLs (e.g., "*example.com/blog/*"). Supports glob-style wildcards. Default: "*" (all URLs) live_check: Whether to perform HEAD requests to verify URL liveness. Default: False extract_head: Whether to fetch and parse section for metadata extraction. Required for BM25 relevance scoring. Default: False max_urls: Maximum number of URLs to discover. Use -1 for no limit. Default: -1 concurrency: Maximum concurrent requests for live checks/head extraction. Default: 1000 hits_per_sec: Rate limit in requests per second to avoid overwhelming servers. Default: 5 force: If True, bypasses the AsyncUrlSeeder's internal .jsonl cache and re-fetches URLs. Default: False base_directory: Base directory for UrlSeeder's cache files (.jsonl). If None, uses default ~/.crawl4ai/. Default: None llm_config: LLM configuration for future features (e.g., semantic scoring). Currently unused. Default: None verbose: Override crawler's general verbose setting for seeding operations. Default: None (inherits from crawler) query: Search query for BM25 relevance scoring (e.g., "python tutorials"). Requires extract_head=True. Default: None score_threshold: Minimum relevance score (0.0-1.0) to include URL. Only applies when query is provided. Default: None scoring_method: Scoring algorithm to use. Currently only "bm25" is supported. Future: "semantic". Default: "bm25" filter_nonsense_urls: Filter out utility URLs like robots.txt, sitemap.xml, ads.txt, favicon.ico, etc. Default: True cache_ttl_hours: Hours before sitemap cache expires. Set to 0 to disable TTL (only lastmod validation). Default: 24 validate_sitemap_lastmod: If True, compares sitemap's with cache timestamp and refetches if sitemap is newer. Default: True N)rrrrrrrrrrr/rrrrrr)r&rrrrrrrrrrr/rrrrrrs r<rUSeedingConfig.__init__6sgv  $( &( ,$  .,$8!.(@%r?rcURR5VVs0sHupUS:wdUcMX_M snn$s snnf)Nr)__dict__r0)r&rFrGs r<rSeedingConfig.to_dicts8!%!4!4!6]!6!|:Kq!6]]]s;;r4c[S0UD6$)Nrf)rvrs r<rVSeedingConfig.from_kwargss&v&&r?c nUR5nURU5 [RU5$r()rrrvrVrs r<rSeedingConfig.clones,lln 6"((55r?)rrrrrrrrrrrrrrrrr/)z sitemap+cc*FFirFNNNNNbm25TT)rCrgrhrirrr"rrrmrrUr rrrrVrrlrfr?r<rvrv2sz #!$ "(,*."&#+/$%)!)-%KAKA#KA KA  KA  KAKAKAKA! KAY'KA$KA}KA"%KAKA #!KA"#KA$#'%KA\^c3h^'DcN'''6c6o6r?rv)F)Fr1rZrrrIryrerrrrrr r r r r user_agent_generatorrrrrrrrrmarkdown_generation_strategyrrcontent_scraping_strategyrr deep_crawlingrrrr cache_contextrproxy_strategyrrVtypingrrrr r!r"r#enumr$rrrYr`rbrrrrrrsrurrrr{rmrvrfr?r<rsw     :J>^W,M$1HHH3#-tE#xt ?T:T4U/VV W JZDHcH$HVBBBJ#$.8.8`}2}2~qqqf ++Z\8\8~D5D5Lc 9c 9c 9J^2^2@\6\6r?