Mi$NSSKJr SSKJr SSKrSSKrSSKrSSK7 SSKrSSK J r \RR\RR\R"5\RR\555r\"5S5r\"5S5r\"5S 5rS r\"5S 5r\"5S 5r\"5SS \4Sjj5r\"5S5r\"5S5r\"5S5r\"5S5rSSjrSr\ S:Xa\"5 gg)) lru_cache)PathN)*)MODEL_REPO_BRANCHcSSKnURS:Xa%URRU5R$URS:Xagg)Nrcudampsl0)torchtyper get_device_properties total_memory)devicer s o/var/www/eduai.edurigo.com/storigo/production/storigo_env/lib/python3.13/site-packages/crawl4ai/model_loader.pyget_available_memoryr s> {{fzz//7DDD  c[U5nURS:XagURS;aUS:agUS:agUS:ag g g) Ncpu)r r lll@ )rr )ravailable_memorys rcalculate_batch_sizersO+F3 {{e  ' | +  -  ,rc SSKnURR5(aURS5nU$URR R5(aURS5nU$URS5nU$)Nrr r r)r r is_availablerbackendsr )r rs r get_devicer,sr zz  f% M    ( ( * *e$ Me$ Mrc>[5nURU5 X4$)N)rto)modelrs rset_model_devicer#9s \F HHV =rc [RR[R"S[R "55S5n[R "USS9 [R "US3SS9 [R "US3SS9 U$)NCRAWL4_AI_BASE_DIRECTORYz .crawl4aiT)exist_okz/cachez/models)ospathjoingetenvrhomemakedirs) home_folders rget_home_folderr.?sk'',, ,diik:KKKK d+KK;-v&6KK;-w'$7 rcSSKJnJn URSSS9nURSSS9nUR 5 [ U5up4X#4$)Nr) BertTokenizer BertModelzbert-base-uncasedresume_download) transformersr0r1from_pretrainedevalr#)r0r1 tokenizerr"rs rload_bert_base_uncasedr8JsP5--.ASW-XI  % %&94 % PE JJL$U+ME  rreturncSSKJnJn URUSS9nURUSS9nUR 5 [ U5upEX44$)zLoad the Hugging Face model for embedding. Args: model_name (str, optional): The model name to load. Defaults to "BAAI/bge-small-en-v1.5". Returns: tuple: The tokenizer and model. r) AutoTokenizer AutoModelNr2)r4r;r<r5r6r#) model_namer;r<r7r"rs rload_HF_embedding_modelr>UsO6--j$-OI  % %j$ % GE JJL$U+ME  rcSSKJnJn SSKJn UR S5nUR S5nUR 5 [ U5upEU"SXCS9nU$)Nr)r;"AutoModelForSequenceClassification)pipelinez1dstefa/roberta-base_topic_classification_nyt_newsztext-classification)r"r7)r4r;r@rAr5r6r#)r;r@rAr7r"rpipes rload_text_classifierrChsYN%--;I / > >; E JJL$U+ME ) LD Krc^^^^^^ SSKJnJn SSKJm SSKm SnUR USS9mUR USS9mTR5 [T5ummTRRmSUUUUUU 4SjjnUT4$)Nr)r@r;)expitzcardiffnlp/tweet-topic-21-multir2c>T"USSSUS9nUR5VVs0sHupEXERT5_M nnnTR5 T"S0UD6nSSS5 WRR 5R 5R 5nT"U5nXq:S-n/n UH>n [U 5V V s/sHupU S:XdM TU PM n n n U RU 5 M@ U $s snnf!,(df  N=fs sn n f)NptT)return_tensorspadding truncation max_lengthr) itemsr!no_gradlogitsdetachrnumpy enumerateappend)texts thresholdrKtokenskeyvaloutputscores predictions batch_labels predictionivaluelabels class_mappingrrEr"r7r s r _classifier4load_text_multilabel_classifier.._classifiers !  17  0>HCC   ]]__V_F%%'++-335v*a/  %J1::1F1FXQ%ST* a 1F     ' & % _sC- C3: D D3 D)g?r) r4r@r; scipy.specialrEr r5r6r#configid2label) r@r;MODELrbrarrEr"r7r s @@@@@@rload_text_multilabel_classifierrhysN# .E--eT-JI . > > t ? E JJL$U+ME6LL))M8  rcSSKnURRS5 URRS5$![a UR S5 N8f=f)Nrztokenizers/punktpunkt)nltkdatafind LookupErrordownload)rks rload_nltk_punktrpsN )* 99>>, --  gs<AAc `SSKnSn[5n[U5U- nUR5(a[ UR 55(GdSn[ n[U5S- n[S5 UR5(aB[R"U5 UR5(a[R"U5 [R"SS S XT[U5/[R[RS S 9 [U5S - nURS S S9 US - S- n[R "X5 [R"U5 [S5 UR'[U55$![a* [S5 [SU35 [SU35 gf=f![R"an [SU 35 Sn A gSn A f[$an [SU 35 Sn A gSn A ff=f![$an [SU 35 Sn A gSn A ff=f)Nrmodels/reutersz)https://github.com/unclecode/crawl4ai.gitcrawl4aiu7[LOG] ⏬ Downloading Spacy model for the first time...zh[WARNING] Unable to remove existing folders. Please manually delete the following folders and try again:z- gitclonez-bT)stdoutstderrcheckmodels)parentsr&reutersu-[LOG] ✅ Spacy Model downloaded successfullyz0An error occurred while cloning the repository: zAn error occurred: zError loading spacy model: )spacyr.rexistsanyiterdirrprintshutilrmtreePermissionError subprocessrunstrDEVNULLmkdircopytreeCalledProcessError Exceptionload) r|namer- model_folderrepo_urlbranch repo_folder models_folder source_folderes rload_spacy_modelrs D!#K $t+L    ! !c,*>*>*@&A&A>";'*4  GH       k*&&((MM,/  NNv[9IJ!))!))  !-8M   t  <((2Y>M OOM 8 MM+ & A Bzz#l+,,K# ~;-()<.)*  :,,  DQCH I  's+ ,   +A3/0sP AF BF<+H 1F98F9<HG## H0HH H-H((H-cU(a[S5 [5n[RR US5[RR US5/nUH9n[ U5R 5(dM#[R"U5 M; [S5 [S5 [5upE[SU35 [S5 [5 [S5 g ) z*Download all models required for Crawl4AI.z![LOG] Removing existing models...rrryz[LOG] Existing models removed.z$[LOG] Downloading text classifier...z [LOG] Text classifier loaded on z,[LOG] Downloading custom NLTK Punkt model...u-[LOG] ✅ All models downloaded successfully.N) rr.r'r(r)rr}rrrhrp)remove_existingr- model_foldersfolder_rs rdownload_all_modelsrs 12%' GGLL&6 7 GGLLh / $FF|""$$ f%$ ./ 01/1IA ,VH 56 89 9:rc[S5 [S5 [R"SS9nURSSSS9 UR 5n[ UR S 9 g) Nz/[LOG] Welcome to the Crawl4AI Model Downloader!zE[LOG] This script will download all the models required for Crawl4AI.zCrawl4AI Model Downloader) descriptionz--remove-existing store_truez)Remove existing models before downloading)actionhelp)r)rargparseArgumentParser add_argument parse_argsrr)parserargss rmainrs_ ;< QR  $ $1L MF  8    D(<(<=r__main__)zBAAI/bge-small-en-v1.5)F)! functoolsrpathlibrrr'r model_loaderrcrawl4ai.configrr(realpathr)getcwddirname__file__ __location__rrrr#r.r8tupler>rCrhrprrr__name__rLrrrsX -ww RYY["''//(:S TU     &          E $      3 3l . . = =@;: > zFr