Mi SSKrSSKJrJr SSKrSSKJrJr SSKJ r SSK J r S\ S\ 4Sjr S \ 4S jrSS \ S\\ S\ S \ 4S jjrSrg)N)ListOptional) MarianMTModelMarianTokenizer) sent_tokenize)chunk_by_attention_window source_lang target_langcSUSU3$)zfConstructs the name of the MarianMT machine translation model based on the source and target language.zHelsinki-NLP/opus-mt--)r r s y/var/www/eduai.edurigo.com/storigo/production/storigo_env/lib/python3.13/site-packages/unstructured/cleaners/translate.py_get_opus_mt_model_namer s#;-q >> language_codecj[U[5(a[U5S:wa[SUS35eg)NzInvalid language code: z,. Language codes must be two letter strings.) isinstancestrlen ValueError)rs r_validate_language_coders; mS ) )S-?1-D%m_4` a  .ErtextreturncUR5S:XaU$UbUO[R"U5nURS5(aSn[ U5 [ U5 X#:XaU$[ X25n[ SU35 [R"U5n[R"U5n[X[S9n/nUHn UR[XU55 M SR!U5$![a [SUS35ef=f)aTranslates the foreign language text. If the source language is not specified, the function will attempt to detect it using langdetect. Parameters ---------- text: str The text to translate target_lang: str The two letter language code for the target langague. Defaults to "en". source_lang: Optional[str] The two letter language code for the language of the input text. If source_lang is not provided, the function will try to detect it. zhz Using model: z2Transformers could not find the translation model z>. The requested source/target language combo is not supported.)split_function )strip langdetectdetect startswithrrprintrfrom_pretrainedrOSErrorrrrappend_translate_textjoin) rr r _source_lang model_name tokenizermodelchunkstranslated_chunkschunks rtranslate_textr1s zz|r '2'> JDUDUVZD[Lt$$ K(L)" (CJ M* &' #33J? --j92$R_`F#%  i!HI 88% &&  @ MK K   s ,C22D c [R"5 [R"S5 UR"S 0U"U/SSSS9D6nSSS5 WVs/sHoBR USSS9PM snS$!,(df  N2=fs snf) z8Translates text using the specified model and tokenizer.ignoreptT)return_tensorspadding truncationNi)max_new_tokensskip_special_tokensrr )warningscatch_warnings simplefiltergeneratedecode)rr-r, translatedts rr(r(Ks  "h'^^ tTdS  # Xb bWaRS  Qs  MWa b   # " cs1A3B3 B)Nen)r:typingrrr! transformersrrunstructured.nlp.tokenizer unstructured.staging.huggingfacerrrrr1r(r rrrFsb!73F??3?  3 0'0'8C=0'c0']`0'f r