jSrSSKrSSKrSSKrSSKrSSKJr SSKJr SSK J r SSK J r J r JrJrJr S\S\4SjrS \S\4S jrS\4S jrS rS r\S:Xa\"5 gg)a training_doc.py =============== Training module for DOC/DOCX document formats. Handles text extraction from Word documents (.doc, .docx), semantic chunking, and FAISS embedding creation. - .docx: Extracted directly using python-docx - .doc: Converted to .docx via LibreOffice, then extracted Shared utilities (semantic chunking, embedding creation, FAISS merging) are imported from training.py to maintain DRY principles. N)Document)OllamaEmbeddings) split_text_with_semantic_chunkercount_tokens_in_documentssave_documents_to_txtcreate_and_save_embeddingsmerge_all_faiss file_pathreturnc[U5n/nURH7nURR5nU(dM&UR U5 M9 UR H]nUR HJnURH7nURR5nU(dM&UR U5 M9 ML M_ SRU5$)z Extract all text content from a .docx file using python-docx. Iterates through all paragraphs and table cells to capture the complete document content.  ) DocxDocument paragraphstextstripappendtablesrowscellsjoin) r doc text_parts paragraphstrippedtablerowcell cell_texts G/var/www/eduai.edurigo.com/question_generate/production/training_doc.py_extract_text_from_docxr 's y !CJ^^ >>'') 8   h '$ ::C  IIOO- 9%%i0" 99Z  doc_file_pathc X[RRU5=(d Sn[R"SSSSSUU/SSSS 9nUR S :wa%[ S UR S UR35e[RR[RRU55S n[RRXS35n[RRU5(d[ SU35eU$![a [ S 5ef=f)z Convert a legacy .doc file to .docx using LibreOffice headless mode. Returns the path to the newly created .docx file. Raises RuntimeError if LibreOffice conversion fails. . libreofficez --headlessz --convert-todocxz--outdirTx)capture_outputrtimeoutrz)LibreOffice conversion failed (exit code z): zYLibreOffice is not installed or not in PATH. It is required to convert legacy .doc files..docxz:LibreOffice conversion produced no output file. Expected: ) ospathdirname subprocessrun returncode RuntimeErrorstderrFileNotFoundErrorsplitextbasenamerexists)r" output_dirresult base_name docx_paths r_convert_doc_to_docxr;Bs./63J J         !;F>) $ $"  %  !   ;   s ADD)cr[RRU5SR5nSnUS:Xa [ U5nO,US:Xa[ U5n[ U5nO[ SUS35eUR5(da[SU35 /U(aJ[RRU5(a%[R"U5 [SU35 $$$[UXS .S 9/n[S US [U5S 35 UU(aJ[RRU5(a%[R"U5 [SU35 $$$!U(aJ[RRU5(a%[R"U5 [SU35 fff=f)a, Load a DOC or DOCX file and return a list of LangChain Document objects. For .docx files, text is extracted directly using python-docx. For .doc files, the document is first converted to .docx via LibreOffice. Returns: list[Document]: List of LangChain Document objects with page_content set. Nr*z.doczUnsupported file type: z$. Only .doc and .docx are supported.z(Warning: No text content extracted from z%Cleaned up temporary converted file: )sourceformat) page_contentmetadatazSuccessfully loaded z document: z characters extracted.) r+r,r4lowerr r; ValueErrorrprintr6removerlen)r file_extensionconverted_docx_pathr documentss rload_docrJvsWW%%i0399;N Q W $*95D v %"6y"A *+>?D).)9:56  zz|| >2E#F#F II) * 9:M9NO P$G  !$-H   $^$4KD {J`ab 277>>2E#F#F II) * 9:M9NO P$G  277>>2E#F#F II) * 9:M9NO P$G sA"E#()E##AF6clSnUH+nURR5nU[U5- nM- U$)zx Count total tokens across all Document objects. Uses whitespace tokenization for consistency with the existing system. r)r@splitrF)rI total_tokensrtokenss rcount_tokens_in_doc_documentsrOs= L!!'')F #  r!cSnSnSnSUSU3n[R"5n[SU35 [U5nU(d [S5 g[U5n[S U35 [ S S 9n[ XW5n[ X5 [XU5 [X5 [R"5n [S X- S S35 [SU35 g)z/Test the DOC/DOCX training pipeline standalone.ztest_document.docx test_clienttest_refztemp/_z Start Time: zNo documents loaded. Exiting.NzTotal tokens: znomic-embed-text)modelzTraining process took z.2fz secondszTotal tokens processed: ) timerDrJrOrrrrr ) r client_id reference_idr7 start_timedocs token_count embeddingssplit_documentsend_times rmainr^s$IIL 1\N3JJ L %& I D  -./5K N;- ()"(:;J6tHO/6<HI,yy{H "8#8"=X FG $[M 23r!__main__)__doc__r+r.tempfilerUr&rrlangchain_core.documentslangchain_ollamartrainingrrrrr strr r;rJrOr^__name__r!rrhs   )--!s!s!6---h-Q-Qh  "4J zFr!