7|h{/BddlmZddlZddlZddlmZmZddlmZddl m Z ddl m Z m Z mZmZmZmZmZmZmZmZmZmZddlmZmZej6eZedd ZGd d eeZGd d eZ Gdde!e Z"edGddZ#ddZ$y)) annotationsN)ABCabstractmethod) dataclass)Enum) AbstractSetAnyCallable CollectionIterableListLiteralOptionalSequenceTypeTypeVarUnion)BaseDocumentTransformerDocumentTS TextSplitter)boundceZdZdZddedddf ddZeddZ d dd Zdd Z dd Z dd Z e dd Z e ddedf ddZ ddZy)rz)Interface for splitting text into chunks.iFTc||kDrtd|d|d||_||_||_||_||_||_y)adCreate a new TextSplitter. Args: chunk_size: Maximum size of chunks to return chunk_overlap: Overlap in characters between chunks length_function: Function that measures the length of given chunks keep_separator: Whether to keep the separator and where to place it in each corresponding chunk (True='start') add_start_index: If `True`, includes chunk's start index in metadata strip_whitespace: If `True`, strips whitespace from the start and end of every document zGot a larger chunk overlap (z) than chunk size (z), should be smaller.N) ValueError _chunk_size_chunk_overlap_length_function_keep_separator_add_start_index_strip_whitespace)self chunk_size chunk_overlaplength_functionkeep_separatoradd_start_indexstrip_whitespaces \/var/www/html/test/engine/venv/lib/python3.12/site-packages/langchain_text_splitters/base.py__init__zTextSplitter.__init__!sd* : %.}o><46 &+ /- /!1cy)z$Split text into multiple components.N)r#texts r* split_textzTextSplitter.split_textBsr,Nc |xsigt|z}g}t|D]\}}d}d}|j|D]} tj||} |j r>||z|j z } |j| td| }|| d<t| }t| | } |j| |S)z&Create documents from a list of texts.r start_index) page_contentmetadata) len enumerater0copydeepcopyr!rfindmaxrappend) r#texts metadatas _metadatas documentsir/indexprevious_chunk_lenchunkr4offsetnew_docs r*create_documentszTextSplitter.create_documentsFs32$U"3   ' *GAtE!" . *==A7(("%77$:M:MMF IIeSF^>B>N>N=OQ{#a'//+yAC C( $"5"55 [9IA9M STU**+!AI!6!6{1~!F-0-=-AMq"'2!"o  $"5"55 [9IA9M STU**+!AI   q ! Tc+.>.B]J JE9 K:ook95 ? KK  r,c  ddlm}t|s tddfd }|dd|i|S#t$r tdwxYw) z>Text splitter that uses HuggingFace tokenizer to count length.r)PreTrainedTokenizerBasezATokenizer received was not an instance of PreTrainedTokenizerBasec8tj|SNr5encoder/ tokenizers r*_huggingface_tokenizer_lengthzNTextSplitter.from_huggingface_tokenizer.._huggingface_tokenizer_lengths9++D122r,z`Could not import transformers python package. Please install it with `pip install transformers`.r&r/strreturnintr.) transformersr] isinstancer ImportError)clsrckwargsr]rds ` r*from_huggingface_tokenizerz'TextSplitter.from_huggingface_tokenizerse  <i)@A W 3K#@KFKK  E  s #0Agpt2allc  ddl}||j| n|j| d fd }t |t r ||d}i||}|dd|i|S#t$r tdwxYw) z9Text splitter that uses tiktoken encoder to count length.rNzCould not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with `pip install tiktoken`.c>tj|SN)allowed_specialdisallowed_specialr`)r/rtruencs r*_tiktoken_encoderz=TextSplitter.from_tiktoken_encoder.._tiktoken_encoders* $3'9 r,) encoding_name model_namertrur&rer.)tiktokenrkencoding_for_model get_encoding issubclassTokenTextSplitter) rlrxryrtrurmrzrw extra_kwargsrvs `` @r*from_tiktoken_encoderz"TextSplitter.from_tiktoken_encoders    !--j9C'' 6C  c, -!.(#2&8 L 0/,/F?#4???= A  s AA2c 6|jt|S)z2Transform sequence of documents by splitting them.)rIlist)r#r?rms r*transform_documentsz TextSplitter.transform_documentss##DO44r,)r$rhr%rhr&zCallable[[str], int]r'z$Union[bool, Literal['start', 'end']]r(boolr)rrgNoner/rfrg List[str]r_)r<rr=zOptional[List[dict]]rgList[Document])r?zIterable[Document]rgr)rNrrOrfrg Optional[str])rUz Iterable[str]rOrfrgr)rcr rmr rgr)rlzType[TS]rxrfryrrt'Union[Literal['all'], AbstractSet[str]]ru&Union[Literal['all'], Collection[str]]rmr rgr)r?Sequence[Document]rmr rgr)__name__ __module__ __qualname____doc__r5r+rr0rFrIrPr[ classmethodrnsetrrr.r,r*rrsQ3 03?D %!%222. 2 = 2  22 2B33CG+? (A(TLL($$(CF5EJ )@ )@)@")@A )@ C )@  )@ )@)@V5+57:5 5r,cVeZdZdZddedf dfd ZddZxZS) r~z/Splitting text to tokens using model tokenizer.roNrpc t|di| ddl}||j |}n|j |}||_||_||_y#t$r tdwxYw)zCreate a new TextSplitter.rNzCould not import tiktoken python package. This is needed in order to for TokenTextSplitter. Please install it with `pip install tiktoken`.r.) superr+rzrkr{r| _tokenizer_allowed_special_disallowed_special) r#rxryrtrurmrzrv __class__s r*r+zTokenTextSplitter.__init__s "6"    !--j9C'' 6C /#5  A  s AA&cdfd }tjjjj|}t ||S)aSplits the input text into smaller chunks based on tokenization. This method uses a custom tokenizer configuration to encode the input text into tokens, processes the tokens in chunks of a specified size with overlap, and decodes them back into text chunks. The splitting is performed using the `split_text_on_tokens` function. Args: text (str): The input text to be split into smaller chunks. Returns: List[str]: A list of text chunks, where each chunk is derived from a portion of the input text based on the tokenization and chunking rules. chjj|jjSrs)rrarr)_textr#s r*_encodez-TokenTextSplitter.split_text.._encode s4??)) $ 5 5#'#;#;* r,)r%tokens_per_chunkdecoderarb)rrfrgz List[int]) Tokenizerrrrrsplit_text_on_tokens)r#r/rrcs` r*r0zTokenTextSplitter.split_textsE  --!--??))  $CCr,) rxrfryrrtrrurrmr rgrr)rrrrrr+r0 __classcell__)rs@r*r~r~s]9$$(CF5EJ 66"6A 6 C 6  6 66Dr,r~cxeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZy)Languagez"Enum of the programming languages.cppgojavakotlinjstsphpprotopythonrstrubyrustscalaswiftmarkdownlatexhtmlsolcsharpcobolcluaperlhaskellelixir powershellN)rrrrCPPGOJAVAKOTLINJSrPHPPROTOPYTHONRSTRUBYRUSTSCALASWIFTMARKDOWNLATEXHTMLSOLCSHARPCOBOLCLUAPERLHASKELLELIXIR POWERSHELLr.r,r*rrs, C B D F B B C E F C D D E EH E D C F E A C DG FJr,rT)frozenc@eZdZUdZded< ded< ded< ded<y ) rzTokenizer data class.rhr%rzCallable[[List[int]], str]rzCallable[[str], List[int]]raN)rrrr__annotations__r.r,r*rr;s)*, &&= &&=r,rcg}|j|}d}t||jzt|}|||}|t|kr|j |j ||t|k(r |S||j|j z z }t||jzt|}|||}|t|kr|S)z6Split incoming text and return chunks using tokenizer.r)raminrr5r;rr%)r/rcrU input_ids start_idxcur_idx chunk_idss r*rrIsF  &II)i888#i.IG)G,I c)n $ i&&y12 c)n $  M Y//)2I2III i)"<"<rs" #!    G   8 $ T(|5*C|5~<D <D~sD> $ > > >r,