7|h\ddlmZddlZddlmZmZmZmZmZddl m Z m Z Gdde Z d dZ Gdd e Zy) ) annotationsN)AnyListLiteralOptionalUnion)Language TextSplitterc>eZdZdZ d dfd ZddZxZS)CharacterTextSplitterz(Splitting text that looks at characters.c @t|di|||_||_y)Create a new TextSplitter.N)super__init__ _separator_is_separator_regex)self separatoris_separator_regexkwargs __class__s a/var/www/html/test/engine/venv/lib/python3.12/site-packages/langchain_text_splitters/character.pyrzCharacterTextSplitter.__init__ s$ "6"##5 c|jr |jntj|j}t |||j }|j rdn |j}|j ||S)&Split incoming text and return chunks.)rrreescape_split_text_with_regex_keep_separator _merge_splits)rtextrsplitsrs r split_textz CharacterTextSplitter.split_textsc $77DOORYYt=W (i9M9MN//RT__ !!&*55r) F)rstrrboolrrreturnNoner#r'r) List[str])__name__ __module__ __qualname____doc__rr% __classcell__rs@rr r s72CH66;?6SV6 66rr c|r|rtjd|d|}|dk(r3tdt|dz dDcgc]}||||dzzc}n/tdt|dDcgc]}||||dzzc}}t|dzdk(r||ddz }|dk(r ||dgzn|dg|z}n"tj||}n t |}|Dcgc] }|dk7s | c}Scc}wcc}wcc}w) N()endrr)rsplitrangelenlist)r#rkeep_separator_splitsir$ss rr r s# hh9+Q/6G"U*8=QG q@PRS7TU!'!*wq1u~-U)r&  rNr)rr _separatorsr)r separatorsr>rrrs rrz'RecursiveCharacterTextSplitter.__init__As0 AA&A%@)@#5 rcg}|d}g}t|D]S\}}|jr|ntj|}|dk(r|}n$tj||sI|}||dzd}n|jr|ntj|}t |||j } g} |j rdn|}| D]} |j| |jkr| j| 3| r%|j| |} |j| g} |s|j| n|j| |} |j| | r#|j| |} |j| |S)rr9rr7N) enumeraterrrsearchr r!_length_function _chunk_sizeappendr"extend _split_text)rr#rH final_chunksrnew_separatorsr@_srr$ _good_splitsrA merged_text other_infos rrPz*RecursiveCharacterTextSplitter._split_textMsw rN z* EAr#77RYYr]JRx yyT* !+AEG!4 #'":":Y )@T 'j$:N:NO //RY  4A$$Q'$*:*::##A&"&"4"4\:"NK '' 4#%L% ''*!%!1!1!^!DJ '' 3 4 ,,\:FK    ,rc:|j||jS)zSplit the input text into smaller chunks based on predefined separators. Args: text (str): The input text to be split. Returns: List[str]: A list of text chunks obtained after splitting. )rPrG)rr#s rr%z)RecursiveCharacterTextSplitter.split_textusd&6&677rc :|j|}|d|dd|S)aReturn an instance of this class based on a specific language. This method initializes the text splitter with language-specific separators. Args: language (Language): The language to configure the text splitter for. **kwargs (Any): Additional keyword arguments to customize the splitter. Returns: RecursiveCharacterTextSplitter: An instance of the text splitter configured for the specified language. T)rHrr)get_separators_for_language)clslanguagerrHs r from_languagez,RecursiveCharacterTextSplitter.from_languages) 44X> LjTLVLLrc|tjk(s|tjk(rgdS|tjk(rgdS|tjk(rgdS|tj k(rgdS|tj k(rgdS|tjk(rgdS|tjk(rgdS|tjk(rgdS|tjk(rgd S|tjk(rgd S|tjk(rgd S|tjk(rgd S|tjk(rgd S|tjk(rgdS|tj k(rgdS|tj"k(rgdS|tj$k(rgdS|tj&k(rgdS|tj(k(rgdS|tj*k(rgdS|tj,k(rgdS|tj.k(rgdS|tj0k(rgdS|tj2k(rgdS|tj4vrt7d|dt7d|dt9t)a Retrieve a list of separators specific to the given language. Args: language (Language): The language for which to get the separators. Returns: List[str]: A list of separators appropriate for the specified language. ) class z void z int z float z double  if  for  while  switch  case r&rErFr)  func  var  const  type r_r`rbrcr&rErFr)r^ public protected private  static r_r`rarbrcr&rErFr)r^rhrirjz internal z companion z fun  val rer_r`raz when rc else r&rErFr) function rf let rer^r_r`rarbrc default r&rErFr) enum interface z namespace rgr^rnrfrorer_r`rarbrcrpr&rErFr) rnr^r_ foreach ra do rbrcr&rErFr) z message z service rqz option  import z syntax r&rErFr)r^ def z def r&rErFr)z =+ z -+ z \*+ z .. * r&rErFr) rvr^r_ unless rar`rtz begin z rescue r&rErFr)rvz defp z defmodule z defprotocol z defmacro z defmacrop r_rwrarcz cond z with r`rtr&rErFr) z fn rfror_rar`z loop  match rfr&rErFr)r^z object rvrlrer_r`rarxrcr&rErFr)rdr^ struct rqr_r`rartrbrcr&rErFr) z #{1,6} z``` z \*\*\*+ z ---+ z ___+ r&rErFr)z \\chapter{z \\section{z \\subsection{z \\subsubsection{z \\begin{enumerate}z \\begin{itemize}z \\begin{description}z \\begin{list}z \\begin{quote}z \\begin{quotation}z \\begin{verse}z \\begin{verbatim}z \\begin{align}z$$$rFr)z (## H % : ' > %  )) )%% L,, , .44 4y 2JKL LH:&&&*8n%57 r)NTF) rHzOptional[List[str]]r>$Union[bool, Literal['start', 'end']]rr(rrr)r*)r#r'rHr,r)r,r+)r[r rrr)rC)r[r r)r,) r-r.r/r0rrPr% classmethodr\ staticmethodrYr1r2s@rrCrC:s+/?C#( 6' 6= 6! 6  6  6&P 8MM+.M 'MM$||rrC)r#r'rr'r>rr)r,) __future__rrtypingrrrrrlangchain_text_splitters.baser r r r rCrrrrsS" 66@6L6,* **/S**6V \V r