|h;FddlmZddlmZddlmZmZddlZddlm Z ddl m Z ddl m Z ddlmZ ddlZGd d e j&ZGd d eZGd deZGddeZddedej2defdZy#e$re j$dddlZYiwxYw))abstractmethod)Path)ListUnionN)Image)checks)smart_inference_modez+git+https://github.com/ultralytics/CLIP.gitcBeZdZdZfdZedZedZxZS) TextModela Abstract base class for text encoding models. This class defines the interface for text encoding models used in vision-language tasks. Subclasses must implement the tokenize and encode_text methods to provide text tokenization and encoding functionality. Methods: tokenize: Convert input texts to tokens for model processing. encode_text: Encode tokenized texts into normalized feature vectors. c"t|y)z$Initialize the TextModel base class.N)super__init__)self __class__s X/var/www/html/test/engine/venv/lib/python3.12/site-packages/ultralytics/nn/text_model.pyrzTextModel.__init__!s cy)z3Convert input texts to tokens for model processing.Nrtextss rtokenizezTextModel.tokenize% rcy)z7Encode tokenized texts into normalized feature vectors.Nrrrdtypes r encode_textzTextModel.encode_text*rr) __name__ __module__ __qualname____doc__rrrr __classcell__rs@rr r s5     rr ceZdZdZdedej ddffd Zdeee efdejfdZ e ejfdejd ejdejfd Ze ejfd eej ejfd ejdejfd ZxZS) CLIPa Implements OpenAI's CLIP (Contrastive Language-Image Pre-training) text encoder. This class provides a text encoder based on OpenAI's CLIP model, which can convert text into feature vectors that are aligned with corresponding image features in a shared embedding space. Attributes: model (clip.model.CLIP): The loaded CLIP model. device (torch.device): Device where the model is loaded. Methods: tokenize: Convert input texts to CLIP tokens. encode_text: Encode tokenized texts into normalized feature vectors. Examples: >>> import torch >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu") >>> clip_model = CLIP(size="ViT-B/32", device=device) >>> tokens = clip_model.tokenize(["a photo of a cat", "a photo of a dog"]) >>> text_features = clip_model.encode_text(tokens) >>> print(text_features.shape) sizedevicereturnNct|tj||\|_|_|j |||_|jy)ao Initialize the CLIP text encoder. This class implements the TextModel interface using OpenAI's CLIP model for text encoding. It loads a pre-trained CLIP model of the specified size and prepares it for text encoding tasks. Args: size (str): Model size identifier (e.g., 'ViT-B/32'). device (torch.device): Device to load the model on. Examples: >>> import torch >>> clip_model = CLIP("ViT-B/32", device=torch.device("cuda:0")) >>> text_features = clip_model.encode_text(["a photo of a cat", "a photo of a dog"]) )r&N) r rcliploadmodelimage_preprocesstor&eval)rr%r&rs rrz CLIP.__init__HsE ,0IId6,J) D)   rrc^tj|j|jS)a Convert input texts to CLIP tokens. Args: texts (str | List[str]): Input text or list of texts to tokenize. Returns: (torch.Tensor): Tokenized text tensor with shape (batch_size, context_length) ready for model processing. Examples: >>> model = CLIP("ViT-B/32", device="cpu") >>> tokens = model.tokenize("a photo of a cat") >>> print(tokens.shape) # torch.Size([1, 77]) )r)rr-r&rs rrz CLIP.tokenize^s"}}U#&&t{{33rrc|jj|j|}||jdddz }|S)a Encode tokenized texts into normalized feature vectors. This method processes tokenized text inputs through the CLIP model to generate feature vectors, which are then normalized to unit length. These normalized vectors can be used for text-image similarity comparisons. Args: texts (torch.Tensor): Tokenized text inputs, typically created using the tokenize() method. dtype (torch.dtype, optional): Data type for output features. Returns: (torch.Tensor): Normalized text feature vectors with unit length (L2 norm = 1). Examples: >>> clip_model = CLIP("ViT-B/32", device="cuda") >>> tokens = clip_model.tokenize(["a photo of a cat", "a photo of a dog"]) >>> features = clip_model.encode_text(tokens) >>> features.shape torch.Size([2, 512]) Tpdimkeepdimr+rr-norm)rrr txt_featss rrzCLIP.encode_textosB,JJ**5144U;  D II rimagec.t|tjr9|j|jdj |j }|j j|j |}||jdddz }|S)a Encode preprocessed images into normalized feature vectors. This method processes preprocessed image inputs through the CLIP model to generate feature vectors, which are then normalized to unit length. These normalized vectors can be used for text-image similarity comparisons. Args: image (PIL.Image | torch.Tensor): Preprocessed image input. If a PIL Image is provided, it will be converted to a tensor using the model's image preprocessing function. dtype (torch.dtype, optional): Data type for output features. Returns: (torch.Tensor): Normalized image feature vectors with unit length (L2 norm = 1). Examples: >>> from ultralytics.nn.text_model import CLIP >>> from PIL import Image >>> clip_model = CLIP("ViT-B/32", device="cuda") >>> image = Image.open("path/to/image.jpg") >>> image_tensor = clip_model.image_preprocess(image).unsqueeze(0).to("cuda") >>> features = clip_model.encode_image(image_tensor) >>> features.shape torch.Size([1, 512]) rr1r2Tr3) isinstancerr, unsqueezer-r&r+ encode_imager8)rr:r img_featss rr>zCLIP.encode_imagesz4 eU[[ )))%0::1=@@MEJJ++E255e<  D II r)rrrr strtorchr&rrrTensorrr float32rrrr>r!r"s@rr$r$0s.S%,,4,4eCcN34 4"DIMMekkV[VbVb2Y^YfYf% U\\(A"B5;;kpkwkwrr$ceZdZdZddddddZdedejd d ffd Zd e ed ejfd Z e ejfd ejdejd ejfdZxZS) MobileCLIPaD Implement Apple's MobileCLIP text encoder for efficient text encoding. This class implements the TextModel interface using Apple's MobileCLIP model, providing efficient text encoding capabilities for vision-language tasks with reduced computational requirements compared to standard CLIP models. Attributes: model (mobileclip.model.MobileCLIP): The loaded MobileCLIP model. tokenizer (callable): Tokenizer function for processing text inputs. device (torch.device): Device where the model is loaded. config_size_map (dict): Mapping from size identifiers to model configuration names. Methods: tokenize: Convert input texts to MobileCLIP tokens. encode_text: Encode tokenized texts into normalized feature vectors. Examples: >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu") >>> text_encoder = MobileCLIP(size="s0", device=device) >>> tokens = text_encoder.tokenize(["a photo of a cat", "a photo of a dog"]) >>> features = text_encoder.encode_text(tokens) s0s1s2b)rFrGrHrIbltr%r&r'Nc: ddl}|j5|jdtddl}dddt|%|j|}d|d}t|jsddl m}|d |jd||| d|_|j#d||_|j'|||_|j+y#1swYxYw#t $rt jdddl}YwxYw) aK Initialize the MobileCLIP text encoder. This class implements the TextModel interface using Apple's MobileCLIP model for efficient text encoding. Args: size (str): Model size identifier (e.g., 's0', 's1', 's2', 'b', 'blt'). device (torch.device): Device to load the model on. Examples: >>> import torch >>> model = MobileCLIP("s0", device=torch.device("cpu")) >>> tokens = model.tokenize(["a photo of a cat", "a photo of a dog"]) >>> features = model.encode_text(tokens) rNignore)categoryz1git+https://github.com/ultralytics/mobileclip.git mobileclip_z.pt)downloadzHhttps://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/) pretrainedr&)warningscatch_warningsfilterwarnings FutureWarning mobileclip ImportErrorrcheck_requirementsr rconfig_size_mapris_file ultralyticsrOcreate_model_and_transformsr+ get_tokenizer tokenizerr-r&r.) rr%r&rQrUconfigfilerOrs rrzMobileCLIP.__init__s  ((* "''='I! " %%d+TF#&Dz!!# , _`d_ef g;;k&>> model = MobileCLIP("s0", "cpu") >>> tokens = model.tokenize(["a photo of a cat", "a photo of a dog"]) r]r-r&rs rrzMobileCLIP.tokenize"~~e$'' 44rrc|jj|j|}||jdddz}|S)a Encode tokenized texts into normalized feature vectors. Args: texts (torch.Tensor): Tokenized text inputs. dtype (torch.dtype, optional): Data type for output features. Returns: (torch.Tensor): Normalized text feature vectors with L2 normalization applied. Examples: >>> model = MobileCLIP("s0", device="cpu") >>> tokens = model.tokenize(["a photo of a cat", "a photo of a dog"]) >>> features = model.encode_text(tokens) >>> features.shape torch.Size([2, 512]) # Actual dimension depends on model size r1r2Tr3r7)rrr text_featuress rrzMobileCLIP.encode_textsE& ..u588? ++aR+FF r)rrrr rXr@rAr&rrrBrr rCrrr!r"s@rrErEs."TCPO'S'%,,'4'R5d3i5ELL5 DIMMekkV[VbVbrrEceZdZdZdej ffd ZdeedejfdZ e ejfdejdejdejfdZxZS) MobileCLIPTSa Load a TorchScript traced version of MobileCLIP. This class implements the TextModel interface using Apple's MobileCLIP model in TorchScript format, providing efficient text encoding capabilities for vision-language tasks with optimized inference performance. Attributes: encoder (torch.jit.ScriptModule): The loaded TorchScript MobileCLIP text encoder. tokenizer (callable): Tokenizer function for processing text inputs. device (torch.device): Device where the model is loaded. Methods: tokenize: Convert input texts to MobileCLIP tokens. encode_text: Encode tokenized texts into normalized feature vectors. Examples: >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu") >>> text_encoder = MobileCLIPTS(device=device) >>> tokens = text_encoder.tokenize(["a photo of a cat", "a photo of a dog"]) >>> features = text_encoder.encode_text(tokens) r&ct|ddlm}tj j |d||_tjj|_ ||_ y)a% Initialize the MobileCLIP TorchScript text encoder. This class implements the TextModel interface using Apple's MobileCLIP model in TorchScript format for efficient text encoding with optimized inference performance. Args: device (torch.device): Device to load the model on. Examples: >>> model = MobileCLIPTS(device=torch.device("cpu")) >>> tokens = model.tokenize(["a photo of a cat", "a photo of a dog"]) >>> features = model.encode_text(tokens) r)attempt_download_assetzmobileclip_blt.ts) map_locationN) r rultralytics.utils.downloadsrhrAjitr*encoderr)rr]r&)rr&rhrs rrzMobileCLIPTS.__init__,sK Fyy~~&<=P&Q`f~g ++ rrr'cV|j|j|jS)a Convert input texts to MobileCLIP tokens. Args: texts (List[str]): List of text strings to tokenize. Returns: (torch.Tensor): Tokenized text inputs with shape (batch_size, sequence_length). Examples: >>> model = MobileCLIPTS("cpu") >>> tokens = model.tokenize(["a photo of a cat", "a photo of a dog"]) rars rrzMobileCLIPTS.tokenizeBrbrrcB|j|j|S)a Encode tokenized texts into normalized feature vectors. Args: texts (torch.Tensor): Tokenized text inputs. dtype (torch.dtype, optional): Data type for output features. Returns: (torch.Tensor): Normalized text feature vectors with L2 normalization applied. Examples: >>> model = MobileCLIPTS(device="cpu") >>> tokens = model.tokenize(["a photo of a cat", "a photo of a dog"]) >>> features = model.encode_text(tokens) >>> features.shape torch.Size([2, 512]) # Actual dimension depends on model size )rlr-rs rrzMobileCLIPTS.encode_textRs(||E"%%e,,r)rrrr rAr&rrr@rBrr rCrrr!r"s@rrfrfsu,u||,5d3i5ELL5 DIMM--ekk-V[VbVb--rrfvariantr&r'c|jd\}}|dk(r t||S|dk(r t|Std|d)a Build a text encoding model based on the specified variant. Args: variant (str): Model variant in format "base:size" (e.g., "clip:ViT-B/32" or "mobileclip:s0"). device (torch.device, optional): Device to load the model on. Returns: (TextModel): Instantiated text encoding model. Examples: >>> model = build_text_model("clip:ViT-B/32", device=torch.device("cuda")) >>> model = build_text_model("mobileclip:s0", device=torch.device("cpu")) :r)rUzUnrecognized base model: 'z/'. Supported base models: 'clip', 'mobileclip'.)splitr$rf ValueError)ror&baser%s rbuild_text_modelruisSs#JD$ v~D&!!  F##5dV;jkllr)N)abcrpathlibrtypingrrrAtorch.nnnnPILrultralytics.utilsrultralytics.utils.torch_utilsr r)rVrWModuler r$rErfr@r&rurrrrs $>    6w9wthhVQ-9Q-hmcm5<<m9ms FKLsBB B