|hddlmZddlZddlmcmZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZdd lmZmZdd lmZmZd ZGd dej2ZGddejj2Zy))ListN)nn) trunc_normal_)MLP)LOGGER)SAM2TwoWayTransformer) MaskDecoderSAM2MaskDecoder)ImageEncoderViT PromptEncoder)get_1d_sine_peselect_closest_cond_framesgc deZdZUdZdZeed< d dedede de ede ed d f fd Z d Z xZ S)SAMModela Segment Anything Model (SAM) for object segmentation tasks. This class combines image encoders, prompt encoders, and mask decoders to predict object masks from images and input prompts. Attributes: mask_threshold (float): Threshold value for mask prediction. image_encoder (ImageEncoderViT): Backbone for encoding images into embeddings. prompt_encoder (PromptEncoder): Encoder for various types of input prompts. mask_decoder (MaskDecoder): Predicts object masks from image and prompt embeddings. pixel_mean (torch.Tensor): Mean values for normalizing pixels in the input image. pixel_std (torch.Tensor): Standard deviation values for normalizing pixels in the input image. Methods: set_imgsz: Set image size to make model compatible with different image sizes. Examples: >>> image_encoder = ImageEncoderViT(...) >>> prompt_encoder = PromptEncoder(...) >>> mask_decoder = MaskDecoder(...) >>> sam_model = SAMModel(image_encoder, prompt_encoder, mask_decoder) >>> # Further usage depends on SAMPredictor class Notes: All forward() operations are implemented in the SAMPredictor class. mask_threshold image_encoderprompt_encoder mask_decoder pixel_mean pixel_stdreturnNc(t|||_||_||_|j dt j|jdddd|j dt j|jddddy)a Initialize the SAMModel class to predict object masks from an image and input prompts. Args: image_encoder (ImageEncoderViT): The backbone used to encode the image into image embeddings. prompt_encoder (PromptEncoder): Encodes various types of input prompts. mask_decoder (MaskDecoder): Predicts masks from the image embeddings and encoded prompts. pixel_mean (List[float]): Mean values for normalizing pixels in the input image. pixel_std (List[float]): Standard deviation values for normalizing pixels in the input image. Examples: >>> image_encoder = ImageEncoderViT(...) >>> prompt_encoder = PromptEncoder(...) >>> mask_decoder = MaskDecoder(...) >>> sam_model = SAMModel(image_encoder, prompt_encoder, mask_decoder) >>> # Further usage depends on SAMPredictor class Notes: All forward() operations moved to SAMPredictor. rrFrN) super__init__rrrregister_buffertorchTensorview)selfrrrrr __class__s a/var/www/html/test/engine/venv/lib/python3.12/site-packages/ultralytics/models/sam/modules/sam.pyrzSAMModel.__init__;s8 *,( \5<< +C+H+HQPQ+RTYZ [%,,y*A*F*Fr1a*PRWXct|jdr|jj|||j_|Dcgc]}|dz c}|j_|d|j_ycc}w)CSet image size to make model compatible with different image sizes. set_imgszrN)hasattrrr(rinput_image_sizeimage_embedding_sizeimg_sizer"imgszxs r$r(zSAMModel.set_imgsz^si 4%%{ 3    ( ( //4,EJ3KAG3K0&+Ah#4Ls A:))g33333^@gR]@gRY@)g(\2M@g(\L@gL@)__name__ __module__ __qualname____doc__rfloat__annotations__r r r rrr( __classcell__r#s@r$rrsp8 NE#>> model = SAM2Model(image_encoder, memory_attention, memory_encoder) >>> image_batch = torch.rand(1, 3, 512, 512) >>> features = model.forward_image(image_batch) >>> track_results = model.track_step(0, True, features, None, None, None, {}) rruse_multimask_token_for_obj_ptrpred_obj_scorespred_obj_scores_mlpfixed_no_obj_ptrsoft_no_obj_ptruse_mlp_for_obj_ptr_projno_obj_embed_spatialcompile_image_encoderc$Vt$|||_| |_| rdnd|_||_||_|r(tjjdddd|_ ||_ |r|sJ||_ ||_ ||_||_|j |_||_|j"|_t)|j$drRt)|j$j*dr2|j$j*j,j.d|_||_tjj3tj4|dd|j&|_t9|j6d tjj3tj4dd|j"|_tjj3tj4dd|j"|_t9|j:d t9|j<d | |_||_ ||_!| |_"||_#||_$| |_%||_&||_'||_(||_)||_*||_+||_,||_-|"|_.||_/||_0||_1||_2|jbr|j^sJ|j sJ|j^re|j rYtjj3tj4d|j"|_3t9|jfd | |_4d |_5|!rYtjj3tj4d|j&|_5t9|jjd |jm| |_7|#rRtqjrd tjt|jjvd d d|j_;y y )a Initialize the SAM2Model for video object segmentation with memory-based tracking. Args: image_encoder (nn.Module): Visual encoder for extracting image features. memory_attention (nn.Module): Module for attending to memory features. memory_encoder (nn.Module): Encoder for generating memory representations. num_maskmem (int): Number of accessible memory frames. image_size (int): Size of input images. backbone_stride (int): Stride of the image backbone output. sigmoid_scale_for_mem_enc (float): Scale factor for mask sigmoid probability. sigmoid_bias_for_mem_enc (float): Bias factor for mask sigmoid probability. binarize_mask_from_pts_for_mem_enc (bool): Whether to binarize sigmoid mask logits on interacted frames with clicks during evaluation. use_mask_input_as_output_without_sam (bool): Whether to directly output the input mask without using SAM prompt encoder and mask decoder on frames with mask input. max_cond_frames_in_attn (int): Maximum number of conditioning frames to participate in memory attention. directly_add_no_mem_embed (bool): Whether to directly add no-memory embedding to image feature on the first frame. use_high_res_features_in_sam (bool): Whether to use high-resolution feature maps in the SAM mask decoder. multimask_output_in_sam (bool): Whether to output multiple masks for the first click on initial conditioning frames. multimask_min_pt_num (int): Minimum number of clicks to use multimask output in SAM. multimask_max_pt_num (int): Maximum number of clicks to use multimask output in SAM. multimask_output_for_tracking (bool): Whether to use multimask output for tracking. use_multimask_token_for_obj_ptr (bool): Whether to use multimask tokens for object pointers. iou_prediction_use_sigmoid (bool): Whether to use sigmoid to restrict IoU prediction to [0-1]. memory_temporal_stride_for_eval (int): Memory bank's temporal stride during evaluation. non_overlap_masks_for_mem_enc (bool): Whether to apply non-overlapping constraints on object masks in memory encoder during evaluation. use_obj_ptrs_in_encoder (bool): Whether to cross-attend to object pointers from other frames in the encoder. max_obj_ptrs_in_encoder (int): Maximum number of object pointers from other frames in encoder cross-attention. add_tpos_enc_to_obj_ptrs (bool): Whether to add temporal positional encoding to object pointers in the encoder. proj_tpos_enc_in_obj_ptrs (bool): Whether to add an extra linear projection layer for temporal positional encoding in object pointers. use_signed_tpos_enc_to_obj_ptrs (bool): Whether to use signed distance in the temporal positional encoding in the object pointers. only_obj_ptrs_in_the_past_for_eval (bool): Whether to only attend to object pointers in the past during evaluation. pred_obj_scores (bool): Whether to predict if there is an object in the frame. pred_obj_scores_mlp (bool): Whether to use an MLP to predict object scores. fixed_no_obj_ptr (bool): Whether to have a fixed no-object pointer when there is no object present. soft_no_obj_ptr (bool): Whether to mix in no-object pointer softly for easier recovery and error mitigation. use_mlp_for_obj_ptr_proj (bool): Whether to use MLP for object pointer projection. no_obj_embed_spatial (bool): Whether add no obj embedding to spatial frames. sam_mask_decoder_extra_args (dict | None): Extra arguments for constructing the SAM mask decoder. compile_image_encoder (bool): Whether to compile the image encoder for faster inference. Examples: >>> image_encoder = ImageEncoderViT(...) >>> memory_attention = SAM2TwoWayTransformer(...) >>> memory_encoder = nn.Sequential(...) >>> model = SAM2Model(image_encoder, memory_attention, memory_encoder) >>> image_batch = torch.rand(1, 3, 512, 512) >>> features = model.forward_image(image_batch) >>> track_results = model.track_step(0, True, features, None, None, None, {}) r) kernel_sizestrideout_projweightrg{Gz?)stdNzFImage encoder compilation is enabled. First forward pass will be slow.z max-autotuneTF)mode fullgraphdynamic)r? no_obj_ptrr@rA_build_sam_headsmax_cond_frames_in_attnrinfocompileforward)%r"rrXr[r^rprqrerfrgrjrurdrNrkrlrmrnr;rorirhrPrQrTrUrVrWr<r=r>r?r@rArrrBr#s% r$rzSAM2Model.__init__smB +,H)'C!'>$'>$ "$)88??1aQq?#QD (@% $+ ++)B&/N,2T/!1*22- 4&& 3@S@S@\@\^f8g..77>>DDQGDL& % 2 25;;{AqRVR^R^3_ `d++6!HH..u{{1a/QR#hh00Q4??1STd''T2d))t4)B&*C&(@%2T/-J*/N,5Y1'>$$8!$8!-J*/N,*D'%.+F(.#6 0.  '' ''// //   D$@$@#hh00Q1PQDO $//t 4(@%$(! (-(:(:5;;q$,,;W(XD % $33 > '>$ ! KK` a).""**# *D   & !r%cHt|jjS)z=Return the device on which the model's parameters are stored.)next parametersdevicer"s r$r|zSAM2Model.deviceosDOO%&---r%ctd)zWProcess image and prompt inputs to generate object masks and scores in video sequences.zPlease use the corresponding methods in SAM2VideoPredictor for inference.See notebooks/video_predictor_example.ipynb for an example.)NotImplementedError)r"argskwargss r$rxzSAM2Model.forwardts! J  r%c |j|_|j|jz|_t |j|j|jf|j|jfd|_td dtd|jdd|jdd|j|j|j|j|jd |jxsi|_|j rwt"j$j'|j|j|_|j*rUt-|j|j|jd|_n#t"j$j/|_|j0r:t"j$j'|j|j2|_y t"j$j/|_y ) zMBuild SAM-style prompt encoder and mask decoder for image segmentation tasks.r)) embed_dimr,r+ mask_in_chansrDi)depth embedding_dimmlp_dim num_heads) num_multimask_outputs transformertransformer_dimiou_head_depthiou_head_hidden_dimuse_high_res_featuresror<r=r;N)rZsam_prompt_embed_dimrprqsam_image_embedding_sizer sam_prompt_encoderr r rNror<r=r;rrsam_mask_decoderrPrrLinear obj_ptr_projr@rIdentityrUr\obj_ptr_tpos_projr}s r$rtzSAM2Model._build_sam_heads{s$(OO!(,4;O;O(O%#0//----"#oot?# !0! "#-"77  !55 #"&"C"C'+'F'F 00 $ 8 8,0,P,P! //52!! $  ' ' % QD ,,$'$//[\$]! % 1 1 3D   ) )&+XX__T__dll%SD "%*XX%6%6%8D "r%c |jd}|j}|jd|jk(sJ|jd|jk(sJ|jd|jk(sJ|4|d}|d} |jd|k(r| jd|k(sCJt j |dd|}t j |dtj|  } |t|jd k(r|jdd|dfk(sJ|jd d|jjk7r=tj|j|jjd d d} n|} nd} |j|| fd| \} } |j||jj!| | |d |\} }}}|j"r(|dkD}t j$|ddddf| t&} | j} tj| |j(|j(fd d }|dddf}|rvt j*|d}t j,||}| ||fj/d}|||fj/d}|jddkDr |||f}n| |}}|j1|}|j"rS|j2r|j5}nj}|j6r||z}|d|z |j8zz}| ||||||fS)a{ Forward pass through SAM prompt encoders and mask heads. This method processes image features and optional point/mask inputs to generate object masks and scores. Args: backbone_features (torch.Tensor): Image features with shape (B, C, H, W). point_inputs (Dict[str, torch.Tensor] | None): Dictionary containing point prompts. 'point_coords': Tensor of shape (B, P, 2) with float32 dtype, containing absolute pixel-unit coordinates in (x, y) format for P input points. 'point_labels': Tensor of shape (B, P) with int32 dtype, where 1 means positive clicks, 0 means negative clicks, and -1 means padding. mask_inputs (torch.Tensor | None): Mask of shape (B, 1, H*16, W*16), float or bool, with the same spatial size as the image. high_res_features (List[torch.Tensor] | None): List of two feature maps with shapes (B, C, 4*H, 4*W) and (B, C, 2*H, 2*W) respectively, used as high-resolution feature maps for SAM decoder. multimask_output (bool): If True, output 3 candidate masks and their IoU estimates; if False, output only 1 mask and its IoU estimate. Returns: low_res_multimasks (torch.Tensor): Tensor of shape (B, M, H*4, W*4) with SAM output mask logits. high_res_multimasks (torch.Tensor): Tensor of shape (B, M, H*16, W*16) with upsampled mask logits. ious (torch.Tensor): Tensor of shape (B, M) with estimated IoU for each output mask. low_res_masks (torch.Tensor): Tensor of shape (B, 1, H*4, W*4) with the best low-resolution mask. high_res_masks (torch.Tensor): Tensor of shape (B, 1, H*16, W*16) with the best high-resolution mask. obj_ptr (torch.Tensor): Tensor of shape (B, C) with object pointer vector for the output mask. object_score_logits (torch.Tensor): Tensor of shape (B) with object score logits. Examples: >>> backbone_features = torch.rand(1, 256, 32, 32) >>> point_inputs = {"point_coords": torch.rand(1, 2, 2), "point_labels": torch.tensor([[1, 0]])} >>> mask_inputs = torch.rand(1, 1, 512, 512) >>> results = model._forward_sam_heads(backbone_features, point_inputs, mask_inputs) >>> ( ... low_res_multimasks, ... high_res_multimasks, ... ious, ... low_res_masks, ... high_res_masks, ... obj_ptr, ... object_score_logits, ... ) = results rrrrDN point_coords point_labelsr|)dtyper|rEFbilinearTsize align_cornersrK antialias)pointsboxesmasks)image_embeddingsimage_pesparse_prompt_embeddingsdense_prompt_embeddingsmultimask_output repeat_imagehigh_res_features)rrKrrdim)rr|rrrr`onesint32lenr]rmask_input_sizeF interpolater5r get_dense_per<where NO_OBJ_SCORErpargmaxarange unsqueezerr?sigmoidr>rs)r"backbone_features point_inputs mask_inputsrrBr|sam_point_coordssam_point_labelssam_mask_promptsparse_embeddingsdense_embeddingslow_res_multimasksioussam_output_tokensobject_score_logitsis_obj_appearinghigh_res_multimaskssam_output_token best_iou_inds batch_inds low_res_maskshigh_res_masksobj_ptrlambda_is_obj_appearings r$_forward_sam_headszSAM2Model._forward_sam_headssh  " "1 %")) %%a(D,E,EEEE %%a(D,I,IIII %%a(D,I,IIII  #+N; +N; #((+q05E5J5J15MQR5R RR %{{1a6B  % 1au{{6 RR   "{(()Q.;3D3DRa3HQPQF3R RR  %)@)@)P)PP"#--%%'00@@"'#" ##.#O.2.E.E$&67!/F/ ++ LPK`K`.,,99;%6$4-/LaL HD"35H   2Q6 "'-=atm-LN`bn!o 0557mm //4??3  -QT2 !LL26Ma7J.z=/HISSTUVM0]1JKUUVWXN %%a(1,#4Z5N#O ,>@S>M##$45   ##*=*E*E*G'*:*@*@*B'$$1G;%rs)r"rrr out_scaleout_biasmask_inputs_floatrrrr_rrrs r$_use_mask_as_outputzSAM2Model._use_mask_as_outputCs* 8'--/*Y6A  %%b)Q.0C0CB0G10LM  ##K$4$4Q$7;AAC++kk+"2"21"5t{OaOabG)-(?(?"3 001BC"3)@) %Aq!Q7A!99[%8%8%;%A%A%Cc%IqQ+I6"2"8"8":'*AAHL   $$1G;%|#| }(tA|(|&z |'}(|jC|(}(|(jEdjGd| |j<}(n&|%jIt |#| |j<}(|j<| kro|%jKd| | |j<z|j<}%|%j ddddj!dd}%|(jM| |j<zd}(|j|%|j|(|%jNd}nd}n|jPr9|d|jRz})|)j dddj | | | | })|)S|jRjGd| |j<g}|jTjGd| |j<g}t5jV|d}*t5jV|d}+|jY|||*|+|})|)j dddj | | | | })|)Scc}wcc}}wcc}}w)zePrepare memory-conditioned features by fusing current frame's visual features with previous memories.rrrrcond_frame_outputsnon_cond_frame_outputsNmaskmem_featuresT)r| non_blockingmaskmem_pos_encrrrrD)currcurr_posmemory memory_posnum_obj_ptr_tokens)-rrZr|r^rr!rrruvaluestrainingrirangegetappendtorrarPminrQrWitemsrVabsziprstackrTrUr\tensorrrrexpand new_zerosreshaperepeat_interleaver]rdrbrccatrX),r" frame_idxis_init_cond_framecurrent_vision_featscurrent_vision_pos_embedsr output_dict num_framestrack_in_reverserCHWr|r tpos_sign_mul to_cat_memoryto_cat_memory_pos_embed cond_outputsselected_cond_outputsunselected_cond_outputsoutt_pos_and_prevsrt_post_relprev_frame_idxprevfeats maskmem_encrQtptr_cond_outputs pos_and_ptrst_diffpos_list ptrs_listobj_ptrs t_diff_maxtpos_dimobj_pospix_feat_with_memrmemory_pos_embeds, r$$_prepare_memory_conditioned_featuresz.SAM2Model._prepare_memory_conditioned_featuress ! $ ) )! , OO"~1%b)00   q '+33Aq!<AA!Q1M M.A !572M{#789A= ==&'; : !#:4I3O3O3QRC3xROR ]](L(LAq$"2"23 5((50A::JY%6PY\aPaN)(11}&:a%?N%3uqyAo%EN*3Q'71'<%=%AN%3uqyAo%EN!":;??PTU;255ndKC&&s|4- 50 / < t</0336PT3U$$U]]1%5%=%=aA%FG"#45b9<ANAN3($( (=$#3"8"8": 3 $CC']m;!$Y]!3I   $A'>?FF.> F*IPVDVA1u!7AO%&>?CCAG^GbGbcdfjGklC$++VS^,DE F *-|*<'Hi${{9!CCAq!QO  ASd( s W?!X1X c|djd}|j}|d\}} |djdddj|||| } |jr|j s|j |}|jxr|} | r |j s|dkDj} ntj|} |jdk7r| |jz} |jdk7r| |jz} |j| | d} | d }| d }|jE|dkDj}|d|d z |jd j|j zz }||fS) zXEncode frame features and masks into a new memory representation for video segmentation.rrrr?rT)skip_mask_sigmoidvision_featuresr).NN)rrZrr!rhr"_apply_non_overlapping_constraintsrgr5rrrerfr[rArr])r"r rpred_masks_high_resris_mask_from_ptsrrrrpix_featbinarize mask_for_mem maskmem_outrrrs r$_encode_new_memoryzSAM2Model._encode_new_memory/s ! $ ) )! , OO"~1'+33Aq!<AA!Q1M  - -dmm#'"I"IJ]"^ ::O?O DMM/!3::>@  %5o%F!FK$JcJcKf&,,K.!. .  00r%c j||d} t|dkDrft|dd|ddD cgc]H\} }| jdddj| j d| j dg|J}} }nd}|W|j rK|djddd}|jd|j g|d}|j|||}nT|j|||dd|dd|dd|| | }| ||J| }|j||}|j|||||}| |||fScc}} w) hPerform a single tracking step, updating object masks and memory features based on current frame inputs.)rrrNrrr)r r r rrrrr)rrrrr) rrrr!rrjrZrr0_use_multimaskr)r"r r r rrrrrrrprev_sam_mask_logits current_outr0srr8 sam_outputsrs r$ _track_stepzSAM2Model._track_step[s(4KP # $q ( 4Sb 9:cr?K!Aq( !Q"''q 166!9AqA! ! !%   "t'P'P,B/771a@H$x}}RJ:b>JH228=NP[\K@@##5%9"#%>*CBC*H%bc?'%!1A H$/#/K4GGG2 #223E|T 11"*)'"3!1 2KK):HDDO!sA D/c|r7|jdkDr(|}|j|||||du\} } | |d<| |d<yd|d<d|d<y)z^Run memory encoder on predicted mask to encode it into a new memory feature for future frames.rN)r rr6rr7rr)r^r<) r"r rrrun_mem_encoderrrrAhigh_res_masks_for_mem_encrrs r$_encode_memory_in_outputz"SAM2Model._encode_memory_in_outputsx t//!3)7 &040G0G%9%$>$7".d": 1H1 - o/?K* +- pred_masksr6rr)rDrrH)r"r r r rrrrrrrrFr@rArCrrrrrs r$ track_stepzSAM2Model.track_steps,*.)9)9   %        * & [!QP[L1a9L$1 L!-; )*!( I}}2EK- . %%       r%c|dn|djd}|jxr6|xs |jxr$|j|cxkxr|jkScS)zaDetermine whether to use multiple mask outputs in the SAM head based on configuration and inputs.rrr)rrkrnrlrm)r"r rnum_ptss r$r?zSAM2Model._use_multimasksk#+!n1M1R1RST1U  ( ( T#It'I'I T**gR9R9RR S r%c |jd}|dk(r|S|j}tj|dd}tj||dddddf}||k(}tj ||tj |d}|S) z\Apply non-overlapping constraints to masks, keeping the highest scoring object per location.rrT)rkeepdimrNr)max)rr|rrrrclamp)rJ batch_sizer| max_obj_indsbatch_obj_indskeeps r$r5z,SAM2Model._apply_non_overlapping_constraintss __Q' ? ""||JAtD j@D$PTATU~-[[z5;;zu3UV r%c||_y)z Set binarize for VideoPredictor.N)rg)r"r9s r$ set_binarizezSAM2Model.set_binarizes 2:/r%c|d|_||j_|Dcgc]}|dz c}|j_ycc}w)r'rr)N)rprr+r,r.s r$r(zSAM2Model.set_imgsz s;(380IN7OAR7O47Os ?) ir)r2rFFrFFFrrFFFrFFr)TFFFFFFFFFNF)NNNF)F)FTN)r1r2r3r4rr5r6boolrpropertyr|rxrtrrrr rrr0r<rDrHrKr? staticmethodr5rWr(r7r8s@r$r:r:gsDL NE"%!$+0-2 ""'%* %&+05#(()&+ % "!%"'(-+0 %$)!& %).%*$(&+I}&*.'}:;}<"=}>?}@A}B#'C}D#E}H $I}~.. -9d W r- ^u|| I0b!H*1X9Ev2J !'8t ";Pr%r:)typingrrtorch.nn.functionalr functionalr torch.nn.initrultralytics.nn.modulesrultralytics.utilsrblocksr decodersr r encodersr r utilsrrrModulerr:rr%r$rhs] '&$)24= H/ryyH/VfPfPr%