|h dddlmZmZmZmZddlZddlmZddlmZm Z GddejZ GddejZ y) )ListOptionalTupleTypeN)nn)MLP LayerNorm2dceZdZdZdej ddfdedejdedeejded ed d ffd Z d e jde jde jde jde d e e je jff dZd e jde jde jde jd e e je jff dZxZS) MaskDecodera Decoder module for generating masks and their associated quality scores using a transformer architecture. This class predicts masks given image and prompt embeddings, utilizing a transformer to process the inputs and generate mask predictions along with their quality scores. Attributes: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. iou_token (nn.Embedding): Embedding for the IoU token. num_mask_tokens (int): Number of mask tokens. mask_tokens (nn.Embedding): Embedding for the mask tokens. output_upscaling (nn.Sequential): Neural network sequence for upscaling the output. output_hypernetworks_mlps (nn.ModuleList): Hypernetwork MLPs for generating masks. iou_prediction_head (nn.Module): MLP for predicting mask quality. Methods: forward: Predict masks given image and prompt embeddings. predict_masks: Internal method for mask prediction. Examples: >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> masks, iou_pred = decoder( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, multimask_output=True ... ) >>> print(f"Predicted masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") transformer_dim transformernum_multimask_outputs activationiou_head_depthiou_head_hidden_dimreturnNc t|||_||_||_t j d||_|dz|_t j |j||_ t jt j||dzddt|dz|t j|dz|dzdd||_ t jt|jDcgc]}t!|||dzdc}|_t!|||j||_ycc}w)a Initialize the MaskDecoder module for generating masks and their associated quality scores. Args: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. activation (Type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality. Examples: >>> transformer = nn.TransformerEncoder(nn.TransformerEncoderLayer(d_model=256, nhead=8), num_layers=6) >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer) >>> print(decoder)  kernel_sizestrider N)super__init__rrrr Embedding iou_tokennum_mask_tokens mask_tokens SequentialConvTranspose2dr output_upscaling ModuleListrangeroutput_hypernetworks_mlpsiou_prediction_head) selfrrrrrr_ __class__s f/var/www/html/test/engine/venv/lib/python3.12/site-packages/ultralytics/models/sam/modules/decoders.pyrzMaskDecoder.__init__)s$2 .&%:"a94q8<<(<(>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> image_emb = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_emb = torch.rand(1, 2, 256) >>> dense_emb = torch.rand(1, 256, 64, 64) >>> masks, iou_pred = decoder(image_emb, image_pe, sparse_emb, dense_emb, multimask_output=True) >>> print(f"Masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") )r.r/r0r1rNr) predict_masksslice) r*r.r/r0r1r2masksiou_pred mask_slices r-forwardzMaskDecoder.forwardYsk>,,-%=$; - x(8U1d^U1a[ aQ)*AzM*hc tj|jj|jjgd}|j dj |jddd}tj||fd}tj||jdd}||z}tj||jdd}|j\} } } } |j|||\} }| dddddf}| dddd|jzddf}|jddj| | | | }|j|}t|jDcgc]!}|j||dd|ddf#}}tj |d}|j\} } } } ||j| | | | zzj| d| | }|j#|}||fScc}w)z`Predict masks and quality scores using image and prompt embeddings via transformer architecture.rdimrNr)torchcatr weightr" unsqueezeexpandshaperepeat_interleaverr! transposeviewr%r'r(stackr))r*r.r/r0r1 output_tokenstokenssrcpos_srcbchwhs iou_token_outmask_tokens_outupscaled_embeddingi hyper_in_listhyper_inr6r7s r-r4zMaskDecoder.predict_maskss 4>>#8#8$:J:J:Q:Q"RXYZ %//299:R:X:XYZ:[]_acd M+CD!L%%&6 QQO++))(FLLOKYY 1a""38C1a7 QQ)=)=%= >ABmmAq!&&q!Q2!2237QVW[WkWkQl- LM -D * *1 -oaAg.F G- - ;;}!4'-- 1a.33Aq!a%@@FFq"aQRS++M:h- s3&H)__name__ __module__ __qualname____doc__rGELUintModulerrr?Tensorboolrr9r4 __classcell__r,s@r-r r s2B&'&(gg#&.s.sYY.s # .s O .s  .s!.s .s`+,,+,,+#(,, + "' +  + u||U\\) *+Z%,,%,,%#(,, % "' % u||U\\) * %r:r ceZdZdZdej ddddddddddf dedejd ed eejd ed ed e de de de ddffd Z dde jde jde jde jde de de ee jdee je je je jffdZ dde jde jde jde jde de ee jdee je je je jffdZdZdZxZS) SAM2MaskDecodera Transformer-based decoder for predicting instance segmentation masks from image and prompt embeddings. This class extends the functionality of the MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Attributes: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. iou_token (nn.Embedding): Embedding for IOU token. num_mask_tokens (int): Total number of mask tokens. mask_tokens (nn.Embedding): Embedding for mask tokens. pred_obj_scores (bool): Whether to predict object scores. obj_score_token (nn.Embedding): Embedding for object score token. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. output_upscaling (nn.Sequential): Upscaling layers for output. use_high_res_features (bool): Whether to use high-resolution features. conv_s0 (nn.Conv2d): Convolutional layer for high-resolution features (s0). conv_s1 (nn.Conv2d): Convolutional layer for high-resolution features (s1). output_hypernetworks_mlps (nn.ModuleList): List of MLPs for output hypernetworks. iou_prediction_head (MLP): MLP for IOU prediction. pred_obj_score_head (nn.Linear | MLP): Linear layer or MLP for object score prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. Methods: forward: Predict masks given image and prompt embeddings. predict_masks: Predict instance segmentation masks from image and prompt embeddings. _get_stability_scores: Compute mask stability scores based on IoU between thresholds. _dynamic_multimask_via_stability: Dynamically select the most stable mask output. Examples: >>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) r r Fg?g\(\?rrrrrruse_high_res_featurespred_obj_scorespred_obj_scores_mlpuse_multimask_token_for_obj_ptrrNc 4t|||_||_||_t j d||_|dz|_t j |j||_ | |_ |jrt j d||_ ||_ t jt j||dzddt|dz|t j|dz|dzdd||_||_|rBt j$||dzdd|_t j$||dzdd|_t j*t-|jDcgc]}t/|||dzdc}|_t/|||j|||_|jr0t j4|d|_| rt/||dd|_| |_| |_| |_ycc}w) a Initialize the SAM2MaskDecoder module for predicting instance segmentation masks. This decoder extends the functionality of MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Args: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. activation (Type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality. use_high_res_features (bool): Whether to use high-resolution features. iou_prediction_use_sigmoid (bool): Whether to use sigmoid for IOU prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. pred_obj_scores (bool): Whether to predict object scores. pred_obj_scores_mlp (bool): Whether to use MLP for object score prediction. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. Examples: >>> transformer = nn.TransformerEncoder(nn.TransformerEncoderLayer(d_model=256, nhead=8), num_layers=6) >>> decoder = SAM2MaskDecoder(transformer_dim=256, transformer=transformer) >>> print(decoder) rrrrrr )sigmoidN)rrrrrrrr r!r"rfobj_score_tokenrhr#r$r r%reConv2dconv_s0conv_s1r&r'rr(r)Linearpred_obj_score_headdynamic_multimask_via_stability!dynamic_multimask_stability_delta"dynamic_multimask_stability_thresh)r*rrrrrrreiou_prediction_use_sigmoidrqrrrsrfrgrhr+r,s r-rzSAM2MaskDecoder.__init__sX .&%:"a94q8<<(<(>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) )r.r/r0r1rurvNrr)r4rqtraining _dynamic_multimask_via_stabilityrh) r*r.r/r0r1r2rurvr6r7rSobject_score_logitssam_tokens_outs r-r9zSAM2MaskDecoder.forward8sLAE@R@R-%=$;%/ ASA =x*= !QRA+&E12H  1 1$--"CCE8TOE8!QqS!Q,'E1Q3'H  D D,QU3N-Q!V4Nh0CCCr:c d}|jrYtj|jj|j j|j jgd}d}nAtj|j j|j jgd}|jdj|jddd}tj||fd} |r&tj|| jdd} n#|jd| jdk(sJ|} | |z} |jddk(sJdtj|| jdd} | j\} } }}|j| | | \}} |dd|ddf}|dd|dz|dz|jzddf}| jddj| | ||} |j s|j#| }n?|j"\}}}}}|\}}|||| |z}||||z}t%|jDcgc]!}|j&||dd|ddf#}}tj(|d}|j\} } }}||j| | ||zzj| d||}|j+|}|jr#|dk(sJ|j-|dddddf} n"d|j/|jddz} |||| fScc}w) zYPredict instance segmentation masks from image and prompt embeddings using a transformer.rr<rr>z@image_pe should have size 1 in batch dim (from `get_dense_pe()`)Nrg$@)rfr?r@rkrAr r"rBrCsizerErDrr!rFrGrer%r'r(rHr)rpnew_ones)!r*r.r/r0r1rurvsrIrJrKrLrMrNrOrPrQrRrSrTdc1ln1act1dc2act2feat_s0feat_s1rUrVrWr6r7rzs! r-r4zSAM2MaskDecoder.predict_masks}s^    !II((//NN))$$++  MA!IIt~~'<'N>N>U>U&V\]^M%//299:R:W:WXY:Z\^`bc M+CD!L ))*:FLLOQRSC#))!, Q? ??"C++}}Q1$h&hh$))(FLLOKYY 1a""38C1a7 QAQ1E1E)E FIJmmAq!&&q!Q2))!%!6!6s!; (,(=(= %CdC0 GW!%c#c(W*<&=!> !%c*<&=&G!H RWW[WkWkQl- LM -D * *1 -oaAg.F G- - ;;}!4'-- 1a.33Aq!a%@@FFq"aQRS++M:   6M6"&":":2aAg;"G #'):):8>>!;La)P"P h1DDD!- s/&Mc|jd}|j}tj||kDdj }tj|| kDdj }tj |dkD||z dS)zNCompute mask stability scores based on IoU between upper and lower thresholds.r>r<rg?)flattenrrr?sumfloatwhere)r* mask_logitsstability_deltaarea_iarea_us r-_get_stability_scoresz%SAM2MaskDecoder._get_stability_scoressw!))"- @@;8bAGGI;/)99rBHHJ{{6A:v<>> decoder = SAM2MaskDecoder(...) >>> all_mask_logits = torch.rand(2, 4, 256, 256) # 2 images, 4 masks each >>> all_iou_scores = torch.rand(2, 4) >>> mask_logits, iou_scores = decoder._dynamic_multimask_via_stability(all_mask_logits, all_iou_scores) >>> print(mask_logits.shape, iou_scores.shape) torch.Size([2, 1, 256, 256]) torch.Size([2, 1]) Nrr>r<r)device).NN) r?argmaxaranger}rrBrrsr expand_as)r*all_mask_logitsall_iou_scoresmultimask_logitsmultimask_iou_scoresbest_scores_inds batch_indsbest_multimask_logitsbest_multimask_iou_scoressinglemask_logitssinglemask_iou_scoresstability_scores is_stablemask_logits_outiou_scores_outs r-ryz0SAM2MaskDecoder._dynamic_multimask_via_stabilitysL6+1ab!Q;7-ae4 <<(<"E\\"6";";A">~G\G\] 0=M1M N 5 ? ? B$8EU9U$V!$=$G$G$J!,AqsAqL9 .q!A#v 6556GH$(O(OO  ++ o & 0 01B C  !      5 6 ! %  ..r:)N)rXrYrZr[rr\r]r^rr`rr?r_rrrr9r4rryrarbs@r-rdrds*`&'&(gg#&&+#((-*.+/ %$)05[U[UYY[U # [U O [U  [U![U $[U[U"[U*.[U ![UJ;?CD,,CD,,CD#(,, CD "' CD  CDCD$D$67CD u||U\\5<<E FCDX;?EE,,EE,,EE#(,, EE "' EE  EE$D$67EE u||U\\5<<E FEEN=5/r:rd) typingrrrrr?rultralytics.nn.modulesrr r^r rdr:r-rs;/. 3`"))`FS/biiS/r: