'}hLc UddlZddlZddlmZmZmZmZmZmZddl Z ddl m Z ddl m Z ddgZe s:ddlZGdd Zd Zeej$d _eej$d _ydd lmZmZmZmZmZmZmZmZmZej>e Z!er dd l"m#Z#GddZ&e&Z'e&e(d<dde)fdZ*GddZddde)dee+dfdeee)dfdefdZy#e$$re!jKdY\wxYw)N)DictListOptionalTuple TYPE_CHECKINGUnion) is_available)not_noneinit_device_mesh DeviceMeshc eZdZy)_DeviceMeshStubN)__name__ __module__ __qualname__\/var/www/html/test/engine/venv/lib/python3.12/site-packages/torch/distributed/device_mesh.pyrrs rrcyNrrrr_init_device_mesh_stubrs rztorch.distributed.device_mesh) _find_pg_by_ranks_and_tag_get_default_group_get_group_tagget_rankget_world_sizeinit_process_groupis_initialized new_group ProcessGroup) ArrayLikezCDeviceMesh requires numpy >= 1.21 to be installed for type checkingceZdZddZddZdddededdfd Zdddedfd Z dddeefd Z e d edefd Z e d edefdZ dddedefdZy)_MeshEnvreturnNc.g|_i|_i|_yr) mesh_stackchild_to_parent_mappingparent_to_child_mappingselfs r__init__z_MeshEnv.__init__<s02DOIKD (TVD (rr cft|jdk(r td|jdS)Nrz#No device mesh is currently active!)lenr' RuntimeErrorr*s rget_current_meshz_MeshEnv.get_current_meshAs.4??#q("#HII??2& &r device_meshmesh_dim mesh_dim_namec|jj|}|r|j|}|r|S|j}|jj d|j d|jj |}|D]#}t|j||fd}||vs"|} %|j|g _ ||j| <| |jj|i|<| S)Nr.Fmesh_dim_names _init_backend) r)getrmeshswapdimsreshapesizer device_type_dim_group_infosr( setdefault) r+r2r3r4child_mesh_mappingssub_meshcur_rankpg_ranks_by_dimmesh_1d res_sub_meshs rcreate_child_meshz_MeshEnv.create_child_meshFs#'">">"B"B;"O ".22=A#O#++-H)..77HEMMK$$))(3O+ ,%++$1#3"'  w&#+L ,.9-I-I(-S,TL )9DD ( ( 6  ( ( 3 3K D  rc:|jj|dSr)r(r9)r+r2s rget_parent_meshz_MeshEnv.get_parent_meshis//33KF Frc|j|}|j}|r.|r,t|dk(sJd|d}|j||Sy)z Return the index of the mesh dim in the parent mesh. The device_mesh passed in needs to be sliced out from a parent mesh. z%The child mesh can only be a 1D mesh.rN)rIr7r/get_mesh_dim_by_name)r+r2 parent_meshchild_mesh_dim_nameschild_mesh_dim_names rget_parent_mesh_dimz_MeshEnv.get_parent_mesh_dimlsf ..{;K#.#=#= 3,-2;:;2&:1&=#00>QRRrr>c4t|jSr)_get_device_handle device_countr>s rnum_devices_per_hostz_MeshEnv.num_devices_per_host{s%k2??A ArcBttj|zSr)rr$rUrTs r num_hostsz_MeshEnv.num_hostss"#x'D'D['QQ Qrc|jt|jdk(r td||jvrtd|dd|jt|jj |S)NrzNo `mesh_dim_names` found.zMesh dimension 'z' does not exist.z.Available mesh dimensions are: mesh_dim_names=)r7r/KeyErrorr index)r+r2r4s rrLz_MeshEnv.get_mesh_dim_by_names**2{112a70K$>$>>&}o5FGD[E_E_D`aK66<<]KL Lrr%Nr%r )rrrr,r1intstrrGrrIrP staticmethodrUrWrLrrrr$r$;s W  ' ! +! 7:! KN! ! F G| G@V G < HSM   Bc Bc B  B  R3 R3 R  R  M+ Mc$tt|dS)a: Get the module corresponding to the device_type which is cuda or cuda-like device. For example, when the device_type is cuda, the module `torch.cuda` is returned. Return None when there is no corresponding module for device_type, otherwise return the corresponding module. N)getattrtorchrTs rrRrRsuk400rc eZdZUdZeed<ejed<ee edfed<ddddede ejd fdee edfd e d df d Z d Z dZddZd dZd efdZdZded e fdZded dfdZ d!dee eefd e eeeffdZd!deed efdZed efdZed e edffdZd efdZd!dee eefd efdZd eeefdZ y)"r a DeviceMesh represents a mesh of devices, where layout of devices could be represented as a n-d dimension array, and each value of the n-d dimensional array is the global id of the default process group ranks. DeviceMesh could be used to describe the layout of devices across the cluster, and serves as a proxy for communication among the device lists within the cluster. DeviceMesh can be used as a context manager. .. note:: DeviceMesh follows SPMD programming model, which means the same PyTorch Python program is running on all processes/ranks in the cluster. Therefore, users need to make sure the `mesh` array (which describes the layout of devices) should be identical across all ranks. Inconsistent `mesh` will lead to silent hang. Args: device_type (str): The device type of the mesh. Currently supports: "cpu", "cuda/cuda-like". mesh (ndarray): A multi-dimensional array or an integer tensor describing the layout of devices, where the IDs are global IDs of the default process group. Returns: DeviceMesh: A :class:`DeviceMesh` object representing the device layout. The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. A reduction over the first dimension of mesh will reduce across columns (0, 4), .. and (3, 7), a reduction over the second dimension of mesh reduces across rows (0, 1, 2, 3) and (4, 5, 6, 7). Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) r>r:.r7NTr6r"r8r%cf||_t|tjr'|jj dk7rt d|t|tjr|jjn$tj|tj|_ ||_ t|jjj|_t#|j |jj$t'|f|_|dk7r|r |j+|j-|jt/k(j1}|j3ddvsJ|j3ddkDr|djnd|_yy)Ncpuz!`mesh` must be a CPU tensor, got )dtypexlar)rrK)r> isinstancercTensordevicetype ValueErrordetachrftensorr]r:r7tupleflattentolist_flatten_mesh_listhashshapeid_hash_get_or_create_default_group_init_process_groupsrnonzeror=_coordinate_on_dim)r+r>r:r7r8 rank_coordss rr,zDeviceMesh.__init__s[ +D $ -$++2B2Be2K #DTF!KLLdELL1 !!#\\$eii8 I #1D ',DII,=,=,?,F,F,H&ID #t66 DRSDJe#!557--/ $yyHJ6??A "''*f444/:/?/?/BQ/FKN))+D'$rc t}|s tt}|jj |kDr't d|jj dt |j}|sZ|rX|j}||kDr'||zdk7rt d|d|d|jd|jt|ztS)Nz=Mesh should not be bigger than default world size, but found z ranks!rz8DeviceMesh only support homogeneous hardware, but found z ranks and  z devices!) rrrr:numelr0rRr>rS set_devicerr)r+default_initialized world_size device_handlerUs rrxz'DeviceMesh._get_or_create_default_groups"0"2 &"$')Jyy :-"STXT]T]TcTcTeSffmn/t/?/?@M&=(5'A'A'C$!55"%99Q>&R%,k2F1GqIYIYHZZce((6J)JK%' 'rc 6g}|jjdk(r~|jjtk(rY|j t t tttt jf||_yt|jjD]}|jjd|jd|jj|}|D]}|j}t|}|j|vs2t!||kDrt#d|jd|d|j t t%|||jf||_y)NrKr.)rankszFEach device mesh dimension should get only one process group, but got z in !)r:ndimrrappendrrlistrange group_namer;r<r=rrr rr/r0r r?)r+dim_group_infosdimrDdim_meshsubgroup_ranks dim_groups rryzDeviceMesh._init_process_groupssxACOyy~~"tyy'8N#345*,77N%4D !=!0C'+ii&8&8S&A&I&IDIINN3/'O %4)1): %.N$C  ==?n<"?3c9&2&lmqmzmzl{|**8)9%<'"!",22$28I3F$G$2$-$8$8!"<%4D !rcDtjj||Sr)r`r'rr*s r __enter__zDeviceMesh.__enter__Gs  & & - -d 3Krc@tjjyr)r`r'pop)r+exc_type exc_value exc_tracebacks r__exit__zDeviceMesh.__exit__Ms  & & * * ,rc|js d|jjd}|Sd|jjd|jd}|S)Nz DeviceMesh()z, mesh_dim_names=)r7r:rr)r+device_mesh_reprs r__repr__zDeviceMesh.__repr__Qsn**dii..013  $ ##499#3#3#5"66GH[H[G\\]^  $ #rc|jSr)rwr*s r__hash__zDeviceMesh.__hash__Ys :: rotherc t|tsyt|jt|jk(ry|jj|jjk(xr|j |j k(S)NFT)rir rvr:rurs)r+rs r__eq__zDeviceMesh.__eq__\sceZ0$))}5::. 5::#3#33H++u/G/GG rr4c|jjdk(r/|jr||jdk(r|Std|dtj ||}tj |||}|S)a Slice the current DeviceMesh based on the mesh_dim_name given to create a child DeviceMesh. Args: mesh_dim_name (str): the name of the mesh dimension of the parent DeviceMesh to create a child DeviceMesh for. Returns: A :class:`DeviceMesh` object The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. Calling mesh["tp"] on rank 0, 1, 2, 3 would return a 1D child DeviceMesh:([0, 1, 2, 3]). Calling mesh["tp"] on rank 4, 5, 6, 7 would return a 1D child DeviceMesh:([4, 5, 6, 7]). Calling mesh["dp"] on rank 0, 4 would return a 1D child DeviceMesh:([0, 4]). Calling mesh["dp"] on rank 1, 5 would return a 1D child DeviceMesh:([1, 5]). Calling mesh["dp"] on rank 2, 6 would return a 1D child DeviceMesh:([2, 6]). Calling mesh["dp"] on rank 3, 7 would return a 1D child DeviceMesh:([3, 7]). Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) rKrzInvalid mesh_dim_name z specified.)r:rr7r0r`rLrG)r+r4r3submeshs r __getitem__zDeviceMesh.__getitem__fs{8yy~~"&&=DV Vrc.|jjSr)r:rr*s rrzDeviceMesh.ndims99>> !rc@t|jjSr)rpr:rur*s rruzDeviceMesh.shapes) )rctS)z: Returns the current global rank. )rr*s rrzDeviceMesh.get_ranks : rc|jdkDr&|$td|jjdd|d}t|j |}t |t sJdtt|S)a{ Returns the local rank of the given mesh_dim of the DeviceMesh. Args: mesh_dim (str/int, optional): it can be the name of the mesh dimension or the index of the mesh dimension. Default is None. Returns: An integer denotes the local rank. The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 0, 1, 2, 3 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 4, 5, 6, 7 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 0, 4 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 1, 5 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 2, 6 would return 2. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 3, 7 would return 3. Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) rKzFound the DeviceMesh have z dimensionszJOptional kwarg `mesh_dim` needs to be specified when device_mesh.ndim > 1.rz1We expect ProcessGroup before calling `get_rank`!)rr0r:r rrir!r)r+r3mesh_dim_groups rget_local_rankzDeviceMesh.get_local_ranks8yy1}!1"00@ L`!%dnnX&>?N  CB CH^45 5rc6|jr |jSdS)z Return the relative indices of this rank relative to all dimensions of the mesh. If this rank is not part of the mesh, return None. N)r{r*s rget_coordinatezDeviceMesh.get_coordinates /3.E.E4** O4 Orr\r[r)!rrr__doc__r^__annotations__rcrjrrrboolr,rxryrrrrobjectrrr]r!rrr=propertyrrurrrrrrr r s% Nll sCx119="& % %  k12%  %U38_5 %   % % N (:4 4l   - $c $   4 & S& \& R9=) "$U38_5) " <l!33 4) "V W# W# W  "# "  "  *5c? *  * c  ( 68E#s(O+D( 6PS( 6T PHT#Y$7 Pr)r7 mesh_shape.r7r%c j|ktt|t|k7rtdd|t|t|k7r%tddt|dt|dtjt j |j|}t|||}|S)aG Initializes a `DeviceMesh` based on `device_type`, `mesh_shape`, and `mesh_dim_names` parameters. This creates a DeviceMesh with an n-dimensional array layout, where `n` is the length of `mesh_shape`. If `mesh_dim_names` is provided, each dimension is labeled as `mesh_dim_names[i]`. .. note:: `init_device_mesh` follows SPMD programming model, meaning the same PyTorch Python program runs on all processes/ranks in the cluster. Ensure `mesh_shape` (the dimensions of the nD array describing device layout) is identical across all ranks. Inconsistent `mesh_shape` may lead to hanging. .. note:: If no process group is found, init_device_mesh will initialize distributed process group/groups required for distributed communications behind the scene. Args: device_type (str): The device type of the mesh. Currently supports: "cpu", "cuda/cuda-like". mesh_shape (Tuple[int]): A tuple defining the dimensions of the multi-dimensional array describing the layout of devices. mesh_dim_names (Tuple[str], optional): A tuple of mesh dimension names to assign to each dimension of the multi-dimensional array describing the layout of devices. Its length must match the length of `mesh_shape`. Each string in `mesh_dim_names` must be unique. Returns: DeviceMesh: A :class:`DeviceMesh` object representing the device layout. Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import init_device_mesh >>> >>> mesh_1d = init_device_mesh("cuda", mesh_shape=(8,)) >>> mesh_2d = init_device_mesh("cuda", mesh_shape=(2, 8), mesh_dim_names=("dp", "tp")) z"Each mesh_dim_name must be unique.z/Found repeated mesh_dim_name in mesh_dim_names z6mesh_shape and mesh_dim_names should have same length!zFound len(mesh_dim_names): z and len(mesh_shape):.)r>r:r7) r/setr0rcarangemathprodviewr )r>rr7r:r2s rr r sP  %3~&'3~+>>"8EnEUV :#n"55"L1#n2E1FF[\_`j\k[llmn ||DIIj1277 C #) r)cuda),loggingrtypingrrrrrrrctorch.distributedr utils._typing_utilsr __all__sysrrmodulesr r "torch.distributed.distributed_c10drrrrrrrr r! getLoggerrlogger numpy.typingr" ImportErrorwarningr$r`rr^rRr]rrrrs7 DD ** | ,~   ?NCKK/0;0KK'   W  x (F  . YMYMv!) OX*11XPXP| 59 <<#s(O<!sCx1 <  <K  NNU  sC$$C=<C=