T^(hdZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZdd lmZmZdd lmZd dlmZej:eZdZ eGddeZ!Gdde jDZ#Gdde jDZ$Gdde jDZ%Gdde jDZ&Gdde jDZ'Gdde jDZ(Gdde jDZ)dCd Z*Gd!d"e jDZ+Gd#d$e jDZ,Gd%d&e jDZ-e j\j^dDd'e0d(e1fd)Z2Gd*d+e jDZ3Gd,d-e jDZ4Gd.d/e jDZ5Gd0d1e jDZ6Gd2d3e jDZ7Gd4d5e jDZ8Gd6d7e jDZ9Gd8d9e jDZ:Gd:d;e jDZ;Gd<d=eZZ=d?Z>ed@e=GdAdBe<Z?dBd=gZ@y)EzPyTorch ZoeDepth model.N) dataclass)ListOptionalTupleUnion)nn)ACT2FN)add_start_docstrings%add_start_docstrings_to_model_forwardreplace_return_docstrings)DepthEstimatorOutput)PreTrainedModel) ModelOutputlogging) load_backbone)ZoeDepthConfigrceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<y) ZoeDepthDepthEstimatorOutputa Extension of `DepthEstimatorOutput` to include domain logits (ZoeDepth specific). Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. predicted_depth (`torch.FloatTensor` of shape `(batch_size, height, width)`): Predicted depth for each pixel. domain_logits (`torch.FloatTensor` of shape `(batch_size, num_domains)`): Logits for each domain (e.g. NYU and KITTI) in case multiple metric heads are used. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, num_channels, height, width)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlosspredicted_depth domain_logits. hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrrl/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/zoedepth/modeling_zoedepth.pyrr,s2)-D(5$$ %,37OXe//0715M8E--.5=AM8E%"3"3S"89:A:>Ju00#567>r$rcfeZdZdZfdZdeejdeejfdZxZ S)ZoeDepthReassembleStageaE This class reassembles the hidden states of the backbone into image-like feature representations at various resolutions. This happens in 3 stages: 1. Map the N + 1 tokens to a set of N tokens, by taking into account the readout ([CLS]) token according to `config.readout_type`. 2. Project the channel dimension of the hidden states according to `config.neck_hidden_sizes`. 3. Resizing the spatial dimensions (height, width). Args: config (`[ZoeDepthConfig]`): Model configuration class defining the model architecture. c Nt||j|_tj|_t |j|jD],\}}|j jt|||.|jdk(rtj|_ |j}|jD]Y}|jjtjtjd|z|t|j [yy)N)channelsfactorproject)super__init__ readout_typer ModuleListlayerszipneck_hidden_sizesreassemble_factorsappendZoeDepthReassembleLayerreadout_projectsbackbone_hidden_size SequentialLinearr hidden_act)selfconfigneck_hidden_sizer* hidden_size_ __class__s r%r.z ZoeDepthReassembleStage.__init__^s "//mmo (+F,D,DfF_F_(` j $ f KK  6vHXagh i j   ) +$&MMOD ! 55K-- %%,,MM"))A O["I6RXRcRcKde  ,r$rreturncN|djd}tj|d}|dddf|ddddf}}|j\}}}|j||||}|j ddddj }|j dk(rZ|jdj d}|jdj|} tj|| fd }n#|j d k(r||jd z}g} t|j|dD]t\} } |j dk(r|j| | } | j dddj|d ||} |j| | } | j| v| S) z Args: hidden_states (`List[torch.FloatTensor]`, each of shape `(batch_size, sequence_length + 1, hidden_size)`): List of hidden states from the backbone. rdimNrr r,r+)rr,radd)shaper catreshapepermute contiguousr/flatten unsqueeze expand_as enumeratesplitr7r1r5) r<r patch_height patch_width batch_size cls_tokentotal_batch_sizesequence_length num_channelsreadoutout stage_idx hidden_states r%forwardzZoeDepthReassembleStage.forwardos #1%++A.  -Q7 #0A#6 ae8L= :G:M:M7/<%--.> k[gh %--aAq9DDF    ))11!4<CM   % ')I,?,?,CCM'01D1DZUV1D1W'X % #I|  I-?t44Y? M (//1a8@@RQ]_jkL14;;y1,?L JJ| $ % r$ rrrrr.rr Tensorr] __classcell__rAs@r%r'r'Ns6 "&T%,,%7&W[\a\h\hWi&r$r'c$eZdZfdZdZxZS)r6c ^t||j}tj||d|_|dkDr tj ||||d|_y|dk(rtj|_y|dkr,tj||dtd|z d|_yy)Nr) in_channels out_channels kernel_sizerrfstridepaddingr ) r-r.r8rConv2d projectionConvTranspose2dresizeIdentityint)r<r=r)r*r?rAs r%r.z ZoeDepthReassembleLayer.__init__s 11 )) (`ab A:,,XxV\blmnDK q[++-DK aZ))HhAcRSV\R\oghiDKr$cJ|j|}|j|}|SN)rkrmr<r\s r%r]zZoeDepthReassembleLayer.forwards$|4 {{<0 r$rrrr.r]r`ras@r%r6r6s j r$r6c$eZdZfdZdZxZS)ZoeDepthFeatureFusionStagect|tj|_t t |jD]&}|jjt|(yrq) r-r.rr0r1rangelenr3r5ZoeDepthFeatureFusionLayer)r<r=r@rAs r%r.z#ZoeDepthFeatureFusionStage.__init__sT mmo s63345 CA KK  9&A B Cr$c|ddd}g}d}t||jD]*\}}| ||}n |||}|j|,|S)NrF)r2r1r5)r<rfused_hidden_statesfused_hidden_stater\layers r%r]z"ZoeDepthFeatureFusionStage.forwardsq%dd+  !#&}dkk#B ; L%!)%*<%8"%*+=|%L"  & &'9 :  ;#"r$rsras@r%rurusC #r$rucZeZdZdZfdZdej dej fdZxZS)ZoeDepthPreActResidualLayerz ResidualConvUnit, pre-activate residual unit. Args: config (`[ZoeDepthConfig]`): Model configuration class defining the model architecture. ct||j|_|j |jn |j }t j |_t j|j|jddd||_ t j |_ t j|j|jddd||_ |jrat j|j|j|_t j|j|j|_yy)Nr r)rfrhribias)eps)r-r.!use_batch_norm_in_fusion_residualuse_batch_normuse_bias_in_fusion_residualrReLU activation1rjfusion_hidden_size convolution1 activation2 convolution2 BatchNorm2dbatch_norm_eps batch_norm1 batch_norm2)r<r=rrAs r%r.z$ZoeDepthPreActResidualLayer.__init__s $FF11=  . .((( $ 779II  % %  % %,  779II  % %  % %,     !~~f.G.GVMbMbcD !~~f.G.GVMbMbcD  r$r\rBc |}|j|}|j|}|jr|j|}|j |}|j |}|jr|j |}||zSrq)rrrrrrrr<r\residuals r%r]z#ZoeDepthPreActResidualLayer.forwards'' 5 ((6   ++L9L'' 5 ((6   ++L9Lh&&r$) rrrrr.r r_r]r`ras@r%rrs* dD'ELL'U\\'r$rc,eZdZdZdfd ZddZxZS)rya8Feature fusion layer, merges feature maps from different stages. Args: config (`[ZoeDepthConfig]`): Model configuration class defining the model architecture. align_corners (`bool`, *optional*, defaults to `True`): The align_corner setting for bilinear upsample. ct|||_tj|j |j dd|_t||_t||_ y)NrT)rfr) r-r. align_cornersrrjrrkrresidual_layer1residual_layer2)r<r=rrAs r%r.z#ZoeDepthFeatureFusionLayer.__init__sT *))F$=$=v?X?Xfgnrs:6B:6Br$c|l|j|jk7r?tjj||jd|jdfdd}||j |z}|j |}tjj|dd|j }|j|}|S)Nr,r bilinearFsizemoder scale_factorrr)rHr functional interpolaterrrrkrs r%r]z"ZoeDepthFeatureFusionLayer.forwards  !!X^^3==44L$6$6q$9<;M;Ma;P#QXbrw5($*>*>x*HHL++L9 }}00 qzI[I[1 |4 r$)Trqrrrrr.r]r`ras@r%ryrysCr$rycfeZdZdZfdZdeejdeejfdZxZ S) ZoeDepthNeckaO ZoeDepthNeck. A neck is a module that is normally used between the backbone and the head. It takes a list of tensors as input and produces another list of tensors as output. For ZoeDepth, it includes 2 stages: * ZoeDepthReassembleStage * ZoeDepthFeatureFusionStage. Args: config (dict): config dict. c t|||_|j |jjdvrd|_nt ||_tj|_ |jD]?}|jjtj||jdddAt||_y)N)swinv2r rF)rfrir)r-r.r=backbone_config model_typereassemble_stager'rr0convsr3r5rjrru fusion_stage)r<r=channelrAs r%r.zZoeDepthNeck.__init__:s    ! ! -&2H2H2S2SWa2a$(D !$;F$CD !]]_ // sG JJ  bii1J1JXYcdkpq r s7v>r$rrBct|ttfs tdt |t |j j k7r td|j|j|||}t|Dcgc]\}}|j||}}}|j|}||dfScc}}w)z Args: hidden_states (`List[torch.FloatTensor]`, each of shape `(batch_size, sequence_length, hidden_size)` or `(batch_size, hidden_size, height, width)`): List of hidden states from the backbone. z2hidden_states should be a tuple or list of tensorszOThe number of hidden states should be equal to the number of neck hidden sizes.rF) isinstancetuplelist TypeErrorrxr=r3 ValueErrorrrPrr)r<rrRrSifeaturefeaturesoutputs r%r]zZoeDepthNeck.forwardKs -%7PQ Q } T[[%B%B!C Cno o  , 11-{[M=F}=UVzq'MDJJqM'*VV""8,x|## WsB?r^ras@r%rr-s6 ?"$T%,,%7$W[\a\h\hWi$r$rc`eZdZdZfdZdeejdejfdZxZ S)#ZoeDepthRelativeDepthEstimationHeada Relative depth estimation head consisting of 3 convolutional layers. It progressively halves the feature dimension and upsamples the predictions to the input resolution after the first convolutional layer (details can be found in DPT's paper's supplementary material). ct||j|_d|_|jrt j ddddd|_|j}t j ||dzddd|_t jddd |_ t j |dz|jddd|_ t j |jdddd |_ y) N)r r )rrrgr,r rrTrr)r-r. head_in_indexrkadd_projectionrrjrconv1Upsampleupsamplenum_relative_featuresconv2conv3)r<r=rrAs r%r.z,ZoeDepthRelativeDepthEstimationHead.__init__js #11   iiSfV]cdDO,,YYxQAaYZ[  SWX YYx1}f.J.JXYbcmno YYv;;QAVWabc r$rrBc||j}|j+|j|}tj|}|j |}|j |}|j |}tj|}|}|j|}tj|}|jd}||fS)NrrD) rrkrrrrrrsqueeze)r<rrrs r%r]z+ZoeDepthRelativeDepthEstimationHead.forwardys%d&8&89 ?? & OOM:M%BGGIm4M =1  m4  =1 ! -0   =1 ! -0 '//A/6((r$r^ras@r%rrcs. d)T%,,%7)ELL)r$rc||z}||z}|tj|z|tj|zz ||z tj||z |zzz S)z%log(nCk) using stirling approximation)r log)nkrs r% log_binomrsX CA CA uyy| a%))A,. .!a%599QUS[;Q1Q QQr$c@eZdZdejffd ZddZxZS)LogBinomialSoftmaxrc @t|||_||_|j dt j d|jddddd|j dt j|jdz gjdddddy) a7Compute log binomial distribution for n_classes Args: n_classes (`int`, *optional*, defaults to 256): Number of output classes. act (`torch.nn.Module`, *optional*, defaults to `torch.softmax`): Activation function to apply to the output. k_idxrrrFF) persistent k_minus_1N) r-r.ractregister_bufferr arangeviewtensor)r< n_classesrrAs r%r.zLogBinomialSoftmax.__init__s  Well1i&@&E&EaQPQ&R_de [%,, |*D*I*I!RQRTU*Vchir$c|jdk(r|jd}tjd|z |d}tj||d}t |j |j |j tj|zz|j |j z tj|zz}|j||z dS)aCompute the log binomial distribution for probabilities. Args: probabilities (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Tensor containing probabilities of each class. temperature (`float` or `torch.Tensor` of shape `(batch_size, num_channels, height, width)`, *optional*, defaults to 1): Temperature of distribution. eps (`float`, *optional*, defaults to 1e-4): Small number for numerical stability. Returns: `torch.Tensor` of shape `(batch_size, num_channels, height, width)`: Log binomial distribution logbinomial(p;t). r rrD) ndimrNr clamprrrrr)r< probabilities temperaturerone_minus_probabilitiesys r%r]zLogBinomialSoftmax.forwards    ")33A6M"'++a-.?a"H M3: dnndjj 1jj599]33 4~~ *eii8O.PP Q xxKQx//r$)?-C6?)rrrr softmaxr.r]r`ras@r%rrs!$%-- j0r$rc*eZdZ dfd ZdZxZS)%ZoeDepthConditionalLogBinomialSoftmaxc t|||z|z}tjtj||z|dddtj tj|ddddtj |_d|_|j|_ |j|_ t|tj|_y)aPer-pixel MLP followed by a Conditional Log Binomial softmax. Args: in_features (`int`): Number of input channels in the main feature. condition_dim (`int`): Number of input channels in the condition feature. n_classes (`int`, *optional*, defaults to 256): Number of classes. bottleneck_factor (`int`, *optional*, defaults to 2): Hidden dim factor. rrrgr)rN)r-r.rr9rjGELUSoftplusmlpp_epsmax_tempmin_temprr rlog_binomial_transform)r<r= in_features condition_dimrbottleneck_factor bottleneckrAs r%r.z.ZoeDepthConditionalLogBinomialSoftmax.__init__s* !M16GG == IIkM1:1UV`a b GGI IIj%Qq! L KKM     &8 &V#r$c|jtj||fd}|dddddf|dddddf}}||jz}|ddddf|ddddf|ddddfzz }||jz}|ddddf|ddddf|ddddfzz }|j d}|j |j z |z|j z}|j||S)az Args: main_feature (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Main feature. condition_feature (torch.Tensor of shape `(batch_size, num_channels, height, width)`): Condition feature. Returns: `torch.Tensor`: Output log binomial distribution rrDNr,.r)rr concatrrNrrr)r< main_featurecondition_featureprobabilities_and_temperaturerrs r%r]z-ZoeDepthConditionalLogBinomialSoftmax.forwards)-|M^>_ef1g(h% )!RaR* 5 )!QR* 5# & 2 %aCi0M!Q)4L}]^`acf]fOg4gh !DJJ. !!Q), Aq#I0FUVXY[^U^I_0_` !++A. }}t}}4 CdmmS **=+FFr$)rr,rsras@r%rrs  #WJGr$rc&eZdZdfd ZdZxZS)ZoeDepthSeedBinRegressorct||j|_|j|_||_||_tj|j|ddd|_ tjd|_ tj||ddd|_ |jdk(rtjd|_ytj|_y)adBin center regressor network. Can be "normed" or "unnormed". If "normed", bin centers are bounded on the (min_depth, max_depth) interval. Args: config (`int`): Model configuration. n_bins (`int`, *optional*, defaults to 16): Number of bin centers. mlp_dim (`int`, *optional*, defaults to 256): Hidden dimension. min_depth (`float`, *optional*, defaults to 1e-3): Min depth value. max_depth (`float`, *optional*, defaults to 10): Max depth value. rrTinplacenormedN)r-r.bottleneck_featuresrbin_centers_type min_depth max_depthrrjrract1rract2)r<r=n_binsmlp_dimrrrAs r%r.z!ZoeDepthSeedBinRegressor.__init__s" !55 & 7 7""YYt//!QB GGD) YYw1a8 -1-B-Bh-NBGGD) TVT_T_Ta r$c|j|}|j|}|j|}|j|}|jdk(r|dz}||j ddz }|j |jz |z}tjj|dd|j}tj|d }d |d d d d d f|d d dd d fzz}||fS||fS)z] Returns tensor of bin_width vectors (centers). One vector b for every pixel rMbP?rTrEkeepdim)rrrrrrconstant)rvaluerDg?NrF.) rrrrrsumrrrrpadr cumsum)r<x bin_centersbin_widths_normed bin_widths bin_edgess r%r]z ZoeDepthSeedBinRegressor.forward#s JJqM IIaL JJqMiil  H ,%,K +koo!To.R R ..4>>9=NNJ**:7IPZbfbpbp*qJ ZQ7I1crc3;!7)Aqr3J:O!OPK$k1 1 + +r$)rr rsras@r%rrsb:,r$ralphagammacN|jd||j|zzS)a:Inverse attractor: dc = dx / (1 + alpha*dx^gamma), where dx = a - c, a = attractor point, c = bin center, dc = shift in bin center This is the default one according to the accompanying paper. Args: dx (`torch.Tensor`): The difference tensor dx = Ai - Cj, where Ai is the attractor point and Cj is the bin center. alpha (`float`, *optional*, defaults to 300): Proportional Attractor strength. Determines the absolute strength. Lower alpha = greater attraction. gamma (`int`, *optional*, defaults to 2): Exponential Attractor strength. Determines the "region of influence" and indirectly number of bin centers affected. Lower gamma = farther reach. Returns: torch.Tensor: Delta shifts - dc; New bin centers = Old bin centers + dc r)divpow)dxrrs r% inv_attractorr=s%" 66!ebffUm++ ,,r$c0eZdZ dfd ZddZxZS)ZoeDepthAttractorLayerct ||j|_|j|_|j |_||_||_ ||_ ||_ ||_ |jx}}tj||ddd|_tj"d|_tj||dzddd|_tj"d|_y)zq Attractor layer for bin centers. Bin centers are bounded on the interval (min_depth, max_depth) rrTrr,N)r-r.attractor_alpharattractor_gammagemmaattractor_kindkind n_attractorsrrrmemory_efficientbin_embedding_dimrrjrrrrr r<r=rr rrr!rrrAs r%r.zZoeDepthAttractorLayer.__init__Rs ++ ++ )) ( "" 0!' 8 88 gYY{GQ1= GGD) YYw q(8!QB GGD) r$c|7|r0tjj||jdddd}||z}|j |}|j |}|j |}|j|}|dz}|j\}}}} |j||jd|| }|dddddd f} tjj||| fdd} |jsctjtjd |j} | t| j!d| j!d z d } ntj"| | j$ } t'|jD]*}| t| dd|d fj!d | z z } ,|jdk(r| |jz } | | z}|j(|j*z |z|j*z} tj,| d \} }tj.| |j*|j(} || fS)ao The forward pass of the attractor layer. This layer predicts the new bin centers based on the previous bin centers and the attractor points (the latter are predicted by the MLP). Args: x (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Feature block. prev_bin (`torch.Tensor` of shape `(batch_size, prev_number_of_bins, height, width)`): Previous bin centers normed. prev_bin_embedding (`torch.Tensor`, *optional*): Optional previous bin embeddings. interpolate (`bool`, *optional*, defaults to `True`): Whether to interpolate the previous bin embeddings to the size of the input features. Returns: `Tuple[`torch.Tensor`, `torch.Tensor`]: New bin centers normed and scaled. NrTrrrr,r.meanrrrDdevicer()rrrrHrrrrrr r!r r(rrrrN zeros_liker*rwrrsortclip)r<r prev_binprev_bin_embeddingr attractorsrTr@heightwidthattractors_normedr funcdelta_crbin_new_centerss r%r]zZoeDepthAttractorLayer.forwardqs%&  )%']]%>%>& :UY&?&"&&A JJqM IIaL JJqMYYq\ $& '1'7'7$ Avu__Z1B1BAvuU 'q!Q|4mm//65/PZjn/o $$!JJuyy9$))DD=):)D)DQ)G+J_J_`aJb)bcijkG&&{;;M;MNG4,,- b=):1a9)E)O)OPQ)RU`)`aa byyF"!D$5$55%/~~6/IDNNZ KQ7 QjjdnndnnM  ++r$)rrrFNTrsras@r%rrQs *><,r$rc0eZdZ dfd ZddZxZS)ZoeDepthAttractorLayerUnnormedct |||_||_||_||_|j |_|j |_|j|_ ||_ |jx}}tj||ddd|_tj d|_tj||ddd|_tj&|_y)zL Attractor layer for bin centers. Bin centers are unbounded rrTrN)r-r.r rrrrrrrrr!r"rrjrrrrrrr#s r%r.z'ZoeDepthAttractorLayerUnnormed.__init__s ( ""++ ++ )) 0 & 8 88 gYY{GQ1= GGD) YYw aA> KKM r$c`|7|r0tjj||jdddd}||z}|j |}|j |}|j |}|j|}|jdd\}}tjj|||fdd}|jsctjtjd|j} | t|jd|jdz d } ntj||j } t#|j$D]*} | t|dd| d fjd|z z } ,|jd k(r| |j$z } || z} | }| |fS) a The forward pass of the attractor layer. This layer predicts the new bin centers based on the previous bin centers and the attractor points (the latter are predicted by the MLP). Args: x (`torch.Tensor` of shape (batch_size, num_channels, height, width)`): Feature block. prev_bin (`torch.Tensor` of shape (batch_size, prev_num_bins, height, width)`): Previous bin centers normed. prev_bin_embedding (`torch.Tensor`, *optional*): Optional previous bin embeddings. interpolate (`bool`, *optional*, defaults to `True`): Whether to interpolate the previous bin embeddings to the size of the input features. Returns: `Tuple[`torch.Tensor`, `torch.Tensor`]: New bin centers unbounded. Two outputs just to keep the API consistent with the normed version. Nr%rTr&r'r,rrDr).r()rrrrHrrrrr!r r(rrrrNr+r*rwr ) r<r r.r/rr0r1r2r r4r5rr6s r%r]z&ZoeDepthAttractorLayerUnnormed.forwards&  )%']]%>%>& :UY&?&"&&A JJqM IIaL JJqMYYq\ "((- mm//65/PZjn/o $$!JJuyy9$))DD=)=)=a)@;CXCXYZC[)[\bcdG&&{;;M;MNG4,,- [=Aq#I)>)H)H)Kk)YZZ [yyF"!D$5$55%/%  ++r$)rrrTr7rsras@r%r9r9s ":3,r$r9cXeZdZdfd Zdej dej fdZxZS)ZoeDepthProjectorct|tj||ddd|_tj d|_tj||ddd|_y)aProjector MLP. Args: in_features (`int`): Number of input channels. out_features (`int`): Number of output channels. mlp_dim (`int`, *optional*, defaults to 128): Hidden dimension. rrTrN)r-r.rrjrrrr)r<r out_featuresrrAs r%r.zZoeDepthProjector.__init__sP YY{GQ1= 774(YYw aA> r$r\rBcl|j|}|j|}|j|}|Srq)rrrrrs r%r]zZoeDepthProjector.forwards2zz,/ xx - zz,/ r$))rrrr.r r_r]r`ras@r%r=r=s#?"ELLU\\r$r=ceZdZdZfdZdej dej fdZ d dej dej dej d eejd ee de ej f d Z xZ S) ZoeDepthMultiheadAttentionzKEquivalent implementation of nn.MultiheadAttention with `batch_first=True`.ct|||zdk7rtd|d|d||_t ||z |_|j|j z|_tj||j |_ tj||j |_ tj||j |_ tj|||_ tj||_y)NrzThe hidden size (z6) is not a multiple of the number of attention heads ())r-r.rnum_attention_headsroattention_head_size all_head_sizerr:querykeyrout_projDropoutdropout)r<r?rFrMrAs r%r.z#ZoeDepthMultiheadAttention.__init__#s  , , 1#K=1-.a1  $7 #&{5H'H#I !558P8PPYY{D,>,>? 99[$*<*<=YY{D,>,>?  +{; zz'* r$r rBc|jdd|j|jfz}|j|}|j ddddS)NrFrr,rr )rrFrGrrK)r<r  new_x_shapes r%transpose_for_scoresz/ZoeDepthMultiheadAttention.transpose_for_scores7sLffhsmt'?'?AYAY&ZZ FF; yyAq!$$r$querieskeysvaluesattention_maskoutput_attentionsc|j|j|}|j|j|}|j|j|}t j ||j dd} | tj|jz } || |z} tjj| d} |j| } t j | |} | jddddj} | j!dd|j"fz} | j%| } |j'| } |r| | f} | S| f} | S)NrFr%rDrr,rr )rPrIrJrr matmul transposemathsqrtrGrrrrMrKrLrrHrrK)r<rQrRrSrTrU query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputss r%r]z"ZoeDepthMultiheadAttention.forward<sY// 70CD --dhhtn= // 60BC !<< Y5H5HR5PQ+dii8P8P.QQ  %/.@ --//0@b/I,,7 _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**+BC  m4 6G=/2O\M]r$)NF)rrrrr.r r_rPrr!boolrr]r`ras@r%rCrCsU+(%ell%u||%7;,1 %%ll% % !!2!23 % $D> % u|| %r$rCcJeZdZdfd Z ddeej fdZxZS)ZoeDepthTransformerEncoderLayerct||j}|j}|j}t ||||_tj|||_ tj||_ tj|||_ tj||_tj||_tj||_tj||_t$||_y)N)rM)r-r.patch_transformer_hidden_size#patch_transformer_intermediate_size%patch_transformer_num_attention_headsrC self_attnrr:linear1rLrMlinear2 LayerNormnorm1norm2dropout1dropout2r activation)r<r=rMrrr?intermediate_sizerFrAs r%r.z(ZoeDepthTransformerEncoderLayer.__init__es :: "FF$JJ3KAT^efyy.?@ zz'* yy!2K@ \\+. \\+.  7+  7+  ,r$src_maskc N|x}}|j||||d}||j|z}|j|}|j|j |j |j |}||j|z}|j|}|S)N)rQrRrSrTr) rjrprnrlrMrrrkrqro)r<srcrtrQrRsrc2s r%r]z'ZoeDepthTransformerEncoderLayer.forwardys $~~gDU]~^_`aDMM$''jjo||DLLc9J)KLMDMM$''jjo r$)g?relurq) rrrr.rr r_r]r`ras@r%rereds%-.,0 5<<( r$recDeZdZfdZdej fdZdZxZS)ZoeDepthPatchTransformerEncoderct||j}tjt |j Dcgc] }t|c}|_tj||jddd|_ ycc}w)zViT-like transformer block Args: config (`ZoeDepthConfig`): Model configuration class defining the model architecture. rrrgN) r-r.rrr0rwnum_patch_transformer_layersretransformer_encoderrjrgembedding_convPxP)r<r=rdr@rAs r%r.z(ZoeDepthPatchTransformerEncoder.__init__sw 00 #%==>CFDgDg>h i ,V 4 i$  "$ ==1UV`a"  jsB cpuc tjd|||jd}tjd|d||jd}tj|tjtj d| |z z}||z} tj tj| tj| gd} | jdj|dd} | S)zGenerate positional encodings Args: sequence_length (int): Sequence length embedding_dim (int): Embedding dimension Returns: torch.Tensor: Positional encodings. r)dtyper*rr,g@r)rD) r rrNexprrrIsincosrepeat) r<rTrW embedding_dimr*rpositionindexdiv_term pos_encodings r%positional_encoding_1dz6ZoeDepthPatchTransformerEncoder.positional_encoding_1ds<<?%OYYZ[\ Q qfMWWXYZ99Uuyygf1U'V&VYf&fgh(* yy%))L"9599\;R!SYZ[ #--!-4;;J1M r$cp|j|jd}tjj |d}|j ddd}|j \}}}||j||||j|jz}tdD]}|j||}|S)zForward pass Args: x (torch.Tensor - NCHW): Input feature tensor Returns: torch.Tensor - Transformer output embeddings of shape (batch_size, sequence_length, embedding_dim) r,)rrrr)r*rr) r~rMrrrrKrHrr*rrwr})r<r  embeddingsrTrWrrs r%r]z'ZoeDepthPatchTransformerEncoder.forwards++A.66q9 ]]&&z6: ''1a0 5?5E5E2 O]$"="= z?P?PXbXhXh#>#  q AA411!4Z@J Ar$) rrrr.r float32rr]r`ras@r%rzrzs" &Y^ejerer$r$rzc&eZdZdfd ZdZxZS)ZoeDepthMLPClassifierct||}tj|||_tj |_tj|||_yrq)r-r.rr:rkrrrrl)r<rr?hidden_featuresrAs r%r.zZoeDepthMLPClassifier.__init__sD %yyo> '')yy,? r$cl|j|}|j|}|j|}|Srq)rkrrrl)r<r\rs r%r]zZoeDepthMLPClassifier.forwards2||L1 |4  \2 r$)rBNrsras@r%rrs@r$rc(eZdZdZfdZdZxZS)*ZoeDepthMultipleMetricDepthEstimationHeadszn Multiple metric depth estimation heads. A MLP classifier is used to route between 2 different heads. ct ||j}|j}|j|_|j |_|j }tj||ddd|_ t||_ tdd|_ |j dk(rt}n|j dk(rt}tj |jDcic]"}|d t#||d |dz|d |d  $c}|_t'|||dz|_tj*t-dDcgc]}t'|j.||dzc}|_tj |jD cic]N}|d tj*t-t3|D cgc]} ||| |d |d c} Pc} }|_|j6} tj |jDcic]}|d t9|| ||d dc}|_ycc}wcc}wcc} wcc} }wcc}w)NrrrgrAr,rr?rsoftplusnamerrr)rrrr)rr?rrrrr)r)r-r.r"num_attractorsbin_configurationsrrrrjrrzpatch_transformerrmlp_classifierrr9 ModuleDictrseed_bin_regressorsr=seed_projectorr0rwr projectorsrxr0rrconditional_log_binomial) r<r=r"r r Attractorconfr@ configurationrlast_inrAs r%r.z3ZoeDepthMultipleMetricDepthEstimationHeads.__init__s "44,, "(";"; & 7 7%88YY24GUV_`jkl "A!H3RST  H ,.I  " "j 06I$&==#55 V 6>-2";/";/   $  0+:KUfjkUk --q    " & 9 9!2-2   --&,%>%> "f%r}}"'s<'8!9 ""#/?&3K&@&3K&@  (   "..(* &,%>%> "f%'L%!(+&' (  ) %]    & s*'I"I .I >I I II cr|j|}|j|dddddf}|j|}tj|j ddd}|j D cgc]} | d } } | tj|djj} |j D cgc] } | d| k(s | c} d} | d }| d }|j| }||\}}|jd vr ||z ||z z }n|}|j|}|j| }t!|j"||D]!\}}}||}||||d \}}|}|}#|}t$j&j)|j*d ddd}t$j&j)|j*d ddd}|j,| }|||}tj ||zdd}||fScc} wcc} w#t$rtd| dwxYw)NrTrrFrDrzbin_configurations_name z! not found in bin_configurationssrrrhybrid2rr%rr&r)rrrr rrrargmaxritem IndexErrorrrrrr0r2rrrrrHr)r<outconv_activationrfeature_blocksrelative_depthr  embeddingr domain_voternamesbin_configurations_namer=rrrseed_bin_regressorr@seed_bin_centersr.r/r0 projector attractorr bin_embeddingbinr lastrrZs r%r]z2ZoeDepthMultipleMetricDepthEstimationHeads.forward1sv JJz "**1-aAg6 ++I6 mmM$5$5!T$5$JPRS =A>9dnnt~~>]^H'H!003.1$//Sa-b 7 )Iy'%g.M(BTbfg Cyy{H!.!4!4!6   7 "!/ 8 8 ; " 9 9 ! 12ZW[!:! yy$ 56A> 11-BCWaqu1v  ) )$ >mm// QWWRS\PZjn/o iiKQ=Dyr$rsras@r%rres9 v"r$rc&eZdZdZeZdZdZdZdZ y)ZoeDepthPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. zoedepth pixel_valuesTct|tjtjtjfrm|j j jd|jj|j%|jj jyyt|tjrJ|jj j|j j jdyy)zInitialize the weightsg)r(stdNr)rrr:rjrlweightdatanormal_r=initializer_rangerzero_rmfill_)r<modules r% _init_weightsz%ZoeDepthPreTrainedModel._init_weightss fryy"))R5G5GH I MM   & &CT[[5R5R & S{{&   &&('  - KK   " " $ MM   $ $S ).r$N) rrrrr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingrr#r$r%rrs$ "L"$O&*# *r$raE This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ViTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`DPTImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. zU ZoeDepth model with one or multiple metric depth estimation head(s) on top. ceZdZfdZeeeee d de jde e jde e de e de e deee j eff d ZxZS) ZoeDepthForDepthEstimationc6t||t||_t |jj drkt |jj drK|jj j |_|jj j|_n tdt||_ t||_ t|jdkDr t!|n t#||_|j'y)Nr? patch_sizezXZoeDepth assumes the backbone's config to have `hidden_size` and `patch_size` attributesr)r-r.rbackbonehasattrr=r?r8rrrneckr relative_headrxrrr metric_head post_init)r<r=rAs r%r.z#ZoeDepthForDepthEstimation.__init__s  %f- 4=='' 7GDMMDXDXZf*>*J*JF '"mm22==DOj !( @H6,,-1 7v >26:  r$) output_typerrlabelsrUoutput_hidden_states return_dictrBcd}| td||n|jj}||n|jj}||n|jj}|j j |||}|j}|j\} } } } |j} | | z} | | z}|j|| |\}}|g|z}|j|\}}|g|z}|j|d|d|dd|\}}|jd}|s | ||f|ddz}n |f|ddz}||f|zS|St||||j|j S) a labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth depth estimation maps for computing the loss. Returns: Examples: ```python >>> from transformers import AutoImageProcessor, ZoeDepthForDepthEstimation >>> import torch >>> import numpy as np >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("Intel/zoedepth-nyu-kitti") >>> model = ZoeDepthForDepthEstimation.from_pretrained("Intel/zoedepth-nyu-kitti") >>> # prepare image for the model >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> # interpolate to original size >>> post_processed_output = image_processor.post_process_depth_estimation( ... outputs, ... source_sizes=[(image.height, image.width)], ... ) >>> # visualize the prediction >>> predicted_depth = post_processed_output[0]["predicted_depth"] >>> depth = predicted_depth * 255 / predicted_depth.max() >>> depth = depth.detach().cpu().numpy() >>> depth = Image.fromarray(depth.astype("uint8")) ```NzTraining is not implemented yet)rrUrrr,)rrrrrD)rrrrr)NotImplementedErrorr=use_return_dictrrUrforward_with_filtered_kwargs feature_mapsrHrrrrrrrr)r<rrrUrrrrbrr@r1r2rrRrSrrZr metric_depthrrs r%r]z"ZoeDepthForDepthEstimation.forwards`  %&GH H%0% ] 'tn ] d^ ]  uU\\"$88 9] _F] r$r)gHz>)i,r,)ArrY dataclassesrtypingrrrrr torch.utils.checkpointr activationsr file_utilsr r r modeling_outputsrmodeling_utilsrutilsrrutils.backbone_utilsrconfiguration_zoedepthr get_loggerrloggerrrModuler'r6rurryrrrrrrjitscriptfloatrorrr9r=rCrerzrrrrZOEDEPTH_START_DOCSTRINGrr__all__r#r$r%rsM !// ! 5-)12   H %# ?;? ?BGbiiGTbii0##2;'"));'~""J3$2993$l))"))))XR)0)0X@GBII@GF5,ryy5,p-U---&\,RYY\,~Q,RYYQ,h 6BBJ!bii!H<bii<~BII"J"J"Z^ ^F*o*0 "  y !8y  y x ()B Cr$