T^(h" dZddlZddlmZddlmZmZmZddlZddlm Z ddl m Z m Z ddl mZdd lmZmZdd lmZmZmZd d lmZej0eZd ZdZdeej:deej:deej:fdZdej:dededej:fdZ dej:dej:dej:dedej:f dZ!dej:dej:dedej:fdZ"d edej:fd!Z#eGd"d#eZ$Gd$d%e jJZ&Gd&d'e jJZ'Gd(d)e jJZ(Gd*d+e jJZ)d,e(iZ*Gd-d.e jJZ+Gd/d0e jJZ,Gd1d2e jJZ-Gd3d4e jJZ.Gd5d6e Z/d7Z0d8Z1e d9e0Gd:d;e/Z2d6d;gZ3y) keypointsheightwidthctj||g|j|jd}|dz }|j ddj dz}||dddddfz |dddddfz S)a Normalize keypoints locations based on image image_shape Args: keypoints (`torch.Tensor` of shape `(batch_size, num_keypoints, 2)`): Keypoints locations in (x, y) format. height (`int`): Image height. width (`int`): Image width. Returns: Normalized keypoints locations of shape (`torch.Tensor` of shape `(batch_size, num_keypoints, 2)`). )devicedtypeNrT)keepdimgffffff?)rtensorr#r$maxvalues)rr r!sizecenterscalings rnormalize_keypointsr-5sv << 0@0@  XY] ^D AXFhhq$h'..4G q$z* *gaqj.A AAlog_cost_matrixlog_source_distributionlog_target_distributionnum_iterationscrtj|}tj|}t|D]Z}|tj||j dzdz }|tj||j dzdz }\||j dz|j dzS)az Perform Sinkhorn Normalization in Log-space for stability Args: log_cost_matrix (`torch.Tensor` of shape `(batch_size, num_rows, num_columns)`): Logarithm of the cost matrix. log_source_distribution (`torch.Tensor` of shape `(batch_size, num_rows)`): Logarithm of the source distribution. log_target_distribution (`torch.Tensor` of shape `(batch_size, num_columns)`): Logarithm of the target distribution. Returns: log_cost_matrix (`torch.Tensor` of shape `(batch_size, num_rows, num_columns)`): Logarithm of the optimal transport matrix. r%rdim)r zeros_likerange logsumexp unsqueeze)r/r0r1r2 log_u_scaling log_v_scaling_s rlog_sinkhorn_iterationsr=Js*$$%<=M$$%<=M > "w/%///TaTkTklmTnBntu2vv /%///TaTkTklmTnBntu2vv w ]44Q7 7-:Q:QRS:T TTr.scores reg_param iterationsc|j\}}}|jd}||zj|||zj|}}|j||d} |j|d|} |j|dd}t j t j || gdt j | |gdgd} ||zj } t j | j||j d| zg} t j | j||j d| zg}| dj|d|dj|d}} t| | ||}|| z }|S)a Perform Differentiable Optimal Transport in Log-space for stability Args: scores: (`torch.Tensor` of shape `(batch_size, num_rows, num_columns)`): Cost matrix. reg_param: (`torch.Tensor` of shape `(batch_size, 1, 1)`): Regularization parameter. iterations: (`int`): Number of Sinkhorn iterations. Returns: log_optimal_transport_matrix: (`torch.Tensor` of shape `(batch_size, num_rows, num_columns)`): Logarithm of the optimal transport matrix. r%N)r2)shape new_tensortoexpandrrlogr=)r>r?r@ batch_sizenum_rows num_columns one_tensornum_rows_tensornum_columns_tensorsource_reg_paramtarget_reg_param couplingslog_normalizationr0r1log_optimal_transport_matrixs rlog_optimal_transportrSgs )/ %J+""1%J+3j+@*D*DV*L{]gOgNkNklrNs'O '' Ha@ '' A{C  Q2I 599f.>%?DeiiQaclPmoqFrsuvwI),>>CCEE#ii  ! !( +-?-C-C-Ed-KN_-_`$ii  ! !+ .0C0C0Ed0KN_0_` %,,Z<%,,Z<5 $;*,CT^$ $@BS#S ''r.r5cb|j|j|jddz SNrr%)new_onesrCcumsum)xr5s r arange_likerYs) ::aggcl # * *1 - 11r.c6eZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZeeejed<dZeeejed <y) KeypointMatchingOutputa, Base class for outputs of keypoint matching models. Due to the nature of keypoint detection and matching, the number of keypoints is not fixed and can vary from image to image, which makes batching non-trivial. In the batch of images, the maximum number of matches is set as the dimension of the matches and matching scores. The mask tensor is used to indicate which values in the keypoints, matches and matching_scores tensors are keypoint matching information. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*): Loss computed during training. mask (`torch.IntTensor` of shape `(batch_size, num_keypoints)`): Mask indicating which values in matches and matching_scores are keypoint matching information. matches (`torch.FloatTensor` of shape `(batch_size, 2, num_matches)`): Index of keypoint matched in the other image. matching_scores (`torch.FloatTensor` of shape `(batch_size, 2, num_matches)`): Scores of predicted matches. keypoints (`torch.FloatTensor` of shape `(batch_size, num_keypoints, 2)`): Absolute (x, y) coordinates of predicted keypoints in a given image. hidden_states (`Tuple[torch.FloatTensor, ...]`, *optional*): Tuple of `torch.FloatTensor` (one for the output of each stage) of shape `(batch_size, 2, num_channels, num_keypoints)`, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) attentions (`Tuple[torch.FloatTensor, ...]`, *optional*): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, 2, num_heads, num_keypoints, num_keypoints)`, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) Nlossmatchesmatching_scoresrmask hidden_states attentions)__name__ __module__ __qualname____doc__r\rr FloatTensor__annotations__r]r^rr_ IntTensorr`rrar.rr[r[s6)-D(5$$ %,+/GXe'' (/37OXe//07-1Ix))*1&*D(5?? #*8output_hidden_statesc|jd}tj||gd}|rdnd}|jD]}||}|s||fz}||fS)Nrr4ri)r9rrr)ryrr>rr{all_hidden_stateslayers rrz SuperGlueKeypointEncoder.forwardsp !!!$yy)V!4!< "6BD\\ HE .L#$5$G! H...r.)F rbrcrdr rrrrrboolrrrrs@rrrsn - -4 -$05 /<< /  /'tn / u||XeELL&9:: ; /r.rcPeZdZd fd Zdej dej fdZ ddej deejdeejdeejd eejd ee e ejd ee de ej fd Z xZ S)SuperGlueSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|_|xs t#|dd|_|j$dk(s|j$d k(rF|j&|_tj(d |j&zd z |j|_|j,|_y) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()position_embedding_typeabsolute relative_keyrelative_key_queryrr%)rqrrrnum_attention_headshasattr ValueErrorrattention_head_size all_head_sizerrsquerykeyvalueDropoutattention_probs_dropout_probdropoutgetattrrmax_position_embeddings Embeddingdistance_embedding is_decoderryrlrrzs rrrzSuperGlueSelfAttention.__init__s     : : :a ?PVXhHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD # ++r.rXrc|jdd|j|jfz}|j|}|j ddddS)NrBrrr%r )r*rrviewpermute)ryrX new_x_shapes rtranspose_for_scoresz+SuperGlueSelfAttention.transpose_for_scores sLffhsmt'?'?AYAY&ZZ FF; yyAq!$$r.r`attention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsc$|j|}|du} | r||d} |d} |}n| rC|j|j|} |j|j|} |}n|y|j|j|} |j|j|} t j |d| gd} t j |d| gd} n@|j|j|} |j|j|} |j|} |du} |j r| | f}t j| | jdd}|jdk(s|jdk(r| jd| jd}}| rDt j|dz tj|j jdd}n@t j|tj|j jdd}t j|tj|j jdd}||z }|j!||j"zdz }|j%| j& }|jdk(rt j(d | |}||z}nE|jdk(r6t j(d | |}t j(d | |}||z|z}|t+j,|j.z }|||z}t0j2j5|d}|j7|}|||z}t j|| }|j9dddd j;}|j=dd|j>fz}|j|}|r||fn|f}|j r||fz}|S)Nrr%rr4rBr}rr)r$r#r$zbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rrrrrrrmatmulr~rrCr'longr#rarangerrrEr$einsummathsqrtrr functionalsoftmaxrr contiguousr*r)ryr`rrrrrrmixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss rrzSuperGlueSelfAttention.forwards!JJ}5 3$> ."<&q)I(+K3N 11$((;P2QRI33DJJ?T4UVK3N  '11$((=2IJI33DJJ}4MNK >!#4i"@aHI))^A%6 $D!LK11$((=2IJI33DJJ}4MNK//0AB "$. ??(5N!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC 6G=/2mM] ?? 11Gr.rpNNNNNF) rbrcrdrrrrrrrfrrrrrs@rrrs,4%ell%u||%7;15=A>BDH,1c||c!!2!23cE--. c ((9(9: c !)):): ; c!uU->->'?!@Ac$D>c u|| cr.rc\eZdZdeffd Zdej dej fdZxZS)SuperGlueSelfOutputrlct|tj|j|j|_yrp)rqrrrrsrdense)ryrlrzs rrrzSuperGlueSelfOutput.__init__us, YYv1163E3EF r.r`rc(|j|}|Srp)r)ryr`argss rrzSuperGlueSelfOutput.forwardys =1 r. rbrcrdr rrrrrrrs@rrrts,GGU\\U\\r.reagerceZdZd fd ZdZ d dej deejdeejdeejdeejdee e ejd ee d e ej fd Z xZ S)SuperGlueAttentionct|t|j|||_t ||_t|_y)N)r) rqrr SUPERGLUE_SELF_ATTENTION_CLASSES_attn_implementationryroutputset pruned_headsrs rrrzSuperGlueAttention.__init__sC 4V5P5PQ ,C  *&1 Er.c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrr%r4)rr ryrrrr rrrrrrunion)ryheadsindexs r prune_headszSuperGlueAttention.prune_headss u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r.r`rrrrrrrc p|j|||||||}|j|d|} | f|ddz} | SrU)ryr) ryr`rrrrrr self_outputsattention_outputrs rrzSuperGlueAttention.forwardsWyy    ! "    ;;|A F#% QR(88r.rpr) rbrcrdrrrrrrrfrrrrrs@rrrs";*7;15=A>BDH,1||!!2!23E--.  ((9(9:  !)):): ; !uU->->'?!@A$D> u|| r.rceZdZdeddffd Z d dej deej deej deej d ed ede ej ee ej ee ej ffd Z xZ S) SuperGlueAttentionalPropagationrlrNc vt||j}t||_|dz|dz|g}t dt |dz Dcgc]}t|||dz ||}}|jtj|d|dtj||_ ycc}w)Nrr%r}rB) rqrrrr attentionr7rrkrrrsrmlp)ryrlr mlp_channelsrrrzs rrrz(SuperGlueAttentionalPropagation.__init__s (( +F3#aq+F 1c,/!34  *&,q1u2E|TU W    bii R 0,r2BCD==(  sB6 descriptorsrrrrrc|j|||||}|d}|dd} tj||gd} |rdnd} |jD]} | | } |s| | fz} | | | fS)N)rrrrrr%rr4ri)rrrr) ryrrrrrrattention_outputsrrr{rrs rrz'SuperGlueAttentionalPropagation.forwards!NN )"7#9/ + #1%%ab) yy+v!6A> "6BDXX HE .L#$5$G! H . 99r.)NNNFFrrs@rrrs ) )4 )268<9="'%*:\\:!.: ( 5 : !) 6 :  :#: u||XeELL&9:HU5<N>NOT!WWZ8H8HI&'LLQ1m<AA!DLLZYZ\]_lm' $#&;'=%9"3 K NE#$5 A$F! !/+a.!@%-K= .>-~==r.)NFFrrs@rrrsvv4v(,"'/4 ->\\->u||$-> -> 'tn -> u||Xe_huo= > ->r.rc`eZdZdeddffd Zdej dej fdZxZS)SuperGlueFinalProjectionrlrNctt||j}tj||d|_y)NT)bias)rqrrrrrs final_proj)ryrlrrzs rrrz!SuperGlueFinalProjection.__init__s- (( ))K4Hr.rc$|j|Srp)r)ryrs rrz SuperGlueFinalProjection.forwards{++r.rrs@rr r s4II4I ,5<<,ELL,r.r c@eZdZdZeZdZdZdejddfdZ y)SuperGluePreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. superglue pixel_valuesmodulerNct|tjtjtjfrm|j j jd|jj|j%|jj jyyt|tjrJ|jj j|j j jdyt|tr5tjj!|j"jdyy)zInitialize the weightsg)meanstdN?) isinstancerrsConv2dConv1dweightdatanormal_rlinitializer_rangerzero_ LayerNormfill_rkinit constant_rt)ryrs r _init_weightsz&SuperGluePreTrainedModel._init_weights-s fryy"))RYY? @ MM   & &CT[[5R5R & S{{&   &&('  - KK   " " $ MM   $ $S )  = > GG  fmm00# 6?r.) rbrcrdrer config_classbase_model_prefixmain_input_namerModuler'rir.rrr#s0 #L#$O 7BII 7$ 7r.raO This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SuperGlueConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`SuperGlueImageProcessor`]. See [`SuperGlueImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zLSuperGlue model taking images as inputs and outputting the matching of them.ceZdZdZdeddffd Z ddejdejdejd ed ed e ejd e e d e e de ejeje e ffdZ e e ddejde ej d e e d e e de e dee eff dZxZS)SuperGlueForKeypointMatchingaSuperGlue feature matching middle-end Given two sets of keypoints and locations, we determine the correspondences by: 1. Keypoint Encoding (normalization + visual feature and location fusion) 2. Graph Neural Network with multiple self and cross-attention layers 3. Final projection layer 4. Optimal Transport Layer (a differentiable Hungarian matching algorithm) 5. Thresholding matrix based on mutual exclusivity and a match_threshold The correspondence ids use -1 to indicate non-matching points. Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning Feature Matching with Graph Neural Networks. In CVPR, 2020. https://arxiv.org/abs/1911.11763 rlrNctt||tj|j|_t ||_t||_ t||_ tjjtjd}|j!d||j#y)Nr bin_score)rqrrr from_configkeypoint_detector_configkeypoint_detectorrkeypoint_encoderrgnnr final_projectionrr Parameterr'register_parameter post_init)ryrlr/rzs rrrz%SuperGlueForKeypointMatching.__init__ls  !>!J!J6KjKj!k 8 @*62 8 @HH&&u||C'89   Y7 r.rrr>r r!r_rrc D &'|rdnd} |rdnd} |jddk(rD|jdd} |j| dtj|j | | | fS|j\&} '} |j &dz'd}|j &dz'|j j}|j &dz'}||j &dz'nd}t|||}|j|||} | d}||z}|#|j}|j||}n#tj&'f|j}|j|||| }|d}|j|}|j &d'|j j}|dddf}|ddd f}||j!d dz}||j jd zz }|Q|j &d'}|dddfj#dj%dd'}|j'|dk(d }t)||j*|j j, }|ddddddfj/d}|ddddddfj/d }|j0}|j0}t3|d d|j5d |k(}t3|d d|j5d |k(}|j7d}tj8||j:j=|}tj8||j j>kD||}tj8||j5d ||}|||kDz} || j5d |z}!tj8| ||j7d}"tj8|!||j7d}#tj@|"|#gj &dd}$tj@||gj &dd}%|r+| | d z} | |d z} | |fz} tC&'fd| D} |r| |dz} tC&'fd| D} |$|%| | fS)a= Perform keypoint matching between two images. Args: keypoints (`torch.Tensor` of shape `(batch_size, 2, num_keypoints, 2)`): Keypoints detected in the pair of image. descriptors (`torch.Tensor` of shape `(batch_size, 2, descriptor_dim, num_keypoints)`): Descriptors of the keypoints detected in the image pair. scores (`torch.Tensor` of shape `(batch_size, 2, num_keypoints)`): Confidence scores of the keypoints detected in the image pair. height (`int`): Image height. width (`int`): Image width. mask (`torch.Tensor` of shape `(batch_size, 2, num_keypoints)`, *optional*): Mask indicating which values in the keypoints, matches and matching_scores tensors are keypoint matching information. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors. Default to `config.output_attentions`. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. Default to `config.output_hidden_states`. Returns: matches (`torch.Tensor` of shape `(batch_size, 2, num_keypoints)`): For each image pair, for each keypoint in image0, the index of the keypoint in image1 that was matched with. And for each keypoint in image1, the index of the keypoint in image0 that was matched with. matching_scores (`torch.Tensor` of shape `(batch_size, 2, num_keypoints)`): Scores of predicted matches for each image pair all_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of each stage) of shape `(1, 2, num_keypoints, num_channels)`. all_attentions (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for each layer) of shape `(1, 2, num_heads, num_keypoints, num_keypoints)`. riNrrrBr)r)r#)r_rrr%g?ge)r@c3fK|](}|jddjdd*yw)rrBr}N)rr~.0rXrHrs r zASuperGlueForKeypointMatching._match_image_pair..s2&RS *a;EEb"M&s.1c3HK|]}|jddyw)rrBN)rr;s rr=zASuperGlueForKeypointMatching._match_image_pair..s#"vbc199ZB }#]"vs")"rCnew_fullrr new_zerosrrlrr-r3r*get_extended_attention_maskonesr#r4r5r~r9rF masked_fillrSr/sinkhorn_iterationsr(indicesrYgatherrDwherer)expmatching_thresholdrr)(ryrrr>r r!r_rrrrrCr<encoded_keypointslast_hidden_state input_shapeextended_attention_maskr projected_descriptorsfinal_descriptorsfinal_descriptors0final_descriptors1mask0max0max1indices0indices1mutual0mutual1zeromatching_scores0matching_scores1valid0valid1matches0matches1r]r^rHrs( @@r_match_image_pairz.SuperGlueForKeypointMatching._match_image_pairzsX#7BD0d ??1  "OOCR(E""5"EII">##E*!  +4//' A}a%%j1nmQG !))*q.-I`I`a  Q >>B>Nt||JNM:TX( 65A  11)VZn1o-a0"$55  %**,K&*&F&Ft[&Y #&+jj*m1LU^UeUe&f #hh (!5/  "!n !% 5 5k B299*aX\XcXcXoXop.q!t4.q!t4$&8&B&B1a&HH$++11366  << A}=DAJ((,33BMJE'' D9F'vt~~$++JiJija"crck"&&q)a"crck"&&q)<<<<h*40HOOAx4PPh*40HOOAx4PP  # ;;w 0A4H ;;'7$++:X:X'XZjlpq ;;w0@0G0G80TVZ[,t346==H55;;vx1D1DR1HI;;vx1D1DR1HI))Xx0199*aL))%57G$HIQQR\^_acd  14Ea4H H  1KN B  15J4L L  %&Wh&!  +k!n>> from transformers import AutoImageProcessor, AutoModel >>> import torch >>> from PIL import Image >>> import requests >>> url = "https://github.com/magicleap/SuperGluePretrainedNetwork/blob/master/assets/phototourism_sample_images/london_bridge_78916675_4568141288.jpg?raw=true" >>> image1 = Image.open(requests.get(url, stream=True).raw) >>> url = "https://github.com/magicleap/SuperGluePretrainedNetwork/blob/master/assets/phototourism_sample_images/london_bridge_19481797_2295892421.jpg?raw=true" >>> image2 = Image.open(requests.get(url, stream=True).raw) >>> images = [image1, image2] >>> processor = AutoImageProcessor.from_pretrained("magic-leap-community/superglue_outdoor") >>> model = AutoModel.from_pretrained("magic-leap-community/superglue_outdoor") >>> with torch.no_grad(): >>> inputs = processor(images, return_tensors="pt") >>> outputs = model(**inputs) ```Nz9SuperGlue is not trainable, no labels should be provided.r%rzOInput must be a 5D tensor of shape (batch_size, 2, num_channels, height, width)rBr)r_rrc3$K|]}|| ywrpri)r<vs rr=z7SuperGlueForKeypointMatching.forward..Qs=s)r\r]r^rr_r`ra)rrlrruse_return_dictndimr*rCrr2rErcloner`rr[)ryrrarrrbr\rHr<channelsr r!keypoint_detectionsrr>rr_absolute_keypointsr]r^r`ras rrz$SuperGlueForKeypointMatching.forward sG>  XY Y1B1N-TXT_T_TqTq$8$D $++JjJj &1%no o1=1C1C. Ax#++JNHfeT "44\B/B2A/F, 6;%%j!R;>>|L  Ar255lC!))*aT[[=T=TUXXYef ||J2.&__.);Aq!QJ)G%)O1aA:&);Aq!QJ)G&)P1aA:&>B>T>T     /!5?U ? ;-)T=Zde  &+'!  r.)NNN)NNNN)rbrcrdrer rrrrrrrrr`rSUPERGLUE_INPUTS_DOCSTRINGrf LongTensorrr[rrrs@rr-r-VsT "  4 *(,,0/3O <<O \\O  O  O  O u||$O $D>O 'tnO  u||U\\5%7 8O b++EF.2,0/3&* S ''S ))*S $D> S 'tn S d^ S  u,, -S GS r.r-)4rer dataclassesrtypingrrrrr transformersrr 5transformers.models.superglue.configuration_supergluer pytorch_utilsr r utilsrrrautor get_loggerrblogger_CONFIG_FOR_DOC__CHECKPOINT_FOR_DOC_rrrr-r=rSrYr[r+rkrrrrrrrr rSUPERGLUE_START_DOCSTRINGrnr-__all__rir.rr}sR !)) >QQPP0   H %$> lell 3 lE%,,DW l\abgbnbn\o l B5<<BBSBU\\B*U\\U"\\U#\\U U  \\ U:*(%,,*(5<<*(UX*(]b]i]i*(Z22 2 ":[": ":J BII  /ryy/@CRYYCL")) #$ 00f(:bii(:V4>bii4>n,ryy,772  RE #;E E P &'E Fr.