T^(h(dZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZmZmZmZdd lmZdd lmZmZmZdd lmZmZmZmZd d l m!Z!ejDe#Z$dZ%GddejLZ'GddejLZ(GddejLZ)GddejLZ*GddejLZ+GddejLZ,GddejLZ-GddejLZ.Gdd ejLZ/Gd!d"ejLZ0Gd#d$eZ1d%Z2d&Z3ed'e2Gd(d)e1Z4ed*e2Gd+d,e1Z5ed-e2Gd.d/e1Z6Gd0d1ejLZ7ed2e2Gd3d4e1Z8gd5Z9y)6zPyTorch LiLT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) LiltConfigrc:eZdZfdZ ddZdZdZxZS)LiltTextEmbeddingsc8t|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)ddt+|dd|_|j |_tj|j|j |j.|_ y) N padding_idxeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandgetattrr%rselfconfig __class__s d/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/lilt/modeling_lilt.pyr(zLiltTextEmbeddings.__init__-s2 !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$"..#%<<  * *F,>,>DL\L\$  cD|I|6|j||jj|j}n|j |}||j }n|j dd}|:t j|t j|jj}||j|}|j|}||z}|jdk(r|j|}||z }|j|}|j|}||fS)Nr#dtypedevicer&)"create_position_ids_from_input_idsrtorE&create_position_ids_from_inputs_embedssizer8zeroslongr"r-r1r%r/r2r6) r= input_idstoken_type_idsr" inputs_embeds input_shaper1 embeddingsr/s r@forwardzLiltTextEmbeddings.forwardDs!  $#FFyRVRbRbcff$$   $JJ=Y  #..*K',,.s3K  !"[[EJJtO`O`OgOghN   00;M $ : :> J"%::  ' ': 5"&":":<"H  - -J^^J/ \\*- <''rAc|j|j}tj|dj ||z}|j |zS)a Args: Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. x: torch.Tensor x: Returns: torch.Tensor rdim)neintr8cumsumtype_asrK)r=rLrmaskincremental_indicess r@rFz5LiltTextEmbeddings.create_position_ids_from_input_idshsP||K(,,.$||Da8@@F$N"'')K77rAc|jdd}|d}tj|jdz||jzdztj|j }|j dj|S)z Args: We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids.: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr#rrCr)rIr8r9rrKrE unsqueezer:)r=rNrOsequence_lengthr"s r@rHz9LiltTextEmbeddings.create_position_ids_from_inputs_embedsus$((*3B/ %a.||   q /D4D4D"Dq"HPUPZPZcpcwcw %%a(// <">|"L&AD[&[#&*nn5P&Q#&*ll3N&O#**3 cZ[ab b csA,D&& E/ D;;E)NN)r^r_r`r(rQrarbs@r@rdrds >*+rArdc6eZdZdfd ZddZ ddZxZS)LiltSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|jz|j|jz|_tj|j|jz|j|jz|_tj|j|jz|j|jz|_tj$|j&|_|xs t+|dd|_|j,dk(s|j,d k(rF|j.|_tj0d |j.zd z |j|_|j|_y) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r%r& relative_keyrelative_key_queryrsr)r'r(r+num_attention_headshasattr ValueErrorrVattention_head_size all_head_sizerrpquerykeyvaluern layout_query layout_key layout_valuer4attention_probs_dropout_probr6r;r%r.r)distance_embedding)r=r>r%r?s r@r(zLiltSelfAttention.__init__sN     : : :a ?PVXhHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #$*$?$?!rAc|jdd|j|j|zfz}|j|}|j ddddS)Nr#rrsrr )rIrrviewpermute)r=xr new_x_shapes r@transpose_for_scoresz&LiltSelfAttention.transpose_for_scoressTffhsmt'?'?AYAY]^A^&__ AFFK yyAq!$$rAc|j|j||j}|j|j||j}|j|j ||j}|j |} |j|j |} |j|j|} |j| } tj| | jdd} tj||jdd}|jdk(s|jdk(rF|jd}tj|tj|jj!dd}tj|tj|jj!dd}||z }|j#||j$zdz }|j'| j(}|jdk(rtj*d | |}| |z} nE|jdk(r6tj*d | |}tj*d | |}| |z|z} | t-j.|j0z }|t-j.|j0|jzz }||z} ||z}|||z}t3j4d |}|j7|}|||z}tj||}|j9d d ddj;}|jdd|j<|jzfz}|j |}|| |z} t3j4d | }|j7|}|||z}tj|| }|j9d d ddj;}|jdd|j<fz}|j |}|r||f|f}|S||ff}|S)N)rr#rrrrC)rDzbhld,lrd->bhlrzbhrd,lrd->bhlrrSrrsr )rrrnrrrrrr8matmul transposer%rIr9rKrErrr.rGrDeinsummathsqrtrrSoftmaxr6r contiguousr)r= hidden_states layout_inputsattention_mask head_maskoutput_attentionslayout_value_layerlayout_key_layerlayout_query_layermixed_query_layer key_layer value_layer query_layerattention_scoreslayout_attention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keytmp_attention_scorestmp_layout_attention_scoreslayout_attention_probslayout_context_layernew_context_layer_shapeattention_probs context_layeroutputss r@rQzLiltSelfAttention.forwardsd"66t7H7H7W[_[t[t6u44T__]5SW[WpWp4q!66t7H7H7W[_[t[t6u JJ}5--dhh}.EF // =0IJ //0AB  << Y5H5HR5PQ"',,/ACSC]C]^`bdCe"f  ' '> 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||,,7  - 9O _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD !1 2O D  !"679  rAN)rNNF)r^r_r`r(rrQrarbs@r@r~r~s "@H% \rAr~cneZdZfdZdej dej dej fdZxZS)LiltSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr ) r'r(rrpr+denser2r3r4r5r6r<s r@r(zLiltSelfOutput.__init__Hs` YYv1163E3EF f&8&8f>S>STzz&"<"<= rAr input_tensorreturncr|j|}|j|}|j||z}|Srrr6r2r=rrs r@rQzLiltSelfOutput.forwardN7 =1  ]3 }|'CD rAr^r_r`r(r8TensorrQrarbs@r@rrG1> U\\RWR^R^rArceZdZd fd ZdZ d dej dej deejdeejdee de ej f d Z xZ S) LiltAttentionct|t|||_t ||_t |_|j}|j|jz|_t ||_ ||_y)N)r%) r'r(r~r=routputset pruned_headsr+rn layout_output)r=r>r%ori_hidden_sizer?s r@r(zLiltAttention.__init__Vsl %fF]^ $V, E ,,#//63N3NN+F3,rAc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)NrrrS)lenrr=rrrrrrrrrrunion)r=headsindexs r@ prune_headszLiltAttention.prune_headsbs u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rArrrrrrc|j|||||}|j|dd|}|j|dd|}||ff|ddz} | S)Nrr)r=rr) r=rrrrr self_outputsattention_outputlayout_attention_outputrs r@rQzLiltAttention.forwardtszyy        ;;|Aq'9=I"&"4"4\!_Q5G"W$&=>@  u|| rArcVeZdZfdZdej dej fdZxZS)LiltIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r'r(rrpr+intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr<s r@r(zLiltIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rArrcJ|j|}|j|}|Sr)rr)r=rs r@rQzLiltIntermediate.forwards& =1 00? rArrbs@r@rrs#9U\\ellrArcneZdZfdZdej dej dej fdZxZS) LiltOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r'r(rrprr+rr2r3r4r5r6r<s r@r(zLiltOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rArrrcr|j|}|j|}|j||z}|Srrrs r@rQzLiltOutput.forwardrrArrbs@r@rrrrArceZdZfdZ d dej dej deejdeejdeede ej f dZ d Z d Z xZ S) LiltLayerct||j|_d|_t ||_t ||_t||_ |j}|j}|j|jz|_ |j|jz|_ t ||_ t||_||_ ||_ y)Nr)r'r(chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr+rrnlayout_intermediater)r=r>rori_intermediate_sizer?s r@r(zLiltLayer.__init__s '-'E'E$&v.,V4 (  ,, & 8 8#//63N3NN#)#;#;v?Z?Z#Z #3F#; '/,#8 rArrrrrrc|j|||||}|dd}|dd}|dd} t|j|j|j|} t|j |j|j|} | | ff| z} | S)N)rrr)rrfeed_forward_chunkrrlayout_feed_forward_chunk) r=rrrrrself_attention_outputsrrr layer_outputlayout_layer_outputs r@rQzLiltLayer.forwards"&    / "0" 2!4Q7"8";A">(,0  # #T%A%A4CSCSUe 8  * *D,H,H$JZJZ\s !"5687BrAcL|j|}|j||}|Sr)rrr=rintermediate_outputrs r@rzLiltLayer.feed_forward_chunks,"//0@A{{#68HI rAcL|j|}|j||}|Sr)rrrs r@rz#LiltLayer.layout_feed_forward_chunks."667GH))*=?OP rAr)r^r_r`r(r8rrrrrrQrrrarbs@r@rrs9*7;15,1 ||||!!2!23  E--.  $D>  u|| > rArceZdZfdZ d dej dej deejdeejdeedeedeed e e ej e ffd Z xZ S) LiltEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) r'r(r>r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r=r>_r?s r@r(zLiltEncoder.__init__sN  ]]uVE]E]?^#_!If$5#_` &+#$`sA#rrrrroutput_hidden_states return_dictrc |rdnd}|rdnd} t|jD]w\} } |r||fz}||| nd} |jr-|jr!|j | j |||| |} n | |||| |} | dd}| dd}|so| | dfz} y|r||fz}|st d||| fDSt||| S)Nrrc3$K|]}|| ywrr).0vs r@ z&LiltEncoder.forward..s = s)last_hidden_stater attentions) enumerater r training_gradient_checkpointing_func__call__tupler )r=rrrrrr r all_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r@rQzLiltEncoder.forwards4#7BD$5b4(4 POA|#$58H$H!.7.CilO**t}} $ A A ))!!"#% ! !-!!"#% ! *!,Q/M)!,Q/M &9]1=M < 'tn< d^<  uU\\"O3 4< rArcVeZdZfdZdej dej fdZxZS) LiltPoolerct|tj|j|j|_tj |_yr)r'r(rrpr+rTanh activationr<s r@r(zLiltPooler.__init__-s9 YYv1163E3EF '')rArrc\|dddf}|j|}|j|}|SNr)rr%)r=rfirst_token_tensor pooled_outputs r@rQzLiltPooler.forward2s6+1a40 #56  6 rArrbs@r@r"r",s#$ U\\ellrAr"c&eZdZdZeZdZdZgZdZ y)LiltPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. liltTct|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyy)zInitialize the weightsg)meanstdNg?)rrrpweightdatanormal_r>initializer_rangebiaszero_r)rr2fill_)r=modules r@ _init_weightsz!LiltPreTrainedModel._init_weightsFs fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).rAN) r^r_r`__doc__r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modulesr8rrAr@r+r+;s% L&*#*rAr+a= This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`LiltConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a- Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) bbox (`torch.LongTensor` of shape `({0}, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare LiLT Model transformer outputting raw hidden-states without any specific head on top.ceZdZdfd ZdZdZdZeejde e e  dde ejde ejd e ejd e ejd e ejd e ejd e ejde ede ede edeeeje ffdZxZS) LiltModelct||||_t||_t ||_t||_|r t|nd|_ |jyr) r'r(r>rrPrdlayout_embeddingsrencoderr"pooler post_init)r=r>add_pooling_layerr?s r@r(zLiltModel.__init__sX   ,V4!5f!="6* ,=j(4  rAc.|jjSrrPr-)r=s r@get_input_embeddingszLiltModel.get_input_embeddingss...rAc&||j_yrrG)r=rs r@set_input_embeddingszLiltModel.set_input_embeddingss*/'rAc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrBr rr)r=heads_to_pruner rs r@ _prune_headszLiltModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B CrAbatch_size, sequence_length output_typer:rLrvrrMr"rrNrr r rc ||n|jj}| | n|jj} | | n|jj} | | t d|#|j |||j } n!||j dd} n t d| \} } | |jn |j}|)tj| dztj|}|tj| | f|}|pt|jdr4|jjddd| f}|j| | }|}n&tj| tj|}|j!|| }|j#||jj$}|j|||| \}}|j'|| }|j)|||||| | }|d }|j*|j+|nd}| s ||f|d dzSt-|||j.|j0S)a Returns: Examples: ```python >>> from transformers import AutoTokenizer, AutoModel >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModel.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer#z5You have to specify either input_ids or inputs_embeds)rC)rErM)rLr"rMrN)rvr")rrrr r rr)r pooler_outputrr)r>rr use_return_dictr%warn_if_padding_and_no_attention_maskrIrEr8rJrKonesrrPrMr:get_extended_attention_mask get_head_maskrrArBrCr rr)r=rLrvrrMr"rrNrr r rO batch_sizerrEbuffered_token_type_ids buffered_token_type_ids_expandedextended_attention_maskembedding_outputlayout_embedding_outputencoder_outputssequence_outputr)s r@rQzLiltModel.forwardssL2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T <;;{T1FSD  !"ZZ*j)A6RN  !t(89*.//*H*HKZK*X'3J3Q3QR\^h3i0!A!&[ SY!Z150P0PQ_al0m&&y$++2O2OP )-%)' *9* &,#'"8"8dQ]"8"^,,  #2/!5#' *!,8< 8O O4UY #]3oab6II I)-')77&11   rA)T) NNNNNNNNNN)r^r_r`r(rHrJrNrLILT_INPUTS_DOCSTRINGformatrr _CONFIG_FOR_DOCrr8rrrrrQrarbs@r@r?r?sI /0C++@+G+GHe+fg+ETcd-1'+1515/3,004,0/3&*m ELL)m u||$m !. m !. m u||, m ELL)m   -m $D>m 'tnm d^m  uU\\"$>> ?m ehm rAr?z LiLT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZeej deee  dde e jde e jde e jde e jde e jd e e jd e e jd e e jd e ed e ede edeee jeffdZxZS)LiltForSequenceClassificationct|||j|_||_t |d|_t ||_|jyNF)rE) r'r( num_labelsr>r?r,LiltClassificationHead classifierrDr<s r@r(z&LiltForSequenceClassification.__init__:sJ   ++ f> 08 rArOrPrLrvrrMr"rrNlabelsrr r rc V| | n|jj} |j|||||||| | |  } | d} |j| }d}||j |j }|jj |jdk(rd|j_nl|jdkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt}|jdk(r&||j|j}n|||}n|jj dk(r=t}||jd|j|jd}n,|jj dk(rt!}|||}| s|f| d dz}||f|zS|St#||| j$| j& S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForSequenceClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_idx = outputs.logits.argmax(-1).item() >>> predicted_class = model.config.id2label[predicted_class_idx] ```N rvrrMr"rrNrr r rr regressionsingle_label_classificationmulti_label_classificationr#rslosslogitsrr)r>rUr,rkrGrE problem_typerirDr8rKrVr squeezerrrrrrr=rLrvrrMr"rrNrlrr r rrartrsloss_fctrs r@rQz%LiltForSequenceClassification.forwardEsX&1%] 'tn] d^]  uU\\"$<< =] ch] rArfz Lilt Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. ceZdZfdZeej deee  dde e jde e jde e jde e jde e jd e e jd e e jd e e jd e ed e ede edeee j"effdZxZS)LiltForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyrh)r'r(rir?r,classifier_dropoutr5rr4r6rpr+rkrDr=r>r~r?s r@r(z#LiltForTokenClassification.__init__s   ++f> )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rArOrPrLrvrrMr"rrNrlrr r rc | | n|jj} |j|||||||| | |  } | d} |j| } |j | }d}|W|j |j }t}||jd|j|jd}| s|f| ddz}||f|zS|St||| j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Returns: Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForTokenClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForTokenClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_indices = outputs.logits.argmax(-1) ```Nrnrr#rsrr) r>rUr,r6rkrGrErrrirrrrws r@rQz"LiltForTokenClassification.forwards R&1%K 'tnK d^K  uU\\"$99 :K `hK rAr|c(eZdZdZfdZdZxZS)rjz-Head for sentence-level classification tasks.cZt|tj|j|j|_|j |j n |j}tj||_ tj|j|j|_ yr) r'r(rrpr+rr~r5r4r6riout_projrs r@r(zLiltClassificationHead.__init__s YYv1163E3EF )/)B)B)NF % %TZTnTn zz"45  &"4"4f6G6GH rAc |dddddf}|j|}|j|}tj|}|j|}|j |}|Sr')r6rr8tanhr)r=featureskwargsrs r@rQzLiltClassificationHead.forwardsY Q1W  LLO JJqM JJqM LLO MM! rA)r^r_r`r9r(rQrarbs@r@rjrjs7IrArjz Lilt Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZfdZeej deee  dde e jde e jde e jde e jde e jd e e jd e e jd e e jd e e jd e ede ede edeee j"effdZxZS)LiltForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyrh) r'r(rir?r,rrpr+ qa_outputsrDr<s r@r(z!LiltForQuestionAnswering.__init__.sU   ++f> ))F$6$68I8IJ rArOrPrLrvrrMr"rrNstart_positions end_positionsrr r rc *| | n|jj} |j|||||||| | |  } | d}|j|}|j dd\}}|j dj }|j dj }d}|| t|jdkDr|j d}t| jdkDr| j d} |jd}|jd|}| jd|} t|}|||}||| }||zdz }| s||f| ddz}||f|zS|St|||| j| j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Returns: Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForQuestionAnswering >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForQuestionAnswering.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> predicted_answer = tokenizer.decode(predict_answer_tokens) ```Nrnrrr#rS) ignore_indexrs)rs start_logits end_logitsrr)r>rUr,rsplitrvrrrIclamprrrr)r=rLrvrrMr"rrNrrrr r rrartrr total_loss ignored_indexrx start_lossend_lossrs r@rQz LiltForQuestionAnswering.forward8sj&1%d 'tnd d^d  uU\\"$@@ Ad ghd rAr)rrfr|r?r+):r9rtypingrrrr8torch.utils.checkpointrtorch.nnrrr activationsr modeling_outputsr r rrrmodeling_utilsr pytorch_utilsrrrutilsrrrrconfiguration_liltr get_loggerr^loggerrdModulerrdr~rrrrrrr"r+LILT_START_DOCSTRINGrbr?rfr|rjr__all__rrAr@rs )) AA!.lltt*   H %V=V=r5+2995+pF FT RYY 1BII1j ryy   9 9xD "))D P  */*8 6rdK #K K \ l $7l l ^ ] !4] ] BRYY, r 2r r j rA