T^(h7dZddlZddlZddlmZmZmZmZddlZ ddl Z ddl Z ddl m Z ddl mZddlmZddlmZdd lmZmZdd lmZmZmZmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#d dl$m%Z%e"jLe'Z(dZ)dZ*de jVde,de,fdZ-Gdde j\Z/Gdde j`Z1Gdde j`Z2de1iZ3Gdde j`Z4GddeZ5d Z6d!Z7d"Z8Gd#d$e5Z9Gd%d&e5Z:e d'e6Gd(d)e5Z;e d*e6Gd+d,e5eZ<Gd-d.e5Z=Gd/d0e5eZ>gd1Z?y)2z=PyTorch MarianMTModel model, ported from the Marian C++ repo.N)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)GenerationMixin)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)add_end_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) MarianConfigrzHelsinki-NLP/opus-mt-en-de input_ids pad_token_iddecoder_start_token_idc|j|j}|ddddfj|ddddf<||dddf<| td|j |dk(||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrrshifted_input_idss h/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/marian/modeling_marian.pyshift_tokens_rightr'7s}"++IOO<(CRC0668ae4adLMM""#4#arangelongr9rOr1forward)r3rLrMbszseq_len positionsr4s r&rRz+MarianSinusoidalPositionalEmbedding.forward\sT'r* WLL "$:W$DEJJ_c_j_j_q_q wy))r(r0)r)__name__ __module__ __qualname____doc__intrr2rKr>no_gradSizeTensorrR __classcell__r4s@r&r*r*GsnN7c7#7HUXM7ei7 =U]]_*uzz*3*W\WcWc**r(r*ceZdZdZ ddededededededeeffd Z d e jd ed efd Z dde jdee jdee e jdee jdee jdede e jee jee e jffdZxZS)MarianAttentionz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsdropout is_decoderbias is_causalconfigc t|||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ tj||||_ tj||||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rf)r1r2rbrcrdhead_dimrhr#scalingrergrLineark_projv_projq_projout_proj) r3rbrcrdrerfrgrhr4s r&r2zMarianAttention.__init__js "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r(tensorrTrSc|j|||j|jjddj S)Nrr6)viewrcrk transpose contiguous)r3rrrTrSs r&_shapezMarianAttention._shapes7{{3GQQRSUVWbbddr( hidden_stateskey_value_statespast_key_valueattention_masklayer_head_maskoutput_attentionsr.c |du}|j\}} } |j||jz} |r0|.|djd|jdk(r |d} |d} n |rE|j |j |d|} |j |j |d|} n|}|j |j |d|} |j |j |d|} tj|d| gd} tj|d| gd} nD|j |j |d|} |j |j |d|} |jr| | f}||jzd|jf}|j | | |j|} | j|} | j|} | jd}tj| | jdd}|j||jz| |fk7r/t!d||jz| |fd|j|{|j|d| |fk7r#t!d |d| |fd|j|j||j| ||z}|j||jz| |}t"j$j'|d}||j|jfk7r*t!d |jfd|j|jdddd|j||j| |z}|j||jz| |}|r?|j||j| |}|j||jz| |}nd}t"j$j)||j(|j* }tj|| }|j||jz| |jfk7r9t!d ||jz| |jfd|j|j||j| |j}|jdd}|j|| |j,}|j/|}|||fS) z#Input shape: Batch x Time x ChannelNrr6rrrEz$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )sizerprlr!rwrnror>catrercrkrtreshapebmmrur#r functionalsoftmaxrdrrbrq)r3rxryrzr{r|r}is_cross_attentionrStgt_len_ query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r&rRzMarianAttention.forwards.T9',,.Wa{{=1DLL@ *q!''*.>.D.DQ.GG(*J)!,L T[[1A%BBLJ;;t{{3C'Db#NL  'T[[%?SIJ;;t{{='A2sKLN1$5z#BJJ 99nQ&7%FANLT[[%?SIJ;;t{{='A2sKL ??),7NDNN*B > Ct{{<#>CCZP 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVZVdVdfmov?wwL',,S4>>-A7GTL  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<AAr()FTFN)NNNNF)rVrWrXrYrZfloatboolrrr2r>r]rwrrRr^r_s@r&raragsMG  )-CCC C  C  CC&C>eU\\eCece 488<1526"'vB||vB#5<<0vB!u||!45 vB !. vB "%,,/ vB vB u||Xell3XeELL>Q5RR SvBr(raceZdZdeffd Z d dej dej dej deede ej eej ff dZ xZ S) MarianEncoderLayerrhct||j|_t |j |j|j |j||_tj|j|_ |j|_ t|j|_|j |_tj"|j|j$|_tj"|j$|j|_tj|j|_y)N)rbrcrdrh)r1r2d_modelrbMARIAN_ATTENTION_CLASSES_attn_implementationencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrdr activation_function activation_fnactivation_dropoutrmencoder_ffn_dimfc1fc2final_layer_normr3rhr4s r&r2zMarianEncoderLayer.__init__s 1&2M2MNnn44,,   %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r(rxr{r|r}r.c|}|j||||\}}}tjj||j|j}||z}|j |}|}|j |j|}tjj||j|j}|j|}tjj||j|j}||z}|j|}|jtjk(rtj|js#tj |jrEtj"|jj$dz }tj&|| |}|f} |r| |fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rxr{r|r}ri)minmax)rrrrdrrrrrrrr7r>float16isinfanyisnanfinforclamp) r3rxr{r|r}residualrr clamp_valueoutputss r&rRzMarianEncoderLayer.forwards$!)-')+/ *8* & |Q  --mt||VZVcVc-d  =0 11-@  **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0 --m<   %-- / KK & * * , M0J0N0N0P++m&9&9:>>EK!KK K<[YM "    &Gr()F) rVrWrXrr2r>r@rrrrRr^r_s@r&rrsz=|=.-2 0((0))0** 0 $D> 0 u  (5+<+<"== > 0r(reagerc~eZdZdeffd Z ddej deej deej deej deej deej d eeej d ee d ee d eejeeejejfffd Z xZ S)MarianDecoderLayerrhcNt||j|_t |j |j|j |jdd||_|j|_ t|j|_ |j|_ tj|j|_t |j |j|j |jd||_tj|j|_tj&|j|j(|_tj&|j(|j|_tj|j|_y)NT)rbrcrdrergrh)rdrerh)r1r2rrbrrdecoder_attention_headsrrrdr rrrrrr encoder_attnencoder_attn_layer_normrmdecoder_ffn_dimrrrrs r&r2zMarianDecoderLayer.__init__Qs0 1&2M2MNnn44,,  ~~ #F$>$>?"(";";$&LL$@!4V5P5PQ NN  * *,,  (*||DNN'C$99T^^V-C-CD99V33T^^D " T^^ <r(rxr{encoder_hidden_statesencoder_attention_maskr|cross_attn_layer_head_maskrzr} use_cacher.c t|} ||ddnd} |j|| |||\}} } tjj||j|j}| |z}|j |}d}d}|w|} ||ddnd}|j ||||||\}}}tjj||j|j}| |z}|j|}| |z} |} |j|j|}tjj||j|j}|j|}tjj||j|j}| |z}|j|}|f}|r|| |fz }| r|| fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. Nr6)rxrzr{r|r}r)rxryr{r|rzr}) rrrrdrrrrrrrrr)r3rxr{rrr|rrzr}rrself_attn_past_key_valueself_attn_weightspresent_key_valuecross_attn_present_key_valuecross_attn_weightscross_attn_past_key_valuers r&rRzMarianDecoderLayer.forwardns <!:H9S>"1#5Y] >Bnn'3)+/ ?M? ; (*; --mt||VZVcVc-d  =0 11-@ (,$! ,$H@N?Yrs(;_c %NRN_N_+!65 :8"3 O`O KM-/KMM11-4< >G  )+ +Gr()NNNNNNFT) rVrWrXrr2r>r]rrrr@rRr^r_s@r&rrPs=|=@268<9=26=A8<,1$(W||W!.W ( 5 W !) 6 W "%,,/ W%-U\\$:W!u||!45W$D>WD>W u  (51B1BEDUDU1U+V"WW XWr(rcheZdZeZdZdZdeejeje ffdZ e dZy)MarianPreTrainedModelmodelTmodulecX|jj}t|tjrY|j j jd||j%|jj jyyt|tr|jyt|tjrf|j j jd||j2|j j |jjyyy)Nr)meanstd)rhinit_std isinstancerrmr9datanormal_rfzero_r*rK Embeddingr-)r3rrs r& _init_weightsz#MarianPreTrainedModel._init_weightsskk"" fbii ( MM   & &CS & 9{{&   &&('  C D    !  - MM   & &CS & 9!!- ""6#5#56<<>..r(c|jj}tjgddddd|gg|j}|j |||d}|S)N)r r6r r6rO)r{rdecoder_input_ids)rhrr>rrrOne)r3 pad_tokenr dummy_inputss r&rz"MarianPreTrainedModel.dummy_inputssZKK,, LL"2Q2q)4L!MVZVaVab 'll95"!* r(N)rVrWrXr config_classbase_model_prefixsupports_gradient_checkpointingrrrmrr*rpropertyrr(r&rrsHL&*# ?E"))R\\Cf*f$g ?r(raK This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MarianConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. u4 Pytorch version of marian-nmt's transformer.h (c++). Designed for the OPUS-NMT translation checkpoints. Available models are listed [here](https://huggingface.co/models?search=Helsinki-NLP). Examples: ```python >>> from transformers import AutoTokenizer, MarianMTModel >>> src = "fr" # source language >>> trg = "en" # target language >>> model_name = f"Helsinki-NLP/opus-mt-{src}-{trg}" >>> model = MarianMTModel.from_pretrained(model_name) >>> tokenizer = AutoTokenizer.from_pretrained(model_name) >>> sample_text = "où est l'arrêt de bus ?" >>> batch = tokenizer([sample_text], return_tensors="pt") >>> generated_ids = model.generate(**batch) >>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] "Where's the bus stop?" ``` ah Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Marian uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, *optional*: `hidden_states`, *optional*: `attentions`) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. c*eZdZdZddedeejffd ZdZ dZ ddee jdee jd ee jd ee jd eed eed eedeee jeffdZxZS) MarianEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`MarianEncoderLayer`]. Args: config: MarianConfig embed_tokens (nn.Embedding): output embedding rh embed_tokensct|||j|_|j|_|j }|j |_|j|_ |jrtj|nd|_ |||_n0tj |j"||j|_t%|j||j|_tj(t+|j,Dcgc] }t/|c}|_d|_|j5ycc}wNg?F)r1r2rdencoder_layerdrop layerdroprrr-max_position_embeddingsmax_source_positionsscale_embeddingmathsqrt embed_scalerrr vocab_sizer*embed_positions ModuleListr<encoder_layersrlayersgradient_checkpointing post_init)r3rhrrbrr4s r&r2zMarianEncoder.__init__ts  ~~ 11NN !..$*$B$B!393I3I499Y/s  # ,D  " V->-> 4K[K[ \D B  * *It7G7G mmvOdOdIe$fA%7%?$fg &+#  %gsEc|jSr0rr3s r&get_input_embeddingsz"MarianEncoder.get_input_embeddings   r(c||_yr0rr3values r&set_input_embeddingsz"MarianEncoder.set_input_embeddings !r(rr{ head_mask inputs_embedsr}output_hidden_states return_dictr.c(||n|jj}||n|jj}||n|jj}| | t d|8|j |||j }|jd|d}n!||j dd}n t d||j||jz}|j|} || z} tjj| |j|j} |t||j }|rdnd} |rdnd} |[|j dt#|j$k(s2Jdt#|j$d |j dd t'|j$D]\} }|r| | fz} d }|jr&t)j*g}||j,krd }|rd }nW|j.r3|jr'|j1|j2| |||| nd|}n|| |||| nd|}|d} |s| |dfz} |r| | fz} |st5d| | | fDSt7| | | S)a~ Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrrrz&The head_mask should be specified for  layers, but it is for .FT)NN)r|r}rc3&K|] }|| ywr0r.0vs r& z(MarianEncoder.forward..seqWXWdeslast_hidden_staterx attentions)rhr}r use_return_dictr#%warn_if_padding_and_no_attention_maskrrtrrrrrrdrr r7lenr enumerater>randrr_gradient_checkpointing_func__call__tupler)r3rr{rrr}r r  input_shape embed_posrxencoder_statesall_attentionsidx encoder_layerto_dropdropout_probability layer_outputss r&rRzMarianEncoder.forwards\2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K!r;r?;I  &',,.s3KTU U   --i84;K;KKM((5 % 1  --mt||VZVcVc-d   %7 H[H[\N30d  >>#A&3t{{+;< 8T[[9I8JJabkbpbpbrstbuavvwx <#,DKK"8 F C#!/=2B!BG}}&+jjn#&7"G , ..4==$($E$E%..%&+4+@3d) %M%2%&;D;P3VZ*; %M!.a 0  !/=3C2E!EA FD +}.>>Ne]NN$See e+>Vd  r(r0)NNNNNNN)rVrWrXrYrrrrr2rrr> LongTensorr]r@rrrrrRr^r_s@r&rrjs|8BLL;Q2!" 1559,059,0/3&*~ E,,-~ !!1!12~ ELL) ~   1 12 ~ $D> ~ 'tn~ d^~  uU\\"O3 4~ r(rceZdZdZddedeejffd ZdZ dZ ddee jdee jd ee jd ee jd ee jd ee jd eeee jdee jdeedeedeedeedeee jeffdZxZS) MarianDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MarianDecoderLayer`] Args: config: MarianConfig embed_tokens (nn.Embedding): output embedding rhrct|||j|_|j|_|j |_|j|_|jrtj|jnd|_ |||_n:tj |j"|j|j |_t%|j|j|j |_tj(t+|j,Dcgc] }t/|c}|_d|_|j5ycc}wr)r1r2rddecoder_layerdroprrr-rmax_target_positionsrrrrrrrrdecoder_vocab_sizer*rrr<decoder_layersrrrr)r3rhrrr4s r&r2zMarianDecoder.__init__s  ~~ 11!..$*$B$B!8>8N8N499V^^4TW  # ,D  " V-F-FX\XhXh iD B  * *FNND