T^(h|dZddlZddlZddlZddlmZddlmZmZm Z m Z m Z ddl Z ddl m Z ddlmZmZmZddlmZmZdd lmZdd lmZmZmZdd lmZmZdd lmZm Z m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*e'jVe,Z-dZ.dZ/dZ0e jbeeedZ2Gdde jfZ4Gdde jfZ5Gdde jfZ6GddeZ7eGdde#Z8dZ9dZ:e%d e9Gd!d"e7Z;e%d#e9Gd$d%e7eZ<e%d&e9Gd'd(e7Z=e%d)e9Gd*d+e7Z>gd,Z?y)-zPyTorch OpenAI GPT model.N) dataclass)AnyDictOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)gelu_newsilu)GenerationMixin)BaseModelOutputCausalLMOutputSequenceClassifierOutput)PreTrainedModelSequenceSummary)Conv1D find_pruneable_heads_and_indicesprune_conv1d_layer) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)OpenAIGPTConfigzopenai-community/openai-gptr c ddl}ddl}d|vrtjj |}t j d|t|dzdd5}tj|}dddt|d zdd5}tj|}ddd|jD cgc]} |j| c} } td D cgc]} |j|d | d z} } |j|j| d| dd } t| |D cgc]\} } | j!| } } } | Dcgc]}|j#} }|j$j&j(| dj(k7r>+q97CCRHK*>>)%*<*<*C*C*I*I)JKA$$% '   ##))[^-A-AA,U-B-B-I-I-O-O,PQA$$% '  &+%5%5k!n%EE"(-(8(8Q(HE  % IIaLOOAOOA5+./ eABx 9 vdV+@AB BCRyzz# 'F||OV4 hhx8 %h 1~$!'84Q3&!'62Q3&!'84!';q>:;1$+a.)!#, '$ ==EKK '~gmm_@Q@Q uYYut'>T__I\@]^_ (jaH (eC ??dkk9dkkCPUJ>VWkkCJ.  --33E:rctj||}|jr'|tj|j dz }|j ddddd|j dd|j df}||zdd|z zz}|||z}tjj|d}|j|}|||z}tj||g} |r| j|| S)Nr%r'grr) rEmatmulrrmathsqrtsizer,r functionalsoftmaxrwappend) r|qkvattention_mask head_maskoutput_attentionsr-r+outputss r`_attnzAttention._attns LLA  ::DIIaffRj))A IIaLaffRjL,AFF2J,6 7 EDAEN "  %N"A MM ! !! ! ,   a   I A<<1%&  NN1 rc|jddddj}|jdd|jd|jdzfz}|j|S)Nrr.rr r'r%)permute contiguousrrp)r|x new_x_shapes r` merge_headszAttention.merge_headss^ IIaAq ! , , .ffhsmqvvbzAFF2J'>&@@ qvv{##rc|jdd|j|jd|jzfz}|j|}|r|jddddS|jddddS)Nr%rr.r r)rrlrpr)r|rrrs r` split_headszAttention.split_headssoffhsmt{{AFF2J$++4M&NN AFFK  99Q1a( (99Q1a( (rc|j|}|j|jd\}}}|j|}|j|d}|j|}|j ||||||}|d} |j | } |j | } |j| } | g|ddz} | S)Nr.rT)rrr)rsr<rqrrrrtry) r|rrrrquerykeyvalue attn_outputsars r`forwardzAttention.forwards KKNGGDOOG;sE  'sd+  'zz%e^YPab O   Q  KKN   q !# QR((rFNNF) __name__ __module__ __qualname__rkrrrrr __classcell__rs@r`rfrfs!"* ;2$ )rrfc$eZdZfdZdZxZS)MLPct||j}t|||_t|||_t |j|_tj|j|_ yN) rjrkn_embdrc_fcrtACT_FNSafnactr rurxdropout)r|rrNr}rs r`rkz MLP.__init__sY  ]]7B' R) 6::&zz&"4"45 rc|j|j|}|j|}|j|Sr)rrrtr)r|rhh2s r`rz MLP.forwards4 HHTYYq\ " [[^||Brrrrrkrrrs@r`rrs 6 rrc(eZdZdfd ZddZxZS)Blockc t||j}t|||||_t j ||j|_td|z||_ t j ||j|_ y)N)eps) rjrkrrfattnr LayerNormlayer_norm_epsilonln_1rmlpln_2)r|r~rNrrr}rs r`rkzBlock.__init__sk  ]]b+vu= LL)B)BC q2vv&LL)B)BC rc|j||||}|d}|j||z}|j|}|j||z} | g|ddz} | S)N)rrrrr)rrrr) r|rrrrrrrVmrrs r`rz Block.forwardsuyy )/ ! O IIa!e  HHQK IIa!e # QR((rrrrrs@r`rrsDrrc"eZdZdZeZeZdZdZ y)OpenAIGPTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformerct|tjtfrm|jj j d|jj|j%|jj jyyt|tjrz|jj j d|jj|j2|jj |jjyyt|tjrJ|jj j|jj jdyy)zInitialize the weights.g)meanstdN?) isinstancer Linearrr*rGnormal_rNinitializer_ranger,zero_ Embedding padding_idxrfill_)r|modules r` _init_weightsz&OpenAIGPTPreTrainedModel._init_weightss  fryy&1 2 MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).rN) rrr__doc__r config_classraload_tf_weightsbase_model_prefixrrr`rr s #L3O%*rrceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ ee ejed<dZee ejed<y) OpenAIGPTDoubleHeadsModelOutputa Base class for outputs of models predicting if two sentences are consecutive or not. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. mc_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mc_labels` is provided): Multiple choice classification loss. logits (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). mc_logits (`torch.FloatTensor` of shape `(batch_size, num_choices)`): Prediction scores of the multiple choice classification head (scores for each choice before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossmc_losslogits mc_logits hidden_states attentions)rrrrrrrE FloatTensor__annotations__rrrrrrrrr`rr$s2)-D(5$$ %,+/GXe'' (/*.FHU&& '.-1Ix))*18