$%h؉UddlmZddlZddlZddlmZmZmZmZm Z ddl m Z ddl m ZGdde Zed d Zd*d Zd+d Zddgfd,dZd-d.dZddgfd,dZd*dZd*dZd*dZd*dZd*dZd*dZd/dZddgf d0dZd*dZd1dZd2dZ d3dZ!d4dZ"d5d Z#d6d!Z$d7d"Z%d8d#Z&d9d:d$Z'd%Z(d&e)d'<d;d(Z* d.lower_case_column,sT::BaQ6)ABB 47FLLN3T B Cs !AANc|SNr)r%r2necessary_columns rlower_case_column_creatorz=necessary_column_validator..lower_case_column_creator9s(-=>>rz - The `z ` column/key should be lowercasezLower case column name to ``z^` column/key is missing. Please make sure you name your columns/keys appropriately, then retryr5)rrrrr)r% pd.DataFramer/rreturnr8)r%r8r9r8)r+rr-r ) r%r5rrrrr0r6r2s ` @rnecessary_column_validatorr:'s  MLMIrzz)  C1A C C ?5L'(8'99YZM9:J9K1MM,-.LMI  ##!   Ds A<prompt completioncFg}d}d}d}t|jdkDrh|jDcgc] }|vs| }}d}|D]1}|Dcgc] }||vs| } }t| dkDs&|d|d|dz }3d||}d |}d fd }td ||| Scc}wcc}w)zK This validator will remove additional columns from the dataframe. Nr r!rz9 WARNING: Some of the additional columns/keys contain `z<` in their name. These will be ignored, and the column/key `z`` will be used instead. This could also result from a duplicate column/key in the provided file.zh - The input file should contain exactly two columns/keys per row. Additional columns/keys present are: z Remove additional columns/keys: c|Sr4rxfieldss rrz1additional_column_validator..necessary_fn^s V9 radditional_columnrrrrr@rr9r)r$r+r ) r%rAadditional_columnsrrrr0 warn_messageacdupss ` radditional_column_validatorrIKsMML 2::)+GAqaGG $ BB1=!R1WA=D=4y1}"\]_\`a]^`]aaA!BB  BDEWDXYeXfg :;M:NO    ##!  H>s BB BBcd}d}d}|jdjs!|jjrg|dk(|jz}|jj|j }dd|}d fd }dt |dd }td ||| S) zA This validator will ensure that no completion is empty. Nc |dk(S)Nr!rr@s rz+non_empty_field_validator..qs brr!z - `z?` column/key should not contain empty strings. These are rows: c<||dk7jgS)Nr!subset)dropna)r@fields rrz/non_empty_field_validator..necessary_fnvs&QuX^$++E7+; ;rRemove z rows with empty sempty_rCrD)applyanyisnull reset_indexindextolistr$r )r%rRrrr empty_rows empty_indexess ` rnon_empty_field_validatorr^isMLM %y()--/2e93C3C3E3I3I3Ki2o"U)*:*:*<= (..z:AAC w&efsetu  <"#m"4!55FugQO eW ##!  rc4|j}|jj|j}d}d}d}t |dkDr8dt |ddj d|}dt |d }d fd }t d ||| S)zY This validator will suggest to the user to remove duplicate rows if they exist. rONr - There are z duplicated -z sets. These are rows: rSz duplicate rowsc(|jS)NrO)drop_duplicatesr?s rrz.duplicated_rows_validator..optional_fns$$F$3 3rduplicated_rowsrrrrrD) duplicatedrYrZr[r$joinr )r%rArdduplicated_indexesrrrs ` rduplicated_rows_validatorrismm6m2O)//@GGIMLK "(-?)@(AchhW]N^M__vxJwKL  %7!8 9I  4  #!  rcd}d}d}t|}|dk7rBd d|tdkDr(dtdd}dtd }dfd }td ||| S)zW This validator will suggest to the user to remove examples that are too long. Nopen-ended generationc|jdd}|jj|jS)Nc^t|jt|jzdkDS)Ni')r$r;r<rLs rrMzClong_examples_validator..get_long_indexes..s"c!((mc!,,>O.ORW.Wr)axis)rVrYrZr[)d long_exampless rget_long_indexesz1long_examples_validator..get_long_indexess6GG$W^_G`M==?((7>>@ @rrr`z. examples that are very long. These are rows: zf For conditional generation, and for classification the examples shouldn't be longer than 2048 tokens.rSz long examplesc|}|k7r/tjjdt|d|d|j |S)NzeThe indices of the long examples has changed as a result of a previously applied recommendation. The z? long examples to be dropped are now at the following indices:  )sysstdoutwriter$drop)r@long_indexes_to_droprr long_indexess rrz,long_examples_validator..optional_fns'7':$#77JJ$$ABEFZB[A\\[\p[qqstvv233rrqre)rpr8r9rrD)infer_task_typer$r )r%rrrft_typerrrzs @@rlong_examples_validatorr}sMLKb!G)) A(+ | q ,S->,??mnzm{|cdM$S%6$7~FL 4  #!  rcb d}d}d}d}d gd}|D]t}|dk(r4|jjjdjr<|jjj|djrr| n j dd}t |}|d k(r t d Sdd  t|jd } |j| k(jrd| d}t d |S| dk7r| j dd} d| d}t| dkDr |d|dz }|jjdt|  jj| djr |d| dz }nd}| dk(r d|d}d fd }t d||||S)!z This validator will suggest to add a common suffix to the prompt if one doesn't already exist in case of classification or conditional generation. Nz ### => ) ->z ### z === z --- z ===> z ---> rrtFregex\nrk common_suffixrsuffixc |dxx|z cc<|SNr;rr@rs r add_suffixz2common_prompt_suffix_validator..add_suffixs ( v rxfixzAll prompts are identical: `zt` Consider leaving the prompts blank if you want to do open-ended generation, otherwise ensure prompts are differentrrr!z - All prompts end with suffix `r7 R. This suffix seems very long. Consider replacing with a shorter suffix, such as `z5 WARNING: Some of your prompts contain the suffix `zZ` more than once. We strongly suggest that you review your prompts and add a unique suffixa - Your data does not contain a common separator at the end of your prompts. Having a separator string appended to the end of the prompt makes it clearer to the fine-tuned model where the completion should begin. See https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset for more detail and examples. If you intend to do open-ended generation, then you should leave the prompts emptyzAdd a suffix separator `z` to all promptsc|Sr4rr@rsuggested_suffixs rrz3common_prompt_suffix_validator..optional_fna!12 2rcommon_completion_suffixrrrrrr@rrrr9rrD) r;rcontainsrWreplacer{r get_common_xfixallr$) r%rrrrsuffix_options suffix_optiondisplay_suggested_suffixr|rcommon_suffix_new_line_handledrrs @@rcommon_prompt_suffix_validatorrsIMLK(N( E !yy}}%%d+//1 99== ! !-u ! = A A C (  077eDb!G))00$BIIH=M ]"'')2=/Bwx 9EE)6)>)>tU)K&;.remove_common_prefixs#kooc&km4( rz" - All prompts start with prefix `r7z. Fine-tuning doesn't require the instruction specifying the task, or a few-shot example scenario. Most of the time you should only add the input data into the prompt, and the desired output into the completionRemove prefix `z` from all promptsc|Sr4r)r@rrs rrz3common_prompt_prefix_validator..optional_fn!s+A}==rcommon_prompt_prefixre)r@rrrr9rrD)rr;r rr$)r%MAX_PREFIX_LENrrrrrs @@rcommon_prompt_prefix_validatorrsNMLK#BIIH=M00 ]"'')00=m_AN C . . r rM,]O;MNL >  ##!  rc>d}t|jdtdkDxrddk(t|kr tdSdd|jk(j r tdSd d }d d }dfd }td|||S)zh This validator will suggest to remove a common prefix from the completion if a long one exist. rrr rrc\|djt|d|d<|r d|d|d<|S)Nr<rr)r@r ws_prefixs rrz@common_completion_prefix_validator..remove_common_prefix7s=L/--c&km<,  !!L/!23AlOrz& - All completions start with prefix `z_`. Most of the time you should only add the output data into the completion, without any prefixrz` from all completionsc|Sr4r)r@rrrs rrz7common_completion_prefix_validator..optional_fnEs#A}i@@rcommon_completion_prefixre)r@rrrrrr9rrD)rr<r$r r)r%rrrrrrrs @@@r"common_completion_prefix_validatorr,sN#BMMAMM"Q&B=+;s+BI =N*00 &++-00=m_MlmM$]O3IJLA  '#!  rc d}d}d}d}t|}|dk(s|dk(r tdSt|jd}|j|k(j rd|d |d }td| Sd gd }|D];}|jj j |djr9| n jdd} dd |dk7r|jdd} d| d }t|dkDr |d| d z }|jj dt| j j |djr |d|dz }nd}|dk(r d| d}d fd }td||||S)!z This validator will suggest to add a common suffix to the completion if one doesn't already exist in case of classification or conditional generation. Nrkclassificationrrrrz All completions are identical: `zJ` Ensure completions are different, otherwise the model will just repeat `r7rz [END]) rt.z ENDz***z+++z&&&z$$$z@@@z%%%Frrtrc |dxx|z cc<|SNr<rrs rrz6common_completion_suffix_validator..add_suffixvs ,6!rr!z$ - All completions end with suffix `rrz9 WARNING: Some of your completions contain the suffix `zU` more than once. We suggest that you review your completions and add a unique endingaH - Your data does not contain a common ending at the end of your completions. Having a common ending string appended to the end of the completion makes it clearer to the fine-tuned model where the completion should end. See https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset for more detail and examples.zAdd a suffix ending `z` to all completionsc|Sr4rrs rrz7common_completion_suffix_validator..optional_fnrrrrrrD) r{r rr<rrrrWrr$) r%rrrrr|rrrrrrrs @@r"common_completion_suffix_validatorrPsIMLKb!G))W8H-H00#BMMAM &++-6}oFQR_Q``ab 9EE  N( ==   % %m5 % A E E G (  077eD)6)>)>tU)K&?@^?__`a }  " qsKrLLMN NM ==  2M 22 3 7 7 @ @V[ @ \ ` ` b YZgYhi~ Md ./G.HH\]  3  '#!  rcd d}d}d}d}|jjddjdk7s|jjdddk7rd}d}|}t d||| S) z This validator will suggest to add a space at the start of the completion if it doesn't already exist. This helps with tokenization. c6|djd|d<|S)Nr<c8|jdrd|zSd|zS)Nrr!) startswith)rTs rrMzLcompletions_space_start_validator..add_space_start..s"cAR2_`:`X[_`:`r)rVrLs radd_space_startz:completions_space_start_validator..add_space_starts L///0`a,rNrnrrz - The completion should start with a whitespace character (` `). This tends to produce better results due to the tokenization we use. See https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset for more detailsz=Add a whitespace character to the beginning of the completioncompletion_space_startrerD)r<rnuniquevaluesr )r%rrrrs r!completions_space_start_validatorrs LKM }}!$$&!+r}}/C/CA/Fq/IS/PB V %  %#!  rcd fd }|jdj}|jdj}|dz|kDrtddddd d | Sy )zt This validator will suggest to lowercase the column values, if more than a third of letters are uppercase. cH|jj|<|Sr4)rr-)r@r/s r lower_casez(lower_case_validator..lower_cases"fIMM'')& rc&td|DS)Nc3dK|](}|js|js%d*ywrnN)isalphaisupper.0r0s r z9lower_case_validator....$0]qQRQZQZQ\0] 000sumrLs rrMz&lower_case_validator..S0]A0]-]rc&td|DS)Nc3dK|](}|js|js%d*ywr)rislowerrs rrz9lower_case_validator....rrrrLs rrMz&lower_case_validator..rrr rz - More than a third of your `z%` column/key is uppercase. Uppercase zs tends to perform worse than a mixture of case encountered in normal language. We recommend to lower case the data if that makes sense in your domain. See https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset for more detailsz'Lowercase all your data in column/key `r7reNrD)rVrr )r%r/r count_upper count_lowers ` rlower_case_validatorrs V*""#]^bbdKV*""#]^bbdKQ$;F8ChiohpqhiB6(!L"   rcd}d}d}d}d}tjj|r |jj ds|jj dr_|jj drdnd\}}d|d}d|d }t j ||t jd }nV|jj d rdd }d}t j|} | j} t| dkDr|dz }t j|tjd }n|jj drud}d}t|d5} | j} t j| j!dD cgc]} d | gc} |tjd }dddn?|jj drit j"|dtjd }t|dk(r/d}d}t j"|tjd }nn|jj drj t j"|dtjd }t|dk(r+t j"|tjd }nd}d}n.d}d |vr|d!|d"|j!d d#d$z }n |d!|d%z }nd*|d+}t+d,|||-}||fScc} w#1swY(xYw#t$$r-t j"|tjd }Y`wxYw#t$t&f$r1|j!d d#j)}d&|d'|d(|d)}YwxYw).z This function will read a file saved in .csv, .json, .txt, .xlsx or .tsv format using pandas. - for .xlsx it will read the first sheet - for .txt it will assume completions and split on newline Nz.csvz.tsv)CSV,)TSV z= - Based on your file extension, your file is formatted as a z filez Your format `z` will be converted to `JSONL`)sepdtyper!z.xlsxzH - Based on your file extension, your file is formatted as an Excel filez/Your format `XLSX` will be converted to `JSONL`rnz - Your Excel file contains more than one sheet. Please either save as csv or ensure all data is present in the first sheet. WARNING: Reading only the first sheet...)rz.txtz9 - Based on your file extension, you provided a text filez.Your format `TXT` will be converted to `JSONL`rrt)r+r.jsonlT)linesrz^ - Your JSONL file appears to be in a JSON format. Your file will be converted to JSONL formatz/Your format `JSON` will be converted to `JSONL`z.jsonz^ - Your JSON file appears to be in a JSONL format. Your file will be converted to JSONL formatz]Your file must have one of the following extensions: .CSV, .TSV, .XLSX, .TXT, .JSON or .JSONLrz Your file `z` ends with the extension `.z` which is not supported.z` is missing a file extension.z Your file `z!` does not appear to be in valid z9 format. Please ensure your file is formatted as a valid z file.zFile z does not exist.read_any_format)rrrr)ospathisfiler-endswithpdread_csvrfillna ExcelFile sheet_namesr$ read_excelopenread DataFramesplit read_json ValueError TypeErrorupperr )fnamerA remediationrrrr%file_extension_str separatorxlssheetsfcontentlines rrrsKMMI B ww~~e< v{{}%%f-1G1G1O@E @V@VW]@^ dq-"ITUgThhmn#00B/CCa b [[ISAHHL''0 k Q ll5)v;?!&NNM]]54;;B?''/ \ P %%!ffhG07 d0CD"dD &!fRj !!''1\\%t3?FFrJr7a<%FM$UMe37>>rBB''0 Ce4sCJJ2NB2w!|\\%s;BB2F)J (Y t%<<w6RSXS^S^_bScdfSgRhiB"CCI<w6T!UUIE7"23  ## K {?cE!!6"Ce37>>rBBCI& v!&S!1"!5!;!;!= %eW,MN`Mab[\n[oouvI vsbEN;3L>. L9 9L>B1NA(M 0/N9L>>MN 3N=N?NN=OOcLt|}d}|dk(rd|d}td|S)z This validator will infer the likely fine-tuning format of the data, and display it to the user if it is classification. It will also suggest to use ada and explain train/validation split benefits. NrzK - Based on your data it seems like you're trying to fine-tune a model for z - For classification, we recommend you try one of the faster and cheaper models, such as `ada` - For classification, you can estimate the expected model performance by keeping a held out dataset, which is not used for trainingr"r#)r{r )r%r|rs rformat_inferrer_validatorrsA b!GM""fgnfopUV N- HHrc`|jOtjjd|jd|jdtj d|j )tjj|j |j|j|}|S)zs This function will apply a necessary remediation to a dataframe, or print an error message if one exists. z ERROR in z validator: z Aborting...rn) rrustderrrwrexitrrvr)r%rs rapply_necessary_remediationr(s( =)9)9(:,{G\G\F]]lmn    , 223+  % %b ) Irctjj||r tjjdytj dk7S)NzY Tn)rurvrwinputr-) input_text auto_accepts raccept_suggestionr 6s?JJZ   7==?c !!rcd}d|jd}|j-t||r!|jJ|j|}d}|j-tj j d|jd||fS)zc This function will apply an optional remediation to a dataframe, based on the user input. Fz- [Recommended] z [Y/n]: Tz- [Necessary] rt)rr rrrurvrw)r%rr optional_appliedr s rapply_optional_remediationr>s #K$<$<#=XFJ+ Z 5**6 66((,B#   , >+*C*C)DBGH  rct|}d}|dk(rt|}|dz}n%|jdj}|dz}d d}||dz}tj j d |d y ) z? Estimate the time it'll take to fine-tune the dataset g?rg ףp= ?T)rZg|?5^?c|dkrt|ddS|dkrt|dz ddS|dkrt|dz ddSt|dz ddS) N<r z secondsiz minutesiQz hoursz days)round)times r format_timez.estimate_fine_tuning_time..format_time]sv "9D!n%X. . D[D2Iq)*(3 3 E\D4K+,F3 3D5L!,-U3 3rz:Once your model starts training, it'll approximately take z~ to train a `curie` model, and less for `ada` and `babbage`. Queue will approximately take half an hour per job ahead of you. N)rfloatr9r)r{r$ memory_usagerrurvrw)r% ft_format expected_timer"sizer time_strings restimate_fine_tuning_timerPs #IM$$2w $t+ T*..0v 4mc12KJJ D[MRQ Rrc|rddgndg}d} |dkDrd|dnd}|Dcgc],}tjj|dd||d.}}td |Ds|S|d z }^cc}w) N_train_validr!rz () _preparedrc3ZK|]#}tjj|%ywr4)rrr)rrs rrz get_outfnames..ss?277>>!$?s)+rn)rrsplitextrW)rrsuffixesi index_suffixrcandidate_fnamess r get_outfnamesr(ms',(#2$H A $%EA3ayr owxekrww//6q9:)F8L>Y_`xx?.>??# # Q xs1A+c|jj}d}|dk(r'|jjjd}||fS)Nr r)r<r value_countsrZ)r% n_classes pos_classs rget_classification_hyperparamsr-xsH %%'IIA~MM..066q9 i rc &t|}t|jd}t|jd}d}d}|dk(rt ||rd}d} |j dd } |j dd } t | d kDrd | d nd} d }|s:|s8tjjd|d| d| d| d t|yt ||rt||} |rt | dk(rd| d vrd| dvsJd}tt ||z tt |dz}|j|d}|j|j }|ddgj#| d dddd|ddgj#| dddddt%|\}}| dz } |dk(r | d |dz } n7| d!|z } n.t | dk(sJ|ddgj#| d dddd|rd"ndd#zd$j'| z}|r d%| ddnd}t | d k(rdnd&| d}tjjd'|d(| d d|| d)|| d t|ytjjd*y)+aQ This function will write out a dataframe to a file, if the user would like to proceed, and also offer a fine-tuning command with the newly created file. For classification it will optionally ask the user if they would like to split the data into train/valid files, and modify the suggested command to include the valid set. rrFzQ- [Recommended] Would you like to split into training and validation set? [Y/n]: rTr!rtrrz Make sure to include `stop=["z;"]` so that the generated texts ends at the expected place.z@ Your data will be written to a new JSONL file. Proceed [Y/n]: zK You can use your file for fine-tuning: > openai api fine_tunes.create -t ""ue After you’ve fine-tuned a model, remember that your prompt has to end with the indicator string `zX` for the model to start generating completions, rather than continuing with the prompt.r trainvalidrnig?*)r random_stater;r<recordsN)rorient force_asciiindentz! --compute_classification_metricsz" --classification_positive_class "z --classification_n_classes rTz to `z` and `z -v "ucAfter you’ve fine-tuned a model, remember that your prompt has to end with the indicator string `z Wrote modified filezd` Feel free to take a look! Now use that file when fine-tuning: > openai api fine_tunes.create -t "z z#Aborting... did not write the file )r{rr;r<r rr$rurvrwrr(maxintsamplerxrZto_jsonr-rg)r%rany_remediationsr rcommon_prompt_suffixrrr additional_params%common_prompt_suffix_new_line_handled)common_completion_suffix_new_line_handledoptional_ending_stringfnamesMAX_VALID_EXAMPLESn_traindf_traindf_validr+r, files_string valid_stringseparator_reminders rwrite_out_filerJs  #I*2998D.r}}8L EdJ$$ Z 5E,@,H,Hu,U)0H0P0PQUW\0]- 8 9A = ))R(STO P  VJ E [\a[bbcducvw^_D^EE]^t]uuw x  ""% :{ 3ue, v;!#6!9(<FSTIAU UU!% #b'$66CGcM8JKGyy7y>&;QR /4vayk+" 89Q> vx]w^^vw  #L>2Z[abc[dZeefgsftuFtGGKL^K_`v_wwy z  ""% ?@rcd}t|jjjdk(ryt|jj t||z kryy)z> Infer the likely fine-tuning task type from the data rrkrzconditional generation)rr;rr$r<unique)r%CLASSIFICATION_THRESHOLDs rr{r{sU ! 299==   1$& 2==   !"SW/G%GG #rcd} |dk(r|jt|dz dn|jdt|dz}|jdk7r |S||jdk(r |S|jd}w)zQ Finds the longest common suffix or prefix of all the values in a series r!rrnNr)rr$rr)seriesr common_xfix common_xfixess rrrsK 59X5EFJJ[)A-.0 16::VlX[\gXhklXlKm   "a '   M003 3  (..q1K rz,Callable[[pd.DataFrame], Remediation | None]r Validatorctddttttt ddt ttttgS)Nct|dSrr:rLs rrMz get_validators..s,Q9rct|dSrrVrLs rrMz get_validators..s,Q =rct|dSrrrLs rrMz get_validators..s&q(3rct|dSrrYrLs rrMz get_validators..s&q,7r) r(rIr^rrir}rrrrrrrrget_validatorsr[s99=#!!!37&&**) rc0g}||j||D]*}||}||j|t||},t|Dcgc]}|j |j|c}}t|Dcgc]}|j|c}} d} |r=t j jd|D]}t|||\}} | xs| } nt j jd| xs| } |||| |ycc}wcc}w)NFz? Based on the analysis we will perform the following actions: z No remediations found. ) appendrrWrrrurvrwr) r%rr validatorsr write_out_file_funcoptional_remediations validator&any_optional_or_necessary_remediationsany_necessary_appliedany_optional_appliedr !any_optional_or_necessary_applieds rapply_validatorsrfsB02$$[1> m  " ! ( ( 5,R=B > .1 5 ''3{7P7P7\  .* (=gAZAZAfg!- ]^0 LK#=b+{#[ B #7#K;K  L 78(<(U@U%E#DkR+  hs!D?DD)r%r8r9r )r%r8r5rr9r )r%r8rA list[str]r9r )r<)r%r8rRrr9r )r%r8r/rr9Remediation | None)rrrArgr9z'tuple[pd.DataFrame | None, Remediation])r%rrr r9r)r rr boolr9ri)r%r8rr r rir9ztuple[pd.DataFrame, bool])r%r8r9None)rrrrir9rg)r%r8r9ztuple[int, object]) r%r8rrr<rir rir9rj)r%r8r9r)r)rPrrrr9r)r9list[Validator])r%r8rrrrhr^rkr rir_zCallable[..., Any]r9rj), __future__rrrutypingrrrrrtyping_extensionsr _extrasr rr rr(r:rIr^rir}rrrrrrrrrr rrr(r-rJr{rrSrr[rfrrrrpsx" ??'"$*$19QR I!HHPQ]F^<4FN|D\2"JAH$N!HAH2.&.|$<V V!V,Vr I "  #. =A  $: HAV $$F 9E('S'S 'S$'S 'S  'S , 'S 'Sr