S^(hNddlZddlmZddlmZmZddlmZddlm Z ddl m Z m Z ddl mZmZddlmZdd lmZmZmZe d Zgd ZGd d eZdededefdZde fdZdZdededeeeeeeeffdZ d1dededede edef dZ!dede"dede#fdZ$dede"defd Z% d2dededede"de ede#defd!Z&d"edefd#Z'd"edefd$Z(d%efd&Z)e*d'k(rYeZ+e+jYZ-ee-j\j_e-_. e0d(e&e-jbe-jde-j\e-jfe-jhe-jje-je-jPrUeee-jbd)k(re0d*e0d+e'e-j\e-_6e(e-jle-_7e-jprRe0d,e)e-j\e9e-d-re)e-jle9e-d.re)e-jnyyyy#e:$rZ;e0d/e;e= than 2Gbz --quantizez/Quantize the neural network to be run with int8output)super__init__ add_argumentstrSUPPORTED_PIPELINESint)self __class__s `/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/convert_graph_to_onnx.pyr$z$OnnxConverterArgumentParser.__init__3s )*  '(   G  -c8rs  4L2  )#r@ST  7   #4   B  (#)__name__ __module__ __qualname____doc__r$ __classcell__)r*s@r+rr.s&$&$r,rfilename identifierreturnc|jj|j|zj|jS)aE Append a string-identifier at the end (before the extension, if any) to the provided filepath Args: filename: pathlib.Path The actual path object we would like to add an identifier suffix identifier: The suffix to add Returns: String with concatenated identifier at the end of the filename )parentjoinpathstem with_suffixsuffix)r2r3s r+generate_identified_filenamer;\s1 ?? # #HMMJ$> ? K KHOO \\r,minimum_versionc ddl}t|j}|tkrt d|jd|dy#t$r t dwxYw)z Check onnxruntime is installed and if the installed version match is recent enough Raises: ImportError: If onnxruntime is not installed or too old version is found rNz*We found an older version of onnxruntime (z&) but we require onnxruntime to be >= zp to enable all the conversions options. Please update onnxruntime by running `pip install --upgrade onnxruntime`zonnxruntime doesn't seem to be currently installed. Please install the onnxruntime by running `pip install onnxruntime` and relaunch the conversion.) onnxruntimer __version__ORT_QUANTIZE_MINIMUM_VERSION ImportError)r<r> ort_versions r+check_onnxruntime_requirementsrCis K334  5 5<[=T=TAActd|jjj}gg}}|ddD];}||vr&|j ||j ||-t|dntd||t |fS)a Ensure inputs are presented in the correct order, without any Non Args: model: The model used to forward the input data tokens: BatchEncoding holding the input data input_names: The name of the inputs Returns: Tuple z$Ensuring inputs are in correct orderNz, is not present in the generated input list.zGenerated inputs order: )printforward__code__ co_varnamesappendtuple)modeltokens input_namesmodel_args_name model_argsordered_input_namesarg_names r+ensure_valid_inputrSs 01mm,,88O&("#J#AB' { "  & &x 0   fX. / XJJK L   $%8$9 :; j 1 11r,nlp frameworkc :dtdtdtffd |jd|}|jj d}|dk(r|j d i|n|j |}t|tr|j}t|ttfs|f}t|j}|jDcic]\}}|||d |}}}g} |D];} t| ttfr| j| +| j| =t!t#| D cgc]} d |  } } t%| | Dcic]\}}|||d |} }}t'|fi| }|| ||fScc}}wcc} wcc}}w) a? Attempt to infer the static vs dynamic axes for each input and output tensors for a specific model Args: nlp: The pipeline object holding the model to be exported framework: The framework identifier to dispatch to the correct inference scheme (pt/tf) Returns: - List of the inferred input variable names - List of the inferred output variable names - Dictionary with input/output variables names as key and shape tensor as value - a BatchEncoding reference which was used to infer all the above information nameis_inputseq_lenc Ft|ttfr|Dcgc] } ||||c}St|jDcgc] \}}|dk(s |c}}ddi}|r@t |jdk(rd|d<ntt dt |jdt|jD cgc] \}} | |k(s |} }} |jtj| dtd|rd nd d |d ||Scc}wcc}}wcc} }w) NrErbatchsequencezUnable to infer tensor axes ()zFound inputr" z with shape: ) isinstancerKlist enumerateshapelen ValueErrorupdatedictfromkeysrF) rWtensorrXrYtaxisnumelaxesdimrdseq_axesbuild_shape_dicts r+rqz&infer_shapes..build_shape_dicts fudm ,JPQQ$T1h@Q Q.7v||-DSkdEQR TSTUVX_`Dv||$)(DG$'DSEVDWWX%YZZ2;FLL2I^JCUV]M]C^^ DMM(J?@ (w94& dVTU RT_sD DD> D DzThis is a sample output)return_tensorsrToutput_F)r&boolr( tokenizer input_idsrdrLrar to_tuplerbrKkeysitemsextendrJrangereziprh)rTrUrMrYoutputs input_varskvinput_dynamic_axes outputs_flatr"i output_namesoutput_dynamic_axes dynamic_axesrqs @r+ infer_shapesrs sdS&]]4Y] OF$$R(G%.$%6icii!&!CIIf 2Gb /!\rN)r?zUsing framework TensorFlow: z , tf2onnx: r)rW)r output_pathzCannot import z6 required to convert TF model to ONNX. Please install z first. )rrrF tensorflowtf2onnxr?versionVERSIONrrLpredictdatar{ TensorSpec from_tensorconvert from_kerasrrArW)rTrr"rrt2ovrNrrrMkeyrjinput_signature model_proto_es r+convert_tensorflowr&s/  fgg RS / ,RZZ-?-?,@ D6RS;GsD:Q7 \< &++&Z`ZfZfZhi;3PV2==44V#4Fii 33 IIeAR4  Qj  QVVH$Z[\[a[aZbbjkljm n   s*A Load the model at the specified path and let onnxruntime look at transformations on the graph to enable all the optimizations possible Args: onnx_model_path: filepath where the model binary description is stored Returns: Path where the optimized model binary description has been saved rInferenceSessionSessionOptionsz -optimizedz$Optimized model has been written at : ✔zY/!\ Optimized model contains hardware specific operators which might not be portable. /!\)r>rrr;roptimized_model_filepathrF)rrropt_model_path sess_optionrs r+optimizersf=2/<PN "K+9+B+B+DK(113[AA 00@@V WX gh r,cddl}ddl}ddlm}ddlm}ddlm}ddlm }|j|j}t|jtdkr td|}|j|t|jtd kr'||d d |j d d d dddt#| } n&||d d |j d d d dddt#| } | j%t'|d} td| d|j)| j*j*| j| S)z Quantize the weights of the model from float32 to in8 to allow very efficient inference on modern CPU Args: onnx_model_path: Path to location the exported ONNX model is stored Returns: The Path generated for the quantized rN) ModelProto)QuantizationMode) ONNXQuantizer)IntegerOpsRegistryz1.5.0zpModels larger than 2GB will fail to quantize due to protobuf constraint. Please upgrade to onnxruntime >= 1.5.0.z1.13.1FT) rL per_channel reduce_rangemodestatic weight_qType input_qType tensors_rangenodes_to_quantizenodes_to_excludeop_types_to_quantize) rLrrrrractivation_qTyperrrrz -quantizedz$Quantized model has been written at r)onnxr> onnx.onnx_pbronnxruntime.quantizationr'onnxruntime.quantization.onnx_quantizerr!onnxruntime.quantization.registryrloadrr r?rFCopyFrom IntegerOpsrbquantize_modelr; save_modelrL) rrr>rrrr onnx_model copy_model quantizerquantized_model_paths r+quantizersO'9ED?3356J T  w/  6 J #  [ $ $%h7!!,,"!!%&8!9  "!,,""!!%&8!9  8V 01E0FF\ ]^OOIOO))+?+H+H+JK r,pathcddlm}m}ddlm}t d|d |}||j |dg}t d|d y#|$r}t d |d Yd}~yd}~wwxYw) Nrr)RuntimeExceptionz"Checking ONNX model loading from: z ...CPUExecutionProvider) providerszModel u correctly loaded: ✔zError while loading the model u: ✘)r>rr+onnxruntime.capi.onnxruntime_pybind11_staterrFr)rrrr onnx_optionsrres r+verifyrsz<L .tfD 9:I%' T]]_lG]F^ _ tfCDE I .rd2FGHHIs0AA.A))A.__main__z' ====== Converting model to ONNX ======raV Using TensorFlow might not provide the same optimization level compared to PyTorch. For TensorFlow users you can try optimizing the model directly through onnxruntime_tools. For more information, please refer to the onnxruntime documentation: https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers z$ ====== Optimizing ONNX model ======z+ ====== Check exported ONNX model(s) ======optimized_outputquantized_outputz"Error while converting the model: rE)N)NFr)=rargparserosrrpathlibrtypingrpackaging.versionrr transformers.pipelinesr r transformers.tokenization_utilsr transformers.utilsr rrr@r'rr&r;rCrSrKrbrhrrr(rvrrrrrrr-parser parse_argsargsr"absoluterFrUrLrrwrrr check_loadinghasattrrrexitrur,r+rs%# ,59OO %W~ +$.+$\ ]4 ]S ]T ] G :28=:h=:3=:5cDItUb9b3c=:BPTvv#&v/2v?G}v v>" " #" t" RV" J% H% S% $% Z $ %-//// // //  // } //  ////ddt2H dH tH V I I z ( *F    Dt{{#,,.DK/ 89 NN JJ KK JJ NN  $ $ MM  == *+G H~~%w 9 :%-T[[$9D !%-T-B-B$CD !    @ A 4;; t/0t,,-t/0t,,-1 Uh  21#67 QsDH88I=II