T^(h>ddlZddlmZddlmZmZmZmZmZm Z ddl m Z ddl m Z ddlmZerdd lmZdd lmZmZmZmZmZmZmZmZer ddlZdd lmZej<eZ Gd d e Z!y)N)cached_property) TYPE_CHECKINGAnyDictListOptionalUnion)version) HfQuantizer)get_module_from_name)PreTrainedModel)ACCELERATE_MIN_VERSIONis_accelerate_availableis_bitsandbytes_availableis_torch_availableis_torch_hpu_availableis_torch_npu_availableis_torch_xpu_availablelogging)Conv1DcLeZdZdZdZdZdZddgZfdZdZ d d Z d d d d de de e e fdef dZ d!d d d d de ddde e e fdeee f dZde e eee ffde e eee fffdZd"dZdZ d!d d deee fdZd#dZd!dZedefdZedefdZdZxZ S)$Bnb4BitHfQuantizera 4-bit quantization from bitsandbytes.py quantization method: before loading: converts transformer layers into Linear4bit during loading: load 16bit weight and pass to the layer object after: quantizes individual weights in Linear4bit into 4bit at the first .cuda() call saving: from state dict, as usual; saves weights and `quant_state` components loading: need to locate `quant_state` components and pass to Param4bit constructor TF bitsandbytes acceleratec t||fi||jj|jj|_yyN)super__init__quantization_configllm_int8_skip_modulesmodules_to_not_convert)selfr!kwargs __class__s h/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/quantizers/quantizer_bnb_4bit.pyr zBnb4BitHfQuantizer.__init__@sB ,77  # # 9 9 E*.*B*B*X*XD ' Fc tstdtdts tdddlm}ddlm}|}|d|jd d s|jd d r td |jd d}|t|tr|jjs|jDcic]}||jvs|||}}t!|j#dhk(r|rn/d|j#vsd|j#vr tdt%j&t(j*j%dt%j&dkr tdycc}w)NzWUsing `bitsandbytes` 4-bit quantization requires Accelerate: `pip install 'accelerate>=z'`zrUsing `bitsandbytes` 4-bit quantization requires the latest version of bitsandbytes: `pip install -U bitsandbytes`r)!validate_bnb_backend_availability)'is_bitsandbytes_multi_backend_availableT)raise_exceptionfrom_tfF from_flaxzConverting into 4-bit or 8-bit weights from tf/flax weights is currently not supported, please make sure the weights are in PyTorch format. device_mapcpudiskaSome modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `llm_int8_enable_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details. rz0.39.0zYou have a version of `bitsandbytes` that is not compatible with 4bit inference and training make sure you have the latest version of `bitsandbytes` installed)r ImportErrorrr integrationsr*utilsr+get ValueError isinstancedictr! llm_int8_enable_fp32_cpu_offloadkeysr#setvaluesr parse importlibmetadata) r$argsr%r*r+bnb_multibackend_is_enabledr/keydevice_map_without_lm_heads r'validate_environmentz'Bnb4BitHfQuantizer.validate_environmentFs&(ikAjBBDE )*E  EC&M&O#)$? ::i '6::k5+I;  ZZ d3  ":t,,,MM1;0A*),SPTPkPkEkZ_$* &*:$$&'E727R4;;==KeKlKlKnAn ) ==++33NC Dw}}U]G^ ^U  _*s F F returnctjtjjdtjdkDr:ddlm}|t jk7rtjd|jStd)Nrz0.19.0r) CustomDtypezXtarget_dtype {target_dtype} is replaced by `CustomDtype.INT4` for 4-bit BnB quantizationaUYou are using `device_map='auto'` on a 4bit loaded version of the model. To automatically compute the appropriate device map, you should upgrade your `accelerate` library,`pip install --upgrade accelerate` or install it from source to support fp4 auto device mapcalculation. You may encounter unexpected behavior, or pass your own device map) r r=r>r?accelerate.utilsrGtorchint8loggerinfoINT4r6)r$ target_dtyperGs r'adjust_target_dtypez&Bnb4BitHfQuantizer.adjust_target_dtypewsi ==++33LA BW]]S[E\ \ 4uzz) vw## #b r(modelr param_valuez torch.Tensor param_name state_dictc ddl}t||\}}t|jj |d|j j ryt||j jr|dk(ryy)NrTbiasF)rr r7 _parametersr5nn Params4bit Linear4bit) r$rPrQrRrSr%bnbmodule tensor_names r'check_quantized_paramz(Bnb4BitHfQuantizer.check_quantized_paramse #25*E f((,,[$?ARAR S  1 1 2{f7Lr( target_devicez torch.deviceunexpected_keyscddl}t||\}} | |jvrt|d| dt || } t |t rtrd|}| dk(r`|| j|} n|j|} tjj| | j} | |j| <yt |j| |jjs td| jtjd k(r,|d tjd fvr|t| d |d|jr|j s td |d z|vr|d z|vrtd|di} |j#D]+\} }|dz| vs|| | <|| |vs|j%| -i}|j&r||d<|jjj(d|| d|d|} nq|jd} t+|j,t.r | j0} | j2}|jj| fddi|j|} | |j| <y)z combines logic from _load_state_dict_into_meta_model and .integrations.bitsandbytes.py::set_module_quantized_tensor_to_device() rNz- does not have a parameter or a buffer named .npu:rU) requires_gradz0this function only loads `Linear4bit components`metaz7 is on the meta device, we need a `value` to put in on zDetected int4 weights but the version of bitsandbytes is not compatible with int4 serialization. Make sure to download the latest `bitsandbytes` version. `pip install --upgrade bitsandbytes`.z.quant_state.bitsandbytes__fp4z.quant_state.bitsandbytes__nf4zSupplied state dict for zT does not contain `bitsandbytes__*` and possibly other `quantized_stats` components.r[F)dataquantized_statsrcdevicer0rc)rr rVr6getattrr7intrtorIrW ParameterrcrXrg pre_quantizedis_serializableitemsremove$is_bnb_supports_quant_storage_modulefrom_prequantized issubclass source_clsrT__dict__)r$rPrQrRr^rSr_rZr[r\ old_value new_valuerfkv param_kwargsr%s r'create_quantized_paramz)Bnb4BitHfQuantizer.create_quantized_params #25*E f00 0x'TU`Taabcd dFK0  mS ).D.F"=/2M & "%LL7 'NN=9 **9ID[D[*\I.7F  { + &,,[9366;L;LMOP P    V 4 4fell6.B%CC# },cdqcrrstu u   '' u ==ZO==ZO .zl;OP!O"((* 21#q()*OA&&2qO7K'..q1  2 L88)/ X&;));; /#$   I$u-I&++V4%KK ''F))))S5SFSVVWdeI*3;'r( max_memoryc^|jDcic] \}}||dz }}}|Scc}}w)Ng?)ro)r$r}rBvals r'adjust_max_memoryz$Bnb4BitHfQuantizer.adjust_max_memorys66@6F6F6HI(#sc3:oI IJs)cV|&tjd|tj}|S)NaOverriding torch_dtype=%s with `torch_dtype=torch.float16` due to requirements of `bitsandbytes` to enable model loading in 8-bit or 4-bit. Pass your own torch_dtype to specify the dtype of the remaining non-linear layers or pass torch_dtype=torch.float16 to remove this warning.)rKrLrIfloat16)r$ torch_dtypes r'update_torch_dtypez%Bnb4BitHfQuantizer.update_torch_dtypes0   KKE   --Kr(c|tjjr!dtjji}nt r$ddtj ji}n`t r$ddtjji}n2tr$ddtjji}nddi}tjd|d|S)Nrbzhpu:zxpu:r0z:The device_map was not initialized. Setting device_map to zL. If you want to use the model for inference, please set device_map ='auto' ) rIcuda is_availablecurrent_devicernpurhpurxpurKrL)r$r/s r'update_device_mapz$Bnb4BitHfQuantizer.update_device_map s  zz&&( %**";";"=> ') D)A)A)C(D"EF ') D)A)A)C(D"EF ') D)A)A)C(D"EF  %[ KK))3 5]]  r(keep_in_fp32_modulesc &ddlm}|jj}|j ||jj ||_t|tryt|jdkDr]|jDcgc] \}}|dvs |} }}t| dkDr |s td|j j| |||j |j}|j|j_ycc}}w)Nr)replace_with_bnb_linearr )r1r0rzIf you want to offload some keys to `cpu` or `disk`, you need to set `llm_int8_enable_fp32_cpu_offload=True`. Note that these modules will not be converted to 8-bit but kept in 32-bit.)r#r!)r3rr!r9get_modules_to_not_convertr"r#r7r8lenr:ror6extendconfig) r$rPr/rr%rr9rBvalue keys_on_cpus r'$_process_model_before_weight_loadingz7Bnb4BitHfQuantizer._process_model_before_weight_loadings ;+/+C+C+d+d(&*&E&E 4++AACW' # j$ 'C 0A,BQ,F1;1A1A1C`:3uP_G_3`K`;!#,L >  ' ' . .{ ;' $*E*E[_[s[s  ,0+C+C (as  D D c >d|_|j|_|SNT)is_loaded_in_4bitrnis_4bit_serializable)r$rPr%s r'#_process_model_after_weight_loadingz6Bnb4BitHfQuantizer._process_model_after_weight_loadingBs "&%)%9%9%;" r(ctjtjjdtjdk\}|stj dyy)Nrz0.41.3zYou are calling `save_pretrained` to a 4-bit converted model, but your `bitsandbytes` version doesn't support it. If you want to save 4-bit models, make sure to have `bitsandbytes>=0.41.3` installed.FT)r r=r>r?rKwarning)r$safe_serialization_is_4bit_serializables r'rnz"Bnb4BitHfQuantizer.is_serializableGsQ ' i.@.@.H.H.X Y]d]j]jks]t t$ NNh r(ctjtjjdtjdk\S)z determines if the current version of bitsandbytes supports the `module` parameter in `Params4bit.from_prequantized` :return: rz0.43.3)r r=r>r?r$s r'rqz7Bnb4BitHfQuantizer.is_bnb_supports_quant_storage_moduleSs3}}Y//77GHGMMZbLcccr(cyrrhrs r' is_trainablezBnb4BitHfQuantizer.is_trainable\sr(cPddlm}|||j|j}|S)Nr)dequantize_and_replace)r!)r3rr#r!)r$rPrs r' _dequantizezBnb4BitHfQuantizer._dequantize`s)9& 4..DD\D\  r()rN torch.dtyperErr)rrrEr)rPr)!__name__ __module__ __qualname____doc__use_keep_in_fp32_modules requires_parameters_quantizationrequires_calibrationrequired_packagesr rDrOstrrrboolr]rrr|r rjrrrrrrnrrqpropertyrr __classcell__)r&s@r'rr/s $'+$ '6Y /b  $  cN  804Z4 Z4$Z4 Z4 & Z4 cN Z4"$s),Z4zDeCHo1E,F4PSUZ[^`c[cUdPdKe 059 D  D'tCy1 DF dddddr(r)"r> functoolsrtypingrrrrrr packagingr baser quantizers_utilsr modeling_utilsrr4rrrrrrrrrI pytorch_utilsr get_loggerrrKrrhr(r'rsa%BB20   &   H %wwr(