ctheodoris

/

Geneformer

like 234

Hello, so I faced some errors whle attempting to LoRA fine-tune the classifier:

1. for classifier.py, documentation states that custom quantize configs will be use if passed in the parameter, but when I examine the code it just passes it through the load_model() method in purturber_utils.py, where the method simply takes it as a boolean indicator and use default lora and bnb config. Would be nice if this can be fixed with checking if quantize is a dictionary or boolean value to ensure custom configs passes through:
   if isinstance(quantize, dict):
   quantize_config = quantize.get("bnb_config", None)
   peft_config = quantize.get("peft_config", None)
   elif quantize:
   if inference_only:
   quantize_config = BitsAndBytesConfig(load_in_8bit=True)
   peft_config = None

...

2. the default setting for LoraConfig in this method is possibly outdated, and would be nice if it's updated soon:
   peft_config = LoraConfig(
   lora_alpha=128,
   lora_dropout=0.1,
   r=64,
   bias="none",
   task_type="TokenClassification",
   )

ValueError: Invalid task type: 'TokenClassification'. Must be one of the following task types: SEQ_CLS, SEQ_2_SEQ_LM, CAUSAL_LM, TOKEN_CLS, QUESTION_ANS, FEATURE_EXTRACTION.

Thanks for your time.