microsoft

/

phi-1_5

like 1.36k

Follow

Microsoft 20.2k

did these changes just break all peft finetuned models? I'm now getting

microsoft/phi-1_5 does not appear to have a file named configuration_mixformer_sequential.py. 
Checkout 'https://huggingface.co/microsoft/phi-1_5/main' for available files.

when I run ArtifactAI/phi-arxiv-physics-instruct. Swapping in the new config results in gibberish

No, they are not merged yet. I think this issue is related to problem you are facing.

I just proposed these changes to be able to transfer the weights to their official repo.

BTW you can now load phi from the transformers library itself :)

got it thank you!

I've completed a review here - the port looks good, and outputs should be unchanged compared to the original version. The main difference is that past_key_values is returned by default now, but this isn't a significant issue I think.

I'd support merging this (and the Phi-1 PR) once we do the next release of transformers and add library support for the class! The next release should be due either this or next week.

Thanks for the work everyone! I agree with what you said, and let's merge as soon as transformers release their 4.36+ version.

Hi @gugarosa , transformers have released their 4.36 version.

Thanks @susnato !

We are also going to open a PR to propose some extra functionality and we should be able to merge everything to a unique code. It should be done in the next weeks.

Hi @gugarosa ,

Will this extra functionality be added to the code on the Hub or directly to the transformers library?

Also, let me know if I could be of any help.

I think transformers will be better, so we can gradually move away from having two distinct codebases.

It is pretty simple to be honest, it is only the support to use MQA/GQA because we might need that soon.

Hi @gugarosa , if it's ok with you, I can open a PR to the library to add support for MQA and GQA.

Hi @gugarosa , if it's ok with you, I can open a PR to the library to add support for MQA and GQA.

Of course! Can we work together on https://github.com/huggingface/transformers/pull/28163? I added this morning a starting point which should make our lives easier :D

Even with the updated PR & installing transformers from source, i get the same error:

OSError: microsoft/phi-1_5 does not appear to have a file named configuration_mixformer_sequential.py. Checkout 'https://huggingface.co/microsoft/phi-1_5/main' for available files.

Same here
OSError: microsoft/phi-1_5 does not appear to have a file named configuration_mixformer_sequential.py. Checkout 'https://huggingface.co/microsoft/phi-1_5/main' for available files.