IEEE

non-profit

Verified

https://ieee.org

IEEEorg

Activity Feed Request to join this org

AI & ML interests

None defined yet.

Recent Activity

alanmartinezieee published a Space about 2 months ago

IEEE/README

fatemi authored a paper 3 months ago

Concise Reasoning via Reinforcement Learning

jiayuanz3 authored a paper 4 months ago

Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation

View all activity

sourcesync

posted an update about 1 month ago

Post

607

Hi! I'm having an issue converting a Phi3 model to a torch scripted model. Code, error, and versions below. Any guidance would be super helpful, thanks!

Here is the code:

from transformers import AutoModelForCausalLM
import torch

base_model = "microsoft/Phi-3-mini-4k-instruct"

model = AutoModelForCausalLM.from_pretrained(
        base_model,
        return_dict=True,
        low_cpu_mem_usage=True,
        torch_dtype=torch.float16,
        device_map="auto",
        trust_remote_code=True,
        torchscript=True
)

scripted_model = torch.jit.script(model)

Here is the error:

File ".../anaconda3/envs/qcomm_phi3.5instruct_aihub_py_310/lib/python3.10/site-packages/torch/jit/frontend.py", line 910, in <listcomp>
    args = [build_expr(ctx, py_arg) for py_arg in expr.args]
  File ".../anaconda3/envs/qcomm_phi3.5instruct_aihub_py_310/lib/python3.10/site-packages/torch/jit/frontend.py", line 417, in __call__
    return method(ctx, node)
  File ".../anaconda3/envs/qcomm_phi3.5instruct_aihub_py_310/lib/python3.10/site-packages/torch/jit/frontend.py", line 1255, in build_GeneratorExp
    return ExprBuilder.build_ListComp(ctx, stmt)
  File ".../anaconda3/envs/qcomm_phi3.5instruct_aihub_py_310/lib/python3.10/site-packages/torch/jit/frontend.py", line 1244, in build_ListComp
    raise NotSupportedError(r, "Comprehension ifs are not supported yet")
torch.jit.frontend.NotSupportedError: Comprehension ifs are not supported yet:
  File ".../.cache/huggingface/modules/transformers_modules/microsoft/Phi-3-mini-4k-instruct/0a67737cc96d2554230f90338b163bc6380a2a85/modeling_phi3.py", line 1148
            next_cache = next_decoder_cache.to_legacy_cache() if use_legacy_cache else next_decoder_cache
        if not return_dict:
            return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)
        return BaseModelOutputWithPast(
            last_hidden_state=hidden_states