Manipulating output of osmosis model
Hello!
I am trying to use the osmosis 4b model for mcp related project. I'm trying to generate just the json/tool call when the model is prompted with a tool-related query, but the model keeps outputting think statements as well.
Here is a segment of the output:
...
Use appropriate context from the user\'s query. Avoid simply repeating the raw data.
Please use only the tools that are explicitly defined above.
IF TOOL USE REQUIRED, ONLY OUTPUT JSON. NOTHING ELSE AT ALL./no_think
Please send a message to Joe on Teams saying hello how are you/no_think. Okay, I need to send a message to Joe on Teams.
Let me check the send_message tools available. The send_message tool is the right one here.
...
Requirement 1: How can I get the model to not output think statements?
Requirement 2: How can I get the model to only output the json/tool call?
Requirement 3: How can I get the model to not output the prompt in addition to the response?
what framework are you using to run this model? most have flags to disable this. Or the output can be parsed out if you were to remove the think blocks. If you are desperate not to have these outputs, perhaps consider a grammar
I'm using HuggingFace, particularly the transformer pipeline:
from fastapi import FastAPI
from pydantic import BaseModel
from typing import Optional
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
app = FastAPI()
model_id = "osmosis-ai/osmosis-mcp-4b"
# Load Tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", token=HF_TOKEN)
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)
...
# FastAPI endpoint exposed to receive requests
@app
.post("/generate")
def generate(query: Query):
# Model inference
prompt = query.prompt
result = generator(prompt)
return {"response": result}