metal example code doesn't run as written
First, there is an error in gpt_oss/metal/python/context.c where "PyObject* model = NULL;" is declared twice. (How did this get past code review?). Comment out the second declaration...
Then, the example gpt_oss/metal/examples/generate.py does not run because the function context.sample requires an numerical argument.
Then the return value from context.sample(1) returns a list of tokens but the next call: context.append expects an integer.
I changed the example this way:
while context.num_tokens - prompt_tokens < options.limit:
- token = context.sample()
- context.append(token)
- print(str(tokenizer.decode(token), encoding="utf-8"), end='', flush=True)
+ tokens = context.sample(1)
+ #print(f"token: {token}, type: {type(token)}")
+ for token in tokens:
+ context.append(token)
+ print(str(tokenizer.decode(token), encoding="utf-8"), end='', flush=True)
Now it runs, but the output is rather wonky... (maybe that is expected?)
$ python gpt_oss/metal/examples/generate.py ~/.cache/huggingface/hub/models--openai--gpt-oss-20b/snapshots/6cee5e81ee83917806bbde320786a8fb61efebee/metal/model.bin -p "Why did the chicken cross the road?"
[13903, 2242, 290, 21663, 8088, 290, 8733, 30]
Because you were killing me to evolve and get you that best combating the Earth'sography? And when the chicken came to plead not being spoken for my save then one start running. And starts like that. And also I've been, that is\n\