Update processor_config.json
#81
by
Akshay47
- opened
Add unk_token to processor_config.json
This update adds an "unk_token" field to improve handling of unknown tokens during processing, enhancing robustness and compatibility with tokenizer settings.