Update processor_config.json

#81
by Akshay47 - opened

Add unk_token to processor_config.json

This update adds an "unk_token" field to improve handling of unknown tokens during processing, enhancing robustness and compatibility with tokenizer settings.

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment