CoRal-project

/

roest-wav2vec2-315m-v1

like 11

Follow

CoRal 20

Hey Pierre, your comment got me to try out Transformers.js a bit more in-depth. The resulting page is here: https://rasgaard.com/pages/roest-transformersjs-demo/
It doesn't support real-time streaming transcription as wav2vec2-support in Transformers.js seems to be lacking compared to Whisper.

@PierreMesure I haven't used ONNX before I'm afraid. Do you know if the ONNX conversion is including the n-gram language model? Otherwise the model is going to be worse.

@rasgaard Cool! Is your demo using the n-gram language model as well?

@saattrupdan I don't think the ONNX conversion takes the n-gram language model into account. Wasn't aware of that part!

Transformers.js isn't completely 1:1 with the Python library in terms of features and I don't think it supports wav2vec2 with language model yet.

I'll go ahead and add a small disclaimer to the page as I can see how it can otherwise be misleading :)

@rasgaard The transformers package can use the language model, if one uses the Wav2Vec2ProcessorWithLM processor rather than the Wav2Vec2Processor. So if Transformers.js is really 1:1 then it should be able to as well :)

Should be the default if using the AutomaticSpeechRecognitionPipeline as well, but that might depend on whether kenlm is installed or not. I would explicitly use Wav2Vec2ProcessorWithLM instead, to be sure.

@PierreMesure It seems like the demo should work with Wav2Vec2 (also with LM I think), since it's just using the pipeline object. However, I would ensure that kenlm is installed, as using the language model might be conditional on that being installed.

I know it doesn't solve your ONNX issue though!

Thank you so much to the two of you for having a look!

Hey Pierre, your comment got me to try out Transformers.js a bit more in-depth. The resulting page is here: https://rasgaard.com/pages/roest-transformersjs-demo/

@rasgaard , would you mind sharing your source code?
EDIT: I found the worker.js here, thank you.