bug with tokenizer and gibberish output

[evilsocket]

the tokenizer has issues resolving a few tokens including special ones (they will be shown in the output as ), which is causing all sorts of gibberish output ... it's probably a matter of parsing the model/tokenizer.json properly

[evilsocket]

the model responds well when prompt template is not used (https://github.com/evilsocket/cake/blob/main/cake-core/src/models/llama3/llama.rs#L266)

[evilsocket]

this is also happening in candle llama example code -> huggingface/candle#2341

[caohuilong]

Has this been fixed? I met the same problem.

[evilsocket]

@caohuilong the issue is open, the bug has not been fixed, waiting for the candle team to respond