tokenize_encode

tokenize_encode#

tokenize_encode(input, token_path, io_config, pattern, special_tokens, use_special_tokens)[source]#

Decodes each list of integer tokens into a string using a tokenizer.