tokenize_decode

tokenize_decode#

tokenize_decode(input, token_path, io_config, pattern, special_tokens, use_special_tokens)[source]#

Encodes each string as a list of integer tokens using a tokenizer.