tokenizer.perl error

utf8 does not map to Unicode at /usr/local/scripts/tokenizer/tokenizer.perl

 

solution:

change file to utf-8

iconv -f ISO-8859-1 -t UTF8 file-original.txt -o file-converted.txt