While converting a csv file to UTF8, using the below command I am getting the error like:

iconv: illegal input sequence at position 5857689

Command :

$ iconv -f ASCII -t UTF-8 EDMTradesFeed_20210428.csv -o EDMTradesFeed.csv

What would have gone wrong here? Can you please help?

Thanks

It is failing with huge files. Is there a way to avoid data loss?

Hi Aaron,

File is not that big, it is failing for number of records more than 700, and no of columns will be 5-7 and our file is having Latin America and Chinese data as well. We have done as below, Kindly suggest if we can discuss over google chat.

Our Script 
#!/bin/bash
cd $SRCFILEDIR

for f in *txt
do

encoding=$(file -b --mime-encoding $f)

        if [ utf-8 != $encoding ] && [ us-ascii != $encoding ] && [ binary != $encoding ]; then
                iconv -f $encoding -t utf-8 $f -o $f
        else
                echo $encoding
        fi
done 

We get below error :

us-ascii -- converted successfully
us-ascii -- converted successfully
binary -- converted successfully
./testing.sh: line 6:  1156 Bus error               (core dumped) iconv -f $encoding -t utf-8 $f -o $f
./testing.sh: line 6:  1161 Bus error               (core dumped) iconv -f $encoding -t utf-8 $f -o $f

@Manjeet

How big is the file you are working with? The script should work with huge files.