Different Ways to Read File in Bash Script Using While Loop

This article is all about how to read files in bash scripts using a while loop. Reading a file is a common operation in programming. You should be familiar with different methods and which method is more efficient to use. In bash, a single task can be achieved in many ways but there is always an optimal way to get the task done and we should follow it.

Before seeing how to read file contents using while loop, a quick primer on how while loop works. While loop evaluates a condition and iterates over a given set of codes when the condition is true.

while [ CONDITION ]
do
    code block
done

Let’s break down while loop syntax.

  • while loop should start with a while keyword followed by a condition.
  • A condition should be enclosed within [ ] or [[ ]]. The condition should always return true for the loop to be executed.
  • The actual block of code will be placed between do and done.
NUMBER=0

while [[ $NUMBER -le 10 ]]
do
    echo " Welcome ${NUMBER} times "
    (( NUMBER++ ))
done
While Loop
While Loop

This is a very simple example, where the loop executes until NUMBER is not greater than 10 and prints the echo statement.

Along with while we will use the read command to read the contents of a file line by line. Below is the syntax of how while and read commands are combined. Now there are different ways to pass the file as input and we will see them all.

# SYNTAX
while read VARIABLE
do
    code
done

Piping in Linux

Normally we will use the cat command to view the contents of the file from the terminal. Also, we will pipe the output of the cat command to other commands like grep, sort, etc.

Similarly, we will use the cat command here to read the content of the file and pipe it to a while loop. For demonstration, I am using /etc/passwd file but it is not advisable to mess with this file so take a backup copy of this file and play with it if you desire so.

cat /etc/passwd | while read LREAD
do
    echo ${LREAD}
done
Piping in Linux
Piping in Linux

Let’s break down what will happen when the above code is submitted.

  • cat /etc/passwd will read the contents of the file and pass it as input through the pipe.
  • read command reads each line passed as input from cat command and stores it in the LREAD variable.
  • read command will read file contents until EOL is interpreted.

You can also use other commands like head, tail, and pipe it to while loop.

head -n 5 /etc/passwd | while read LREAD
do
    echo ${LREAD}
done
Head Command
Head Command

Input Redirection in Linux

We can redirect the content of the file to while loop using the Input redirection operator (<).

while read LREAD
do
    echo ${LREAD}
done < /etc/passwd | head -n 5
Input Redirection
Input Redirection

You can also store the file name to a variable and pass it through a redirection operator.

FILENAME="/etc/passwd"

while read LREAD
do
    echo ${LREAD}
done < ${FILENAME}
Store Filename in Variable
Store Filename in Variable

You can also pass file names as an argument to your script.

while read LREAD
do
    echo ${LREAD}
done < $1 | head -n 5
Store Filename as Argument
Store Filename as Argument

Internal Field Separator

You may work with different types of file formats (CSV, TXT, JSON) and you may want to split the contents of the file based on a custom delimiter. In this case, you can use “Internal field separator (IFS)” to split the content of the file and store it in variables.

Let me demonstrate how it works. Take a look at the /etc/passwd file which has a colon (:) as the delimiter. You can now split each word from a line and store it in a separate variable.

In the below example, I am splitting /etc/passwd file with a colon as my separator and storing each split into different variables.

while IFS=":" read A B C D E F G
do
    echo ${A}
    echo ${B}
    echo ${C}
    echo ${D}
    echo ${E}
    echo ${F}
    echo ${G}
done < /etc/passwd
Internal Field Separator
Internal Field Separator

I displayed just one line split in the above screenshot considering screenshot size.

Empty Lines in Linux

Empty lines are not ignored when you loop through the file content. To demonstrate this I have created a sample file with the below content. There are 4 lines and few empty lines, leading whitespace, trailing white space, tab characters in line 2, and some escape characters (\n and \t).

File with Empty Lines
File with Empty Lines
while read LREAD
do
    echo ${LREAD}
done < testfile
Blank Line Not Ignored
Blank Line Not Ignored

See the result, blank line is not ignored. Also, an interesting thing to note is how white spaces are trimmed by the read command. A simple way to ignore blank lines when reading the file content is to use the test operator with the -z flag which checks if the string length is zero. Now let’s repeat the same example but this time with a test operator.

while read LREAD
do
    if [[ ! -z $LREAD ]]
    then
        echo ${LREAD} 
    fi
done < testfile
Blank Lines Ignored
Blank Lines Ignored

Now from the output, you can see empty lines are ignored.

Escape Characters

Escape characters like \n, \t, \c will not be printed when reading a file. To demonstrate this I am using the same sample file which has few escape characters.

File with Escape Characters
File with Escape Characters
while read LREAD
do
    echo ${LREAD}
done < testfile
Escape Character in Linux
Escape Character in Linux

You can see from the output escape characters have lost their meaning and only n and t are printed instead of \n and \t. You can use -r to prevent backslash interpretation.

while read -r LREAD
do
    echo ${LREAD}
done < testfile
Prevent Backslash Interpretation
Prevent Backslash Interpretation

That’s it for this article. We would love to hear back from you if there are any feedbacks or tips. Your feedback is what helps us to create better content. Keep reading and keep supporting.

Karthick
A passionate software engineer who loves to explore new technologies. He is a public speaker and loves writing about technology, especially about Linux and open source.

Each tutorial at TecMint is created by a team of experienced Linux system administrators so that it meets our high-quality standards.

Join the TecMint Weekly Newsletter (More Than 156,129 Linux Enthusiasts Have Subscribed)
Was this article helpful? Please add a comment or buy me a coffee to show your appreciation.

Got something to say? Join the discussion.

Thank you for taking the time to share your thoughts with us. We appreciate your decision to leave a comment and value your contribution to the discussion. It's important to note that we moderate all comments in accordance with our comment policy to ensure a respectful and constructive conversation.

Rest assured that your email address will remain private and will not be published or shared with anyone. We prioritize the privacy and security of our users.