How to Use Awk and Regular Expressions to Filter Text or String in Files

When we run certain commands in Unix/Linux to read or edit text from a string or file, we most times try to filter output to a given section of interest. This is where using regular expressions comes in handy.

Read Also: 10 Useful Linux Chaining Operators with Practical Examples

What are Regular Expressions?

A regular expression can be defined as a strings that represent several sequence of characters. One of the most important things about regular expressions is that they allow you to filter the output of a command or file, edit a section of a text or configuration file and so on.

Features of Regular Expression

Regular expressions are made of:

  1. Ordinary characters such as space, underscore(_), A-Z, a-z, 0-9.
  2. Meta characters that are expanded to ordinary characters, they include:
    1. (.) it matches any single character except a newline.
    2. (*) it matches zero or more existences of the immediate character preceding it.
    3. [ character(s) ] it matches any one of the characters specified in character(s), one can also use a hyphen (-) to mean a range of characters such as [a-f], [1-5], and so on.
    4. ^ it matches the beginning of a line in a file.
    5. $ matches the end of line in a file.
    6. \ it is an escape character.

In order to filter text, one has to use a text filtering tool such as awk. You can think of awk as a programming language of its own. But for the scope of this guide to using awk, we shall cover it as a simple command line filtering tool.

The general syntax of awk is:

# awk 'script' filename

Where 'script' is a set of commands that are understood by awk and are execute on file, filename.

It works by reading a given line in the file, makes a copy of the line and then executes the script on the line. This is repeated on all the lines in the file.

The 'script' is in the form '/pattern/ action' where pattern is a regular expression and the action is what awk will do when it finds the given pattern in a line.

How to Use Awk Filtering Tool in Linux

In the following examples, we shall focus on the meta characters that we discussed above under the features of awk.

A simple example of using awk:

The example below prints all the lines in the file /etc/hosts since no pattern is given.

# awk '//{print}'/etc/hosts
Awk Prints all Lines in a File

Awk Prints all Lines in a File

Use Awk with Pattern:

I the example below, a pattern localhost has been given, so awk will match line having localhost in the /etc/hosts file.

# awk '/localhost/{print}' /etc/hosts 
Awk Print Given Matching Line in a File

Awk Print Given Matching Line in a File

Using Awk with (.) wild card in a Pattern

The (.) will match strings containing loc, localhost, localnet in the example below.

That is to say * l some_single_character c *.

# awk '/l.c/{print}' /etc/hosts
Use Awk to Print Matching Strings in a File

Use Awk to Print Matching Strings in a File

Using Awk with (*) Character in a Pattern

It will match strings containing localhost, localnet, lines, capable, as in the example below:

# awk '/l*c/{print}' /etc/localhost
Use Awk to Match Strings in File

Use Awk to Match Strings in File

You will also realize that (*) tries to a get you the longest match possible it can detect.

Let look at a case that demonstrates this, take the regular expression t*t which means match strings that start with letter t and end with t in the line below:

this is tecmint, where you get the best good tutorials, how to's, guides, tecmint. 

You will get the following possibilities when you use the pattern /t*t/:

this is t
this is tecmint
this is tecmint, where you get t
this is tecmint, where you get the best good t
this is tecmint, where you get the best good tutorials, how t
this is tecmint, where you get the best good tutorials, how tos, guides, t
this is tecmint, where you get the best good tutorials, how tos, guides, tecmint

And (*) in /t*t/ wild card character allows awk to choose the the last option:

this is tecmint, where you get the best good tutorials, how to's, guides, tecmint

Using Awk with set [ character(s) ]

Take for example the set [al1], here awk will match all strings containing character a or l or 1 in a line in the file /etc/hosts.

# awk '/[al1]/{print}' /etc/hosts
Use-Awk to Print Matching Character in File

Use-Awk to Print Matching Character in File

The next example matches strings starting with either K or k followed by T:

# awk '/[Kk]T/{print}' /etc/hosts 
Use Awk to Print Matched String in File

Use Awk to Print Matched String in File

Specifying Characters in a Range

Understand characters with awk:

  1. [0-9] means a single number
  2. [a-z] means match a single lower case letter
  3. [A-Z] means match a single upper case letter
  4. [a-zA-Z] means match a single letter
  5. [a-zA-Z 0-9] means match a single letter or number

Lets look at an example below:

# awk '/[0-9]/{print}' /etc/hosts 
Use Awk To Print Matching Numbers in File

Use Awk To Print Matching Numbers in File

All the line from the file /etc/hosts contain at least a single number [0-9] in the above example.

Use Awk with (^) Meta Character

It matches all the lines that start with the pattern provided as in the example below:

# awk '/^fe/{print}' /etc/hosts
# awk '/^ff/{print}' /etc/hosts
Use Awk to Print All Matching Lines with Pattern

Use Awk to Print All Matching Lines with Pattern

Use Awk with ($) Meta Character

It matches all the lines that end with the pattern provided:

# awk '/ab$/{print}' /etc/hosts
# awk '/ost$/{print}' /etc/hosts
# awk '/rs$/{print}' /etc/hosts
Use Awk to Print Given Pattern String

Use Awk to Print Given Pattern String

Use Awk with (\) Escape Character

It allows you to take the character following it as a literal that is to say consider it just as it is.

In the example below, the first command prints out all line in the file, the second command prints out nothing because I want to match a line that has $25.00, but no escape character is used.

The third command is correct since a an escape character has been used to read $ as it is.

# awk '//{print}' deals.txt
# awk '/$25.00/{print}' deals.txt
# awk '/\.00/{print}' deals.txt
Use Awk with Escape Character

Use Awk with Escape Character

Summary

That is not all with the awk command line filtering tool, the examples above a the basic operations of awk. In the next parts we shall be advancing on how to use complex features of awk. Thanks for reading through and for any additions or clarifications, post a comment in the comments section.

If You Appreciate What We Do Here On TecMint, You Should Consider:

TecMint is the fastest growing and most trusted community site for any kind of Linux Articles, Guides and Books on the web. Millions of people visit TecMint! to search or browse the thousands of published articles available FREELY to all.

If you like what you are reading, please consider buying us a coffee ( or 2 ) as a token of appreciation.

Support Us

We are thankful for your never ending support.

Aaron Kili

Aaron Kili is a Linux and F.O.S.S enthusiast, an upcoming Linux SysAdmin, web developer, and currently a content creator for TecMint who loves working with computers and strongly believes in sharing knowledge.

Your name can also be listed here. Got a tip? Submit it here to become an TecMint author.

RedHat RHCE and RHCSA Certification Book
Linux Foundation LFCS and LFCE Certification Preparation Guide

You may also like...

32 Responses

  1. John says:

    I like this tutorial, but the animated gifs are so annoying that I gave up after the l.c and will look elsewhere.

  2. Gulzhan says:

    Very good tutorial. as a beginner it is very useful, I really appreciate.

    • Ravi Saive says:

      @Gulzhan,

      Thanks for finding this awk tutorial useful and thanks for appreciating our work..all credit goes to our authors for creating such quality articles for our readers like you..

  3. Sh says:

    there is no file named localhost in /etc/ directory…
    # awk ‘/l*c/{print}’ /etc/localhost

    The gif is fine but the command is wrong

  4. Erik Persson says:

    Nice tutorial.

    There are however some things that are not correct. "*" is NOT short for “any number of characters”. "*" in regular expressions means zero or more of the *preceding* character.

    For example /l*c/ matches all strings containing any number of l followed by a c. Thus it matches lc, llc, lllc, llllllc, but since any number of l can be zero it also matches just c.

    Thus /l*c/ is equivalent to all strings containing a c. l* is totally superfluous. All strings matched by /l*c/ will also be matched by /c/, and all strings matched by /c/ will also be matched by /l*c/. /t*t/ matches any string containing a *single* t.

    Again, in this situation the t* will not do anything. If you want to match any character you use a period, ex /t.*t/ matches any string containing two t (a t, followed any number of any character, and then a t).

    If you want to match a string beginning with a t and ending with a t you need “anchors”. The regular expression /^t.*t$/ matches a string starting and ending with a t.

    If you want to match a string containing a word beginning with a t and ending with a t you need word boundaries. I don’t know if you have word boundaries in awk regular expressions. However you do have them in perl regular expressions. You may use white space “\s” to compensate for the lack of word boundaries but then you must also know that the start and end of a string is not considered word boundaries. Thus, if you want to match all strings containing words beginning and ending with a t you need something like /(^|\s)t\S*t(\s|$)/

    /Erik

  5. Suresh Ravanam says:

    This is the best awk tutorial ever. I never understood awk so thoroughly. A simple explanation of awk sysntax(i.e., awk pattern action file) is enough to understand the awk command which i have not read anywhere else. thanks a lot.

  6. erramah says:

    Thank you guys it is really useful website and I appreciate your effort.

  7. Kostyanius says:

    Hi,
    Here is a little typo in tutorial in first awk command.
    awk ‘//{print}’/etc/hosts
    It should be devided to: awk ‘//{print}’ /etc/hosts so as whitespace was missed and this command doesn`t do anything but just waiting for something.

  8. Shashank says:

    The best knowledge full page is this, keep posting

  9. kcdtv says:

    Thanks for this tutorial!
    I always used “grep” but awk seems to do very well the job and the syntax is a bit more friendly to my taste.
    I wonder if it is not even more efficient than “grep” too.
    Correct me if i am wrong, but with grep we always need to use a pipe.
    I assume that awk is a bit more efficient (for file crawling) and that we should try to use it instead of grep (if we can.)
    Am I wrong?
    awk is so powerfull and amazing, i am impatient to read the next chapter :)

    • Aaron Kili K says:

      Thanks for sharing your experience, both grep and awk are great tools to use. As you have mentioned grep sometimes needs a pipe to deal with filtering text of strings.

      But a user always has to find something convenient to use.

    • me says:

      In the case you just want simple search, use grep, it is faster. AWK is full scripting programming language with syntax similar to C and can do tricks you cannot do with simple grep.

      AWK has good support for associative arrays and that is a strong “tool” when you know how to use it (simple key-value DB). AWK is great tool to process TEXT files. It is easy for AWK to do a calculations or reformat file to the form you want to have, like just extract important information, do a statistic, find what is missing in file, etc.

      Is AWK slow? My experience is that it a little bit faster than Python, it depends on task. AWK is great tool to write pipe filters but it can do more. This article is nice way to introduce AWK as tool to “grep” text.

      • Aaron Kili says:

        @me

        Well written, this is a good explanation to summarize the comparison between grep and Awk, as well as uncovering some powerful features of Awk as a text processing language. Thanks for stopping by.

  10. Tomas says:

    Awk is very useful for printing colums, but everything else can be achieved with grep.

    • Aaron Kili K says:

      Yes, grep is a nice tool when dealing with columns, but it will depend on user’s convenience

    • Joseph in Atlanta says:

      The “grep” tool lets you search for a string/pattern easily, but grep is NOT very powerful.

      MANY more things can be done with the logic/code abilities of “awk”… (see examples below)

      Search for NUMBER range in column three:

      awk ‘$3>20 && $3=10 && NR<=800 {if($6 ~"^[+-]?[0-9.]*){vx=$6+0;ss+=vx,nv++} else {nx++;tx[NR]=$3} END{if(!nv){print "No nums in col six"} else {print "Scan",nv,"values. Sum:",ss,"Avg:",ss/nv,"Non Nums:"nx}}}'

Got something to say? Join the discussion.

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.