How to Read Awk Input from STDIN in Linux – Part 7

In the previous parts of the Awk tool series, we looked at reading input mostly from a file(s), but what if you want to read input from STDIN.

In this Part 7 of Awk series, we shall look at few examples where you can filter the output of other commands instead of reading input from a file.

We shall start with the dir utility that works similar to ls command, in the first example below, we use the output of dir -l command as input for Awk to print owner’s username, groupname and the files he/she owns in the current directory:

# dir -l | awk '{print $3, $4, $9;}'
List Files Owned By User in Directory
List Files Owned By User in Directory

Take a look at another example where we employ awk expressions, here, we want to print files owned by the root user by using an expression to filter strings as in the awk command below:

# dir -l | awk '$3=="root" {print $1,$3,$4, $9;} '
List Files Owned by Root User
List Files Owned by Root User

The command above includes the (==) comparison operator to help us filter out files in the current directory which are owned by the root user. This is achieved using the expression $3==”root”.

Let us look at another example of where we use a awk comparison operator to match a certain string.

Here, we have used the cat utility to view the contents of a file named tecmint_deals.txt and we want to view the deals of type Tech only, so we shall run the following commands:

# cat tecmint_deals.txt
# cat tecmint_deals.txt | awk '$4 ~ /tech/{print}'
# cat tecmint_deals.txt | awk '$4 ~ /Tech/{print}'
Use Awk Comparison Operator to Match String
Use Awk Comparison Operator to Match String

In the example above, we have used the value ~ /pattern/ comparison operator, but there are two commands to try and bring out something very important.

When you run the command with pattern tech nothing is printed out because there is no deal of that type, but with Tech, you get deals of type Tech.

So always be careful when using this comparison operator, it is case sensitive as we have seen above.

You can always use the output of another command instead as input for awk instead of reading input from a file, this is very simple as we have looked at in the examples above.

Hope the examples were clear enough for you to understand, if you have any concerns, you can express them through the comment section below and remember to check the next part of the series where we shall look at awk features such as variables, numeric expressions and assignment operators.

Aaron Kili
Aaron Kili is a Linux and F.O.S.S enthusiast, an upcoming Linux SysAdmin, web developer, and currently a content creator for TecMint who loves working with computers and strongly believes in sharing knowledge.

Each tutorial at TecMint is created by a team of experienced Linux system administrators so that it meets our high-quality standards.

Join the TecMint Weekly Newsletter (More Than 156,129 Linux Enthusiasts Have Subscribed)
Was this article helpful? Please add a comment or buy me a coffee to show your appreciation.

17 thoughts on “How to Read Awk Input from STDIN in Linux – Part 7”

  1. Hello Mr Kili, I have two other questions.
    1 – I would like to add some code. Starting from this:
    tail -f /opt/red5-server108m2/log/red5.log | awk ‘{if (/Stream start:/){

    *insertion point*

    }

    … in that marked position I woud like to add a parse of an other file to search the presence of a name. So, if the name is not present no action start.

    2 – at the same position I woul like to operate a sting operation of $9 variable.
    ex: $9 is filled with this value “namex:name”
    How to split to have “name” only? I try several times but with syntax error only.

    Thank you in advance
    (sorry for my bad english)

    Reply
    • In this case, you need to approach the problem using the concept of built-in Awk variables, and that is the FS(input field separator) built-in variable.

      take for example, that the value of $9=”namex:name”

      therefore, you can use two awk commands, first, you can print out the value of $9 then pipe it to a second awk command specifying the FS value as in the command below:

      awk ‘ { print $9 } ‘ | awk -F ‘:’ ‘ { print $2 } ‘

      In the example above, the FS value is :, the value of $2 will be name, from the original value in $9

      I hope this will help, in case it does not, post the question on linuxsay.com and i will answer with some screen shots for better illustration.

      Reply
    • Thank you for fast, kind and comprensive reply to an old noob 60 years old.
      Now I begin to write some simple code about strings operation.

      Have you some gift for the 1st question?

      Giuseppe

      Reply
      • You are welcome @Giuseppe

        If i got the question right, i suppose you want to parse another file then search for a name in that file. The difficulty is therefore reading input from a second file apart from piped input.

        Let me look into the problem and figure out the solution, it will require a little time. I hope you can wait a little more, after all the solution is the most important.

        Reply
        • Hello Mr. Kili, you are correct. In that point marked I want sto scan a second text file where a list of user is stored. So if user is present the stream can start else not.

          Have a good day, Giuseppe

          Reply
          • @Guiseppe

            Sorry for the delay, but i hope this method with help solve your problem.

            Try to use the approach of an Awk script as follows:

            tail -f /opt/red5-server108m2/log/red5.log | awk -f script.awk file

            put awk command in script.awk and use the -f option to read the script as in the command above.

            script.awk

            ————————————————————————————–
            #!/bin/awk -f

            ‘{if (/Stream start:/){

            *insertion point*

            }

            Check this guide for more information: https://www.gnu.org/software/gawk/manual/html_node/Naming-Standard-Input.html

          • Thank you. Going to found a solution out of the pipe.
            Possibly reading the second file before the tail command. So I store results in an array to scan in the second pipe.

            Giuseppe

  2. Ho Mr. Kili,
    The problem was the file lock on the .m3u8 file. So I create a folder and fill it with the stream files. The I remove the folder.
    Here the final script:

    tail -f /opt/red5-server108m2/log/red5.log | awk ‘{if (/Stream start:/ “){

    a=” -nostats -loglevel 0 -i rtmp://xxx.xxx.xxx.xxx:1935/live/”;
    b=” -hls_time 2 -hls_list_size 5 -hls_wrap 10 -start_number 0 -f hls /home/tv/hls/”;
    c=”/home/tv/hls/”;
    system (“mkdir ” c $9);
    system (“/opt/ffmpeg/ffmpeg” a $9 b $9 “/” $9 “.m3u8 < /dev/null &");
    }
    else if (/Stream close:/)
    {
    system ("rm -rf " c $9);
    }
    }'

    Thank you again to point me in the right direction.

    Reply
    • @Guiseppe

      Thanks for letting us know, can you please point out how you solved it. That would be useful in the future to other users in case they encounter a similar issues.

      Many thanks…

      Reply
  3. Hi Mr. Kili

    the problem seem to be the “else” statement. The script erase the .ts files but won’t erase the .m3u8 file. Going crazy!

    Reply
    • Sorry that i did not get back, in the mean time, let me look into the script and follow up. I will get back to you as soon as i figure out and we shall share more. Hope that will help.

      Reply
  4. Here the code:

    tail -f /opt/red5-server108m2/log/red5.log | awk ‘{if (/Stream start:/){
    print “start ” $9 ” streaming”;

    a=” -nostats -loglevel 0 -i rtmp://xxx.xxx.xxx.xxx:1935/live/”;
    b=” -hls_time 2 -hls_list_size 5 -hls_wrap 10 -start_number 0 -f hls /home/tv/hls/”;

    system (“/opt/ffmpeg/ffmpeg” a $9 b $9 “.m3u8 < /dev/null ");

    }
    else if (/Stream close:/)
    {

    print "stop " $9 " streaming";

    c="/home/tv/hls/";

    system ("rm " c $9 "*" );

    }
    }'

    But if I run:
    tail -f /opt/red5-server108m2/log/red5.log | awk '{if (/Stream start:/){
    print "start " $9 " streaming";

    }
    else if (/Stream close:/)
    {

    print "stop " $9 " streaming";

    }

    }'

    work nice.

    Thank you in advance

    Reply
  5. Here the code:

    tail -f /opt/red5-server108m2/log/red5.log | awk ‘{if (/Stream start:/){
    print “start ” $9 ” streaming”;

    a=” -nostats -loglevel 0 -i rtmp://xxx.xxx.xxx.xxx:1935/live/”;
    b=” -hls_time 2 -hls_list_size 5 -hls_wrap 10 -start_number 0 -f hls /home/tv/hls/”;

    system (“/opt/ffmpeg/ffmpeg” a $9 b $9 “.m3u8 < /dev/null ");

    }
    else if (/Stream close:/)
    {

    print "stop " $9 " streaming";

    c="/home/tv/hls/";

    system ("rm " c $9 "*" );

    }
    }'

    Reply
  6. Hi Mr. Kili, I was out of office, back again.

    I wrote a simple bash to start or stop ffmpeg recoding on parsing the red5 server log. All seem to work fine with one session but if I run a second session with another recoding something strange happen.

    Can I add here my code?

    Reply
  7. Hello, first, thank you a lot for your job. I have some problem to get running a little script to run/stop a program parsing a log.

    Reply

Leave a Reply to Aaron Kili K Cancel reply

Thank you for taking the time to share your thoughts with us. We appreciate your decision to leave a comment and value your contribution to the discussion. It's important to note that we moderate all comments in accordance with our comment policy to ensure a respectful and constructive conversation.

Rest assured that your email address will remain private and will not be published or shared with anyone. We prioritize the privacy and security of our users.