Hello! Welcome to your next step in mastering Bash scripting. In this lesson, we will immerse ourselves in the world of text processing with the versatile command-line tool awk. awk is a powerful tool that allows you to manipulate and analyze text files with ease. By the end of this lesson, you’ll be equipped to efficiently handle and process text files, extracting meaningful data and performing relevant computations directly from your Bash scripts.

Let's get started by diving into how we can leverage awk for various text processing tasks.

First, let's create a sample data file to work with. This file will help us learn and practice various awk commands effectively.

The heredoc (short for "here document") is a special syntax in Unix shell scripting that allows you to create a multi-line string. It is particularly useful for creating files or including large blocks of text within your script. The syntax is <<EOF ... EOF, where EOF (End of File) is a marker indicating the beginning and end of the block of text. You can actually use any marker, but EOF is conventionally used.

Let's create a file called data.txt that includes data about computers in inventory.

#!/bin/bash

# Create a sample data file
cat << EOF > data.txt
Brand   Model     RAM
Apple   MacBook    32
Apple   iPad       16
Dell    XPS        32
Dell    Inspiron  128
Lenovo  ThinkPad  128
Lenovo  Yoga      256
Apple   MacBook    64
EOF

Let's break this code down:

• cat << EOF: This starts the heredoc and tells the cat command to begin reading the subsequent lines as a string until it encounters the ending EOF marker.
• > data.txt: This redirects the output of the cat command to a file named data.txt.
• The lines between << EOF and EOF are the content that will be written to data.txt.

The basic syntax of the awk command in Unix-like systems is:

awk options 'selection_criteria {action}' input-file > output-file

Here's a detailed breakdown of each component:

• awk: The command itself.
• options: These are optional flags you can pass to awk to modify its behavior (e.g., -F to specify the field separator).
• selection_criteria: This is an optional condition or pattern that specifies which lines of the input file to process. It can be a regular expression or a logical condition based on field values.
• {action}: This is the block of code to execute for each line that matches the selection criteria. Actions are enclosed in curly braces {}.
• input-file: The file that awk processes.
• > output-file: This optional part redirects the output to a file. If omitted, awk prints the output to the terminal.

With this understanding of awk syntax, let's dive into some examples.

Let's begin with the most basic awk command to print the entire content of the file. The print command is used to output text, fields, or expressions to the terminal or another output stream. It offers flexibility in how the data is displayed and allows custom formatting of text.

#!/bin/bash

# Using `awk` to print the entire file
awk '{print}' data.txt

In this awk command

• There are no options or selection_criteria
• The action is {print} enclosed in curly braces. The print pattern-action statement in awk tells it to print each line of the file.
• There is no output-file, so the result is displayed on the terminal.

Running this command will display all lines of the data.txt file, mirroring the functionality of the cat command.

NR stands for "Number of Records" in awk. It is a built-in variable that keeps track of the current line number being processed in the input file. Each time awk reads a new line, it increments NR by one. This makes NR useful for actions based on the line number, such as skipping headers, processing specific lines, or adding line numbers to output.

Skip Header Line

Suppose we want to print every line, excluding the header line ("Brand Model RAM)". The header line has an NR value of 1. To skip this line, we use the condition NR > 1.

#!/bin/bash

awk 'NR > 1 {print}' data.txt

• NR > 1: This condition skips the first line (header) and prints the remaining lines.

The output is:

Apple   MacBook    32
Apple   iPad       16
Dell    XPS        32
Dell    Inspiron  128
Lenovo  ThinkPad  128
Lenovo  Yoga      256
Apple   MacBook    64

Process a Specific Line

Now let's print only the 3rd line of data.txt.

#!/bin/bash

awk 'NR == 3 {print}' data.txt

The output is:

Apple   iPad       16

Pattern matching is one of the core strengths of awk, allowing you to perform actions only on lines that match specific patterns. The syntax for pattern matching in awk involves enclosing regular expressions within slashes (/pattern/). Suppose we only want to print lines that contain "Apple":

#!/bin/bash

# Using `awk` with pattern matching
awk '/Apple/ {print}' data.txt

• The /Apple/ {print} pattern checks each line for the string "Apple."
• If a line contains "Apple," it is printed.

The output of the command is:

Apple   MacBook    32
Apple   iPad       16
Apple   MacBook    64

The END keyword is used to specify an action to be executed after all lines have been processed. The syntax is:

awk '{action1} END {action2}'

This command will perform action1 for every line of text. After all lines have been processed, action2 is run once.

Now let's write a command that calculates the average RAM across all entries. To do this

• We create a sum variable and count variable.
• For each line, we add the RAM value (column $3) to sum and increment the count variable by 1.
• After all lines have been processed, we print sum/count

#!/bin/bash

# Using `awk` to calculate the average RAM
awk 'NR>1 {sum+=$3; count++} END {print "Average RAM:", sum/count}' data.txt

• NR>1 skips the header line because it does not contain a RAM value.
• {sum+=$3; count++} sums the values of the third field (RAM) and increments the count.
• END {print "Average RAM:", sum/count} executes after processing all lines, printing the calculated average RAM.

The output of the code is:

Average RAM: 93.7143

Great job! In this lesson, you learned how to:

• Create and manipulate data files using heredoc syntax.
• Print the entire content of a file using awk.
• Extract specific columns from a file using field numbers.
• Filter lines based on conditions using awk.
• Perform pattern matching with awk.
• Calculate and display average values using the END block.
• Customize and format output using printf in awk.

Now, it’s time to apply what you’ve learned. Head to the practice section to sharpen your awk skills through hands-on exercises. Happy coding!