When you want to search for the first occurence of a string in a file and exit, the below command may be useful, the file input.txt has the following contents,
Input file: input.txt
Start
Line: 1
Line: 2
Line: 3
Line: 4
Line: 5
End
To search for the first occurence of the word "Line:" from the above file and exit, use the below grep and awk combination,
cat input.txt grep "Line:" awk '{ print $2; exit }' (or) awk '/Line:/ { print $2;exit }' input.txt
Wednesday, February 24, 2010
The grep command in Unix and Linux and its common usage
I have been using the grep command in UNIX virtually everyday in my work but yet sometimes some of the file manipulation tasks annoy me, especially the pattern matching tasks for which I always needed a quick solution, so I decided to do a writeup on grep command and its practical applications, hope this may make the life of UNIX users a little easier.
The grep series of commands.... For a customary introduction, the grep command is used to find/print lines in a file that matches a pattern, it comes in different flavors, there is an enhanced version of the grep command, namely the egrep which does some advanced regular expression pattern matching on text files and the fgrep command which is used to search text in a file without any regular expession interpretation of the search pattern.
Some common grep usage options....
grep pattern filenames - Displays lines in files which matches the pattern.
grep -i pattern filenames - Displays lines in files which matches the pattern, ignores case while matching the pattern.
grep -c pattern filenames - Prints the count of the number of lines which matches the pattern in files.
grep -l pattern filenames - Lists the names of files which matches the pattern along with the lines which matched the pattern.
Similarly egrep command is used with different enhanced pattern matching options used with regular expressions like
egrep (x|y) filename - Displays lines which match the pattern x or y.
egrep (x|y)+ filename - Displays lines which matches atleast one or more text x or y.
fgrep is mainly used to avoid using search patterns as regular expressions or metacharacters
fgrep '*' filename - matches the occurence of pattern * in filenames, note here * doesn't mean a regular expression or a metacharacter.
Applications of grep command:
grep command can be used as a standalone utility to match a pattern in a file or piped to other utility commands like sed or awk to manilupate and extract text fom files, some of the common applications of grep from simple to little tricky ones are shown below.
The text file used to illustrate the below grep commands is shown below.
bash-3.00# cat textfile.txt This is line 1This is line 2This is line 3This is line 4This is line 5bash-3.00#1. Displaying or Saving a file with line numbers
There are several ways to save or display a text file with line numbers, some of them are
bash-3.00# grep -n '^' textfile.txt 1:This is line 12:This is line 23:This is line 34:This is line 45:This is line 5The above command matches the first line of the input file ('^' matches a pattern at the beginning of a line), or it matches all the lines in a file, one can also use grep -n '$' textfile.txt to produce the same output (the regular expression '$' means "ends with").
bash-3.00# grep -n '$' textfile.txt 1:This is line 12:This is line 23:This is line 34:This is line 45:This is line 5Also one can use the Regular expression ".*" to match and print all lines in the file with line numbers.
bash-3.00# grep -n '.*' textfile.txt 1:This is line 12:This is line 23:This is line 34:This is line 45:This is line 5The RE . (dot) is used to match a single character and the pattern * is used to match zero or more occurences, thus printing all line numbers when used with grep -n.
2. Deleting or Removing all lines in a file which matches a pattern
The grep -v command can be used display lines which DON'T match a pattern or in other words remove lines which matches a pattern in a file.
For example in the above text file, to remove or delete lines which matches the pattern "line 1", the command would be
bash-3.00# grep -v "line 1" textfile.txt This is line 2This is line 3This is line 4This is line 5To delete lines which matches the pattern "line 1" or "line 3", use the egrep command
bash-3.00# egrep -v '(line 1
line 3)' textfile.txt This is line 2This is line 4This is line 53. Deleting the first line which matches a pattern in a file
This is an example which uses the grep command piped to other UNIX commands like sed. To delete only the first line which matches a pattern in a file, use the following grep - sed pipe.
The below command deletes the first line which matches the pattern "line" in the above text file.
bash-3.00# grep "line" textfile.txt
sed 1dThis is line 2This is line 3This is line 4This is line 5The sed 1d command deletes the first line in the above grep output.
4. Deleting the last line which matches a pattern in a file
To delete the last line which matches a pattern in a file, use the grep - sed command pipe as shown below.
The below command deletes the last line which matches the pattern "line" in the above text file.
bash-3.00# grep "line" textfile.txt
sed '$d'This is line 1This is line 2This is line 3This is line 4The sed $d command deletes the last line in the above grep output.
Likewise there are countless applications of grep command piped with other utility commands like awk, sed, etc to match patterns in a file and I will add more as and when I find something interesting.
The grep series of commands.... For a customary introduction, the grep command is used to find/print lines in a file that matches a pattern, it comes in different flavors, there is an enhanced version of the grep command, namely the egrep which does some advanced regular expression pattern matching on text files and the fgrep command which is used to search text in a file without any regular expession interpretation of the search pattern.
Some common grep usage options....
grep pattern filenames - Displays lines in files which matches the pattern.
grep -i pattern filenames - Displays lines in files which matches the pattern, ignores case while matching the pattern.
grep -c pattern filenames - Prints the count of the number of lines which matches the pattern in files.
grep -l pattern filenames - Lists the names of files which matches the pattern along with the lines which matched the pattern.
Similarly egrep command is used with different enhanced pattern matching options used with regular expressions like
egrep (x|y) filename - Displays lines which match the pattern x or y.
egrep (x|y)+ filename - Displays lines which matches atleast one or more text x or y.
fgrep is mainly used to avoid using search patterns as regular expressions or metacharacters
fgrep '*' filename - matches the occurence of pattern * in filenames, note here * doesn't mean a regular expression or a metacharacter.
Applications of grep command:
grep command can be used as a standalone utility to match a pattern in a file or piped to other utility commands like sed or awk to manilupate and extract text fom files, some of the common applications of grep from simple to little tricky ones are shown below.
The text file used to illustrate the below grep commands is shown below.
bash-3.00# cat textfile.txt This is line 1This is line 2This is line 3This is line 4This is line 5bash-3.00#1. Displaying or Saving a file with line numbers
There are several ways to save or display a text file with line numbers, some of them are
bash-3.00# grep -n '^' textfile.txt 1:This is line 12:This is line 23:This is line 34:This is line 45:This is line 5The above command matches the first line of the input file ('^' matches a pattern at the beginning of a line), or it matches all the lines in a file, one can also use grep -n '$' textfile.txt to produce the same output (the regular expression '$' means "ends with").
bash-3.00# grep -n '$' textfile.txt 1:This is line 12:This is line 23:This is line 34:This is line 45:This is line 5Also one can use the Regular expression ".*" to match and print all lines in the file with line numbers.
bash-3.00# grep -n '.*' textfile.txt 1:This is line 12:This is line 23:This is line 34:This is line 45:This is line 5The RE . (dot) is used to match a single character and the pattern * is used to match zero or more occurences, thus printing all line numbers when used with grep -n.
2. Deleting or Removing all lines in a file which matches a pattern
The grep -v command can be used display lines which DON'T match a pattern or in other words remove lines which matches a pattern in a file.
For example in the above text file, to remove or delete lines which matches the pattern "line 1", the command would be
bash-3.00# grep -v "line 1" textfile.txt This is line 2This is line 3This is line 4This is line 5To delete lines which matches the pattern "line 1" or "line 3", use the egrep command
bash-3.00# egrep -v '(line 1
line 3)' textfile.txt This is line 2This is line 4This is line 53. Deleting the first line which matches a pattern in a file
This is an example which uses the grep command piped to other UNIX commands like sed. To delete only the first line which matches a pattern in a file, use the following grep - sed pipe.
The below command deletes the first line which matches the pattern "line" in the above text file.
bash-3.00# grep "line" textfile.txt
sed 1dThis is line 2This is line 3This is line 4This is line 5The sed 1d command deletes the first line in the above grep output.
4. Deleting the last line which matches a pattern in a file
To delete the last line which matches a pattern in a file, use the grep - sed command pipe as shown below.
The below command deletes the last line which matches the pattern "line" in the above text file.
bash-3.00# grep "line" textfile.txt
sed '$d'This is line 1This is line 2This is line 3This is line 4The sed $d command deletes the last line in the above grep output.
Likewise there are countless applications of grep command piped with other utility commands like awk, sed, etc to match patterns in a file and I will add more as and when I find something interesting.
Creating ssh tunnel from Windows and Solaris
I was trying to find the easiest way to create an ssh tunnel from Windows to a Solaris box via an authenticated gateway server to access a secure web page in the target server, though there are many ways to do that, I found that the easiest one is to use command line.
Assume that a Host A wants to tunnel HTTPS traffic to host C via an authenticated Gateway server host C (where you have an account), then the following steps would be helpful.
Lets assume that the target's server IP address is 10.2.152.2 and the gateway IP is 10.2.152.1, and you want to access https://10.2.152.2:9443 from the source system.
Creating ssh tunnel from Solaris:
bash-3.00$ ssh -L 9443:10.2.152.2:9443 -X username@10.2.152.1
Password: ********
Last login: Wed Oct 21 20:03:26 2009 from system25
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
bash-3.00$
Now from a firefox browser if you type https://localhost:9443, the HTTPS traffic will be tunneled to https://10.2.152.2:9443.
Creating ssh tunnel from Windows:
Even though there are many ways to do it from Windows using putty, if you are an UNIX user who love command line, its almost similar to that of the above procedure.
1. Download putty (and cd to the directory where you cn find putty.exe).
2. C:\putty_install> putty -L 9443:10.2.152.2:9443 -X username@10.2.152.1
(A putty window opens)
Using username "username".
Using keyboard-interactive authentication.
Password: ********
Last login: Wed Oct 21 20:03:28 2009 from system25
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
bash-3.00$
Now https://localhost:9443 from Windows system will fetch https://10.2.152.2:9443.
Assume that a Host A wants to tunnel HTTPS traffic to host C via an authenticated Gateway server host C (where you have an account), then the following steps would be helpful.
Lets assume that the target's server IP address is 10.2.152.2 and the gateway IP is 10.2.152.1, and you want to access https://10.2.152.2:9443 from the source system.
Creating ssh tunnel from Solaris:
bash-3.00$ ssh -L 9443:10.2.152.2:9443 -X username@10.2.152.1
Password: ********
Last login: Wed Oct 21 20:03:26 2009 from system25
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
bash-3.00$
Now from a firefox browser if you type https://localhost:9443, the HTTPS traffic will be tunneled to https://10.2.152.2:9443.
Creating ssh tunnel from Windows:
Even though there are many ways to do it from Windows using putty, if you are an UNIX user who love command line, its almost similar to that of the above procedure.
1. Download putty (and cd to the directory where you cn find putty.exe).
2. C:\putty_install> putty -L 9443:10.2.152.2:9443 -X username@10.2.152.1
(A putty window opens)
Using username "username".
Using keyboard-interactive authentication.
Password: ********
Last login: Wed Oct 21 20:03:28 2009 from system25
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
bash-3.00$
Now https://localhost:9443 from Windows system will fetch https://10.2.152.2:9443.
Increasing SWAP space in Solaris
Increasing SWAP space is a common task especially when its a requirement to install and run a new application, though its quite trivial, a handy How-to doc is always useful and hence this one.
Assuming that you need to increase swap size in Solaris by 4GB, follow the below procedure.
1. Get the current swap size.
bash-3.00# swap -l
swapfile dev swaplo blocks free
/dev/dsk/c0t2d0s1 31,65 8 16787912 16787912
2. Create a file of size 4G (say swapfile4G)
bash-3.00# mkfile 4g /swapfile4G
bash-3.00#
3. Add the specific file as swap
bash-3.00# swap -a /swapfile4G
bash-3.00#
4. Now the swap -l command should show the new file added as swap
bash-3.00# swap -l
swapfile dev swaplo blocks free
/dev/dsk/c0t2d0s1 31,65 8 16787912 16787912
/swapfile4G - 8 8388600 8388600
Additionally to enable this swap file after system boot, add the following entry in /etc/vfstab.
/swapfile4G - - swap - no -
Assuming that you need to increase swap size in Solaris by 4GB, follow the below procedure.
1. Get the current swap size.
bash-3.00# swap -l
swapfile dev swaplo blocks free
/dev/dsk/c0t2d0s1 31,65 8 16787912 16787912
2. Create a file of size 4G (say swapfile4G)
bash-3.00# mkfile 4g /swapfile4G
bash-3.00#
3. Add the specific file as swap
bash-3.00# swap -a /swapfile4G
bash-3.00#
4. Now the swap -l command should show the new file added as swap
bash-3.00# swap -l
swapfile dev swaplo blocks free
/dev/dsk/c0t2d0s1 31,65 8 16787912 16787912
/swapfile4G - 8 8388600 8388600
Additionally to enable this swap file after system boot, add the following entry in /etc/vfstab.
/swapfile4G - - swap - no -
Subscribe to:
Posts (Atom)