So, in the first part of this series of blogs it was suggested that you learn and be comfortable with using a shell (
bash) and text editor (
vim).
This post follows on from that, giving you some pointers to what utilities you should familiarise yourself with.
Unix Philosophy
As you progress and learn / gain more experience you will realise that the Unix/Linux (*nix) landscape is made up of lots of small little utility programs, each program does something, and does it well - DOTADIW, “do one thing and do it well”.
These tools share common traits,
- They will be accompanied by documentation
- They can take input from STDIN, send errors/warnings to STDERR, and output to STDOUT
- They are modular, and can be combined together to perform amazing things
This post aims to introduce you to a few must know utilities, and how they enshrine the Unix philosophy, and will give you an idea of how they can be combined to great effect.
RTFM
Haven’t come across this acronym before? You will hear and see it a lot. RTFM. Read The F*****g Manual.
If someone says this to you, it’s not with malice, the acronym has good intentions, the intention is that the information is there for you, you just need to help yourself, learn and teach yourself and be self sufficient - teach someone to fish and all that...
Most utilities that are installed on *nix systems will be accompanied with manual pages accessed via a utility called “man”.
Want to see what arguments and/or optional switches a utility can take?
man <command>
will display the manual for said <command>.
For the most part, manual pages are well written and contain a complete guide on how to use the <command>, with examples in the majority of cases. They can be long, and there is a temptation to look up the information elsewhere as a result, but know that the information is there for you to reference.
Why don’t you try starting with
man man
i.e. the manual page for the “man” utility.
No manual page provided for <command>? Try
<command> --help
<command> -?
Either should give you a short brief guide on how to execute said <command>.
For all the utilities that I suggest here, it is well worth spending a bit of time reading the manual pages, there will be some useful command switches that you will simply commit to memory, and soon enough muscle memory.
Navigating the file system
You will be working in a shell A LOT, knowing how to navigate the file system, as well as creating, updating and deleting files & folders is a minimum.
In general, get to know the following commands:
- ls
- list contents of directories
- cd
- change directories
- mkdir
- create directories
- mv
- move / rename files and directories
- rm
- delete files
- rmdir
- remove directories
- cp
- copy files and directories
- find
- find files and directories
- ln
- linking files and directories
Working with files
By now you would already know how to open, edit and delete files with vim. But, what else can you do?
- file - unsure what type a file is, use file
- cat - concatenate and print files [to STDOUT]
- less
- open a file in read-only mode
- more
- same as less
- more or less :-)
- head
- display first lines of a text file
- tail
- display last lines of a text file
- grep
- find text in a file
- tar
- create / extract tar archives
Working with text
Working on the command line you’ll be working almost exclusively with text whether that be directly on the command line or from text files, so it’s important to know what utilities you can call upon to manipulate text.
- tr
- find and replace text
- sed
- a stream editor taking input from files or STDIN, and outputting manipulated text to STDOUT
- awk
- scripting language for manipulating text data and generating reports
- regular expressions (regex) - not a utility, but a concept/language, almost any *nix text tool will have some implementation of regex.
Working with networks
These days it’s almost impossible to do anything without having a network, so knowing what utilities are available to help you find out more about the network that you are on, how to navigate, and lookup information is vitally important - after all, there will be occasions where you or someone working with you will have some network connectivity issues that will require investigation and troubleshooting.
- nslookup
- lookup an IP address of a host
- dig
- DNS lookup utility
- nc
- useful for testing connectivity
- netstat
- show network statistics
- ifconfig
- configure / view network interfaces
- curl
- transfer data from or to a server
- tcpdump
- dump traffic on a network
- ssh
- remote onto another machine securely
What is STDIN / STDOUT / STDERR?
I’ve mentioned these acronyms in previous sections, without actually telling you what they are and what they do.
Simply put they are input / output streams:
- STDIN
- this is where a program can receive input from
- STDOUT
- by default, this is where a program will output information to
- STDERR
- if there are any errors, a program will generally output to this stream
Many of the command utilities mentioned in the blog post will typically take input from STDIN
and output information to STDOUT.
Bring it all together
Input / output streams can be redirected to files, other streams, even other utilities by using “>”, “<”, and “|” operators.
Why is this useful? Let’s say we have a large directory, with 10s or 100s of files and/or directories. Executing the “ls” command will cause your terminal to scroll, you won’t be able to view / search through the directory listing. What you want to be able to do is to view the output in something like more, or less.
So, what you can do is redirect the output, save it to a file, and then open up the file in less.
ls > directory_listing.txt
less directory_listing.txt
Actually, do I even need to save the output of ls to a file before opening it in less? No, I don’t. I can simply “pipe” the output of ls
into less:
ls | less
This is a very simple illustration of the use of I/O stream redirection and the use of pipes (pipes are how you chain commands together), as there is virtually no limit to how many commands you can combine together, the only limit is often your imagination and creativity.
For example, take the below:
date +%s | sha256sum | base64 | head -c 32
This combines 4 utilities (date, sha256sum, base64, and head), seemingly unrelated, with the only commonality that they take input from STDIN
and output to STDOUT, combined together to create a simple password generator. Try executing each of the utilities separately to see what happens, and if you get stuck or get an error (which you will do), then reference the man
page to find out what you may have done wrong and how to fix it.
The use of I/O stream redirection, and especially the use of pipes to combine commands perfectly illustrates the Unix philosophy described before, with each utility DOTADIW, that can be combined to great effect.
You should also know about these utilities:
- tee
- redirects STDIN
to both STDOUT
and one or more files
- xargs
- take STDIN
as an argument to a utility
Read more about I/O stream redirection here, and what other operators are available:
Summary
You’ll use many of these commands on a daily basis, there is no escaping this fact. Many of the commands and option switches will be committed to muscle memory, but for those commands that you won’t be executing on a daily basis, such as sed
and awk, it’s important to know that they are there, what they can do, and how they can be combined.
Do spend a bit of time reading the manual pages for these utilities, get a feel for what problem they are trying to solve, its approach to solving that problem, and how it can be used effectively.
Knowing this will save you large amounts of time, reduce mistakes and therefore, increase your accuracy and productivity - and, you’ll go from zero to hero in no time.