by Michael Phan
•
12 June 2020
So, in the first part of this series of blogs it was suggested that you learn and be comfortable with using a shell ( bash ) and text editor ( vim ). This post follows on from that, giving you some pointers to what utilities you should familiarise yourself with. Unix Philosophy As you progress and learn / gain more experience you will realise that the Unix/Linux (*nix) landscape is made up of lots of small little utility programs, each program does something, and does it well - DOTADIW, “do one thing and do it well”. These tools share common traits, They will be accompanied by documentation They can take input from STDIN , send errors/warnings to STDERR , and output to STDOUT They are modular, and can be combined together to perform amazing things This post aims to introduce you to a few must know utilities, and how they enshrine the Unix philosophy, and will give you an idea of how they can be combined to great effect. RTFM Haven’t come across this acronym before? You will hear and see it a lot. RTFM. Read The F*****g Manual. If someone says this to you, it’s not with malice, the acronym has good intentions, the intention is that the information is there for you, you just need to help yourself, learn and teach yourself and be self sufficient - teach someone to fish and all that... Most utilities that are installed on *nix systems will be accompanied with manual pages accessed via a utility called “ man ”. Want to see what arguments and/or optional switches a utility can take? man
will display the manual for said
. For the most part, manual pages are well written and contain a complete guide on how to use the
, with examples in the majority of cases. They can be long, and there is a temptation to look up the information elsewhere as a result, but know that the information is there for you to reference. Why don’t you try starting with man man i.e. the manual page for the “ man ” utility. No manual page provided for
? Try
--help
-? Either should give you a short brief guide on how to execute said
. For all the utilities that I suggest here, it is well worth spending a bit of time reading the manual pages, there will be some useful command switches that you will simply commit to memory, and soon enough muscle memory. Navigating the file system You will be working in a shell A LOT, knowing how to navigate the file system, as well as creating, updating and deleting files & folders is a minimum. In general, get to know the following commands: ls - list contents of directories cd - change directories mkdir - create directories mv - move / rename files and directories rm - delete files rmdir - remove directories cp - copy files and directories find - find files and directories ln - linking files and directories Working with files By now you would already know how to open, edit and delete files with vim . But, what else can you do? file - unsure what type a file is, use file cat - concatenate and print files [to STDOUT ] less - open a file in read-only mode more - same as less - more or less :-) head - display first lines of a text file tail - display last lines of a text file grep - find text in a file tar - create / extract tar archives Working with text Working on the command line you’ll be working almost exclusively with text whether that be directly on the command line or from text files, so it’s important to know what utilities you can call upon to manipulate text. tr - find and replace text sed - a stream editor taking input from files or STDIN , and outputting manipulated text to STDOUT awk - scripting language for manipulating text data and generating reports regular expressions (regex) - not a utility, but a concept/language, almost any *nix text tool will have some implementation of regex. Working with networks These days it’s almost impossible to do anything without having a network, so knowing what utilities are available to help you find out more about the network that you are on, how to navigate, and lookup information is vitally important - after all, there will be occasions where you or someone working with you will have some network connectivity issues that will require investigation and troubleshooting. nslookup - lookup an IP address of a host dig - DNS lookup utility nc - useful for testing connectivity netstat - show network statistics ifconfig - configure / view network interfaces curl - transfer data from or to a server tcpdump - dump traffic on a network ssh - remote onto another machine securely What is STDIN / STDOUT / STDERR? I’ve mentioned these acronyms in previous sections, without actually telling you what they are and what they do. Simply put they are input / output streams: STDIN - this is where a program can receive input from STDOUT - by default, this is where a program will output information to STDERR - if there are any errors, a program will generally output to this stream Many of the command utilities mentioned in the blog post will typically take input from STDIN and output information to STDOUT . Bring it all together Input / output streams can be redirected to files, other streams, even other utilities by using “ > ”, “ < ”, and “ | ” operators. Why is this useful? Let’s say we have a large directory, with 10s or 100s of files and/or directories. Executing the “ ls ” command will cause your terminal to scroll, you won’t be able to view / search through the directory listing. What you want to be able to do is to view the output in something like more , or less . So, what you can do is redirect the output, save it to a file, and then open up the file in less . ls > directory_listing.txt less directory_listing.txt Actually, do I even need to save the output of ls to a file before opening it in less ? No, I don’t. I can simply “pipe” the output of ls into less : ls | less This is a very simple illustration of the use of I/O stream redirection and the use of pipes (pipes are how you chain commands together), as there is virtually no limit to how many commands you can combine together, the only limit is often your imagination and creativity. For example, take the below: date +%s | sha256sum | base64 | head -c 32 This combines 4 utilities ( date , sha256sum , base64 , and head ), seemingly unrelated, with the only commonality that they take input from STDIN and output to STDOUT , combined together to create a simple password generator. Try executing each of the utilities separately to see what happens, and if you get stuck or get an error (which you will do), then reference the man page to find out what you may have done wrong and how to fix it. The use of I/O stream redirection, and especially the use of pipes to combine commands perfectly illustrates the Unix philosophy described before, with each utility DOTADIW, that can be combined to great effect. You should also know about these utilities: tee - redirects STDIN to both STDOUT and one or more files xargs - take STDIN as an argument to a utility Read more about I/O stream redirection here, and what other operators are available: https://www.tldp.org/LDP/abs/html/io-redirection.html Summary You’ll use many of these commands on a daily basis, there is no escaping this fact. Many of the commands and option switches will be committed to muscle memory, but for those commands that you won’t be executing on a daily basis, such as sed and awk , it’s important to know that they are there, what they can do, and how they can be combined. Do spend a bit of time reading the manual pages for these utilities, get a feel for what problem they are trying to solve, its approach to solving that problem, and how it can be used effectively. Knowing this will save you large amounts of time, reduce mistakes and therefore, increase your accuracy and productivity - and, you’ll go from zero to hero in no time.