Moving around and looking at things

Objectives

• Learn how to navigate through folders.

• Learn how to create, copy, move, rename, and remove files and folders.

• Learn how to see the contents of files in the terminal.

• Learn how to compare files.

Instructor note

• Demo/teaching: 15 min

• Exercise: 15-20 min

Note

• First we will demonstrate a couple of commands and later we will use these in an exercise.

• If you want to type-along with the instructors, you can download and extract the example like this:

  cd
  wget https://gitlab.sigma2.no/training/tutorials/unix-for-hpc/-/raw/master/content/episodes/moving-around/demo-moving-around.tar.gz --no-check-certificate
  tar xvf demo-moving-around.tar.gz
  cd demo-moving-around
  

Navigating in the file tree

I use the following commands often to find out where I am and what files and folders are present in my current path/location:

This command prints the current working directory:

$ pwd

/cluster/home/user/demo-moving-around

With ls we can list the current directory:

$ ls

another-folder	numbers.out  poem.txt  some-folder  unfinished.txt

Most/all Unix commands have many options (“flags”). A flag is entered with a dash infront of the option: -. For example the long format:

$ ls -l

total 139
drwxr-xr-x 2 user user_g      1 Apr 28 21:03 another-folder
-rw-r--r-- 1 user user_g 140164 Apr 28 21:03 numbers.out
-rw-r--r-- 1 user user_g    359 Apr 28 21:03 poem.txt
drwxr-xr-x 2 user user_g      1 Apr 28 21:03 some-folder
-rw-r--r-- 1 user user_g    345 Apr 28 21:03 unfinished.txt

Or human-friendly output:

$ ls -h

total 139K
drwxr-xr-x 2 user user_g    1 Apr 28 21:03 another-folder
-rw-r--r-- 1 user user_g 137K Apr 28 21:03 numbers.out
-rw-r--r-- 1 user user_g  359 Apr 28 21:03 poem.txt
drwxr-xr-x 2 user user_g    1 Apr 28 21:03 some-folder
-rw-r--r-- 1 user user_g  345 Apr 28 21:03 unfinished.txt

We can list files replacing parts of the filename(s) by wildcards. All filenames ending in txt will be listed in this example:

$ ls -l *txt

-rw-r--r-- 1 bast bast_g 359 Apr 28 20:25 poem.txt
-rw-r--r-- 1 bast bast_g 345 Apr 28 20:55 unfinished.txt

All linux commands have a man (manual) page. Let’s have a look at the many options ls offers with the man command (exit with “q” for quit):

$ man ls

With cd we can change directory:

$ pwd
$ cd some-folder
$ pwd
$ cd ..
$ pwd

The .. refers to one directory “up”.

Instead of listing the current directory you can also give ls any other path as argument.

Why do some commands have so short/cryptic/strange names?

Many Unix commands are over 40 years old and when they got introduced, screen space was extremely limited. Some names were also meant to be funny or form funny recursive acronyms. But unfortunately not all command names were designed to be intuitive. Some names haven’t been carefully designed but just evolved from a personal tool that somebody developed over the weekend and their colleagues started using and then quickly became too popular to be changed.

Book recommendation to read about the early days of Unix: “Unix: A History and a Memoir” by Brian Kernighan

How to find back home

When you first connect to a cluster, you land in your home folder (try it and verify the location with pwd). These can tell us where our home folder is located:

$ echo $HOME
$ echo ~

All of these can bring you back to your home folder root:

$ cd $HOME
$ cd ~
$ cd

Creating, copying, renaming, moving, and removing

Note

Before we continue let’s make sure we are back in the demo folder:

cd && cd demo-moving-around

This is also an example of concatenating two commands with &&: first cd is performed to place you in your home folder, then cd demo-moving-around which is a subfolder of your home directory.

Sometimes we need to make new folder/directory:

$ mkdir new-folder

Let’s copy a file:

$ cp poem.txt poem2.txt

We can copy the file into another folder:

$ cp poem.txt new-folder

You can also copy several files at once. The last argument is the destination folder (and must be a folder):

$ cp poem.txt poem2.txt some-folder

The command mv can be used to move files or folders or to rename them (for Unix both are the same thing):

$ mv poem2.txt poem3.txt

Removing a file:

$ rm poem3.txt

Removing a directory (you can check what -r flag does with man rm):

$ rm -r new-folder

Printing and comparing file contents

The command cat can be used to concatenate several files into one but many people use it on one file only to print its contents:

$ cat poem.txt

Let’s try the commands more and less. They behave similarly and allow to page (scroll) through the file. They are useful if you don’t want to see the whole long file in one go. You can scroll with Enter and quit with “q”:

$ more numbers.out
$ less numbers.out

Commands head and tail only print the beginning and the end:

$ head poem.txt
$ tail poem.txt

The command diff is super useful to compare files or to check whether files are identical:

$ diff unfinished.txt poem.txt

1c1
< Working title
---
> Success is counted sweetest

Exercise

Exercise (20 min): Navigate in the file system and look at file content

It is perfectly fine to only do part of the steps below. They may be too many for 15-20 minutes, so if you don’t reach them all, do them as homework later on.

1. Connect to the cluster - check the hostname and locate where in the file system you are:

   $ hostname
   $ pwd
   

2. Download and extract the exercise folder:

   cd
   wget https://gitlab.sigma2.no/training/tutorials/unix-for-hpc/-/raw/master/content/episodes/moving-around/exercise-moving-around.tar.gz --no-check-certificate
   tar xvf exercise-moving-around.tar.gz
   

3. Step into the exercise folder:

   $ cd exercise-moving-around
   

4. Make a drawing on paper (or a mental picture) of the folder structure that you find under exercise-moving-around.

5. Find out in which subfolder are example output files with the ending .output.

6. Let us look at one of the files , computation-a.output in ./exercise-moving-around/results/ and compare the outputs of the following commands:

   $ cat  computation-a.output
   $ more computation-a.output
   $ less computation-a.output
   $ head computation-a.output
   $ tail computation-a.output
   

7. Find a directory called “backup”. Copy all the .output files to the backup folder.

8. In the folder where we found the .output files (not the backup folder), rename the output files to now have the ending .out instead of .output.

9. Among the output files there are two files with the same content (verify with diff), can you remove one of the duplicate files?

10. How would you copy the inputs folder under backup? How would you move the folder?

Solution

1. These commands will produce different outputs depending on where we are. Examples:

   $ hostname
   
   login-4.saga
   

   $ pwd
   
   /cluster/home/user
   

2. This step will hopefully produce the exercise folders. Nothing to change here.

3. Nothing to change here.

4. Drawing or mental picture of the folder structure. Something like this:

   exercise-moving-around
   ├── backup
   │   └── old
   │       └── old-computation.out
   ├── inputs
   │   └── example.in
   ├── results
   │   ├── computation-a.output
   │   ├── computation-b.output
   │   ├── computation-c.output
   │   ├── computation-c2.output
   │   ├── computation-d.output
   │   ├── computation-e.output
   │   └── computation-f.output
   └── scripts
       └── example.sh
   

5. One rudimentary was is to use ls to inspect the contents of the directories - combined with the wildcard symbol *

   $ ls ./*
   $ ls ./*/*
   $ ls ./*/*/*
   

   There are better ways using the find command, which will be taught later. But to spoil the suprise, here it comes:

   $ find . -name "*.output"
   

6. cat prints the whole file. more and less look similar and allow to page (scroll) through the file. You can scroll with Enter and quit with “q”. head and tail only print the beginning and the end.

7. First navigate to the results folder:

   $ cd results
   

   Then copy all files

   $ cp *.output ../backup
   

8. Here is a manual way (later we will learn more elegant ways to rename all in one go):

   $ mv computation-a.output computation-a.out
   $ mv computation-b.output computation-b.out
   
   ...
   

9. These two files have identical content (diff produces no output):

   $ diff computation-c.output computation-c2.output
   

   You can remove one of them like this:

   $ rm computation-c2.output
   

10. We can copy the entire folder like this:

    $ cp -r inputs backup
    

    Or we can move it:

    $ mv inputs backup
    

Keypoints

• With a handful of commands we can navigate, move, rename, copy.

• In Unix there are often many ways to achieve something.

• Commands often have options/”flags”. You can check manual pages with man.