Newer
Older
author: "Laurent Modolo"
---
```{r include = FALSE}
if (!require("fontawesome")) {
install.packages("fontawesome")
}
library(fontawesome)
knitr::opts_chunk$set(echo = TRUE)
knitr::opts_chunk$set(comment = NA)
```
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">
<img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" />
</a>
Objective: Learn basics of batch processing in GNU/Linux
In the previous section, we have seen how to handle streams and text. We can use this knowledge to generate list of command instead of text. This is called batch processing.
In everyday life, you may want to run command sequentiality without using pipes.
To run `CMD1` and then run `CMD2` you can use the `;` operator
```
CMD1 ; CMD2
```
To run `CMD1` and then run `CMD2` if `CMD1` didn’t throw an error, you can use the `&&` operator which is safer than the `;` operator.
```sh
CMD1 && CMD2
```
You can also use the `||` to manage errors and run `CMD2` if `CMD1` failed.
```sh
CMD1 || CMD2
```
The easiest option to execute list of command is to use `xargs`. `xargs` reads arguments from **stdin** and use them as arguments for a command. In UNIX systems the command `echo` send string of character into **stdout**. We are going to use this command to learn more about `xargs`.
In general a string of character differs from a command when it’s placed between quotes.
The two following commands are equivalent, why ?
```sh
echo "file1 file2 file3" | xargs touch
touch file1 file2 file3
```
You can display the command executed by `xargs` with the switch `-t`.
By default the number of arguments sent by `xargs` is defined by the system. You can change it with the option `-n N`, where `N` is the number of arguments sent. Use the option `-t` and `-n` to run the previous command as 3 separate `touch` commands.
<details><summary>Solution</summary>
<p>
```sh
echo "file1 file2 file3" | xargs -t -n 1 touch
```
</p>
</details>
Sometime, the arguments are not separated by space but by other characters. You can use the `-d` option to specify them. Execute `touch`1 time from the following command:
```sh
echo "file1;file2;file3"
```
<details><summary>Solution</summary>
<p>
```sh
echo "file1;file2;file3" | xargs -t -d \; touch
```
</p>
</details>
To reuse the arguments sent to `xargs` you can use the command `-I` which defines a string corresponding to the argument. Try the following command, what does the **man**ual says about the `-c` option of the command `cut` ?
```sh
ls -l file* | cut -c 44- | xargs -t -I % ln -s % link_%
```
Instead of using `ls` the command `xargs` is often used with the command `find`. The command `find` is a powerful command to search for files.
Modify the following command to make a non-hidden copy of all the file with a name starting with *.bash* in your home folder
```sh
find . -name ".bash*" | sed 's|./.||g'
```
<details><summary>Solution</summary>
<p>
```sh
find . -name ".bash*" | sed 's|./.||g' | xargs -t -I % cp .% %
```
</p>
</details>
You can try to remove all the files in the `/tmp` folder with the following command:
Modify this command to remove every folder in the `/tmp` folder.
<details><summary>Solution</summary>
<p>
```sh
find /tmp/ -type d | xargs -t rm -R
```
</p>
</details>
`xargs` It is a simple solution for writing batch commands, but if you want to write more complex command you are going to need to learn `awk`. `awk` is a programming language by itself, but you don’t need to know everything about `awk` to use it.
You can to think of `awk` as a `xargs -I $N` command where `$1` correspond to the first column `$2` to the second column, etc.
There are also some predefined variables that you can use like.
- `$0` Correspond to all the columns.
- `FS` the field separator used
- `NF` the number of fields separated by `FS`
- `NR` the number of records already read
A `awk` program is a chain of commands with the form `motif { action }`
- the `motif` define where there `action` is executed
- The keyword `BEGIN`or `END` (before reading the first line, and after reading the last line)
- a combination of the three separated by `&&` (AND), `||`(OR) and `!` (Negation)
- a range of line `motif_1,motif_2`
With `awk` you can
Modify this command to only display the total number of line with awk (like `wc -l`)
<details><summary>Solution</summary>
<p>
```sh
awk 'END{ print NR }' file
```
</p>
</details>
Convert a tabulated sequences file into fasta format
```sh
awk -vOFS='' '{print ">",$1,"\n",$2,"\n";}' two_column_sample_tab.txt > sample1.fa
```
Modify this command to only get a list of sequence names in a fasta file
<details><summary>Solution</summary>
<p>
```sh
awk -vOFS='' '{print $1 "\n";}' two_column_sample_tab.txt > seq_name.txt
```
</p>
</details>
Convert a multiline fasta file into a single line fasta file
```sh
awk '!/^>/ { printf "%s", $0; n = "\n" } /^>/ { print n $0; n = "" } END { printf "%s", n }' sample.fa > sample1_singleline.fa
```
Convert fasta sequences to uppercase
```sh
awk '/^>/ {print($0)}; /^[^>]/ {print(toupper($0))}' file.fasta > file_upper.fasta
```
Modify this command to only get a list of sequence names in a fasta file un lowercase
<details><summary>Solution</summary>
<p>
```sh
awk '/[^>]/ {print(tolower($0))}' file.fasta > seq_name_lower.txt
```
</p>
</details>
Return a list of sequence_id sequence_length from a fasta file
```sh
awk 'BEGIN {OFS = "\n"}; /^>/ {print(substr(sequence_id, 2)" "sequence_length); sequence_length = 0; sequence_id = $0}; /^[^>]/ {sequence_length += length($0)}; END {print(substr(sequence_id, 2)" "sequence_length)}' file.fasta
```
Count the number of bases in a fastq.gz file
```sh
(gzip -dc $0) | awk 'NR%4 == 2 {basenumber += length($0)} END {print basenumber}'
```
```sh
awk 'BEGIN {OFS = "\n"} {header = $0 ; getline seq ; getline qheader ; getline qseq ; if (length(seq) >= 20){print header, seq, qheader, qseq}}' < input.fastq > output.fastq
```
When you start writing complicated command, you may want to save them to reuse them later.
You can find everything that you are typing in your `bash`in the `~/.bash_history` file, but working with this file can be tedious as it also contains all the command that you mistype. A good solution, for reproducibility is to write `bash` scripts. A bash script is simply a text file that contains a sequence of `bash`commands.
As you use `bash` in your terminal, you can execute a `bash` script with the following command:
It’s usual to write the `.sh` extension for `shell`scripts.
Write a bash script named `download_hg38.sh` that download the [hg38.ncbiRefSeq.gtf.gz](http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/genes/hg38.ncbiRefSeq.gtf.gz) file, then extract it and that says that it has done it.
The `\` character like in regexp cancel the meaning of what follow, you can use it to split your one-liner scripts over many lines to use the `&&` operator.
<details><summary>Solution</summary>
<p>
```sh
wget http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/genes/hg38.ncbiRefSeq.gtf.gz && \
gzip -dc hg38.ncbiRefSeq.gtf.gz && \
echo "download and extraction complete"
```
</p>
</details>
In your first bash script, the only thing saying that your script is a bash script is its extension. But most of the time UNIX system doesn’t care about file extension, a text file is a text file.
To tell the system that your text file is a bash script you need to add a **shebang**. A **shebang** is a special first line that starts with a `#!` followed by the path of the interpreter for your script.
For example, for a bash script in a system where `bash` is installed in `/bin/bash` the **shebang** is:
```bash
```
When you are not sure `which`is the path of the tools available to interpret your script, you can use the following shebang:
```bash
```
You can add a **shebang** to your script and add it the e**x**ecutable right.
Now you can execute your script with the command:
```bash
./download_hg38.sh
```
Congratulations you wrote your first program !
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
Where did they `/usr/bin/env` find the information about your bash ? Why did we have to write a `./` before our script if we are in the same folder ?
This is all linked to the **PATH** bash variable. Like in many programming languages `bash` have what we call *variables*. *variables* are named storage for temporary information. You can print a list of all your environment variables (variables loaded in your `bash` memory), with the command `printenv`.
To create a new variable you can use the following syntax:
```sh
VAR_NAME="text"
VAR_NAME2=2
```
Create a `IDENTIY` variable with your first and last names.
<details><summary>Solution</summary>
<p>
```sh
IDENTITY="First name Last Name"
```
</p>
</details>
It’s good practice to write your `bash` variable in uppercase with `_` in place of spaces.
You can access the value of an existing `bash` variable with the `$VAR_NAME`
To display the value of your `IDENTITY` variable with `echo` you can write:
```sh
echo $IDENTITY
```
When you want to mix variable value and text you can use the two following syntax:
```sh
echo "my name is "$IDENTITY
echo "my name is ${IDENTITY}"
```
Going back to the `printenv` You can see a **PWD** variable that store your current path, a **SHELL** variable that store your current shell, and you can see a **PATH** variable that stores a loot of file path separated by `:`.
The **PATH** variable contains every folder where to look for executable programs. Executable programs can be binary files or text files with a **shebang**.
Display the content of `PATH` with `echo`
<details><summary>Solution</summary>
<p>
```sh
echo $PATH
```
</p>
</details>
You can create a `scripts`folder and move your `download_hg38.sh` script in it. Then we can modify the `PATH` variable to include the `scripts` folder in it.
> Don’t erase your `PATH` variable !
<details><summary>Solution</summary>
<p>
```sh
mkdir ~/scripts
mv `download_hg38.sh` ~/scripts/
PATH=$PATH:~/scripts/
```
</p>
</details>
You can check the result of your command with `echo $PATH`
Try to call your `download_hg38.sh` from anywhere on the file tree. Congratulation you installed your first UNIX program !
You can pass argument to your bash scripts, writing the following command:
```sh
my_script.sh arg1 arg2 arg3
```
Means that from within the script:
- `$0` will give you the name of the script (`my_script.sh`)
- `$1`, `$2`, `$3`, `$n` will give you the value of the arguments (`arg1`, `arg2`, `arg3`, `argn`)
- `$$` the process id of the current shell
- `$#` the total number of arguments passed to the script
- `$@`the value of all the arguments passed to the script
- `$?` the exit status of the last executed command
- `$!`the process id of the last executed command
You can write the following `variables.sh` script in your `scripts` folder:
```sh
echo "Name of the script: $0"
echo "Total number of arguments: $#"
echo "Values of all the arguments: $@"
```
And you can try to call it with some arguments !
> We have used the following commands:
>
> - `echo` to display text
> - `xarg` to execute a chain of commands
> - `awk` to execute complex chain of commands
> - `;` `&&` and `||` to chain commands
> - `source` to load a script
> - `shebang` to specify the language of a script
> - `PATH` to install script
In the next session, we are going to learn how to execute command on other computers with [ssh.](./10_network_and_ssh.html)