Objective: Learn basics of batch processing in GNU/Linux
In the previous section, we have seen how to handle streams and text. We can use this knowledge to generate list of command instead of text. This is called batch processing.
In everyday life, you may want to run command sequentiality without using pipes.
To run `CMD1` and then run `CMD2` you can use the `;` operator
```
CMD1 ; CMD2
```
To run `CMD1` and then run `CMD2` if `CMD1` didn't throw an error, you can use the `&&` operator which is safer than the `;` operator.
```sh
CMD1 && CMD2
```
You can also use the `||` to manage errors and run `CMD2` if `CMD1` failed.
```sh
CMD1 || CMD2
```
## Executing list of commands
The easiest option to execute list of command is to use `xargs`. `xargs` reads arguments from **stdin** and use then as argument for a command. In Unix systems the command `echo` send string of character into **stdout**. We are going to use this command to learn more about `xargs`.
```sh
echo"hello world"
```
In general a string of character differs from a command when it's placed between quotes.
The two following commands are equivalent, why ?
```sh
echo"file1 file2 file3" | xargs touch
touch file1 file2 file3
```
You can display the command executed by `xargs` with the switch `-t`.
By default the number of arguments sent by `xargs` is defined by the system. You can change it with the option `-n N`, where `N` is the number of arguments sent. Use the option `-t` and `-n` to run the previous command as 3 separate `touch` commands.
<details><summary>Solution</summary>
<p>
```sh
echo"file1 file2 file3" | xargs -t-n 1 touch
```
</p>
</details>
Sometime, the arguments are not separated by space but by other characters. You can use the `-d` option to specify them. Execute `touch`1 time from the following command:
```sh
echo"file1;file2;file3"
```
<details><summary>Solution</summary>
<p>
```sh
echo"file1;file2;file3" | xargs -t-d\;touch
```
</p>
</details>
To reuse the arguments sent to `xargs` you can use the command `-I` which defines a string corresponding to the argument. Try the following command, what does the **man**ual says about the `-c` option of the command `cut` ?
You can try to remove every file in the `/tmp` folder with the following command:
```sh
find /tmp/ -type f | xargs -trm
```
Modify this command to remove every directly in the `/tmp` folder.
<details><summary>Solution</summary>
<p>
```sh
find /tmp/ -type d | xargs -trm-R
```
</p>
</details>
## Writing `awk` commands
`xargs` Is a simple solution for writing batch commands, but if you want to write more complex command you are going to need to learn `awk`. `awk` is a programming language by itself, but you don't need to know everything about `awk` to use it.
You can to think of `awk` as a `xargs -I $N` command where `$1` correspond to the first column `$2` to the second column, etc...
There are also some predefined variables that you can use like
-`$0` Correspond to all the columns.
-`FS` the field separator used
-`NF` the number of field separated by `FS`
-`NR` the number for records already read
A `awk` program is a chain of commands with the form `motif { action }`
- the `motif` define where there `action` is executed
- the `action` is what you want to do
The `motif` can be
- a regexp
- The keyword `BEGIN`or `END`
- a comparison like `<`, `<=`, `==`, `>=`, `>` or `!=`
- a combination of the three separated by `&&` (AND), `||`(OR) and `!` (Negation)
- a range of line `motif_1,motif_2`
With `awk` you can
Count the number of line in a file
```sh
awk'{ print NR " : " $0 }' file
```
Convert a tabulated sequences file into fasta format
When you start writing complicated command, you may want to save them to use them later.
You can find everything that you are typing in your `bash`in the `~/.bash_history` but working with this file can be tedious as it also contains all the command that you mistype. A good solution, for reproducibility is to write `bash` scripts. A bash script is simply a text file that contains a sequence of `bash`commands.
To execute a `bash` script you can use the following command:
```bash
source myscrip.sh
```
It's usual to write the `.sh` extension for `shell`scripts.
