diff --git a/.gitignore b/.gitignore index 03854cd0f858632e17961eb51e5593bcba25f349..888590f54820659953ec9d68d8a66d463c973409 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ *.html .DS_Store +.Rproj.user diff --git a/10_network_and_ssh.Rmd b/10_network_and_ssh.Rmd index 7f5244dd4ad2c79133beef96b222e518b0b3e566..ed0f994306861be2b969cb921fc1fff96aadd398 100644 --- a/10_network_and_ssh.Rmd +++ b/10_network_and_ssh.Rmd @@ -1,4 +1,35 @@ -# Ssh +--- +title: Text manipulation +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" +--- + +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') + +``` +## Ssh [](http://creativecommons.org/licenses/by-sa/4.0/) @@ -8,11 +39,11 @@ In the previous section, we have seen how to run scripts and complex commands on Most of the content from this session are from [wikipedia.org](https://wikipedia.org) -## Network +### Network First before talking about how to communicate over a network, we need to define what is a network in computational science. We can distinguish between two types of network, **circuit switching** networks and **packet switching** networks. -### circuit switching +#### circuit switching Circuit switching is the historical telephonic network architecture. When device A wants to communicate with device B, it has to establish a connection over the network. In a circuit switching network, the connections between a chain of nodes (hopefully the shortest chain) are established and fixed. Device A connects to the closest node and ask connection to Device B, this node will do the same thing to the closest node to Device B, so on and so forth until the connection reach Device B. @@ -20,7 +51,7 @@ If you try to call someone, who is already in a phone conversation, the line wil  -### packet switching +#### packet switching Packet switching is a method of grouping data over the network into packets. Each packet has a header and a payload. The header data can be read by each node to direct the packet to its destination. The header data also inform the Host 2 of the packets order. The payload contains the data that we want to transmit over the network. In packet switching, the network bandwidth is not pre-allocated like in circuit switching. Each packet is called a datagram. @@ -30,7 +61,7 @@ Packet switching is a method of grouping data over the network into packets. Eac In a packet switching network when you send a flux of data (video, sound, etc.), you have the illusion of continuity like for process switching handled by the scheduler. -## **Internet Protocol** (IP) +### **Internet Protocol** (IP) > The **Internet Protocol** (**IP**) is the principal [communications protocol](https://en.wikipedia.org/wiki/Communications_protocol) in the [Internet protocol suite](https://en.wikipedia.org/wiki/Internet_protocol_suite) for relaying [datagrams](https://en.wikipedia.org/wiki/Datagram) across network boundaries. Its [routing](https://en.wikipedia.org/wiki/Routing) function enables [internetworking](https://en.wikipedia.org/wiki/Internetworking), and essentially establishes the [Internet](https://en.wikipedia.org/wiki/Internet). @@ -38,7 +69,7 @@ IP has the task of delivering [packets](https://en.wikipedia.org/wiki/Packet_(in The first major version of IP, [Internet Protocol Version 4](https://en.wikipedia.org/wiki/IPv4) (IPv4), is the dominant protocol of the Internet. Its successor is [Internet Protocol Version 6](https://en.wikipedia.org/wiki/IPv6) (IPv6), which has been in increasing [deployment](https://en.wikipedia.org/wiki/IPv6_deployment) on the public Internet since c. 2006. -### IPv4 +#### IPv4 An **IPv4** is composed of 4 digits ranging from 0 to 255 separated by `.` , which gives an address space of 4294967296 (2^32) addresses. Some combinations of **IPv4** are restricted: @@ -61,7 +92,7 @@ An **IPv4** is composed of 4 digits ranging from 0 to 255 separated by `.` , whi | 240.0.0.0/4 | 240.0.0.0–255.255.255.254 | 268435455 | Internet | Reserved for future use.[[15\]](https://en.wikipedia.org/wiki/IPv4#cite_note-rfc3232-15) (Former Class E network). | | 255.255.255.255/32 | 255.255.255.255 | 1 | Subnet | Reserved for the "limited [broadcast](https://en.wikipedia.org/wiki/Broadcast_address)" destination address.[[6\]](https://en.wikipedia.org/wiki/IPv4#cite_note-rfc6890-6)[[16\]](https://en.wikipedia.org/wiki/IPv4#cite_note-rfc919-16) | -### IPv6 +#### IPv6 An **IPv6** is composed of 8 groups of 4 digits long number separated by `:`. The numbers are in hexadecimal format (number of base 16, randing from 0 to 9 and A to F). Compared to **IPv4**, **IPv6** allows for 2^128 = 340,282,366,920,938,463,463,374,607,431,768,211,456 addresses (approximately 3.4×10^38). For example, an IP address is: *2001:0db8:0000:0000:0000:ff00:0042:8329* @@ -69,13 +100,13 @@ To display your VM IP addresses you can use the following command: `ip address s Local **IPv6** addresses start with **fe80::** -### **Domain Name System** (**DNS**) +#### **Domain Name System** (**DNS**) Instead of using IP addresses in your everyday life, you often use the domain name. The DNS is composed of many DNS servers that are hierarchically organized and decentralized. By querying the DNS with a particular domain name, the correct name server will return the corresponding IP address. For most network tools, you can use domain names (URL) or IP addresses.  -### Transmission Control Protocol (**TCP**) +#### Transmission Control Protocol (**TCP**) The **Transmission Control Protocol** (**TCP**) is one of the main [protocols](https://en.wikipedia.org/wiki/Communications_protocol) of the [Internet protocol suite](https://en.wikipedia.org/wiki/Internet_protocol_suite). TCP provide, reliable, ordered, and error-checked delivery of a stream of data between applications running on hosts communincating over an IP network. @@ -86,7 +117,7 @@ The **Transmission Control Protocol** (**TCP**) is one of the main [protocols](h - includes traffic congestion control - Heavtweight (no ordering of messages, no tracking connections, etc. It is a very simple transport layer designed on top of IP) -### **User Datagram Protocol** (**UDP**) +#### **User Datagram Protocol** (**UDP**) UDP uses a simple [connectionless communication](https://en.wikipedia.org/wiki/Connectionless_communication) model with a minimum of protocol mechanisms. @@ -96,7 +127,7 @@ UDP uses a simple [connectionless communication](https://en.wikipedia.org/wiki/C - Multicast (a single datagram packet can be automatically routed without duplication to a group of subscribers) - Lightweight (no ordering of messages, no tracking connections, etc. It is a very simple transport layer designed on top of IP) -### Port +#### Port Higher, communication protocols like TCP and UDP, also define **port**. A **port** is a communication endpoint. When software wants to communicate overt TCP or UDP it will do so using a specific **port**. Each system has **port** numbers ranging from 0 to 65535. **Port** numbered from 0 through 1023 are system **ports** used by well-known processes (you need specific rights to use them). @@ -122,7 +153,7 @@ Here are a list of notable port numbers: Nowadays, **ports** provide multiplexing, which means that multiple service or communication session can use the same **port** number. -## SSH +### SSH There are numerous other protocols ([RTP](https://en.wikipedia.org/wiki/Real-time_Transport_Protocol) for example). But most of them run over the TCP and UDP protocols. **SSH** or **Secure Shell** is one of them. SSH is a [cryptographic](https://en.wikipedia.org/wiki/Cryptography) [network protocol](https://en.wikipedia.org/wiki/Network_protocol) for operating network services securely over an unsecured network. @@ -140,25 +171,25 @@ ps -el | grep "ssh" SSH uses [Public-key cryptography (or asymmetric cryptography](https://en.wikipedia.org/wiki/Public-key_cryptography)), to secure it’s communications. -### Public-key cryptography +#### Public-key cryptography [Public-key cryptography (or asymmetric cryptography](https://en.wikipedia.org/wiki/Public-key_cryptography)), is a cryptographic system which uses pairs of [keys](https://en.wikipedia.org/wiki/Cryptographic_key): *public keys* (which may be known to others), and *private keys* (which may never be known by any except the owner). A cryptographic algorithm is used to generate a pair of *public* and *private* keys from a large random number. Then, the 3 following scheme can be used to secure communication: -### Communicate with the server +#### Communicate with the server The server sent a public key to the client on the first connection.  -### Share a secret +#### Share a secret Can be used to share public keys (see [Diffie-Hellman)](https://fr.wikipedia.org/wiki/%C3%89change_de_cl%C3%A9s_Diffie-Hellman).  -### Authentification +#### Authentification - The server sends a random string of characters to the client - The client crypt the random string and send it back to the server @@ -166,7 +197,7 @@ Can be used to share public keys (see [Diffie-Hellman)](https://fr.wikipedia.org  -## SSH Server +### SSH Server By default, on the IFB, password authentication is disabled to enforce the use of public key based authentication. To learn `ssh` command we are going to enable this option on your VMs. Find the`sshd` configuration file and open it with the editor of your choice. @@ -223,7 +254,7 @@ sudo passwd student Give the password and your IP on the chat. -## SSH client +### SSH client To connect of an SSH server you can use the following command: @@ -241,7 +272,7 @@ Check the content of the `~/.ssh/` folder, where is saved the server public key Congratualtion you are connected on a VM through another VM ! -### Key authentication +#### Key authentication Every time, that you want to connect to the ssh server, you have to type your account password, this password is encrypted and send over the network. Instead you can use a pair of private and public key to authenticate yourself. @@ -279,11 +310,11 @@ ssh login@IP_adress -i ~/.ssh/id_ed25519_otherVM Congratulations, you authenticated yourself on a remote server without sending your password over the network ! -## SSH based tools +### SSH based tools Sometime, you want to do other things than executing commands on a remove computer. For example, you may want to transfer files over the network. -### scp +#### scp The `scp` command comes with the `ssh` client installation you can use it to transfer file from your computer to the ssh sever: @@ -301,7 +332,7 @@ scp login@IP_adress:remote/path local/path To transfer directory you can use the `-r` witch -### rsync +#### rsync `scp` Is a basic command for file transfer. If you want advanced process bar and file integrity checking, you can use the `rsync` command instead. @@ -313,13 +344,13 @@ rsync -auv local/path login@IP_adress:remote/path Will only transfer files from `local/path` not already present in `remote/path`. The `-c` switch will compute a checksum of the file locally and remotely to be certain that they are identical. -### sshfs +#### sshfs You can use the `sshfs` command to mount a remote folder over ssh on your computer. -## SSH tips +### SSH tips -### IFB authentication +#### IFB authentication The default authentication method for the IFB uses keys generated with the `rsa` algorithm @@ -333,7 +364,7 @@ Instead of using the `ssh-copy-id` command, you are going to copy paste your pub You can now use the [RainBio catalogue](https://biosphere.france-bioinformatique.fr/catalogue/) to launch any available VMs and connect to is with SSH from your current VM. -### SSH configuration +#### SSH configuration Long ssh command can be tedious to use. This is why we are now going to explore the last file in the `.ssh` folder: `.ssh/config`. diff --git a/11_install_system_programs.Rmd b/11_install_system_programs.Rmd index 0f9bc158faff07eea5c446cb5c762d102685d18f..5d7c255a638f7cdeba557fa8f253ec34ce425e93 100644 --- a/11_install_system_programs.Rmd +++ b/11_install_system_programs.Rmd @@ -1,4 +1,36 @@ -# Install system programs +--- +title: Text manipulation +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" +--- + +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') + +``` + +## Install system programs [](http://creativecommons.org/licenses/by-sa/4.0/) @@ -8,7 +40,7 @@ As we have seen in the [4 unix file system](http://perso.ens-lyon.fr/laurent.mod Developers don’t want to reinvent the wheel each time they want to write complex instruction in their programs, this is why they use shared library of pre-written complex instruction. This allows for quicker development, fewer bugs (we only have to debug the library once and use it many times), and also [better memory management](http://perso.ens-lyon.fr/laurent.modolo/unix/6_unix_processes.html#processes-tree) (we only load the library once and it can be used by different programs). -## Package Manager +### Package Manager However, interdependencies between programs and libraries can be a nightmare to handle manually this is why most of the time when you install a program you will use a [package manager](https://en.wikipedia.org/wiki/Package_manager). [Package manager](https://en.wikipedia.org/wiki/Package_manager) are system tools that will handle automatically all the dependencies of a program. They rely on **repositories** of programs and library which contains all the information about the trees of dependence and the corresponding files (**packages**). @@ -39,7 +71,7 @@ docker run -it --volume /:/root/chroot alpine sh -c "chroot /root/chroot /bin/ba </p> </details> -### Installing R +#### Installing R **R** is a complex program that relies on loots of dependencies. Your current VM run on Ubuntu, so we are going to use the `apt` tool (`apt-get` is the older version of the `apt` command, `synaptic` is a graphical interface for `apt-get`) @@ -57,7 +89,7 @@ sudo apt install r-base-core What is the **R** version that you installed ? Is there a newer version of **R** ? -### Adding a new repository +#### Adding a new repository You can check the list of repositories that `apt` checks in the file `/etc/apt/sources.list`. @@ -75,7 +107,7 @@ Then you must add the public key of this repository: sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 ``` -### Updating the repository list +#### Updating the repository list You can now use `apt` to update your repository list dans try to reinstall **r-base-core** @@ -111,7 +143,7 @@ sudo apt search linux-image -### Language specific package manager +#### Language specific package manager If it’s not a good idea to have different **package manager** on the same system (they don’t know how the dependencies are handled by the other’s manager). You will also encounter language specific package manager: @@ -138,7 +170,7 @@ Next-time use `pip` with the `--user` switch. -## Manual installation +### Manual installation Sometimes, a specific tool that you want to use will not be available through a **package manager**. If you are lucky, you will find a **package** for your distribution. For `apt` the **package** are `.deb` files. @@ -154,7 +186,7 @@ wget https://github.com/Automattic/simplenote-electron/releases/download/v2.7.0/ You can then use `apt` to install this file. -## From sources +### From sources If the program is open source, you can also [download the sources](https://github.com/Automattic/simplenote-electron/archive/v2.7.0.tar.gz) and build them. diff --git a/12_virtualization.Rmd b/12_virtualization.Rmd index a58622539ae76810eb7db6441190ce2a4af75b08..aadfb4f42776519babc5e82aa934264009f3bf83 100644 --- a/12_virtualization.Rmd +++ b/12_virtualization.Rmd @@ -1,4 +1,36 @@ -# Install system programs +--- +title: Text manipulation +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" +--- + +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') + +``` + +## Install system programs [](http://creativecommons.org/licenses/by-sa/4.0/) @@ -16,21 +48,21 @@ To avoid the overhead of simulating every component of the **guest** system, whi There are different levels of virtualisation which correspond to different levels of isolation between the virtual machine (**guest**) and the real computer (**host**). -## Full virtualization +### Full virtualization A key challenge for full virtualization is the interception and simulation of privileged operations, such as I/O instructions. The effects of every operation performed within a given virtual machine must be kept within that virtual machine – virtual operation cannot be allowed to alter the state of any other virtual machine, the control program, or the hardware. Some machine instructions can be executed directly by the hardware, since their effects are entirely contained within the elements managed by the control program, such as memory locations and arithmetic registers. But other instructions that would "pierce the virtual machine" cannot be allowed to execute directly; they must instead be trapped and simulated. Such instructions either access or affect state information that is outside the virtual machine. -## Paravirtualization +### Paravirtualization In paravitualization, the virtual hardware of the **guest** system is similar to the hardware of the **host**. The goal is to reduce the portion of the **guest** execution time spent to simulate hardware which is the same as the **host** hardware. The paravirtualization provides specially defined **hooks** to allow the **guest** and **host** to request and acknowledge these tasks, which would otherwise be executed in the virtual domain (where execution performance is worse). A hypervisor provides the virtualization of the underlying computer system. In [full virtualization](https://en.wikipedia.org/wiki/Full_virtualization), a guest operating system runs unmodified on a hypervisor. However, improved performance and efficiency is achieved by having the guest operating system communicate with the hypervisor. By allowing the guest operating system to indicate its intent to the hypervisor, each can cooperate to obtain better performance when running in a virtual machine. This type of communication is referred to as paravirtualization. -## OS-level virtualization +### OS-level virtualization **OS-level virtualization** is an [operating system](https://en.wikipedia.org/wiki/Operating_system) paradigm in which the [kernel](https://en.wikipedia.org/wiki/Kernel_(computer_science)) allows the existence of multiple isolated [user space](https://en.wikipedia.org/wiki/User_space) instances. Such instances, called **containers** may look like real computers from the point of view of programs running in them. Programs running inside a container can only see the container's contents and devices assigned to the container. -## VirtualBox +### VirtualBox VirtualBox is own by oracle, you can add the following repository to get the last version: @@ -100,7 +132,7 @@ VBoxManage startvm Ubuntu20.04 Why did this last command fail ? Which kind of virtualisation VirtualBox is using ? -## Docker +### Docker Docker is an **OS-level virtualization** system where the virtualization is managed by the `docker` daemon. @@ -164,7 +196,7 @@ docker rmi rocker/rstudio:3.2.0 Try to run the `mcr.microsoft.com/windows/servercore:ltsc2019` container, what is happening ? -### Building your own container +#### Building your own container You can also create your own container by writing a container recipe. For Docker this file is named `Dockerfile` @@ -196,7 +228,7 @@ docker build ./ -t 'ubuntu_with_htop' -## Singularity +### Singularity Like Docker, Singularity is an **OS-level virtualization**. This main difference with docker is that the user is the same within and outside a container. Singularity is available on the [neuro.debian.net](http://neuro.debian.net/install_pkg.html?p=singularity-container) repository, you can add this source with the following commands: diff --git a/1_understanding_a_computer.Rmd b/1_understanding_a_computer.Rmd index 2ef2d3717b520218a27c5713362226dbd88aa713..9de76d5dbf0b432f3f5c7594e1716828d9a16bc1 100644 --- a/1_understanding_a_computer.Rmd +++ b/1_understanding_a_computer.Rmd @@ -1,56 +1,83 @@ --- -title: First step in a terminal - +title: Understanding a computer +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" --- -# Understanding a computer - -[](http://creativecommons.org/licenses/by-sa/4.0/) +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') + +``` + +<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> +<img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /> +</a> Objective: understand the relations between the different components of a computer ---- +## Which parts are necessary to define a computer ? -# Which parts are necessary to define a computer ? +## Computer components +### CPU (Central Processing Unit) ---- +{width=100%} -# Computer components +### Memory -## CPU (Central Processing Unit) - +#### RAM (Random Access Memory) -## Memory +{width=100%} -### RAM (Random Access Memory) -<img src="./img/220px-Swissbit_2GB_PC2-5300U-555.jpg" alt="RAM" style="zoom:150%;" /> +#### HDD (Hard Disk Drive) / SSD (Solid-State Drive) -### HDD (Hard Disk Drive) / SSD (Solid-State Drive) -<img src="./img/220px-Laptop-hard-drive-exposed.jpg" alt="HDD" style="zoom:150%;" /> -<img src="./img/220px-SSD_Samsung_960_PRO_512GB_-_front_and_back_-_2018-05-27.jpg" alt="SSD" style="zoom:150%;" /> +{width=100%} +{width=100%} -## Motherboard - +### Motherboard -## GPU (Graphical Processing Unit) - +{width=100%} -## Alimentation - +### GPU (Graphical Processing Unit) ---- +{width=100%} + +### Alimentation -# Computer model: universal Turing machine +{width=100%} + +--- +## Computer model: universal Turing machine - +{width=100%} --- -# As simple as a Turing machine ? +## As simple as a Turing machine ? - +{width=100%} - A tape divided into cells, one next to the other. Each cell contains a symbol from some finite alphabet. - A head that can read and write symbols on the tape and move the tape left and right one (and only one) cell at a time. @@ -59,7 +86,7 @@ Objective: understand the relations between the different components of a comput --- -# Basic Input Output System (BIOS) +## Basic Input Output System (BIOS) > Used to perform hardware initialization during the booting process (power-on startup), and to provide runtime services for operating systems and programs. @@ -69,7 +96,7 @@ Objective: understand the relations between the different components of a comput --- -# Unified Extensible Firmware Interface (UEFI) +## Unified Extensible Firmware Interface (UEFI) Advantages: @@ -86,50 +113,54 @@ Disadvantages: --- -# Operating System (OS) +## Operating System (OS) > A system software that manages computer hardware, software resources, and provides common services for computer programs. - The first thing loaded by the BIOS/UEFI - The first thing on the tape of a Turing machine -## Kernel +### Kernel + > The kernel provides the most basic level of control over all of the computer's hardware devices. It manages memory access for programs in the RAM, it determines which programs get access to which hardware resources, it sets up or resets the CPU's operating states for optimal operation at all times, and it organizes the data for long-term non-volatile storage with file systems on such media as disks, tapes, flash memory, etc. - +[Kernel](./img/220px-Kernel_Layout.svg.png){width=100%} --- -# UNIX +## UNIX > Unix is a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, -[](https://upload.wikimedia.org/wikipedia/commons/7/77/Unix_history-simple.svg) +[{width=100%}](https://upload.wikimedia.org/wikipedia/commons/b/b5/Linux_Distribution_Timeline_21_10_2021.svg) The ones you are likely to encounter: + - [macOS](https://en.wikipedia.org/wiki/MacOS) - [BSD (Berkeley Software Distribution) variant](https://www.freebsd.org/) - [GNU/Linux](https://www.kernel.org/) The philosophy of UNIX is to have a large number of small software which do few things but to them well. -# GNU/Linux +## GNU/Linux Linux is the name of the kernel which software, to get a full OS, Linux is part of the [GNU Project](https://www.gnu.org/). The GNU with Richard Stallman introduced the notion of Free Software: + 1. The freedom to run the program as you wish, for any purpose. 2. The freedom to study how the program works, and change it so it does your computing as you wish. Access to the source code is a precondition for this. 3. The freedom to redistribute copies so you can help others. 4. The freedom to distribute copies of your modified versions to others. By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this. - - You can find a [list of software licenses](https://www.gnu.org/licenses/license-list.html) +<video width="100%" controls> + <source src="https://audio-video.gnu.org/video/TEDxGE2014_Stallman05_LQ.webm" type="video/webm"> + Your browser does not support the video tag. +</video> -[Instead of installing GNU/Linux on your computer, you are going to learn to use the IFB Cloud.](./2_using_the_ifb_cloud.html) - - +## session 2 IFB Cloud +[Instead of installing GNU/Linux on your computer, you are going to learn to use the IFB Cloud.](./2_using_the_ifb_cloud.html) diff --git a/2_using_the_ifb_cloud.Rmd b/2_using_the_ifb_cloud.Rmd index 03c6eea56a816975ce99e5b2df7f66b85c5ec2e6..5127154cc319e118d086c34e69bf53adc0ffe9ec 100644 --- a/2_using_the_ifb_cloud.Rmd +++ b/2_using_the_ifb_cloud.Rmd @@ -1,9 +1,40 @@ --- title: IFB (Institu Français de bio-informatique) Cloud - +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" --- -# IFB (Institu Français de bio-informatique) Cloud +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') + +``` + +<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> +<img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /> +</a> + +## IFB (Institu Français de bio-informatique) Cloud [](http://creativecommons.org/licenses/by-sa/4.0/) @@ -11,7 +42,7 @@ Objective: Start and connect to an appliance on the IFB cloud Instead of working on your computer where you don't have an Unix-like OS or have limited right, we are going to use the [IFB (Institu Français de bio-informatique) Cloud]( https://biosphere.france-bioinformatique.fr/). -## Creating an IFB account +### Creating an IFB account 1. Access the [**https://biosphere.france-bioinformatique.fr/**](https://biosphere.france-bioinformatique.fr/) website @@ -29,7 +60,7 @@ Instead of working on your computer where you don't have an Unix-like OS or have 8. Click on  and type **LBMC Unix 2020** 9. You can click on the **+** sign to register and wait to be accepted in the group -## Starting the LBMC Unix 2020 appliance +### Starting the LBMC Unix 2020 appliance To follow this practical you will need to start the **[LBMC Unix 2020](https://biosphere.france-bioinformatique.fr/catalogue/appliance/177/)** appliance from the [IFB Cloud](https://biosphere.france-bioinformatique.fr/) and click on the  button after login with your account. @@ -53,7 +84,7 @@ You will need to start this appliance at the start of each session of this cours The  symbol indicates that your appliance is starting. -## Accessing the LBMC Unix 2020 +### Accessing the LBMC Unix 2020 You can open the **https** link next to the termination button of your appliance in a new tab. You will have the following message diff --git a/3_first_steps_in_a_terminal.Rmd b/3_first_steps_in_a_terminal.Rmd index 71291f8069f1ec5562bee7ac4ea434810b080968..6dc68b8bba92039d016ecf8740a5ecd4824b5831 100644 --- a/3_first_steps_in_a_terminal.Rmd +++ b/3_first_steps_in_a_terminal.Rmd @@ -1,8 +1,40 @@ --- title: First step in a terminal +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" --- -# First step in a terminal +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') + +``` + +<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> +<img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /> +</a> + +## First step in a terminal [](http://creativecommons.org/licenses/by-sa/4.0/) @@ -18,7 +50,7 @@ What is the distribution installed on your VM ? You can go to this distribution website and have a look at the list of firms using it. -## Shell +### Shell A command-line interpreter (or shell), is a software designed to read lines of text entered by a user to interact with an OS. @@ -45,7 +77,7 @@ You can identify the following information from your prompt: **etudiant** is you On Ubuntu 20.04, the default shell is [Bash](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) while on recent version of macOS it’s [zsh](https://en.wikipedia.org/wiki/Z_shell). There are [many different shell](https://en.wikipedia.org/wiki/List_of_command-line_interpreters), for example, Ubuntu 20.04 also has [sh](https://en.wikipedia.org/wiki/Bourne_shell) installed. -## Launching Programs +### Launching Programs You can launch every program present on your computer from the shell. The syntax will always be the following: @@ -95,7 +127,7 @@ cal 2 1999 What is the difference for the parameter value `2` in the first and third command ? -## Moving around +### Moving around For the `cal` program, the position in the file system is not important (it’s not going to change the calendar). However, for most tools that are able to read or write files, it’s important to know where you are. This is the first real difficulty with command line interface: you need to remember where you are. diff --git a/4_unix_file_system.Rmd b/4_unix_file_system.Rmd index f5f9909954735fba7e19c09e55385c67beb3b5ae..8a1f3b9b2e3140fff9fec0e2ffd09f0c447f6cc4 100644 --- a/4_unix_file_system.Rmd +++ b/4_unix_file_system.Rmd @@ -1,8 +1,40 @@ --- title: GNU/Linux file system +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" --- -# GNU/Linux file system +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') + +``` + +<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> +<img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /> +</a> + +## GNU/Linux file system [](http://creativecommons.org/licenses/by-sa/4.0/) @@ -30,7 +62,7 @@ Every file can be accessed by an **absolute path** starting at the root. Your us We can also access file with a **relative path**, using the special folder "**..**". From your home folder, go to the *ubuntu* user home folder without passing by the root (we will see use of the "**.**" folder later). -## File Types +### File Types As you may have guessed, every file type is not the same. We have already seen that common file and folder are different. Here are the list of file types: @@ -50,25 +82,25 @@ ls -la The first column will tell you the type of the file (here we have only the type "**-**" and "**d**" ). We will come back on the other information later. An other less used command to get fine technical information on a file is the command `stat [file_name]`. Can you get the same information as `ls -la` with `stat` ? -## Common Structure +### Common Structure From the root of the system (**/**), most of the Unix-like distribution will share the same folder arborescence. On macOS, the names will be different because when you sell the most advanced system in the world you need to rename things, with more advanced names. -### `/home` +#### `/home` You already know this one. You will find all your file and your configuration files here. Which configuration file can you identify in your home ? -### `/boot` +#### `/boot` You can find the Linux kernel and the boot manager there. What is the name of your boot manager (process by elimination) ? You can see a new type of file here, the type "**l**". What it the version of the **vmlinuz** kernel ? -### `/root` +#### `/root` The home directory of the super user, also called root (we will go back on him later). Can you check its configuration file ? -### `/bin`, `/sbin`, `/usr/bin` and `/opt` +#### `/bin`, `/sbin`, `/usr/bin` and `/opt` The folder containing the programs used by the system and its users. Programs are simple file readable by a computer, these files are often in **bin**ary format which means that it’s extremely difficult for a human to read them. @@ -78,7 +110,7 @@ What is the difference between **/bin** and **/usr/bin** ? **/opt** is where you will find the installation of non-conventional programs (if you don’t follow [the guide of good practice of the LBMC](http://www.ens-lyon.fr/LBMC/intranet/services-communs/pole-bioinformatique/ressources/good_practice_LBMC), you can put your bioinformatics tools with crapy installation procedure there). -### `/lib` and `/usr/lib` +#### `/lib` and `/usr/lib` Those folder contains system libraries. Libraries are a collection of pieces of codes usable by programs. @@ -86,11 +118,11 @@ What is the difference between **/lib** and **/usr/lib**. Search information on the `/lib/gnupg` library on the net. -### `/etc` +#### `/etc` The place where system configuration file and default configuration file are. What is the name of the default configuration file for `bash` ? -### `/dev` +#### `/dev` Contains every peripheric @@ -102,7 +134,7 @@ Using `less` can you visualize the content of the file `urandom` ? What about th What is the content of `/dev/null`? -### `/var` +#### `/var` Storage space for variables and temporary files, like system logs, locks, or file waiting to be printed... @@ -120,11 +152,11 @@ What is the type of the file `autofs.fifo-var-autofs-ifb` in the `run` folder ? There are few examples of the last type of file in the `run` folder, in which color the command `ls -l` color them ? -### `/tmp` +#### `/tmp` Temporary space. **Erased at each shutdown of the system !** -### `/proc` +#### `/proc` Information on the system resources. This file system is virtual. What do we mean by that ? @@ -136,7 +168,7 @@ From the `cpuinfo` file get the brand of the cpu simulated by your VM. From the `meminfo` file retrieve the total size of RAM -## Links +### Links With the command `ls -l` we have seen some links, the command `stat` can give us more information on them @@ -188,7 +220,7 @@ To understand the notion of **Inode** we need to know more about storage systems -## Disk and partition +### Disk and partition On a computer, the data are physically stored on a media (HDD, SSD, USB key, punch card...) diff --git a/5_users_and_rights.Rmd b/5_users_and_rights.Rmd index 0126d52431d09ceb6a8502e3a6345d71b5946a2b..f8039f46c634c3f21ffa6060ee71afe69f39ec8c 100644 --- a/5_users_and_rights.Rmd +++ b/5_users_and_rights.Rmd @@ -1,9 +1,36 @@ --- title: Users and rights - +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" --- -# Users and rights +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') + +``` + +## Users and rights [](http://creativecommons.org/licenses/by-sa/4.0/) @@ -23,7 +50,7 @@ There can also be other users who - belong to groups - the groups also have rights -## File rights +### File rights Each file is associated with a set of rights: @@ -128,7 +155,7 @@ Create a symbolic link (`ln -s`) to your **.bashrc** file, what are the default Can you remove the writing right of this link ? What happened ? -## Users and Groups +### Users and Groups We have seen how to change the right associated with the group, but what about changing the group itself ? The command `chgrp` allows you to do just that: @@ -162,7 +189,7 @@ Copy the `mkdir` tool to your home directory. Can you remove execution rights fo You cannot change the owner of a file, but you can always allow another user to copy it and change the rights on its copy. -## Getting admin access +### Getting admin access Currently you don’t have administrative access to your VM, this means that you don’t have the password to the *root* account. Another way to get administrative access in Linux is to use the `sudo` command. @@ -200,7 +227,7 @@ You can try again the `chown` command with the `sudo` command. Check the content of the file `/etc/shadow` , what is the utility of this file (you can get help from the `man` command). -## Creating Users +### Creating Users You can add a new user to your system with the command `useradd` @@ -227,7 +254,7 @@ sudo su student What append when you don't specify a login with the `su` command ? -## Creating groups +### Creating groups You can add new groups to your system with the command `groupadd` @@ -255,7 +282,7 @@ sudo usermod -G student student Check the results. -## Security-Enhanced Linux +### Security-Enhanced Linux While what you have seen in this section hold true for every Unix system, additionnal rules can be applied to control the rights in Linux. This is what is called [SE Linux](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) (**s**ecurity-**e**nhanced **Linux**) diff --git a/6_unix_processes.Rmd b/6_unix_processes.Rmd index 10f373093180df61eaf716612eb56dce52d3edf8..b9b804baa30f35d0f7312bac4f0d54da4ffd6223 100644 --- a/6_unix_processes.Rmd +++ b/6_unix_processes.Rmd @@ -1,10 +1,36 @@ --- title: Unix Processes +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" +--- +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') ---- +``` -# Unix Processes +## Unix Processes [](http://creativecommons.org/licenses/by-sa/4.0/) @@ -22,7 +48,7 @@ Your shell is a process to manipulate other processes. Some commands in your shell don’t have an associated process, for example there is no `cd` program, it’s a functionality of your shell. The `cd` command tell your `bash` process to do something not to fork another process. -## Process attributes +### Process attributes - PID : the **p**rocess **id**entifier is an integer, at a given time each PID is unique to a process - PPID : the **p**arent **p**rocess **id**entifier is the PID of the process that has stared the current process @@ -69,7 +95,7 @@ ps -l -C systemd Who launched the first `systemd` process ? -## Processes tree +### Processes tree From PPID to PPID, you can guess that like the file system, processes are organized in a tree. The command `pstree` can give you a nice representation of this tree. @@ -116,7 +142,7 @@ Launch the `stress` for 1 cpu and 3600 second. You don’t have a prompt, it means that the last command (`stress`) is running. -## Terminate +### Terminate Instead of taking a nap and come back at the end of this session, we may want to interrupt this command. The first way to do that is to ask the system to terminate the `stress` process. @@ -160,7 +186,7 @@ A process with a PPID of 1 is called a **daemon**, daemons are processes that ru Kill the remaining `stress `processes with the command `pkill`. You can check the **man**ual on how to do that. -## Suspend +### Suspend Launch `htop` then press `ctrl` + `z`. What happened ? @@ -200,7 +226,7 @@ Bring the 2nd `htop` to the foreground. Put it back to the background with `ctrl The command `bg` allow you to resume a job stopped in the background. You can restart your stopped `stress` process with this command. You can use the `kill %N` syntax to kill your two `stress` processes. -## Priority +### Priority We have seen that we can launch a `stress `process to use 100% of a cpu. Launch two `stress` process like that in the background. diff --git a/7_streams_and_pipes.Rmd b/7_streams_and_pipes.Rmd index e36a3fd264cf04bfe25435b86e562cf6585908d0..3f8820ee7b12eb8bd36427feac058230ba1cab2e 100644 --- a/7_streams_and_pipes.Rmd +++ b/7_streams_and_pipes.Rmd @@ -1,11 +1,36 @@ --- title: Unix Streams and pipes +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" +--- +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') +``` ---- - -# Steams and pipes +## Steams and pipes [](http://creativecommons.org/licenses/by-sa/4.0/) @@ -37,7 +62,7 @@ cat .bashrc -## Streams manipulation +### Streams manipulation You can use the `>` character to redirect a flux toward a file. The following command make a copy of your `.bashrc` files. @@ -100,7 +125,7 @@ cal 2&>> my_redirection -## Pipes +### Pipes The last stream manipulation that we are going to see is the pipe which transforms the **stdout** of a process into the **stding** of the next. Pipes are useful to chain multiples simple operations. The pipe operator is `| ` diff --git a/8_text_manipulation.Rmd b/8_text_manipulation.Rmd index c32aea983c86132026f1045f45a6470c8e1eeb5d..43755b7b9a08b303974a51cff1245012b6f47243 100644 --- a/8_text_manipulation.Rmd +++ b/8_text_manipulation.Rmd @@ -1,12 +1,36 @@ --- title: Text manipulation +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" +--- +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') +``` - ---- - -# Text manipulation +## Text manipulation [](http://creativecommons.org/licenses/by-sa/4.0/) @@ -14,7 +38,7 @@ Objective: Learn basics way to work with text file in Unix One of the great thing with command line tools is that they are simple and fast. Which means that they are great for handle large files. And as bioinformaticians you have to handle large file, so you need to use command line tools for that. -## Text search +### Text search The file [hg38.ncbiRefSeq.gtf.gz](http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/genes/hg38.ncbiRefSeq.gtf.gz) contains the RefSeq annotation for hg38 in [GFT format](http://www.genome.ucsc.edu/FAQ/FAQformat.html#format4) @@ -46,7 +70,7 @@ Does the number of *3UTR* match the number of *5UTR* ? How many transcripts does the gene *CCR7* have ? -## Regular expression +### Regular expression When you do a loot text search, you will encounter regular expression (regexp), which allow you to perform fuzzy search. To run `grep` in regexp mode you can use the switch `-E` @@ -68,7 +92,7 @@ There are different special characters in regexp, but you can use `\` to escape gzip -dc hg38.ncbiRefSeq.gtf.gz | head | grep -E "\." ``` -### Character classes and alternatives +#### Character classes and alternatives There are a number of special patterns that match more than one character. You’ve already seen `.`, which matches any character apart from a newline. There are four other useful tools: @@ -89,7 +113,7 @@ gzip -dc hg38.ncbiRefSeq.gtf.gz | head | perl -E "\d\d[A-Z]\d" </p> </details> -### Anchors +#### Anchors By default, regular expressions will match any part of a string. It’s often useful to *anchor* the regular expression so that it matches from the start or end of the string. You can use @@ -106,7 +130,7 @@ gzip -dc hg38.ncbiRefSeq.gtf.gz | head | grep -E "c" gzip -dc hg38.ncbiRefSeq.gtf.gz | head | grep -E "^c" ``` -### Repetition +#### Repetition The next step up in power involves controlling how many times a pattern matches @@ -143,7 +167,7 @@ gzip -dc hg38.ncbiRefSeq.gtf.gz | grep -E "transcript\s.*gene_id\s\"\S{16,}\";" </p> </details> -### Grouping and back references +#### Grouping and back references You can group match using `()`, for example the following regexp match doublet of *12* . @@ -170,7 +194,7 @@ gzip -dc hg38.ncbiRefSeq.gtf.gz | head | sed -E 's|ncbiRefSeq(.*)(transcript_id </details> Regexp can be very complexe see for example [a regex to validate an email on starckoverflow](https://stackoverflow.com/questions/201323/how-to-validate-an-email-address-using-a-regular-expression/201378#201378). When you start you can always use for a given regexp to a more experienced used (just give him the kind of text you want to match and not match). You can test your regex easily with the [regex101 website](https://regex101.com/). -## Sorting +### Sorting GTF files should be sorted by chromosome, starting position and end position. But you can change that with the command `sort` to select the column to sort on you can use the option `-k n,n` where `n` is the column number. @@ -214,7 +238,7 @@ gzip -dc hg38.ncbiRefSeq.gtf.gz | head -n 10000 | sort -k 1,1 -k 4,4n -k 5,5n -c -## Field extractor +### Field extractor Sometime rather than using complex regexp, we want to extract a particular column from a file. You can use the command `cut` to do that. @@ -233,7 +257,7 @@ gzip -dc hg38.ncbiRefSeq.gtf.gz | head | cut -f 2 -f 5 -d ";" ``` </p> </details> -## Concatenation +### Concatenation There are different tools to concatenate files from the command line `cat` for vertical concatenation and `paste` for horizontal concatenation. @@ -249,7 +273,7 @@ gzip -dc hg38.ncbiRefSeq.gtf.gz | head | paste - - -## Text editor +### Text editor You often have access to different text editors from the common line, two of the most popular ones are `vim` and `nano`. diff --git a/9_batch_processing.Rmd b/9_batch_processing.Rmd index e9497a395cb1568c1d5e67c3caa995ece5bca6d9..3bf69dd454b154390c9a2dde2025c5215037244e 100644 --- a/9_batch_processing.Rmd +++ b/9_batch_processing.Rmd @@ -1,4 +1,35 @@ -# Batch processing +--- +title: Text manipulation +author: "Laurent Modolo" +output: + rmdformats::downcute: + self_contain: false + use_bookdown: true + default_style: "light" + lightbox: true + css: "./www/style_Rmd.css" +--- + +```{r include = FALSE} + +if (!require("fontawesome")) { + install.packages("fontawesome") +} +if (!require("klippy")) { + install.packages("remotes") + remotes::install_github("rlesur/klippy") +} +library(fontawesome) +knitr::opts_chunk$set(echo = TRUE) +knitr::opts_chunk$set(comment = NA) +klippy::klippy( + position = c('top', 'right'), + color = "white", + tooltip_message = 'Click to copy', + tooltip_success = 'Copied !') + +``` +## Batch processing [](http://creativecommons.org/licenses/by-sa/4.0/) @@ -26,7 +57,7 @@ You can also use the `||` to manage errors and run `CMD2` if `CMD1` failed. CMD1 || CMD2 ``` -## Executing list of commands +### Executing list of commands The easiest option to execute list of command is to use `xargs`. `xargs` reads arguments from **stdin** and use them as argument for a command. In UNIX systems the command `echo` send string of character into **stdout**. We are going to use this command to learn more about `xargs`. @@ -107,7 +138,7 @@ find /tmp/ -type d | xargs -t rm -R </p> </details> -## Writing `awk` commands +### Writing `awk` commands `xargs` It is a simple solution for writing batch commands, but if you want to write more complex command you are going to need to learn `awk`. `awk` is a programming language by itself, but you don’t need to know everything about `awk` to use it. @@ -209,7 +240,7 @@ awk 'BEGIN {OFS = "\n"} {header = $0 ; getline seq ; getline qheader ; getline q -## Writing a bash script +### Writing a bash script When you start writing complicated command, you may want to save them to reuse them later. @@ -238,7 +269,7 @@ echo "download and extraction complete" </details> -### shebang +#### shebang In your first bash script, the only thing saying that your script is a bash script is its extension. But most of the time UNIX system doesn’t care about file extension, a text file is a text file. @@ -247,13 +278,13 @@ To tell the system that your text file is a bash script you need to add a **sheb For example, for a bash script in a system where `bash` is installed in `/bin/bash` the **shebang** is: ```bash -#!/bin/bash +##!/bin/bash ``` When you are not sure `which`is the path of the tools available to interpret your script, you can use the following shebang: ```bash -#!/usr/bin/env bash +##!/usr/bin/env bash ``` You can add a **shebang** to your script and add it the e**x**ecutable right. @@ -275,7 +306,7 @@ Now you can execute your script with the command: Congratulations you wrote your first program ! -### PATH +#### PATH Where did they `/usr/bin/env` find the information about your bash ? Why did we have to write a `./` before our script if we are in the same folder ? @@ -347,7 +378,7 @@ You can check the result of your command with `echo $PATH` Try to call your `download_hg38.sh` from anywhere on the file tree. Congratulation you installed your first UNIX program ! -### Arguments +#### Arguments You can pass argument to your bash scripts, writing the following command: diff --git a/Makefile b/Makefile index 212dc46f9bd1d37cf370394ce598bba86e371cf2..8ea7ed5711c4ac2d336838aea1347d3e33eb7ad1 100644 --- a/Makefile +++ b/Makefile @@ -15,42 +15,45 @@ all: public/index.html \ public/github-pandoc.css: github-pandoc.css cp github-pandoc.css public/github-pandoc.css + cp -R img public/ + cp -R www public/ public/index.html: index.md github-pandoc.css pandoc -s -c github-pandoc.css index.md -o public/index.html -public/1_understanding_a_computer.html: 1_understanding_a_computer.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 1_understanding_a_computer.md -o public/1_understanding_a_computer.html +public/1_understanding_a_computer.html: 1_understanding_a_computer.Rmd public/github-pandoc.css + Rscript -e 'rmarkdown::render("1_understanding_a_computer.Rmd", output_dir = "public/")' -public/2_using_the_ifb_cloud.html: 2_using_the_ifb_cloud.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 2_using_the_ifb_cloud.md -o public/2_using_the_ifb_cloud.html +public/2_using_the_ifb_cloud.html: 2_using_the_ifb_cloud.Rmd public/github-pandoc.css + Rscript -e 'rmarkdown::render("2_using_the_ifb_cloud.Rmd", output_dir = "public/")' -public/3_first_steps_in_a_terminal.html: 3_first_steps_in_a_terminal.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 3_first_steps_in_a_terminal.md -o public/3_first_steps_in_a_terminal.html +public/3_first_steps_in_a_terminal.html: 3_first_steps_in_a_terminal.Rmd public/github-pandoc.css + Rscript -e 'rmarkdown::render("3_first_steps_in_a_terminal.Rmd", output_dir = "public/")' -public/4_unix_file_system.html: 4_unix_file_system.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 4_unix_file_system.md -o public/4_unix_file_system.html +public/4_unix_file_system.html: 4_unix_file_system.Rmd public/github-pandoc.css + Rscript -e 'rmarkdown::render("4_unix_file_system.Rmd", output_dir = "public/")' -public/5_users_and_rights.html: 5_users_and_rights.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 5_users_and_rights.md -o public/5_users_and_rights.html +public/5_users_and_rights.html: 5_users_and_rights.Rmd public/github-pandoc.css + Rscript -e 'rmarkdown::render("5_users_and_rights.Rmd", output_dir = "public/")' -public/6_unix_processes.html: 6_unix_processes.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 6_unix_processes.md -o public/6_unix_processes.html +public/6_unix_processes.html: 6_unix_processes.Rmd public/github-pandoc.css + Rscript -e 'rmarkdown::render("6_unix_processes.Rmd", output_dir = "public/")' -public/7_streams_and_pipes.html: 7_streams_and_pipes.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 7_streams_and_pipes.md -o public/7_streams_and_pipes.html +public/7_streams_and_pipes.html: 7_streams_and_pipes.Rmd public/github-pandoc.css + Rscript -e 'rmarkdown::render("7_streams_and_pipes.Rmd", output_dir = "public/")' -public/8_text_manipulation.html: 8_text_manipulation.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 8_text_manipulation.md -o public/8_text_manipulation.html +public/8_text_manipulation.html: 8_text_manipulation.Rmd public/github-pandoc.css + Rscript -e 'rmarkdown::render("8_text_manipulation.Rmd", output_dir = "public/")' -public/9_batch_processing.html: 9_batch_processing.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 9_batch_processing.md -o public/9_batch_processing.html +public/9_batch_processing.html: 9_batch_processing.Rmd public/github-pandoc.css + Rscript -e 'rmarkdown::render("9_batch_processing.Rmd", output_dir = "public/")' -public/10_network_and_ssh.html: 10_network_and_ssh.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 10_network_and_ssh.md -o public/10_network_and_ssh.html +public/10_network_and_ssh.html: 10_network_and_ssh.Rmd public/github-pandoc.css + pandoc -s --toc -c github-pandoc.css 10_network_and_ssh.Rmd -o public/10_network_and_ssh.html + Rscript -e 'rmarkdown::render("10_network_and_ssh.Rmd", output_dir = "public/")' -public/11_install_system_programs.html: 11_install_system_programs.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 11_install_system_programs.md -o public/11_install_system_programs.html +public/11_install_system_programs.html: 11_install_system_programs.Rmd public/github-pandoc.css + Rscript -e 'rmarkdown::render("11_install_system_programs.Rmd", output_dir = "public/")' -public/12_virtualization.html: 12_virtualization.md github-pandoc.css - pandoc -s --toc -c github-pandoc.css 12_virtualization.md -o public/12_virtualization.html +public/12_virtualization.html: 12_virtualization.Rmd public/github-pandoc.css + Rscript -e 'rmarkdown::render("12_virtualization.Rmd", output_dir = "public/")'