@@ -40,30 +40,30 @@ A program is a list of instructions to be executed on a computer. These instruct
The execution of a program on the system is represented as one or many processes. The program is the file of instruction while the process is the instructions being read.
`mkdir` is the program, when you type `mkdir my_folder`, you launch an `mkdir` process.
`mkdir` is the program, when you type `mkdir my_folder`, you launch a `mkdir` process.
Your shell is a process to manipulate other processes.
> In multitasking operating systems, processes (running programs) need a way to create new processes, e.g. to run other programs. [Fork](https://en.wikipedia.org/wiki/Fork_(system_call)) and its variants are typically the only way of doing so in Unix-like systems. For a process to start the execution of a different program, it first forks to create a copy of itself. Then, the copy, called the "[child process](https://en.wikipedia.org/wiki/Child_process)", calls the [exec](https://en.wikipedia.org/wiki/Exec_(system_call)) system call to overlay itself with the other program: it ceases execution of its former program in favor of the other.
> In multitasking operating systems, processes (running programs) need a way to create new processes, e.g., to run other programs. [Fork](https://en.wikipedia.org/wiki/Fork_(system_call)) and its variants are typically the only way of doing so in Unix-like systems. For a process to start the execution of a different program, it first forks to create a copy of itself. Then, the copy, called the "[child process](https://en.wikipedia.org/wiki/Child_process)", calls the [exec](https://en.wikipedia.org/wiki/Exec_(system_call)) system call to overlay itself with the other program: it ceases execution of its former program in favor of the other.
Some commands in your shell don’t have an associated process, for example there is no `cd` program, it’s a functionality of your shell. The `cd` command tell your `bash` process to do something not to fork another process.
# Process attributes
- PID : the **p**rocess **id**entifier is an integer, at a given time each PID is unique to a process
- PPID : the **p**arent **p**rocess **id**entifier is the PID of the process that has stared the current process
- UID : the **u**ser **id**entifier is the identifier of the user that has started the process, except for SE Linux, the process will have th same right as the user launching it.
- PGID : the **p**rocess **g**roup **id**entifier (like users, processes have groups)
- **PID** : the **p**rocess **id**entifier is an integer, at a given time each **PID** is unique to a process
- **PPID** : the **p**arent **p**rocess **id**entifier is the **PID** of the process that has stared the current process
- **UID** : the **u**ser **id**entifier is the identifier of the user that has started the process, except for SE Linux, the process will have the same right as the user launching it.
- **PGID** : the **p**rocess **g**roup **id**entifier (like users, processes have groups)
You can use the command `ps` to see the processes launched by your user
You can use the command `ps` to see the processes launched by your user.
```sh
ps
```
Like for the command `ls` you can use the switch `-l` to have more details.
Like for the command, `ls` you can use the switch `-l` to have more details.
Open another tab in your browser to log again on your VM. Keep this tab open, we are going to use both of them in this session.
Open another tab in your browser to log again on your VM. Keep these tabs open, we are going to use both of them in this session.
In this new tab, you are going to launch a `less` process.
...
...
@@ -71,21 +71,21 @@ In this new tab, you are going to launch a `less` process.
less .bashrc
```
Comme back, to your first tab, can you see your `less` process with the command `ps` ?
Come back, to your first tab, can you see your `less` process with the command `ps` ?
The `ps` option `-u [login]` list all the processes with `UID`the `UID`associated with `[login]`
The `ps` option `-u [login]` list all the processes with **UID** the **UID** associated with `[login]`
```sh
ps -l -u etudiant
```
Is the number of `bash` processes consistent with the number of tabs you openned ?
Is the number of `bash` processes consistent with the number of tabs you opened ?
What is the PPID of the `less` process ? Can you identify in which `bash` process `less` is running ?
What is the **PPID** of the `less` process ? Can you identify in which `bash` process `less` is running ?
Did you launch the `systemd` and `(sd-pam)` process ?
`pam` stand for [**p**luggable **a**uthentication **m**odules](https://www.linux.com/news/understanding-pam/), it’s a collection of tools that handle identification and resource access restriction. From the PPID of the `(sd-pam)` can you find which process launched `(sd-pam)` ? What is the PPID of this process ?
**pam** stands for [**p**luggable **a**uthentication **m**odules](https://www.linux.com/news/understanding-pam/), it’s a collection of tools that handle identification and resource access restriction. From the **PPID** of the `(sd-pam)` can you find which process launched `(sd-pam)` ? What is the **PPID** of this process ?
The option `-C` allows you to filter process by name
...
...
@@ -97,23 +97,23 @@ Who launched the first `systemd` process ?
# Processes tree
From PPID to PPID, you can guess that like the file system, processes are organized in a tree. The command `pstree` can give you a nice representation of this tree.
From **PPID** to **PPID**, you can guess that like the file system, processes are organized in a tree. The command `pstree` can give you a nice representation of this tree.
The following `ps` command shows information on the process with PID 1
The following `ps` command shows information on the process with **PID 1**
```sh
ps -l -1
```
Is this output coherent with what you know on PID and the previous `ps` command ? Can you look at the corresponding program (with the command `which`) ?
Is this output coherent with what you know on **PID** and the previous `ps` command ? Can you look at the corresponding program (with the command `which`) ?
Can you look for information on PID 0 ?
Can you look for information on **PID 0** ?
The root of the processes tree is the PID 1.
The root of the processes tree is the **PID 1**.
What is the UID of the `dockerd` process, can you guess why we were able to gain `sudo` access in the previous section by using a `docker` command ?
What is the **UID** of the `dockerd` process, can you guess why we were able to gain `sudo` access in the previous section by using a `docker` command ?
`ps` give you a static snapshot of the processes but processes are dynamic. To see them running you can use the command `top`. While `top` is functional, most system have `htop` with a more accessible interface. You can test `top` and `htop`.
`ps` give you a static snapshot of the processes but processes are dynamic. To see them running you can use the command `top`. While `top` is functional, most systems have `htop` with a more accessible interface. You can test `top` and `htop`.
Like `ps` you can use `-u etduiant` with htop to only display your user processes.
...
...
@@ -124,7 +124,7 @@ With the `F6` key, you can change the column on which to sort your process.
What is the difference between `M_SIZE` (`VIRT` column), `M_RESIDENT` (`RES` column) and `M_SHARE` (`SHR` column) ? To which value, the column `MEM%` corresponds to ?
- `M_SIZE` : The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out and pages that have been mapped but not used (If an application requests 1 GB of memory but uses only 1 MB, then `VIRT` will report 1 GB).
- `M_SIZE` : The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out and pages that have been mapped but not used (if an application requests 1 GB of memory but uses only 1 MB, then `VIRT` will report 1 GB).
- `M_RESIDENT` : what’s currently in the physical memory. This does not include the swapped out memory and some of the memory may be shared with other processes (If a process uses 1 GB of memory and it calls `fork()`, the result of forking will be two processes whose `RES` is both 1 GB but only 1 GB will actually be used since Linux uses copy-on-write).
- `M_SHARE` : The amount of shared memory used by a task. It simply reflects memory that could be potentially shared with other processes.
...
...
@@ -132,13 +132,13 @@ Wait what is swapped out memory ?
> Linux divides its physical RAM (random access memory) into chucks of memory called pages. Swapping is the process whereby a page of memory is copied to the preconfigured space on the hard disk, called swap space, to free up that page of memory. The combined sizes of the physical memory and the swap space is the amount of virtual memory available.
And as you HDD (even your fast SSD) is way slower than your RAM, when you run out of RAM and the system start to swap out memory, things will start to go really slowly on your computer. Generally, you want to avoid swapping. The swap space is often a dedicated partition in the *Linux_swap* format.
And as your HDD (even your fast SSD) is way slower than your RAM, when you run out of RAM and the system start to swap out memory, things will start to go really slowly on your computer. Generally, you want to avoid swapping. The swap space is often a dedicated partition in the *Linux_swap* format.
From the `htop` command, what is the size of the swap space on your VM ?
You have control over all the process launched with your UID. To test this control we are going to use the well-named command `stress`. Check the **man**ual of the `stress` command.
Launch the `stress` for 1 cpu and 3600 second.
Launch the `stress` for 1 CPU and 3600 second.
You don’t have a prompt, it means that the last command (`stress`) is running.
...
...
@@ -148,13 +148,13 @@ Instead of taking a nap and come back at the end of this session, we may want to
From your terminal you can press `ctrl` + `c`. This short cut terminates the current process, it works everywhere except for programs like `man` or `less` which can be closed with the key `q`.
Launch another long `stress` process and switch to your other terminal tab and list your active process.
Launch another long `stress` process and switch to your other terminal tab, then, list your active process.
```sh
ps -l -u etudiant
```
You ask `stress` to launch a worker using 100% of one cpu (you can also see that with `htop`). You can see that the `stress` process you launched (with the PPID of your `bash`) forked another `stress` process.
You ask `stress` to launch a worker using 100% of one CPU (you can also see that with `htop`). You can see that the `stress` process you launched (with the PPID of your `bash`) forked another `stress` process.
Another way to terminate a process is with the command `kill`. `kill` is used to send a signal to a process with the command:
...
...
@@ -164,9 +164,9 @@ kill -15 PID
The `-15` option is the default option for `kill` so you can also write `kill PID`.
We can do the same thing as with the command `ctrl` + `c`: ask nicely the process to terminate itself. The `-15` signal is called the SIGTERM.
We can do the same thing as with the command `ctrl` + `c`: ask nicely the process to terminate itself. The `-15` signal is called the **SIGTERM**.
> On rare occasions a buggy process will not be able to listen to signals anymore. The signal `-9` will kill a process (not nicely at all). The `-9` signal is called the SIGKILL signal. There are 64 different signals that you can send with the `kill` command.
> On rare occasions a buggy process will not be able to listen to signals anymore. The signal `-9` will kill a process (not nicely at all). The `-9` signal is called the **SIGKILL** signal. There are 64 different signals that you can send with the `kill` command.
Use the `kill` command to terminate the worker process of your stress command. Go to the other tab where stress was running. What is the difference with your previous `ctrl` + `c` ?
...
...
@@ -182,7 +182,7 @@ Nicely terminate the intermediate `bash`. What happened ?
Try not nicely. What happened ?
A process with a PPID of 1 is called a **daemon**, daemons are processes that run in the background. Congratulations you created your first daemon.
A process with a **PPID** of **1** is called a **daemon**, daemons are processes that run in the background. Congratulations you created your first daemon.
Kill the remaining `stress `processes with the command `pkill`. You can check the **man**ual on how to do that.
...
...
@@ -196,21 +196,21 @@ ps -l -u etudiant
The manual of the `ps` command say the following about process state
> ```
> D uninterruptible sleep (usually IO)
> I Idle kernel thread
> R running or runnable (on run queue)
> S interruptible sleep (waiting for an event to complete)
> T stopped by job control signal
> t stopped by debugger during the tracing
> W paging (not valid since the 2.6.xx kernel)
> X dead (should never be seen)
> Z defunct ("zombie") process, terminated but not reaped by its parent
> ```
```
D uninterruptible sleep (usually IO)
I Idle kernel thread
R running or runnable (on run queue)
S interruptible sleep (waiting for an event to complete)
T stopped by job control signal
t stopped by debugger during the tracing
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z defunct ("zombie") process, terminated but not reaped by its parent
```
You can use the command `fg` to put `htop` in the **f**ore**g**round.
Close `htop` and type two time de following command (`-i 1` is for simulating **io** **i**nput **o**utput operations) :
Close `htop` and type twice the following command (`-i 1` is for simulating **io** **i**nput **o**utput operations) :
```sh
stress -i 1 -t 3600 &
...
...
@@ -224,19 +224,19 @@ fg %2
Bring the 2nd `htop` to the foreground. Put it back to the background with `ctrl` + `z`. What is now the differences between your two `stress` processes ?
The command `bg` allow you to resume a job stopped in the background. You can restart your stopped `stress` process with this command. You can use the `kill %N` syntax to kill your two `stress` processes.
The command `bg` allows you to resume a job stopped in the background. You can restart your stopped `stress` process with this command. You can use the `kill %N` syntax to kill your two `stress` processes.
# Priority
We have seen that we can launch a `stress `process to use 100% of a cpu. Launch two `stress` process like that in the background.
We have seen that we can launch a `stress `process to use 100% of a CPU. Launch two `stress` processes like that in the background.
What happened ? What can you see with the `htop` command ?
> In Linux the [Scheduler](https://en.wikipedia.org/wiki/Completely_Fair_Scheduler), is a system process that manages the order of execution of the task by the CPU(s). Linux and most Unix systems are also multiprocesses OS which means that your OS is constantly switching between process that has access to the CPU(s). From a universal Turing machine point of view, the head of the machine would be constantly switching back and forth on the tape.
> In Linux the [Scheduler](https://en.wikipedia.org/wiki/Completely_Fair_Scheduler), is a system process that manages the order of execution of the task by the CPU(s). Linux and most Unix systems are also multiprocessors OS which means that your OS is constantly switching between process that has access to the CPU(s). From a universal Turing machine point of view, the head of the machine would be constantly switching back and forth on the tape.
You are working on a computer with a graphical interface, think about the processes drawing your windows, the processes reading and rendering your mouse, checking for your mail, loading and rendering your web pages, reading your keystrokes to send them back over the network to the NSA. The scheduler of your OS has to jungle between everything’s without losing anything (don’t be hard on windows OS).
You are working on a computer with a graphical interface, think about the processes drawing your windows, the processes reading and rendering your mouse, checking for your mail, loading and rendering your web pages, reading your keystrokes to send them back over the network to the NSA. The scheduler of your OS has to jungle between everything’s without losing anything (don’t be too hard on the windows OS).
The **nice** state of a process indicates it’s priority for the scheduler. **nice** value range from -20 (the highest priority) to 19 (the lowest priority). The default **nice** value is 0. The command `renice` allows you to change the **nice** value of a process:
The **nice** state of a process indicates it’s priority for the scheduler. **nice** value ranges from **-20** (the highest priority) to **19** (the lowest priority). The default **nice** value is **0**. The command `renice` allows you to change the **nice** value of a process:
```sh
renice -n N -p PID
...
...
@@ -244,9 +244,9 @@ renice -n N -p PID
With `N` the **nice** value.
Use `renice` to set the **nice** value of the first `stress` process worker to 19. Use the command `htop` to check the result.
Use `renice` to set the **nice** value of the first `stress` process worker to **19**. Use the command `htop` to check the result.
Can we increase the difference between the two processes ? Use re renice command to set the **nice** value of the second `stress` process worker to -20. What happened ?
Can we increase the difference between the two processes ? Use re renice command to set the **nice** value of the second `stress` process worker to **-20**. What happened ?
Only the *root* user can lower the **nice** value of a process. You can also start start a new process with a given **nice** value with the command `nice`:
...
...
@@ -254,7 +254,7 @@ Only the *root* user can lower the **nice** value of a process. You can also sta
nice -n 10 stress -c 1 -t 3600 &
```
Without root access you can only set value lower than 0.
Without root access you can only set value greater than 0.
> We have seen the commands:
>
...
...
@@ -267,7 +267,7 @@ Without root access you can only set value lower than 0.
> - `fg` to bring to the foreground a background processes
> - `jobs` to display background processes
> - `bg` to start a background process
> - `stress` to launch mock computation
> - `stress` to launch a mock computation
> - `nice`/`renince` to change the nice value of a process
[To learn how to articulate processes you can head to the next section.](./7_streams_and_pipes.html)
