initial commit

continuous_integration.md

+# Introduction to continuous integration
+
+## What is Continuous integration (CI) ?
+
+It is about integrating changes to the code of a project continuously and test them frequently through an automated build.
+
+Its main goal is to allow fast detection of errors.
+
+## Advantages
+
+* Reduces the integration risks
+* All participants in a project are informed when a build fails
+* Eases de detection of errors across build versions
+* Improves developers' confidence in themselves and others
+* Improves code quality
+
+
+## Continous integration relies on:
+
+* The usage of a versioning control system (git)
+* A project repository managed by one or many developers
+* Usage of `unit testing` and `integration testing`.
+
+
+## Tools for continuous integration
+
+They are many of them:
+
+* Travis CI (https://travis-ci.org) - well suited to github or gitlab offical repository
+* Gitlab CI (https://docs.gitlab.com/ee/ci/) - well suited to gitlab offical repository or other gitlabs. It can also work with github.
+* Jenkins (https://www.jenkins.io/)
+* Circle CI (https://circleci.com/)
+* And many others.
+
+
+Let's present two of them, `Travis CI` with a github project and `Gitlab CI` with a gitbio project.
+
+
+# Travis CI with github, how it works ?
+
+According to their website:
+
+> Travis CI provides a default build environment and a default set of phases for each programming language. A virtual machine is created with the build environment for your job, your repository is cloned into it, optional addons are installed and then your build phases are run.
+
+## What is a job (short presentation)
+
+Each job is a sequence of phases. The main phases are:
+
+1. `install` - install any dependencies required
+2. `script` - run the build script
+
+Travis CI can run custom commands in the phases:
+
+1. `before_install` - before the install phase
+2. `before_script` - before the script phase
+3. `after_script` - after the script phase.
+4. `after_success` - when the build succeeds (e.g. building documentation), the result is in TRAVIS_TEST_RESULT environment variable
+5. `after_failure` - when the build fails (e.g. uploading log files), the result is in TRAVIS_TEST_RESULT environment variable
+
+They are other optional job phases, their are listed here: https://docs.travis-ci.com/user/job-lifecycle/ 
+
+
+## How to use Travis CI with github ?
+
+* Prerequisites : A github account with a github repository
+
+**Example: Lazyparser**
+
+![lazyparser](./images/lazyparser_repo1.png)
+
+
+1. Go to [Travis-ci.com](https://travis-ci.com/) and click on [Sign up](https://travis-ci.com/signup). The click on Then click on **sign up with github**.
+
+![lazyparser](./images/travis1.png)
+
+2. Accept the authorization of Travis CI. You’ll be redirected to GitHub.
+
+![lazyparser](./images/travis2.png)
+
+3. Click on your profile picture in the top right of your Travis Dashboard, click Settings and then the green **Activate button**, and select the repositories you want to use with Travis CI.
+
+![lazyparser](./images/travis3.png)
+
+4. Add a `.travis.yml` file **to your repository root** to tell Travis CI what to do.
+
+
+---
+ ## Aside
+
+The `.yml` extention corresponds to a YAML file. YAML stands for **Yet Another Markup Language** in its version 1.0 and then became **YAML Ain't Markup Language** in its version 1.1.
+
+It a format used to represent data that can have a hierarchical structure but without but without compromising their visibility. (like with XML for example).
+
+--- 
+
+YAML file used in lazyparser.
+
+```yml
+
+dist: xenial  # route the build to Ubuntu xenial 16.04. Use 'bionic' for Ubuntu 18.04 or focal for Ubuntu 20.04
+language: python  # Usage of python language inside a python virtualenv; Install packages via pip not apt.
+python:  # Make the tests on many python version
+  - "3.5"
+  - "3.6"
+  - "3.7"
+install:  # Python packages to install
+  - pip install pytest-cov
+  - pip install coveralls
+script:  # Command line to run the tests
+  - pytest -v --cov="." test.py
+after_success:
+  - coveralls  # Command to execute after success of the step script: Allow to publish coverage results to https://coveralls.io
+```
+
+To futher customize your configuration file, visit https://docs.travis-ci.com/
+
+
+5. Add the `.travis.yml` file to git, commit and push to trigger a Travis CI build:
+
+```console
+$ git add .travis.yml
+$ git commit -m ".travis.yml: Add Travis Ci configuration file"
+$ git push origin master
+Décompte des objets: 3, fait.
+Delta compression using up to 8 threads.
+Compression des objets: 100% (3/3), fait.
+Écriture des objets: 100% (3/3), 316 bytes | 316.00 KiB/s, fait.
+Total 3 (delta 2), reused 0 (delta 0)
+remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
+To github.com:NFontrodona/Lazyparser.git
+   8b85b1d..076a167  master -> master
+```
+
+6. Check the build status page to see if your build passes or fails by visiting Travis CI and selecting your repository. 
+
+Note: Your job can be queued if there is a heavy activity on Travis CI. So it can start a few minutes after your last commit, and not immediately after.
+
+
+![lazyparser](./images/travis4.png)
+
+
+7. View the logs
+
+![lazyparser](./images/travis8.png)
+
+![lazyparser](./images/travis9.png)
+
+
+8. (Optional) Add a build status badge at the top of your README.md file
+
+![lazyparser](./images/travis5.png)
+
+
+![lazyparser](./images/travis7.png)
+
+
+# Gitlab CI with Gitbio, how it works ?
+
+Official page of presentation of gitlab ci [here](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/).
+
+In the section Architecture, you can see:
+
+> **GitLab CI/CD** is built into GitLab, a web application with an API that stores its state in a database. It manages projects/builds and provides a nice user interface, in addition to other GitLab features.
+
+> **GitLab Runner** is an application that processes builds. It can be deployed separately and works with **GitLab CI/CD through an API**.
+
+> In order to run tests, you need at least one **GitLab instance** and one **GitLab Runner**.
+
+
+* We already have a *Gitlab instance*: gitbio
+* We don't have a *GitLab runner*, we need to install it.
+* We also need *a gitbio account* and a *gitbio repository* to set up Gitlab CI. 
+
+
+## How to install Gitlab runner ?
+
+According to [gitlab documentation](https://docs.gitlab.com/runner/install/), they are 3 ways to install Gitlab runner on Windows, MacOS and GNU/Linux:
+
+* with a binary
+* with docker
+* with a repository for rpm/deb package (for Linux distributions only)
+
+
+## How to install a Gitlab runner with docker ?
+
+You must have docker installed on your device. To see how to install docker, you can go here: https://docs.docker.com/get-docker/
+
+
+1. To keep the runner settings when the container is deleted:
+  * Use **docker volumes** 
+
+```console
+$ # Creation of a volume to keep the runner configurations, even after removing the container
+$ docker volume create gitlab-runner-config
+```
+
+2. Register the runner, otherwise it won't take any jobs.
+
+```console
+$ # Register command
+$ docker run --rm -it -v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner:latest register
+Runtime platform                                    arch=amd64 os=linux pid=6 revision=738bbe5a version=13.3.1
+Running in system-mode.                            
+                                                   
+Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
+https://gitbio.ens-lyon.fr/
+Please enter the gitlab-ci token for this runner:
+```
+
+3. To obtain the token, go on a gitbio repository **containing projects for which you want to set up continuous integration**. Click on settings, then on CI/CD.
+
+![lazyparser](./images/gitbio1.png)
+
+
+4. Expand the **Runner** section, then copy the token
+
+![lazyparser](./images/gitbio2.png)
+
+
+```console
+...
+Please enter the gitlab-ci token for this runner:
+q1KMKy5z
+```
+
+5. Respond to the final questions to complete the configuration
+
+```console
+Please enter the gitlab-ci description for this runner:
+[1a5c0856f1b7]: Test group runner
+Please enter the gitlab-ci tags for this runner (comma separated): 
+runner-test
+Please enter the executor: virtualbox, kubernetes, ssh, docker+machine, docker-ssh+machine, custom, docker, docker-ssh, parallels, shell:
+docker
+Please enter the default Docker image (e.g. ruby:2.6):
+python:3.8-slim
+Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
+```
+
+6. Run the container containing the gitlab runner.
+
+```console
+$ # The next command will download the gitlab runner and launch it, while using the volume previously defined
+$ docker run -d --name gitlab-runner --restart always \
+    -v /var/run/docker.sock:/var/run/docker.sock \
+    -v gitlab-runner-config:/etc/gitlab-runner \
+    gitlab/gitlab-runner:latest
+d11b5c00bbd9c7e26dd51fc1f67ec4433a18033778282e55778cf8ddd9bc39fc
+$ docker container ls  # Display the running containers
+CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                          PORTS                    NAMES
+d11b5c00bbd9        gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   3 minutes ago       Up 3 minutes                                             gitlab-runner
+```
+
+7. Refresh the page CI/CD on gitbio, the newly configured runner should appear:
+
+![lazyparser](./images/gitbio3.png)
+
+8. Edit its configuration by clicking on the pen on the right.
+
+9. Check `Run untagged jobs`. This allows the runner to pick any projects on the folder where it's defined.
+
+![lazyparser](./images/gitbio4.png)
+
+You can go [here](https://docs.gitlab.com/runner/install/docker.html) to see the complete installation procedure.
+
+
+## How to install Gitlab runner with a binary file ?
+
+1. Simply download one of the binaries for your system:
+
+```console
+$ # Linux x86-64
+$ sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
+
+$ # Linux x86
+$ sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-386
+
+$ # Linux arm
+$ sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-arm
+
+$ # Linux arm64
+$ sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-arm64
+
+$ # Linux s390x
+$ sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-s390x
+```
+
+2. Give it permissions to execute:
+
+```console
+$ sudo chmod +x /usr/local/bin/gitlab-runner
+```
+
+3. Create a GitLab CI user:
+
+```console
+$ sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
+```
+
+4. Install and run as service:
+
+```console
+$ sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
+$ sudo gitlab-runner start
+```
+
+5. Register the Runner 
+
+```console
+$ sudo gitlab-runner register
+```
+
+6. Follow the points 3, 4 and 5 of the previous section
+
+
+For more details, visit https://docs.gitlab.com/runner/install/linux-manually.html
+
+
+## Gitlab CI configuration file
+
+Now that the runner is configured and running, we can configure our Gitlab CI pipeline to make it work on a real project !
+
+GitLab CI/CD is configured by a file called `.gitlab-ci.yml` placed **at the repository root**. This file creates a pipeline, which runs for changes to the code in the repository. 
+
+Pipelines:
+* One or more `stages `
+Stages:
+   * Run in order
+   * Contains one or more jobs
+Jobs:
+  * can run in parallel
+
+
+**Example: `gitlab-ci.yml` in Projects_Analyzes/bigWig_visu project**
+
+![lazyparser](./images/gitbio5.png)
+
+```yml
+stages:  # List of the stages, for the jobs to run sequencially 
+  - unittest  # stage that perform unit tests.
+  - quality   # stage that check the code quality
+
+tests:  # Defines a job named tests. It includes the scripts and settings you want to apply to that specific task. 
+  stage: unittest  # Reference to the stage name
+  image: python:3.8-slim  # Docker image used to run the job tests
+  before_script:  # Commands to execute before the script section
+    - apt-get update
+    - apt-get install -y build-essential
+    - apt-get install -y libcurl4-gnutls-dev
+    - apt-get install -y zlib1g
+    - apt-get install -y zlib1g-dev
+    - pip install coverage
+    - pip install matplotlib==3.1.2
+    - pip install numpy==1.17.4
+    - pip install pandas==1.0.3
+    - pip install pyBigWig==0.3.17
+    - pip install seaborn==0.10.1
+    - pip install tqdm==4.47.0
+    - pip install lazyparser==0.2.0
+    - pip install pyfaidx==0.5.7
+    - pip install biopython==1.75
+    - pip install flake8
+  script:  # commands that trigger the tests
+    - coverage run -m unittest discover  # performining unit testing
+    - coverage xml  # generate a file named coverage.xml (will be used later in the quality step)
+    - flake8 --exit-zero --output-file .flake8-report.txt  # Run flake8 and create a report (that will be used in the next step)
+  artifacts: # Use for stage results that are passed between stages.
+    untracked: true  # Every untracked files produced by this job can be shared between different stages
+
+
+sonarqube-check:  # Defines a job named sonarqube-check
+  stage: quality  # Reference to the stage name
+  image: sonarsource/sonar-scanner-cli:latest  # Docker image used to run the job tests
+  variables:  # Envrionment variables to use
+    SONAR_TOKEN: a03f52695660de4b
+    SONAR_HOST_URL: http://192.168.1.78:9000/ 
+    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
+    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
+  dependencies:  # Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. 
+    - tests
+  cache: Caches are used to speed up runs of a given job in subsequent pipelines
+    key: "${CI_JOB_NAME}"  # The cache is shared accross pipelines only for this job.
+    paths:
+      - .sonar/cache  # choose which files or directories to cache. Paths are relative to the project directory
+  script:  # Command to run sonar quality analysis
+    - sonar-scanner -Dsonar.qualitygate.wait=true
+  allow_failure: true  # The CI pipeline is not stopped if the script section fails
+  only:  # The job runs only for merge requests or for the master or dev branch
+    - merge_requests
+    - master
+    - dev
+```
+
+Once the `.gitlab-ci.yml` is present at the root directory of the project, the CI pipeline will be launched each time changes are pushed to gitbio.
+
+```console
+$ git add .gitlab-ci.yml
+$ git commit -m ".gitlab-ci.yml: add gitlab CI configuration file
+$ git push origin master
+```
+
+Then, click on Setting > Pipelines on your project repository on gitbio to monitor what is happening.
+
+
+![gitbio](./images/gitbio6b.png)
+
+![lazyparser](./images/gitbio7.png)
+
+Cick on one of those pipeline to get more information about them.
+
+![lazyparser](./images/gitbio8.png)
+
+Click on `tests` or `sonarqub-check` to see the logs of those jobs.
+
+
+![lazyparser](./images/gitbio9.png)