doc/TP_experimental_biologists.md

@@ -19,6 +19,8 @@ For this we are going to build a small RNASeq analysis pipeline that should run
 - build the index of a reference genome
 - estimate the amount of RNA fragments mapping to the transcripts of this genome
 
+**To do this practical you will need to have [Docker](https://www.docker.com/) installed and running on your computer**
+
 # Initialize your own project
 
 You are going to build a pipeline for you or your team. So the first step is to create your own project.
@@ -55,7 +57,7 @@ And most interestingly for you, the [src](https://gitbio.ens-lyon.fr/LBMC/nextfl
 
 ### `nf_modules`
 
-The `src/nf_modules` folder contains templates of [nextflow](https://www.nextflow.io/) wrappers for the tools available in [Docker](https://www.docker.com/what-docker) and [psmn](http://www.ens-lyon.fr/PSMN/doku.php?id=documentation:tools:psmn). The details of the [nextflow](https://www.nextflow.io/) wrapper will be presented in the next section. Alongside the `.nf` and `.config` files, there is a `tests.sh` script to run test on the tool.
+The `src/nf_modules` folder contains templates of [nextflow](https://www.nextflow.io/) wrappers for the tools available in [Docker](https://www.docker.com/what-docker). The details of the [nextflow](https://www.nextflow.io/) wrapper will be presented in the next section. Alongside the `.nf` and `.config` files, there is a `tests.sh` script to run test on the tool.
 
 # Nextflow pipeline
 
@@ -124,13 +126,6 @@ After writing this first pipeline, you may want to test it. To do that, first cl
 
 You can then run the following commands to download your project on your computer:
 
-If you are on a PSMN PC:
-
-```sh
-pip install cutadapt=1.14
-PATH="/scratch/lmodolo/:$PATH"
-```
-
 and then :
 
 ```sh
@@ -153,6 +148,8 @@ We can run our pipeline with the following command:
 ./nextflow src/fasta_sampler.nf
 ```
 
+
+
 ## Getting your results
 
 Our pipeline seems to work but we don’t know where is the `sample.fasta`. To get results out of a process, we need to tell nextflow to write it somewhere (we may don’t need to get every intermediate file in our results).
@@ -174,6 +171,8 @@ git pull origin master
 
 You can run your pipeline again and check the content of the folder `results/sampling`.
 
+
+
 ## Fasta everywhere
 
 We ran our pipeline on one fasta file. How would nextflow handle 100 of them? To test that we need to duplicate the `tiny_v2.fasta` file: 
@@ -244,8 +243,8 @@ As we are working with paired-end RNASeq data, we tell nextflow to send pairs of
 For the `fastq_sampler.nf` pipeline we used the command `head` present in most base UNIX systems. Here we want to use `cutadapt` which is not. Therefore, we have three main options:
 
 - install cutadapt locally so nextflow can use it
-- launch the process in a Docker container that has cutadapt installed
-- launch the process with psmn while loading the correct module to have cutadapt available
+- launch the process in a [Docker](https://www.docker.com/) container that has cutadapt installed
+- launch the process in a [Singularity](https://singularity.lbl.gov/) container (what we do on the PSMN and CCIN2P3)
 
 We are not going to use the first option which requires no configuration for nextflow but tedious tools installations. Instead, we are going to use existing *wrappers* and tell nextflow about it. This is what the [src/nf_modules/cutadapt/adaptor_removal_paired.config](https://gitbio.ens-lyon.fr/LBMC/nextflow/blob/master/src/nf_modules/cutadapt/adaptor_removal_paired.config) is used for.
 
@@ -331,29 +330,19 @@ login@e5-2667v4comp1
 
 ## Set your environment
 
-Make the LBMC modules available to you:
-
-```sh
-ln -s /Xnfs/lbmcdb/common/modules/modulefiles ~/privatemodules
-echo "module use ~/privatemodules" >> .bashrc
-```
-
 Create and go to your `scratch` folder:
 
 ```sh
-mkdir -p /scratch/<login>
-cd /scratch/<login>
-echo "module use ~/privatemodules" >> .bashrc
+mkdir -p /scratch/Bio/<login>
+cd /scratch/Bio/<login>
 ```
 
 Then you need to clone your pipeline and get the data:
 
 ```sh
-git config --global http.sslVerify false
 git clone https://gitbio.ens-lyon.fr/<usr_name>/nextflow.git
 cd nextflow/data
 git clone https://gitbio.ens-lyon.fr/LBMC/hub/tiny_dataset.git
-git config --global http.sslVerify true
 cd ..
 ```
 

src/.docker_modules/kallistobustools/0.39.3/Dockerfile

@@ -4,7 +4,7 @@ MAINTAINER Laurent Modolo
 ENV B_VERSION=0.39.3
 ENV K_VERSION=0.46.1
 
-RUN apk add  --update --no-cache bash musl-dev linux-headers g++ cmake make build-base hdf5 hdf5-dev zlib-dev autoconf && \
+RUN apk add  --update --no-cache bash musl-dev linux-headers g++ cmake make build-base hdf5 hdf5-dev zlib-dev autoconf bash && \
 wget https://github.com/BUStools/bustools/archive/v${B_VERSION}.tar.gz && \
 tar xvf v${B_VERSION}.tar.gz && \
 cd bustools-${B_VERSION} && \

src/nf_modules/bedtools/fasta_from_bed.config

@@ -3,7 +3,7 @@ profiles {
     docker.temp = "auto"
     docker.enabled = true
     process {
-      withName: fasta_from_bed {
+      withLabel: bedtools {
         container = "lbmc/bedtools:2.25.0"
         cpus = 1
       }
@@ -13,7 +13,7 @@ profiles {
     singularity.enabled = true
     singularity.cacheDir = "./bin/"
     process {
-      withName: fasta_from_bed {
+      withLabel: bedtools {
         container = "lbmc/bedtools:2.25.0"
         cpus = 1
       }
@@ -24,7 +24,7 @@ profiles {
     singularity.cacheDir = "/.singularity_psmn/"
     singularity.runOptions = "--bind /Xnfs,/scratch"
     process{
-      withName: fasta_from_bed {
+      withLabel: bedtools {
         container = "lbmc/bedtools:2.25.0"
         executor = "sge"
         clusterOptions = "-cwd -V"
@@ -40,7 +40,7 @@ profiles {
     singularity.cacheDir = "$baseDir/.singularity_in2p3/"
     singularity.runOptions = "--bind /pbs,/sps,/scratch"
     process{
-      withName: fasta_from_bed {
+      withLabel: bedtools {
         container = "lbmc/bedtools:2.25.0"
         scratch = true
         stageInMode = "copy"

src/nf_modules/bedtools/fasta_from_bed.nf

@@ -24,6 +24,7 @@ Channel
 process fasta_from_bed {
   tag "${bed.baseName}"
   publishDir "results/fasta/", mode: 'copy'
+  label "bedtools"
 
   input:
   file fasta from fasta_files

src/nf_modules/bowtie/indexing.config

@@ -3,7 +3,7 @@ profiles {
     docker.temp = "auto"
     docker.enabled = true
     process {
-      withName: index_fasta {
+      withLabel: bowtie {
         cpus = 4
         container = "lbmc/bowtie:1.2.2"
       }
@@ -13,7 +13,7 @@ profiles {
     singularity.enabled = true
     singularity.cacheDir = "./bin/"
     process {
-      withName: index_fasta {
+      withLabel: bowtie {
         cpus = 4
         container = "lbmc/bowtie:1.2.2"
       }
@@ -24,7 +24,7 @@ profiles {
     singularity.cacheDir = "/.singularity_psmn/"
     singularity.runOptions = "--bind /Xnfs,/scratch"
     process{
-      withName: index_fasta {
+      withLabel: bowtie {
         container = "lbmc/bowtie:1.2.2"
         executor = "sge"
         clusterOptions = "-cwd -V"
@@ -41,7 +41,7 @@ profiles {
     singularity.cacheDir = "$baseDir/.singularity_in2p3/"
     singularity.runOptions = "--bind /pbs,/sps,/scratch"
     process{
-      withName: index_fasta {
+      withLabel: bowtie {
         container = "lbmc/bowtie:1.2.2"
         scratch = true
         stageInMode = "copy"

src/nf_modules/bowtie/indexing.nf

@@ -12,6 +12,7 @@ Channel
 process index_fasta {
   tag "$fasta.baseName"
   publishDir "results/mapping/index/", mode: 'copy'
+  label "bowtie"
 
   input:
     file fasta from fasta_file

src/nf_modules/bowtie/mapping_paired.config

@@ -3,7 +3,7 @@ profiles {
     docker.temp = "auto"
     docker.enabled = true
     process {
-      withName: mapping_fastq {
+      withLabel: bowtie {
         container = "lbmc/bowtie:1.2.2"
         cpus = 4
       }
@@ -13,7 +13,7 @@ profiles {
     singularity.enabled = true
     singularity.cacheDir = "./bin/"
     process {
-      withName: mapping_fastq {
+      withLabel: bowtie {
         cpus = 4
         container = "lbmc/bowtie:1.2.2"
       }
@@ -24,7 +24,7 @@ profiles {
     singularity.cacheDir = "/.singularity_psmn/"
     singularity.runOptions = "--bind /Xnfs,/scratch"
     process{
-      withName: mapping_fastq {
+      withLabel: bowtie {
         container = "lbmc/bowtie:1.2.2"
         executor = "sge"
         clusterOptions = "-cwd -V"
@@ -41,7 +41,7 @@ profiles {
     singularity.cacheDir = "$baseDir/.singularity_in2p3/"
     singularity.runOptions = "--bind /pbs,/sps,/scratch"
     process{
-      withName: mapping_fastq {
+      withLabel: bowtie {
         container = "lbmc/bowtie:1.2.2"
         scratch = true
         stageInMode = "copy"

src/nf_modules/bowtie/mapping_paired.nf

@@ -20,6 +20,7 @@ Channel
 process mapping_fastq {
   tag "$pair_id"
   publishDir "results/mapping/bams/", mode: 'copy'
+  label "bowtie"
 
   input:
   set pair_id, file(reads) from fastq_files

src/nf_modules/bowtie/mapping_single.config

@@ -3,7 +3,7 @@ profiles {
     docker.temp = "auto"
     docker.enabled = true
     process {
-      withName: mapping_fastq {
+      withLabel: bowtie {
         container = "lbmc/bowtie:1.2.2"
         cpus = 4
       }
@@ -13,7 +13,7 @@ profiles {
     singularity.enabled = true
     singularity.cacheDir = "./bin/"
     process {
-      withName: mapping_fastq {
+      withLabel: bowtie {
         cpus = 4
         container = "lbmc/bowtie:1.2.2"
       }
@@ -24,7 +24,7 @@ profiles {
     singularity.cacheDir = "/.singularity_psmn/"
     singularity.runOptions = "--bind /Xnfs,/scratch"
     process{
-      withName: mapping_fastq {
+      withLabel: bowtie {
         container = "lbmc/bowtie:1.2.2"
         executor = "sge"
         clusterOptions = "-cwd -V"
@@ -41,7 +41,7 @@ profiles {
     singularity.cacheDir = "$baseDir/.singularity_in2p3/"
     singularity.runOptions = "--bind /pbs,/sps,/scratch"
     process{
-      withName: mapping_fastq {
+      withLabel: bowtie {
         container = "lbmc/bowtie:1.2.2"
         scratch = true
         stageInMode = "copy"

src/nf_modules/bowtie/mapping_single.nf

@@ -20,6 +20,7 @@ Channel
 process mapping_fastq {
   tag "$file_id"
   publishDir "results/mapping/bams/", mode: 'copy'
+  label "bowtie"
 
   input:
   set file_id, file(reads) from fastq_files

src/nf_modules/bowtie2/indexing.config

@@ -3,7 +3,7 @@ profiles {
     docker.temp = "auto"
     docker.enabled = true
     process {
-      withName: index_fasta {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         cpus = 4
       }
@@ -13,7 +13,7 @@ profiles {
     singularity.enabled = true
     singularity.cacheDir = "./bin/"
     process {
-      withName: index_fasta {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         cpus = 4
       }
@@ -24,7 +24,7 @@ profiles {
     singularity.cacheDir = "/.singularity_psmn/"
     singularity.runOptions = "--bind /Xnfs,/scratch"
     process{
-      withName: index_fasta {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         executor = "sge"
         clusterOptions = "-cwd -V"
@@ -41,7 +41,7 @@ profiles {
     singularity.cacheDir = "$baseDir/.singularity_in2p3/"
     singularity.runOptions = "--bind /pbs,/sps,/scratch"
     process{
-      withName: index_fasta {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         scratch = true
         stageInMode = "copy"

src/nf_modules/bowtie2/indexing.nf

@@ -10,6 +10,7 @@ Channel
 process index_fasta {
   tag "$fasta.baseName"
   publishDir "results/mapping/index/", mode: 'copy'
+  label "bowtie2"
 
   input:
     file fasta from fasta_file

src/nf_modules/bowtie2/mapping_paired.config

@@ -3,7 +3,7 @@ profiles {
     docker.temp = "auto"
     docker.enabled = true
     process {
-      withName: mapping_fastq {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         cpus = 4
       }
@@ -13,7 +13,7 @@ profiles {
     singularity.enabled = true
     singularity.cacheDir = "./bin/"
     process {
-      withName: mapping_fastq {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         cpus = 4
       }
@@ -24,7 +24,7 @@ profiles {
     singularity.cacheDir = "/.singularity_psmn/"
     singularity.runOptions = "--bind /Xnfs,/scratch"
     process{
-      withName: mapping_fastq {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         executor = "sge"
         clusterOptions = "-cwd -V"
@@ -41,7 +41,7 @@ profiles {
     singularity.cacheDir = "$baseDir/.singularity_in2p3/"
     singularity.runOptions = "--bind /pbs,/sps,/scratch"
     process{
-      withName: mapping_fastq {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         scratch = true
         stageInMode = "copy"

src/nf_modules/bowtie2/mapping_paired.nf

@@ -16,6 +16,7 @@ Channel
 process mapping_fastq {
   tag "$pair_id"
   publishDir "results/mapping/bams/", mode: 'copy'
+  label "bowtie2"
 
   input:
   set pair_id, file(reads) from fastq_files

src/nf_modules/bowtie2/mapping_single.config

@@ -3,7 +3,7 @@ profiles {
     docker.temp = "auto"
     docker.enabled = true
     process {
-      withName: mapping_fastq {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         cpus = 4
       }
@@ -13,7 +13,7 @@ profiles {
     singularity.enabled = true
     singularity.cacheDir = "./bin/"
     process {
-      withName: mapping_fastq {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         cpus = 4
       }
@@ -24,7 +24,7 @@ profiles {
     singularity.cacheDir = "/.singularity_psmn/"
     singularity.runOptions = "--bind /Xnfs,/scratch"
     process{
-      withName: mapping_fastq {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         executor = "sge"
         clusterOptions = "-cwd -V"
@@ -41,7 +41,7 @@ profiles {
     singularity.cacheDir = "$baseDir/.singularity_in2p3/"
     singularity.runOptions = "--bind /pbs,/sps,/scratch"
     process{
-      withName: mapping_fastq {
+      withLabel: bowtie2 {
         container = "lbmc/bowtie2:2.3.4.1"
         scratch = true
         stageInMode = "copy"

src/nf_modules/bowtie2/mapping_single.nf

@@ -16,6 +16,7 @@ Channel
 process mapping_fastq {
   tag "$file_id"
   publishDir "results/mapping/bams/", mode: 'copy'
+  label "bowtie2"
 
   input:
   set file_id, file(reads) from fastq_files

src/nf_modules/cutadapt/adaptor_removal_paired.config

@@ -3,7 +3,7 @@ profiles {
     docker.temp = "auto"
     docker.enabled = true
     process {
-      withName: adaptor_removal {
+      withLabel: cutadapt {
         container = "lbmc/cutadapt:2.1"
         cpus = 1
       }
@@ -13,7 +13,7 @@ profiles {
     singularity.enabled = true
     singularity.cacheDir = "./bin/"
     process {
-      withName: adaptor_removal {
+      withLabel: cutadapt {
         container = "lbmc/cutadapt:2.1"
         cpus = 1
       }
@@ -24,7 +24,7 @@ profiles {
     singularity.cacheDir = "/.singularity_psmn/"
     singularity.runOptions = "--bind /Xnfs,/scratch"
     process{
-      withName: adaptor_removal {
+      withLabel: cutadapt {
         container = "lbmc/cutadapt:2.1"
         executor = "sge"
         clusterOptions = "-cwd -V"
@@ -40,7 +40,7 @@ profiles {
     singularity.cacheDir = "$baseDir/.singularity_in2p3/"
     singularity.runOptions = "--bind /pbs,/sps,/scratch"
     process{
-      withName: adaptor_removal {
+      withLabel: cutadapt {
         container = "lbmc/cutadapt:2.1"
         scratch = true
         stageInMode = "copy"

src/nf_modules/cutadapt/adaptor_removal_paired.nf

@@ -8,6 +8,7 @@ Channel
 process adaptor_removal {
   tag "$pair_id"
   publishDir "results/fastq/adaptor_removal/", mode: 'copy'
+  label "cutadapt"
 
   input:
   set pair_id, file(reads) from fastq_files

src/nf_modules/cutadapt/adaptor_removal_single.config

@@ -3,7 +3,7 @@ profiles {
     docker.temp = "auto"
     docker.enabled = true
     process {
-      withName: adaptor_removal {
+      withLabel: cutadapt {
         container = "lbmc/cutadapt:2.1"
         cpus = 1
       }
@@ -13,7 +13,7 @@ profiles {
     singularity.enabled = true
     singularity.cacheDir = "./bin/"
     process {
-      withName: adaptor_removal {
+      withLabel: cutadapt {
         container = "lbmc/cutadapt:2.1"
         cpus = 1
       }
@@ -24,7 +24,7 @@ profiles {
     singularity.cacheDir = "/.singularity_psmn/"
     singularity.runOptions = "--bind /Xnfs,/scratch"
     process{
-      withName: adaptor_removal {
+      withLabel: cutadapt {
         container = "lbmc/cutadapt:2.1"
         executor = "sge"
         clusterOptions = "-cwd -V"
@@ -40,7 +40,7 @@ profiles {
     singularity.cacheDir = "$baseDir/.singularity_in2p3/"
     singularity.runOptions = "--bind /pbs,/sps,/scratch"
     process{
-      withName: adaptor_removal {
+      withLabel: cutadapt {
         container = "lbmc/cutadapt:2.1"
         scratch = true
         stageInMode = "copy"

src/nf_modules/cutadapt/adaptor_removal_single.nf

@@ -8,6 +8,7 @@ Channel
 
 process adaptor_removal {
   tag "$file_id"
+  label "cutadapt"
 
   input:
   set file_id, file(reads) from fastq_files