fusion changes with nextflow git

.gitignore

@@ -22,4 +22,5 @@ nextflow
 .nextflow.log*
 .nextflow/
 work/
-results*
+results
+workspace.code-workspace

doc/TP_experimental_biologists.md

@@ -19,6 +19,8 @@ For this we are going to build a small RNASeq analysis pipeline that should run
 - build the index of a reference genome
 - estimate the amount of RNA fragments mapping to the transcripts of this genome
 
+**To do this practical you will need to have [Docker](https://www.docker.com/) installed and running on your computer**
+
 # Initialize your own project
 
 You are going to build a pipeline for you or your team. So the first step is to create your own project.
@@ -55,7 +57,7 @@ And most interestingly for you, the [src](https://gitbio.ens-lyon.fr/LBMC/nextfl
 
 ### `nf_modules`
 
-The `src/nf_modules` folder contains templates of [nextflow](https://www.nextflow.io/) wrappers for the tools available in [Docker](https://www.docker.com/what-docker) and [psmn](http://www.ens-lyon.fr/PSMN/doku.php?id=documentation:tools:psmn). The details of the [nextflow](https://www.nextflow.io/) wrapper will be presented in the next section. Alongside the `.nf` and `.config` files, there is a `tests.sh` script to run test on the tool.
+The `src/nf_modules` folder contains templates of [nextflow](https://www.nextflow.io/) wrappers for the tools available in [Docker](https://www.docker.com/what-docker). The details of the [nextflow](https://www.nextflow.io/) wrapper will be presented in the next section. Alongside the `.nf` and `.config` files, there is a `tests.sh` script to run test on the tool.
 
 # Nextflow pipeline
 
@@ -124,13 +126,6 @@ After writing this first pipeline, you may want to test it. To do that, first cl
 
 You can then run the following commands to download your project on your computer:
 
-If you are on a PSMN PC:
-
-```sh
-pip install cutadapt=1.14
-PATH="/scratch/lmodolo/:$PATH"
-```
-
 and then :
 
 ```sh
@@ -153,6 +148,8 @@ We can run our pipeline with the following command:
 ./nextflow src/fasta_sampler.nf
 ```
 
+
+
 ## Getting your results
 
 Our pipeline seems to work but we don’t know where is the `sample.fasta`. To get results out of a process, we need to tell nextflow to write it somewhere (we may don’t need to get every intermediate file in our results).
@@ -174,6 +171,8 @@ git pull origin master
 
 You can run your pipeline again and check the content of the folder `results/sampling`.
 
+
+
 ## Fasta everywhere
 
 We ran our pipeline on one fasta file. How would nextflow handle 100 of them? To test that we need to duplicate the `tiny_v2.fasta` file: 
@@ -244,8 +243,8 @@ As we are working with paired-end RNASeq data, we tell nextflow to send pairs of
 For the `fastq_sampler.nf` pipeline we used the command `head` present in most base UNIX systems. Here we want to use `cutadapt` which is not. Therefore, we have three main options:
 
 - install cutadapt locally so nextflow can use it
-- launch the process in a Docker container that has cutadapt installed
-- launch the process with psmn while loading the correct module to have cutadapt available
+- launch the process in a [Docker](https://www.docker.com/) container that has cutadapt installed
+- launch the process in a [Singularity](https://singularity.lbl.gov/) container (what we do on the PSMN and CCIN2P3)
 
 We are not going to use the first option which requires no configuration for nextflow but tedious tools installations. Instead, we are going to use existing *wrappers* and tell nextflow about it. This is what the [src/nf_modules/cutadapt/adaptor_removal_paired.config](https://gitbio.ens-lyon.fr/LBMC/nextflow/blob/master/src/nf_modules/cutadapt/adaptor_removal_paired.config) is used for.
 
@@ -331,29 +330,19 @@ login@e5-2667v4comp1
 
 ## Set your environment
 
-Make the LBMC modules available to you:
-
-```sh
-ln -s /Xnfs/lbmcdb/common/modules/modulefiles ~/privatemodules
-echo "module use ~/privatemodules" >> .bashrc
-```
-
 Create and go to your `scratch` folder:
 
 ```sh
-mkdir -p /scratch/<login>
-cd /scratch/<login>
-echo "module use ~/privatemodules" >> .bashrc
+mkdir -p /scratch/Bio/<login>
+cd /scratch/Bio/<login>
 ```
 
 Then you need to clone your pipeline and get the data:
 
 ```sh
-git config --global http.sslVerify false
 git clone https://gitbio.ens-lyon.fr/<usr_name>/nextflow.git
 cd nextflow/data
 git clone https://gitbio.ens-lyon.fr/LBMC/hub/tiny_dataset.git
-git config --global http.sslVerify true
 cd ..
 ```
 

src/.docker_modules/alntools/dd96682/Dockerfile

+FROM python:3.9-buster as build
+MAINTAINER Laurent Modolo
+
+ENV ALNTOOLS_VERSION=dd96682
+ENV PACKAGES git \
+    ca-certificates \
+    procps
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends ${PACKAGES}\
+    && apt-get clean \
+    && git clone https://github.com/churchill-lab/alntools.git \
+    && cd alntools \
+    && git checkout ${ALNTOOLS_VERSION} \
+    && python setup.py install \
+    && cd .. \
+    && rm -R alntools \
+    && apt-get autoremove --purge -y git ca-certificates
+
+CMD ["bash"] 
\ No newline at end of file

src/.docker_modules/alntools/dd96682/docker_init.sh

+#!/bin/sh
+docker pull lbmc/alntools:dd96682
+docker build src/.docker_modules/alntools/dd96682 -t 'lbmc/alntools:dd96682'
+docker push lbmc/alntools:dd96682

src/.docker_modules/bwa/0.7.17/Dockerfile

-FROM lbmc/sambamba:0.6.9
+FROM lbmc/samtools:1.11
 MAINTAINER Laurent Modolo
 
 ENV BWA_VERSION=0.7.17
 ENV SAMBLASTER_VERSION=0.1.24
 
-ENV PACKAGES curl=7.58* \
-    ca-certificates=20180409 \
-    build-essential=12.4* \
-    zlib1g-dev=1:1.2.11*
+ENV PACKAGES curl \
+  g++
 
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends ${PACKAGES} && \
-    apt-get clean
+RUN apk update && \
+    apk add ${PACKAGES}
 
 RUN curl -k -L https://github.com/lh3/bwa/releases/download/v${BWA_VERSION}/bwa-${BWA_VERSION}.tar.bz2 -o bwa-v${BWA_VERSION}.tar.bz2 && \
 tar xjf bwa-v${BWA_VERSION}.tar.bz2 && \
@@ -21,10 +18,3 @@ cp bwa /usr/bin && \
 cd .. && \
 rm -R bwa-${BWA_VERSION}/
 
-RUN curl -k -L https://github.com/GregoryFaust/samblaster/releases/download/v.${SAMBLASTER_VERSION}/samblaster-v.${SAMBLASTER_VERSION}.tar.gz -o samblaster-v.${SAMBLASTER_VERSION}.tar.gz && \
-tar xvf samblaster-v.${SAMBLASTER_VERSION}.tar.gz && \
-cd samblaster-v.${SAMBLASTER_VERSION}/ && \
-make && \
-cp samblaster /usr/bin && \
-cd .. && \
-rm -R samblaster-v.${SAMBLASTER_VERSION}/

src/.docker_modules/deeptools/3.5.0/Dockerfile

+FROM python:3.9-slim
+MAINTAINER Lauret Modolo
+
+ENV DEEPTOOLS_VERSION=3.5.0
+RUN apt-get update -qq \
+    && apt-get install --no-install-recommends --yes \
+        build-essential \
+        zlib1g-dev \
+        libbz2-dev \
+        liblzma-dev \
+        libcurl4-gnutls-dev \
+        libssl-dev \
+        libncurses5-dev
+RUN pip3 install deeptools==${DEEPTOOLS_VERSION}

src/.docker_modules/deeptools/3.5.0/docker_init.sh

+#!/bin/sh
+docker pull lbmc/deeptools:3.5.0
+docker build src/.docker_modules/deeptools/3.5.0 -t 'lbmc/deeptools:3.5.0'
+docker push lbmc/deeptools:3.5.0

src/.docker_modules/emase-zero/0.3.1/Dockerfile

+FROM debian:buster as build
+MAINTAINER Laurent Modolo
+
+ENV EMASEZERO_VERSION=0.3.1
+ENV PACKAGES build-essential \
+    ca-certificates \
+    zlib1g-dev \
+    git
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends ${PACKAGES} && \
+    apt-get clean
+
+RUN git clone https://github.com/churchill-lab/emase-zero.git \
+    && cd emase-zero/src/ \
+    && make \
+    && mv emase-zero /usr/local/bin/ \
+    && ldd /usr/local/bin/emase-zero
+# linux-vdso.so.1 (0x00007ffe8e35f000)
+# libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fbd358b5000)
+# libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fbd35731000)
+# libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbd355ae000)
+# libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fbd35594000)
+# libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbd353d3000)
+# /lib64/ld-linux-x86-64.so.2 (0x00007fbd35af7000)
+
+FROM debian:buster-slim
+
+COPY --from=build /usr/local/bin/emase-zero /usr/local/bin/
+COPY --from=build /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/
+COPY --from=build /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/
+COPY --from=build /lib/x86_64-linux-gnu/libgcc_s.so.1 /lib/x86_64-linux-gnu/
+
+RUN apt-get update && \
+    apt-get install -y procps bash

src/.docker_modules/emase-zero/0.3.1/docker_init.sh

+#!/bin/sh
+docker pull lbmc/emase-zero:0.3.1
+docker build src/.docker_modules/emase-zero/0.3.1 -t 'lbmc/emase-zero:0.3.1'
+docker push lbmc/emase-zero:0.3.1

src/.docker_modules/fastp/0.20.1/Dockerfile

+FROM quay.io/biocontainers/fastp:0.20.1--h8b12597_0
+MAINTAINER Laurent Modolo

src/.docker_modules/fastp/0.20.1/docker_init.sh

+#!/bin/sh
+docker pull lbmc/fastp:0.20.1
+docker build src/.docker_modules/fastp/0.20.1 -t 'lbmc/fastp:0.20.1'
+docker push lbmc/fastp:0.20.1

src/.docker_modules/g2gtools/0.2.8/Dockerfile

-FROM python:3.8-alpine
+FROM python:3.9-slim
 MAINTAINER Laurent Modolo
 
 ENV G2GTOOLS_VERSION=0.2.8
 
-RUN apk add --update --no-cache bash musl-dev linux-headers g++ cmake make build-base bzip2-dev zlib-dev xz-dev autoconf \
+RUN apt update \
+ && apt install -y wget build-essential zlib1g-dev libbz2-dev liblzma-dev procps \
  && wget https://github.com/churchill-lab/g2gtools/archive/v${G2GTOOLS_VERSION}.tar.gz \
  && tar -xvf v${G2GTOOLS_VERSION}.tar.gz \
  && cd g2gtools-${G2GTOOLS_VERSION} \
  && pip install numpy \
+ && pip install pysam \
  && make install 
+
+
+CMD ["bash"]

src/.docker_modules/kallistobustools/0.39.3/Dockerfile

@@ -4,7 +4,7 @@ MAINTAINER Laurent Modolo
 ENV B_VERSION=0.39.3
 ENV K_VERSION=0.46.1
 
-RUN apk add  --update --no-cache bash musl-dev linux-headers g++ cmake make build-base hdf5 hdf5-dev zlib-dev autoconf && \
+RUN apk add  --update --no-cache bash musl-dev linux-headers g++ cmake make build-base hdf5 hdf5-dev zlib-dev autoconf bash && \
 wget https://github.com/BUStools/bustools/archive/v${B_VERSION}.tar.gz && \
 tar xvf v${B_VERSION}.tar.gz && \
 cd bustools-${B_VERSION} && \

src/.docker_modules/macs3/3.0.0a6/Dockerfile

+FROM python:3.9-slim
+MAINTAINER Laurent Modolo
+
+ENV MACS3_VERSION=3.0.0a6
+RUN apt-get update -qq \
+    && apt-get install --no-install-recommends --yes \
+        build-essential \
+        zlib1g-dev \
+        libbz2-dev \
+        liblzma-dev \
+        libcurl4-gnutls-dev \
+        libssl-dev \
+        libncurses5-dev \
+        procps
+RUN pip install macs3==${MACS3_VERSION}

src/.docker_modules/macs3/3.0.0a6/docker_init.sh

+#!/bin/sh
+docker pull lbmc/macs3:3.0.0a6
+docker build src/.docker_modules/macs3/3.0.0a6 -t 'lbmc/macs3:3.0.0a6'
+docker push lbmc/macs3:3.0.0a6

src/.docker_modules/minimap2/2.17/Dockerfile

+FROM quay.io/biocontainers/samtools:0.1.18--hfb9b9cc_10 AS samtools 
+# /usr/local/bin/samtools
+# / # ldd /usr/local/bin/samtools
+#         /lib64/ld-linux-x86-64.so.2 (0x7efddcdcc000)
+#         libncurses.so.6 => /usr/local/bin/../lib/libncurses.so.6 (0x7efddcfad000)
+#         libtinfo.so.6 => /usr/local/bin/../lib/libtinfo.so.6 (0x7efddcf6f000)
+#         libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7efddcdcc000)
+#         libz.so.1 => /usr/local/bin/../lib/libz.so.1 (0x7efddcf55000)
+#         libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7efddcdcc000)
+
+FROM quay.io/biocontainers/minimap2:2.17--hed695b0_3
+MAINTAINER Laurent Modolo
+ENV MINIMAP2_VERSION=2.17
+
+COPY --from=samtools /usr/local/bin/samtools /usr/local/bin/
+COPY --from=samtools /usr/local//lib/libncurses.so.6 /usr/local/lib/
+COPY --from=samtools /usr/local//lib/libtinfo.so.6 /usr/local/lib/
+
+# /usr/local/bin/minimap2
+# / # ldd /usr/local/bin/minimap2
+#         /lib64/ld-linux-x86-64.so.2 (0x7fe14f5a8000)
+#         libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fe14f5a8000)
+#         libz.so.1 => /usr/local/bin/../lib/libz.so.1 (0x7fe14f7c4000)
+#         libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7fe14f5a8000)
+#         libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fe14f5a8000)
\ No newline at end of file

src/.docker_modules/minimap2/2.17/docker_init.sh

+#!/bin/sh
+docker pull lbmc/minimap2:2.17
+docker build src/.docker_modules/minimap2/2.17 -t 'lbmc/minimap2:2.17'
+docker push lbmc/minimap2:2.17

src/.docker_modules/multiqc/1.9/Dockerfile

+FROM debian:stretch
+MAINTAINER Laurent Modolo
+
+ENV MULTIQC_VERSION=1.9
+ENV PACKAGES build-essential \
+             python3-pip \
+             python3-setuptools \
+             python3-dev \
+             python3-wheel \
+             procps \
+             locales
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends ${PACKAGES} && \
+    apt-get clean
+
+RUN locale-gen en_US.UTF-8
+ENV LC_ALL=en_US.utf-8
+ENV LANG=en_US.utf-8
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+
+
+RUN pip3 install multiqc==${MULTIQC_VERSION}
+

src/.docker_modules/multiqc/1.9/docker_init.sh

+#!/bin/sh
+docker pull lbmc/multiqc:1.9
+docker build src/.docker_modules/multiqc/1.9 -t 'lbmc/multiqc:1.9'
+docker push lbmc/multiqc:1.9

src/.docker_modules/pandoc/2.11/Dockerfile

+FROM alpine:3.13
+MAINTAINER Laurent Modolo
+
+ENV PANDOC_VERSION=2.11
+
+RUN echo "https://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories \
+  && apk update \
+  && apk add pandoc~=${PANDOC_VERSION} make