diff --git a/.gitattributes b/.gitattributes
index 7fe55006f87bb1a423e2cdf70258a55543c2486d..050bb1203530c3ee0b610fe2a05aed5059a3bf19 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -1 +1,3 @@
*.config linguist-language=nextflow
+modules/nf-core/** linguist-generated
+subworkflows/nf-core/** linguist-generated
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
index 4b432d2ac27327fac258792c6570af2149cfb053..383bc0e3d4409ee25f921bddd7b51bffe29f1fae 100644
--- a/.github/CONTRIBUTING.md
+++ b/.github/CONTRIBUTING.md
@@ -68,16 +68,13 @@ If you wish to contribute a new step, please use the following coding standards:
1. Define the corresponding input channel into your new process from the expected previous process channel
2. Write the process block (see below).
3. Define the output channel if needed (see below).
-4. Add any new flags/options to `nextflow.config` with a default (see below).
-5. Add any new flags/options to `nextflow_schema.json` with help text (with `nf-core schema build`).
-6. Add any new flags/options to the help message (for integer/text parameters, print to help the corresponding `nextflow.config` parameter).
-7. Add sanity checks for all relevant parameters.
-8. Add any new software to the `scrape_software_versions.py` script in `bin/` and the version command to the `scrape_software_versions` process in `main.nf`.
-9. Do local tests that the new code works properly and as expected.
-10. Add a new test command in `.github/workflow/ci.yml`.
-11. If applicable add a [MultiQC](https://https://multiqc.info/) module.
-12. Update MultiQC config `assets/multiqc_config.yaml` so relevant suffixes, name clean up, General Statistics Table column order, and module figures are in the right order.
-13. Optional: Add any descriptions of MultiQC report sections and output files to `docs/output.md`.
+4. Add any new parameters to `nextflow.config` with a default (see below).
+5. Add any new parameters to `nextflow_schema.json` with help text (via the `nf-core schema build` tool).
+6. Add sanity checks and validation for all relevant parameters.
+7. Perform local tests to validate that the new code works as expected.
+8. If applicable, add a new test command in `.github/workflow/ci.yml`.
+9. Update MultiQC config `assets/multiqc_config.yaml` so relevant suffixes, file name clean up and module plots are in the appropriate order. If applicable, add a [MultiQC](https://https://multiqc.info/) module.
+10. Add a description of the output files and if relevant any appropriate images from the MultiQC report to `docs/output.md`.
### Default values
@@ -102,27 +99,6 @@ Please use the following naming schemes, to make it easy to understand what is g
If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core bump-version --nextflow . [min-nf-version]`
-### Software version reporting
-
-If you add a new tool to the pipeline, please ensure you add the information of the tool to the `get_software_version` process.
-
-Add to the script block of the process, something like the following:
-
-```bash
-<YOUR_TOOL> --version &> v_<YOUR_TOOL>.txt 2>&1 || true
-```
-
-or
-
-```bash
-<YOUR_TOOL> --help | head -n 1 &> v_<YOUR_TOOL>.txt 2>&1 || true
-```
-
-You then need to edit the script `bin/scrape_software_versions.py` to:
-
-1. Add a Python regex for your tool's `--version` output (as in stored in the `v_<YOUR_TOOL>.txt` file), to ensure the version is reported as a `v` and the version number e.g. `v2.1.1`
-2. Add a HTML entry to the `OrderedDict` for formatting in MultiQC.
-
### Images and figures
For overview images and other documents we follow the nf-core [style guidelines and examples](https://nf-co.re/developers/design_guidelines).
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
deleted file mode 100644
index bc2507dad1513181e12d0d3f6aacf7efc804afde..0000000000000000000000000000000000000000
--- a/.github/ISSUE_TEMPLATE/bug_report.md
+++ /dev/null
@@ -1,63 +0,0 @@
----
-name: Bug report
-about: Report something that is broken or incorrect
-labels: bug
----
-
-<!--
-# nf-core/hic bug report
-
-Hi there!
-
-Thanks for telling us about a problem with the pipeline.
-Please delete this text and anything that's not relevant from the template below:
--->
-
-## Check Documentation
-
-I have checked the following places for your error:
-
-- [ ] [nf-core website: troubleshooting](https://nf-co.re/usage/troubleshooting)
-- [ ] [nf-core/hic pipeline documentation](https://nf-co.re/hic/usage)
-
-## Description of the bug
-
-<!-- A clear and concise description of what the bug is. -->
-
-## Steps to reproduce
-
-Steps to reproduce the behaviour:
-
-1. Command line: <!-- [e.g. `nextflow run ...`] -->
-2. See error: <!-- [Please provide your error message] -->
-
-## Expected behaviour
-
-<!-- A clear and concise description of what you expected to happen. -->
-
-## Log files
-
-Have you provided the following extra information/files:
-
-- [ ] The command used to run the pipeline
-- [ ] The `.nextflow.log` file <!-- this is a hidden file in the directory where you launched the pipeline -->
-
-## System
-
-- Hardware: <!-- [e.g. HPC, Desktop, Cloud...] -->
-- Executor: <!-- [e.g. slurm, local, awsbatch...] -->
-- OS: <!-- [e.g. CentOS Linux, macOS, Linux Mint...] -->
-- Version <!-- [e.g. 7, 10.13.6, 18.3...] -->
-
-## Nextflow Installation
-
-- Version: <!-- [e.g. 21.04.0] -->
-
-## Container engine
-
-- Engine: <!-- [e.g. Conda, Docker, Singularity, Podman, Shifter or Charliecloud] -->
-- version: <!-- [e.g. 1.0.0] -->
-
-## Additional context
-
-<!-- Add any other context about the problem here. -->
diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml
new file mode 100644
index 0000000000000000000000000000000000000000..3505b2a02de8f609e32f1cc50ce686f2e107961e
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/bug_report.yml
@@ -0,0 +1,52 @@
+
+name: Bug report
+description: Report something that is broken or incorrect
+labels: bug
+body:
+
+ - type: markdown
+ attributes:
+ value: |
+ Before you post this issue, please check the documentation:
+
+ - [nf-core website: troubleshooting](https://nf-co.re/usage/troubleshooting)
+ - [nf-core/hic pipeline documentation](https://nf-co.re/hic/usage)
+
+ - type: textarea
+ id: description
+ attributes:
+ label: Description of the bug
+ description: A clear and concise description of what the bug is.
+ validations:
+ required: true
+
+ - type: textarea
+ id: command_used
+ attributes:
+ label: Command used and terminal output
+ description: Steps to reproduce the behaviour. Please paste the command you used to launch the pipeline and the output from your terminal.
+ render: console
+ placeholder: |
+ $ nextflow run ...
+
+ Some output where something broke
+
+ - type: textarea
+ id: files
+ attributes:
+ label: Relevant files
+ description: |
+ Please drag and drop the relevant files here. Create a `.zip` archive if the extension is not allowed.
+ Your verbose log file `.nextflow.log` is often useful _(this is a hidden file in the directory where you launched the pipeline)_ as well as custom Nextflow configuration files.
+
+ - type: textarea
+ id: system
+ attributes:
+ label: System information
+ description: |
+ * Nextflow version _(eg. 21.10.3)_
+ * Hardware _(eg. HPC, Desktop, Cloud)_
+ * Executor _(eg. slurm, local, awsbatch)_
+ * Container engine: _(e.g. Docker, Singularity, Conda, Podman, Shifter or Charliecloud)_
+ * OS _(eg. CentOS Linux, macOS, Linux Mint)_
+ * Version of nf-core/hic _(eg. 1.1, 1.5, 1.8.2)_
diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml
index 887f04598ba6bb758d35d2ca79012ccc8129ff34..379c60ae3294f480310d7e5ebeb2f6641dac2374 100644
--- a/.github/ISSUE_TEMPLATE/config.yml
+++ b/.github/ISSUE_TEMPLATE/config.yml
@@ -1,4 +1,3 @@
-blank_issues_enabled: false
contact_links:
- name: Join nf-core
url: https://nf-co.re/join
diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md
deleted file mode 100644
index 2cec9b3b778f87d420a0d124094557fe5b8efadf..0000000000000000000000000000000000000000
--- a/.github/ISSUE_TEMPLATE/feature_request.md
+++ /dev/null
@@ -1,32 +0,0 @@
----
-name: Feature request
-about: Suggest an idea for the nf-core/hic pipeline
-labels: enhancement
----
-
-<!--
-# nf-core/hic feature request
-
-Hi there!
-
-Thanks for suggesting a new feature for the pipeline!
-Please delete this text and anything that's not relevant from the template below:
--->
-
-## Is your feature request related to a problem? Please describe
-
-<!-- A clear and concise description of what the problem is. -->
-
-<!-- e.g. [I'm always frustrated when ...] -->
-
-## Describe the solution you'd like
-
-<!-- A clear and concise description of what you want to happen. -->
-
-## Describe alternatives you've considered
-
-<!-- A clear and concise description of any alternative solutions or features you've considered. -->
-
-## Additional context
-
-<!-- Add any other context about the feature request here. -->
diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml
new file mode 100644
index 0000000000000000000000000000000000000000..d411b185d23b589888a16fc63b564928ba95afca
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/feature_request.yml
@@ -0,0 +1,11 @@
+name: Feature request
+description: Suggest an idea for the nf-core/hic pipeline
+labels: enhancement
+body:
+ - type: textarea
+ id: description
+ attributes:
+ label: Description of feature
+ description: Please describe your suggestion for a new feature. It might help to describe a problem or use case, plus any alternatives that you have considered.
+ validations:
+ required: true
diff --git a/.github/workflows/awsfulltest.yml b/.github/workflows/awsfulltest.yml
index caa6a1c08a82dfe09fc2e465e4d994ce112c9a5b..dd00a784096880586f67ce18c9c642ea10d376a3 100644
--- a/.github/workflows/awsfulltest.yml
+++ b/.github/workflows/awsfulltest.yml
@@ -14,14 +14,14 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Launch workflow via tower
- uses: nf-core/tower-action@master
+ uses: nf-core/tower-action@v2
# TODO nf-core: You can customise AWS full pipeline tests as required
# Add full size test data (but still relatively small datasets for few samples)
# on the `test_full.config` test runs with only one set of parameters
with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
- bearer_token: ${{ secrets.TOWER_BEARER_TOKEN }}
+ access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_COMPUTE_ENV }}
pipeline: ${{ github.repository }}
revision: ${{ github.sha }}
@@ -30,5 +30,5 @@ jobs:
{
"outdir": "s3://${{ secrets.AWS_S3_BUCKET }}/hic/results-${{ github.sha }}"
}
- profiles: '[ "test_full", "aws_tower" ]'
-
+ profiles: test_full,aws_tower
+ pre_run_script: 'export NXF_VER=21.10.3'
diff --git a/.github/workflows/awstest.yml b/.github/workflows/awstest.yml
index 654e7b637fa1f1ac0a521b2425c9ef3d7c4ea019..cb0c5ee9727a2f7584a70bc35a5e57edfa9008d3 100644
--- a/.github/workflows/awstest.yml
+++ b/.github/workflows/awstest.yml
@@ -11,18 +11,18 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Launch workflow via tower
- uses: nf-core/tower-action@master
+ uses: nf-core/tower-action@v2
with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
- bearer_token: ${{ secrets.TOWER_BEARER_TOKEN }}
+ access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_COMPUTE_ENV }}
pipeline: ${{ github.repository }}
revision: ${{ github.sha }}
workdir: s3://${{ secrets.AWS_S3_BUCKET }}/work/hic/work-${{ github.sha }}
parameters: |
{
- "outdir": "s3://${{ secrets.AWS_S3_BUCKET }}/hic/results-${{ github.sha }}"
+ "outdir": "s3://${{ secrets.AWS_S3_BUCKET }}/hic/results-test-${{ github.sha }}"
}
- profiles: '[ "test", "aws_tower" ]'
-
+ profiles: test,aws_tower
+ pre_run_script: 'export NXF_VER=21.10.3'
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 3c0b4e6f58188d5a5e4fcff1caba3d5371baa773..8ffa52b69fa84dd6d60364e79bb6bd84c1e3a644 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -8,8 +8,9 @@ on:
release:
types: [published]
-# Uncomment if we need an edge release of Nextflow again
-# env: NXF_EDGE: 1
+env:
+ NXF_ANSI_LOG: false
+ CAPSULE_LOG: none
jobs:
test:
@@ -17,20 +18,26 @@ jobs:
# Only run on push if this is the nf-core dev branch (merged PRs)
if: ${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/hic') }}
runs-on: ubuntu-latest
- env:
- NXF_VER: ${{ matrix.nxf_ver }}
- NXF_ANSI_LOG: false
strategy:
matrix:
- # Nextflow versions: check pipeline minimum and current latest
- nxf_ver: ['21.04.0', '']
+ # Nextflow versions
+ include:
+ # Test pipeline minimum Nextflow version
+ - NXF_VER: '21.10.3'
+ NXF_EDGE: ''
+ # Test latest edge release of Nextflow
+ - NXF_VER: ''
+ NXF_EDGE: '1'
steps:
- name: Check out pipeline code
uses: actions/checkout@v2
- name: Install Nextflow
env:
- CAPSULE_LOG: none
+ NXF_VER: ${{ matrix.NXF_VER }}
+ # Uncomment only if the edge release is more recent than the latest stable release
+ # See https://github.com/nextflow-io/nextflow/issues/2467
+ # NXF_EDGE: ${{ matrix.NXF_EDGE }}
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/
diff --git a/.github/workflows/linting_comment.yml b/.github/workflows/linting_comment.yml
index 90f03c6f91ba0c05d12daff7e15840145a63937f..44d72994b0c4b37a2dfcfc3508d00594a1e98a99 100644
--- a/.github/workflows/linting_comment.yml
+++ b/.github/workflows/linting_comment.yml
@@ -15,6 +15,7 @@ jobs:
uses: dawidd6/action-download-artifact@v2
with:
workflow: linting.yml
+ workflow_conclusion: completed
- name: Get PR number
id: pr_number
diff --git a/CHANGELOG.md b/CHANGELOG.md
index fcf522d7d2038c7f1fd424fb82478a325f3db82e..87c2402f589037c624c75cde16bf61cb34bd86b8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,7 +3,7 @@
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
-## v1.3.0 - [date]
+## v1.3.1 - [date]
Initial release of nf-core/hic, created with the [nf-core](https://nf-co.re/) template.
diff --git a/CITATIONS.md b/CITATIONS.md
index da253d248988442de777df56482ff1333cd7dce5..557e1f8a024fa15003b389af094921953a738abf 100644
--- a/CITATIONS.md
+++ b/CITATIONS.md
@@ -12,7 +12,7 @@
* [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/)
-* [MultiQC](https://www.ncbi.nlm.nih.gov/pubmed/27312411/)
+* [MultiQC](https://pubmed.ncbi.nlm.nih.gov/27312411/)
> Ewels P, Magnusson M, Lundin S, Käller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924.
## Software packaging/containerisation tools
diff --git a/README.md b/README.md
index 3dd2b4b3fe48f429a3918f596631749fec36dd95..35b13852b795d9baf704dcb47f8b9b03d549198e 100644
--- a/README.md
+++ b/README.md
@@ -1,11 +1,11 @@
-# 
+#  
[](https://github.com/nf-core/hic/actions?query=workflow%3A%22nf-core+CI%22)
[](https://github.com/nf-core/hic/actions?query=workflow%3A%22nf-core+linting%22)
[](https://nf-co.re/hic/results)
[](https://doi.org/10.5281/zenodo.XXXXXXX)
-[](https://www.nextflow.io/)
+[](https://www.nextflow.io/)
[](https://docs.conda.io/en/latest/)
[](https://www.docker.com/)
[](https://sylabs.io/docs/)
@@ -33,18 +33,21 @@ On release, automated continuous integration tests run the pipeline on a full-si
## Quick Start
-1. Install [`Nextflow`](https://www.nextflow.io/docs/latest/getstarted.html#installation) (`>=21.04.0`)
+1. Install [`Nextflow`](https://www.nextflow.io/docs/latest/getstarted.html#installation) (`>=21.10.3`)
2. Install any of [`Docker`](https://docs.docker.com/engine/installation/), [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/), [`Podman`](https://podman.io/), [`Shifter`](https://nersc.gitlab.io/development/shifter/how-to-use/) or [`Charliecloud`](https://hpc.github.io/charliecloud/) for full pipeline reproducibility _(please only use [`Conda`](https://conda.io/miniconda.html) as a last resort; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles))_
3. Download the pipeline and test it on a minimal dataset with a single command:
```console
- nextflow run nf-core/hic -profile test,<docker/singularity/podman/shifter/charliecloud/conda/institute>
+ nextflow run nf-core/hic -profile test,YOURPROFILE
```
+ Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile (`YOURPROFILE` in the example command above). You can chain multiple config profiles in a comma-separated string.
+
+ > * The pipeline comes with config profiles called `docker`, `singularity`, `podman`, `shifter`, `charliecloud` and `conda` which instruct the pipeline to use the named tool for software management. For example, `-profile test,docker`.
> * Please check [nf-core/configs](https://github.com/nf-core/configs#documentation) to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use `-profile <institute>` in your command. This will enable either `docker` or `singularity` and set the appropriate execution settings for your local compute environment.
- > * If you are using `singularity` then the pipeline will auto-detect this and attempt to download the Singularity images directly as opposed to performing a conversion from Docker images. If you are persistently observing issues downloading Singularity images directly due to timeout or network issues then please use the `--singularity_pull_docker_container` parameter to pull and convert the Docker image instead. Alternatively, it is highly recommended to use the [`nf-core download`](https://nf-co.re/tools/#downloading-pipelines-for-offline-use) command to pre-download all of the required containers before running the pipeline and to set the [`NXF_SINGULARITY_CACHEDIR` or `singularity.cacheDir`](https://www.nextflow.io/docs/latest/singularity.html?#singularity-docker-hub) Nextflow options to be able to store and re-use the images from a central location for future pipeline runs.
+ > * If you are using `singularity` and are persistently observing issues downloading Singularity images directly due to timeout or network issues, then you can use the `--singularity_pull_docker_container` parameter to pull and convert the Docker image instead. Alternatively, you can use the [`nf-core download`](https://nf-co.re/tools/#downloading-pipelines-for-offline-use) command to download images first, before running the pipeline. Setting the [`NXF_SINGULARITY_CACHEDIR` or `singularity.cacheDir`](https://www.nextflow.io/docs/latest/singularity.html?#singularity-docker-hub) Nextflow options enables you to store and re-use the images from a central location for future pipeline runs.
> * If you are using `conda`, it is highly recommended to use the [`NXF_CONDA_CACHEDIR` or `conda.cacheDir`](https://www.nextflow.io/docs/latest/conda.html) settings to store the environments in a central location for future pipeline runs.
4. Start running your own analysis!
diff --git a/assets/multiqc_config.yaml b/assets/multiqc_config.yaml
index 41468cab303a5894aa01e0823790b22cb44c95cd..c05ad146496d1bbabc1b96cdb5ae2511327291df 100644
--- a/assets/multiqc_config.yaml
+++ b/assets/multiqc_config.yaml
@@ -1,7 +1,7 @@
report_comment: >
This report has been generated by the <a href="https://github.com/nf-core/hic" target="_blank">nf-core/hic</a>
analysis pipeline. For information about how to interpret these results, please see the
- <a href="https://github.com/nf-core/hic" target="_blank">documentation</a>.
+ <a href="https://nf-co.re/hic" target="_blank">documentation</a>.
report_section_order:
software_versions:
order: -1000
diff --git a/assets/nf-core-hic_logo.png b/assets/nf-core-hic_logo.png
deleted file mode 100644
index 37461d9a32ae1f73d9090a3a2387cf8997c9a0ed..0000000000000000000000000000000000000000
Binary files a/assets/nf-core-hic_logo.png and /dev/null differ
diff --git a/assets/nf-core-hic_logo_light.png b/assets/nf-core-hic_logo_light.png
new file mode 100644
index 0000000000000000000000000000000000000000..553c19d982fcf9e1e0c9a5f325a8ff290fc087f9
Binary files /dev/null and b/assets/nf-core-hic_logo_light.png differ
diff --git a/assets/sendmail_template.txt b/assets/sendmail_template.txt
index 6213286c8a3c027fb1fb4496c938fbe3ed7909d3..3c7f7236d7a173bc0d6c1dc6c5cd1078d5a83786 100644
--- a/assets/sendmail_template.txt
+++ b/assets/sendmail_template.txt
@@ -12,9 +12,9 @@ $email_html
Content-Type: image/png;name="nf-core-hic_logo.png"
Content-Transfer-Encoding: base64
Content-ID: <nfcorepipelinelogo>
-Content-Disposition: inline; filename="nf-core-hic_logo.png"
+Content-Disposition: inline; filename="nf-core-hic_logo_light.png"
-<% out << new File("$projectDir/assets/nf-core-hic_logo.png").
+<% out << new File("$projectDir/assets/nf-core-hic_logo_light.png").
bytes.
encodeBase64().
toString().
diff --git a/bin/scrape_software_versions.py b/bin/scrape_software_versions.py
deleted file mode 100755
index 322c97c6fc74194e352853b6f5c8f33827a1cbd9..0000000000000000000000000000000000000000
--- a/bin/scrape_software_versions.py
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/usr/bin/env python
-from __future__ import print_function
-import os
-
-results = {}
-version_files = [x for x in os.listdir(".") if x.endswith(".version.txt")]
-for version_file in version_files:
-
- software = version_file.replace(".version.txt", "")
- if software == "pipeline":
- software = "nf-core/hic"
-
- with open(version_file) as fin:
- version = fin.read().strip()
- results[software] = version
-
-# Dump to YAML
-print(
- """
-id: 'software_versions'
-section_name: 'nf-core/hic Software Versions'
-section_href: 'https://github.com/nf-core/hic'
-plot_type: 'html'
-description: 'are collected at run time from the software output.'
-data: |
- <dl class="dl-horizontal">
-"""
-)
-for k, v in sorted(results.items()):
- print(" <dt>{}</dt><dd><samp>{}</samp></dd>".format(k, v))
-print(" </dl>")
-
-# Write out as tsv file:
-with open("software_versions.tsv", "w") as f:
- for k, v in sorted(results.items()):
- f.write("{}\t{}\n".format(k, v))
diff --git a/conf/base.config b/conf/base.config
index 2e7db0c1db87efe04da28a0914de2e9130a1c112..42a37e9e41462e386d3de597e970754764bf391a 100644
--- a/conf/base.config
+++ b/conf/base.config
@@ -54,4 +54,7 @@ process {
errorStrategy = 'retry'
maxRetries = 2
}
+ withName:CUSTOM_DUMPSOFTWAREVERSIONS {
+ cache = false
+ }
}
diff --git a/conf/modules.config b/conf/modules.config
index 0b1bfdec2050c7ba17ab39188d89ced8973ea549..a0506a4db4237df35536159506a9e5175efea415 100644
--- a/conf/modules.config
+++ b/conf/modules.config
@@ -1,32 +1,41 @@
/*
========================================================================================
- Config file for defining DSL2 per module options
+ Config file for defining DSL2 per module options and publishing paths
========================================================================================
Available keys to override module options:
- args = Additional arguments appended to command in module.
- args2 = Second set of arguments appended to command in module (multi-tool modules).
- args3 = Third set of arguments appended to command in module (multi-tool modules).
- publish_dir = Directory to publish results.
- publish_by_meta = Groovy list of keys available in meta map to append as directories to "publish_dir" path
- If publish_by_meta = true - Value of ${meta['id']} is appended as a directory to "publish_dir" path
- If publish_by_meta = ['id', 'custompath'] - If "id" is in meta map and "custompath" isn't then "${meta['id']}/custompath/"
- is appended as a directory to "publish_dir" path
- If publish_by_meta = false / null - No directories are appended to "publish_dir" path
- publish_files = Groovy map where key = "file_ext" and value = "directory" to publish results for that file extension
- The value of "directory" is appended to the standard "publish_dir" path as defined above.
- If publish_files = null (unspecified) - All files are published.
- If publish_files = false - No files are published.
- suffix = File name suffix for output files.
+ ext.args = Additional arguments appended to command in module.
+ ext.args2 = Second set of arguments appended to command in module (multi-tool modules).
+ ext.args3 = Third set of arguments appended to command in module (multi-tool modules).
+ ext.prefix = File name prefix for output files.
----------------------------------------------------------------------------------------
*/
-params {
- modules {
- 'fastqc' {
- args = "--quiet"
- }
- 'multiqc' {
- args = ""
- }
+process {
+
+ publishDir = [
+ path: { "${params.outdir}/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" },
+ mode: 'copy',
+ saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
+ ]
+
+ withName: SAMPLESHEET_CHECK {
+ publishDir = [
+ path: { "${params.outdir}/pipeline_info" },
+ mode: 'copy',
+ saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
+ ]
+ }
+
+ withName: FASTQC {
+ ext.args = '--quiet'
}
+
+ withName: CUSTOM_DUMPSOFTWAREVERSIONS {
+ publishDir = [
+ path: { "${params.outdir}/pipeline_info" },
+ mode: 'copy',
+ pattern: '*_versions.yml'
+ ]
+ }
+
}
diff --git a/conf/test.config b/conf/test.config
index d117e413a738899ff73b5ed900a52ced8c5986f9..5582c1d39a9b69f33b412499672e497c29f660dc 100644
--- a/conf/test.config
+++ b/conf/test.config
@@ -16,8 +16,8 @@ params {
// Limit resources so that this can run on GitHub Actions
max_cpus = 2
- max_memory = 6.GB
- max_time = 6.h
+ max_memory = '6.GB'
+ max_time = '6.h'
// Input data
// TODO nf-core: Specify the paths to your test data on nf-core/test-datasets
diff --git a/docs/images/nf-core-hic_logo.png b/docs/images/nf-core-hic_logo.png
deleted file mode 100644
index 274eb3dc3f3db879c7f3cbc3fd8f49a705a9a3fb..0000000000000000000000000000000000000000
Binary files a/docs/images/nf-core-hic_logo.png and /dev/null differ
diff --git a/docs/images/nf-core-hic_logo_dark.png b/docs/images/nf-core-hic_logo_dark.png
new file mode 100644
index 0000000000000000000000000000000000000000..dee3db82069b61e9a5380d940355f47ff565057c
--- /dev/null
+++ b/docs/images/nf-core-hic_logo_dark.png
@@ -0,0 +1,9 @@
+
+<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
+<HTML><HEAD>
+<TITLE> 508 Resource Limit Is Reached</TITLE>
+</HEAD><BODY>
+<H1>Resource Limit Is Reached</H1>
+The website is temporarily unable to service your request as it exceeded resource limit.
+Please try again later.
+</BODY></HTML>
diff --git a/docs/images/nf-core-hic_logo_light.png b/docs/images/nf-core-hic_logo_light.png
new file mode 100644
index 0000000000000000000000000000000000000000..1993002f28494136e88a278dbbfcb5170b757b3e
--- /dev/null
+++ b/docs/images/nf-core-hic_logo_light.png
@@ -0,0 +1,13 @@
+<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
+<html><head>
+<title>507 Insufficient Storage</title>
+</head><body>
+<h1>Insufficient Storage</h1>
+<p>The method could not be performed on the resource
+because the server is unable to store the
+representation needed to successfully complete the
+request. There is insufficient free space left in
+your storage allocation.</p>
+<p>Additionally, a 507 Insufficient Storage
+error was encountered while trying to use an ErrorDocument to handle the request.</p>
+</body></html>
diff --git a/docs/output.md b/docs/output.md
index e2d35a1b9c9b5bd3b6c2f7948a3cf5bb469e0c62..57dce62bc5c4bdd0fe36cdc8645f518879e70551 100644
--- a/docs/output.md
+++ b/docs/output.md
@@ -60,7 +60,7 @@ Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQ
* `pipeline_info/`
* Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`.
- * Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.tsv`.
+ * Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline.
* Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`.
</details>
diff --git a/docs/usage.md b/docs/usage.md
index 79b33ec2f74790ac27c2c883fe932dfa222f63e6..d06342842f37d6af3182c864d39ac0fabb3d291a 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -181,42 +181,6 @@ process {
> **NB:** We specify just the process name i.e. `STAR_ALIGN` in the config file and not the full task name string that is printed to screen in the error message or on the terminal whilst the pipeline is running i.e. `RNASEQ:ALIGN_STAR:STAR_ALIGN`. You may get a warning suggesting that the process selector isn't recognised but you can ignore that if the process name has been specified correctly. This is something that needs to be fixed upstream in core Nextflow.
-### Tool-specific options
-
-For the ultimate flexibility, we have implemented and are using Nextflow DSL2 modules in a way where it is possible for both developers and users to change tool-specific command-line arguments (e.g. providing an additional command-line argument to the `STAR_ALIGN` process) as well as publishing options (e.g. saving files produced by the `STAR_ALIGN` process that aren't saved by default by the pipeline). In the majority of instances, as a user you won't have to change the default options set by the pipeline developer(s), however, there may be edge cases where creating a simple custom config file can improve the behaviour of the pipeline if for example it is failing due to a weird error that requires setting a tool-specific parameter to deal with smaller / larger genomes.
-
-The command-line arguments passed to STAR in the `STAR_ALIGN` module are a combination of:
-
-* Mandatory arguments or those that need to be evaluated within the scope of the module, as supplied in the [`script`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/modules/nf-core/software/star/align/main.nf#L49-L55) section of the module file.
-
-* An [`options.args`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/modules/nf-core/software/star/align/main.nf#L56) string of non-mandatory parameters that is set to be empty by default in the module but can be overwritten when including the module in the sub-workflow / workflow context via the `addParams` Nextflow option.
-
-The nf-core/rnaseq pipeline has a sub-workflow (see [terminology](https://github.com/nf-core/modules#terminology)) specifically to align reads with STAR and to sort, index and generate some basic stats on the resulting BAM files using SAMtools. At the top of this file we import the `STAR_ALIGN` module via the Nextflow [`include`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/subworkflows/nf-core/align_star.nf#L10) keyword and by default the options passed to the module via the `addParams` option are set as an empty Groovy map [here](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/subworkflows/nf-core/align_star.nf#L5); this in turn means `options.args` will be set to empty by default in the module file too. This is an intentional design choice and allows us to implement well-written sub-workflows composed of a chain of tools that by default run with the bare minimum parameter set for any given tool in order to make it much easier to share across pipelines and to provide the flexibility for users and developers to customise any non-mandatory arguments.
-
-When including the sub-workflow above in the main pipeline workflow we use the same `include` statement, however, we now have the ability to overwrite options for each of the tools in the sub-workflow including the [`align_options`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/workflows/rnaseq.nf#L225) variable that will be used specifically to overwrite the optional arguments passed to the `STAR_ALIGN` module. In this case, the options to be provided to `STAR_ALIGN` have been assigned sensible defaults by the developer(s) in the pipeline's [`modules.config`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/conf/modules.config#L70-L74) and can be accessed and customised in the [workflow context](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/workflows/rnaseq.nf#L201-L204) too before eventually passing them to the sub-workflow as a Groovy map called `star_align_options`. These options will then be propagated from `workflow -> sub-workflow -> module`.
-
-As mentioned at the beginning of this section it may also be necessary for users to overwrite the options passed to modules to be able to customise specific aspects of the way in which a particular tool is executed by the pipeline. Given that all of the default module options are stored in the pipeline's `modules.config` as a [`params` variable](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/conf/modules.config#L24-L25) it is also possible to overwrite any of these options via a custom config file.
-
-Say for example we want to append an additional, non-mandatory parameter (i.e. `--outFilterMismatchNmax 16`) to the arguments passed to the `STAR_ALIGN` module. Firstly, we need to copy across the default `args` specified in the [`modules.config`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/conf/modules.config#L71) and create a custom config file that is a composite of the default `args` as well as the additional options you would like to provide. This is very important because Nextflow will overwrite the default value of `args` that you provide via the custom config.
-
-As you will see in the example below, we have:
-
-* appended `--outFilterMismatchNmax 16` to the default `args` used by the module.
-* changed the default `publish_dir` value to where the files will eventually be published in the main results directory.
-* appended `'bam':''` to the default value of `publish_files` so that the BAM files generated by the process will also be saved in the top-level results directory for the module. Note: `'out':'log'` means any file/directory ending in `out` will now be saved in a separate directory called `my_star_directory/log/`.
-
-```nextflow
-params {
- modules {
- 'star_align' {
- args = "--quantMode TranscriptomeSAM --twopassMode Basic --outSAMtype BAM Unsorted --readFilesCommand zcat --runRNGseed 0 --outFilterMultimapNmax 20 --alignSJDBoverhangMin 1 --outSAMattributes NH HI AS NM MD --quantTranscriptomeBan Singleend --outFilterMismatchNmax 16"
- publish_dir = "my_star_directory"
- publish_files = ['out':'log', 'tab':'log', 'bam':'']
- }
- }
-}
-```
-
### Updating containers
The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. If for some reason you need to use a different version of a particular tool with the pipeline then you just need to identify the `process` name and override the Nextflow `container` definition for that process using the `withName` declaration. For example, in the [nf-core/viralrecon](https://nf-co.re/viralrecon) pipeline a tool called [Pangolin](https://github.com/cov-lineages/pangolin) has been used during the COVID-19 pandemic to assign lineages to SARS-CoV-2 genome sequenced samples. Given that the lineage assignments change quite frequently it doesn't make sense to re-release the nf-core/viralrecon everytime a new version of Pangolin has been released. However, you can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via `-c custom.config`.
diff --git a/lib/NfcoreSchema.groovy b/lib/NfcoreSchema.groovy
index 8d6920dd645644e70e8bce260022e7e70be97788..40ab65f2050555d9b782150ac94c4be6459c62d7 100755
--- a/lib/NfcoreSchema.groovy
+++ b/lib/NfcoreSchema.groovy
@@ -105,9 +105,13 @@ class NfcoreSchema {
// Collect expected parameters from the schema
def expectedParams = []
+ def enums = [:]
for (group in schemaParams) {
for (p in group.value['properties']) {
expectedParams.push(p.key)
+ if (group.value['properties'][p.key].containsKey('enum')) {
+ enums[p.key] = group.value['properties'][p.key]['enum']
+ }
}
}
@@ -155,7 +159,7 @@ class NfcoreSchema {
println ''
log.error 'ERROR: Validation of pipeline parameters failed!'
JSONObject exceptionJSON = e.toJSON()
- printExceptions(exceptionJSON, params_json, log)
+ printExceptions(exceptionJSON, params_json, log, enums)
println ''
has_error = true
}
@@ -202,7 +206,7 @@ class NfcoreSchema {
}
def type = '[' + group_params.get(param).type + ']'
def description = group_params.get(param).description
- def defaultValue = group_params.get(param).default ? " [default: " + group_params.get(param).default.toString() + "]" : ''
+ def defaultValue = group_params.get(param).default != null ? " [default: " + group_params.get(param).default.toString() + "]" : ''
def description_default = description + colors.dim + defaultValue + colors.reset
// Wrap long description texts
// Loosely based on https://dzone.com/articles/groovy-plain-text-word-wrap
@@ -260,13 +264,12 @@ class NfcoreSchema {
// Get pipeline parameters defined in JSON Schema
def Map params_summary = [:]
- def blacklist = ['hostnames']
def params_map = paramsLoad(getSchemaPath(workflow, schema_filename=schema_filename))
for (group in params_map.keySet()) {
def sub_params = new LinkedHashMap()
def group_params = params_map.get(group) // This gets the parameters of that particular group
for (param in group_params.keySet()) {
- if (params.containsKey(param) && !blacklist.contains(param)) {
+ if (params.containsKey(param)) {
def params_value = params.get(param)
def schema_value = group_params.get(param).default
def param_type = group_params.get(param).type
@@ -330,7 +333,7 @@ class NfcoreSchema {
//
// Loop over nested exceptions and print the causingException
//
- private static void printExceptions(ex_json, params_json, log) {
+ private static void printExceptions(ex_json, params_json, log, enums, limit=5) {
def causingExceptions = ex_json['causingExceptions']
if (causingExceptions.length() == 0) {
def m = ex_json['message'] =~ /required key \[([^\]]+)\] not found/
@@ -346,11 +349,20 @@ class NfcoreSchema {
else {
def param = ex_json['pointerToViolation'] - ~/^#\//
def param_val = params_json[param].toString()
- log.error "* --${param}: ${ex_json['message']} (${param_val})"
+ if (enums.containsKey(param)) {
+ def error_msg = "* --${param}: '${param_val}' is not a valid choice (Available choices"
+ if (enums[param].size() > limit) {
+ log.error "${error_msg} (${limit} of ${enums[param].size()}): ${enums[param][0..limit-1].join(', ')}, ... )"
+ } else {
+ log.error "${error_msg}: ${enums[param].join(', ')})"
+ }
+ } else {
+ log.error "* --${param}: ${ex_json['message']} (${param_val})"
+ }
}
}
for (ex in causingExceptions) {
- printExceptions(ex, params_json, log)
+ printExceptions(ex, params_json, log, enums)
}
}
diff --git a/lib/NfcoreTemplate.groovy b/lib/NfcoreTemplate.groovy
index 44551e0a3521cfd9f02501d0a4a1db460dd1ff07..2fc0a9b9b61d85455653e2abbc91391f8b8606b0 100755
--- a/lib/NfcoreTemplate.groovy
+++ b/lib/NfcoreTemplate.groovy
@@ -19,27 +19,16 @@ class NfcoreTemplate {
}
//
- // Check params.hostnames
+ // Warn if a -profile or Nextflow config has not been provided to run the pipeline
//
- public static void hostName(workflow, params, log) {
- Map colors = logColours(params.monochrome_logs)
- if (params.hostnames) {
- try {
- def hostname = "hostname".execute().text.trim()
- params.hostnames.each { prof, hnames ->
- hnames.each { hname ->
- if (hostname.contains(hname) && !workflow.profile.contains(prof)) {
- log.info "=${colors.yellow}====================================================${colors.reset}=\n" +
- "${colors.yellow}WARN: You are running with `-profile $workflow.profile`\n" +
- " but your machine hostname is ${colors.white}'$hostname'${colors.reset}.\n" +
- " ${colors.yellow_bold}Please use `-profile $prof${colors.reset}`\n" +
- "=${colors.yellow}====================================================${colors.reset}="
- }
- }
- }
- } catch (Exception e) {
- log.warn "[$workflow.manifest.name] Could not determine 'hostname' - skipping check. Reason: ${e.message}."
- }
+ public static void checkConfigProvided(workflow, log) {
+ if (workflow.profile == 'standard' && workflow.configFiles.size() <= 1) {
+ log.warn "[$workflow.manifest.name] You are attempting to run the pipeline without any custom configuration!\n\n" +
+ "This will be dependent on your local compute environment but can be achieved via one or more of the following:\n" +
+ " (1) Using an existing pipeline profile e.g. `-profile docker` or `-profile singularity`\n" +
+ " (2) Using an existing nf-core/configs for your Institution e.g. `-profile crick` or `-profile uppmax`\n" +
+ " (3) Using your own local custom config e.g. `-c /path/to/your/custom.config`\n\n" +
+ "Please refer to the quick start section and usage docs for the pipeline.\n "
}
}
@@ -168,7 +157,6 @@ class NfcoreTemplate {
log.info "-${colors.purple}[$workflow.manifest.name]${colors.red} Pipeline completed successfully, but with errored process(es) ${colors.reset}-"
}
} else {
- hostName(workflow, params, log)
log.info "-${colors.purple}[$workflow.manifest.name]${colors.red} Pipeline completed with errors${colors.reset}-"
}
}
diff --git a/lib/Utils.groovy b/lib/Utils.groovy
index 18173e98503206c71e7cfc1615bfbfb6202c1198..1b88aec0ea09349d276643980366e2877c8713be 100755
--- a/lib/Utils.groovy
+++ b/lib/Utils.groovy
@@ -37,11 +37,4 @@ class Utils {
"==================================================================================="
}
}
-
- //
- // Join module args with appropriate spacing
- //
- public static String joinModuleArgs(args_list) {
- return ' ' + args_list.join(' ')
- }
}
diff --git a/lib/WorkflowMain.groovy b/lib/WorkflowMain.groovy
index 92eb1766e1d7763ccc40bcefc92b56a28544a4b1..a86a89134b77bc42cd0833ff7a8d2014b3da442f 100755
--- a/lib/WorkflowMain.groovy
+++ b/lib/WorkflowMain.groovy
@@ -61,6 +61,9 @@ class WorkflowMain {
// Print parameter summary log to screen
log.info paramsSummaryLog(workflow, params, log)
+ // Check that a -profile or Nextflow config has been provided to run the pipeline
+ NfcoreTemplate.checkConfigProvided(workflow, log)
+
// Check that conda channels are set-up correctly
if (params.enable_conda) {
Utils.checkCondaChannels(log)
@@ -69,9 +72,6 @@ class WorkflowMain {
// Check AWS batch settings
NfcoreTemplate.awsBatch(workflow, params)
- // Check the hostnames against configured profiles
- NfcoreTemplate.hostName(workflow, params, log)
-
// Check input has been provided
if (!params.input) {
log.error "Please provide an input samplesheet to the pipeline e.g. '--input samplesheet.csv'"
diff --git a/modules.json b/modules.json
index a68b1c1a6a7a5f41f235094a0055e1f1751c98de..2233ec4ec50f029bff981bc006a101a77d34b32c 100644
--- a/modules.json
+++ b/modules.json
@@ -3,12 +3,15 @@
"homePage": "https://github.com/nf-core/hic",
"repos": {
"nf-core/modules": {
+ "custom/dumpsoftwareversions": {
+ "git_sha": "20d8250d9f39ddb05dfb437603aaf99b5c0b2b41"
+ },
"fastqc": {
- "git_sha": "e937c7950af70930d1f34bb961403d9d2aa81c7d"
+ "git_sha": "9d0cad583b9a71a6509b754fdf589cbfbed08961"
},
"multiqc": {
- "git_sha": "e937c7950af70930d1f34bb961403d9d2aa81c7d"
+ "git_sha": "20d8250d9f39ddb05dfb437603aaf99b5c0b2b41"
}
}
}
-}
+}
\ No newline at end of file
diff --git a/modules/local/functions.nf b/modules/local/functions.nf
deleted file mode 100644
index da9da093d3f6025e328759a12adc2c1c9ede0d03..0000000000000000000000000000000000000000
--- a/modules/local/functions.nf
+++ /dev/null
@@ -1,68 +0,0 @@
-//
-// Utility functions used in nf-core DSL2 module files
-//
-
-//
-// Extract name of software tool from process name using $task.process
-//
-def getSoftwareName(task_process) {
- return task_process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()
-}
-
-//
-// Function to initialise default values and to generate a Groovy Map of available options for nf-core modules
-//
-def initOptions(Map args) {
- def Map options = [:]
- options.args = args.args ?: ''
- options.args2 = args.args2 ?: ''
- options.args3 = args.args3 ?: ''
- options.publish_by_meta = args.publish_by_meta ?: []
- options.publish_dir = args.publish_dir ?: ''
- options.publish_files = args.publish_files
- options.suffix = args.suffix ?: ''
- return options
-}
-
-//
-// Tidy up and join elements of a list to return a path string
-//
-def getPathFromList(path_list) {
- def paths = path_list.findAll { item -> !item?.trim().isEmpty() } // Remove empty entries
- paths = paths.collect { it.trim().replaceAll("^[/]+|[/]+\$", "") } // Trim whitespace and trailing slashes
- return paths.join('/')
-}
-
-//
-// Function to save/publish module results
-//
-def saveFiles(Map args) {
- if (!args.filename.endsWith('.version.txt')) {
- def ioptions = initOptions(args.options)
- def path_list = [ ioptions.publish_dir ?: args.publish_dir ]
- if (ioptions.publish_by_meta) {
- def key_list = ioptions.publish_by_meta instanceof List ? ioptions.publish_by_meta : args.publish_by_meta
- for (key in key_list) {
- if (args.meta && key instanceof String) {
- def path = key
- if (args.meta.containsKey(key)) {
- path = args.meta[key] instanceof Boolean ? "${key}_${args.meta[key]}".toString() : args.meta[key]
- }
- path = path instanceof String ? path : ''
- path_list.add(path)
- }
- }
- }
- if (ioptions.publish_files instanceof Map) {
- for (ext in ioptions.publish_files) {
- if (args.filename.endsWith(ext.key)) {
- def ext_list = path_list.collect()
- ext_list.add(ext.value)
- return "${getPathFromList(ext_list)}/$args.filename"
- }
- }
- } else if (ioptions.publish_files == null) {
- return "${getPathFromList(path_list)}/$args.filename"
- }
- }
-}
diff --git a/modules/local/get_software_versions.nf b/modules/local/get_software_versions.nf
deleted file mode 100644
index 9dc52498ede5820f4590cc1199e1fef34083ef65..0000000000000000000000000000000000000000
--- a/modules/local/get_software_versions.nf
+++ /dev/null
@@ -1,33 +0,0 @@
-// Import generic module functions
-include { saveFiles } from './functions'
-
-params.options = [:]
-
-process GET_SOFTWARE_VERSIONS {
- publishDir "${params.outdir}",
- mode: params.publish_dir_mode,
- saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:'pipeline_info', meta:[:], publish_by_meta:[]) }
-
- conda (params.enable_conda ? "conda-forge::python=3.8.3" : null)
- if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
- container "https://depot.galaxyproject.org/singularity/python:3.8.3"
- } else {
- container "quay.io/biocontainers/python:3.8.3"
- }
-
- cache false
-
- input:
- path versions
-
- output:
- path "software_versions.tsv" , emit: tsv
- path 'software_versions_mqc.yaml', emit: yaml
-
- script: // This script is bundled with the pipeline, in nf-core/hic/bin/
- """
- echo $workflow.manifest.version > pipeline.version.txt
- echo $workflow.nextflow.version > nextflow.version.txt
- scrape_software_versions.py &> software_versions_mqc.yaml
- """
-}
diff --git a/modules/local/samplesheet_check.nf b/modules/local/samplesheet_check.nf
index 760a42609b4490aeb7e3edf9186ed7930e9913c1..31f62fa99c98fdf3938dd5a360c5d4e522479704 100644
--- a/modules/local/samplesheet_check.nf
+++ b/modules/local/samplesheet_check.nf
@@ -1,31 +1,27 @@
-// Import generic module functions
-include { saveFiles } from './functions'
-
-params.options = [:]
-
process SAMPLESHEET_CHECK {
tag "$samplesheet"
- publishDir "${params.outdir}",
- mode: params.publish_dir_mode,
- saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:'pipeline_info', meta:[:], publish_by_meta:[]) }
conda (params.enable_conda ? "conda-forge::python=3.8.3" : null)
- if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
- container "https://depot.galaxyproject.org/singularity/python:3.8.3"
- } else {
- container "quay.io/biocontainers/python:3.8.3"
- }
+ container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
+ 'https://depot.galaxyproject.org/singularity/python:3.8.3' :
+ 'quay.io/biocontainers/python:3.8.3' }"
input:
path samplesheet
output:
- path '*.csv'
+ path '*.csv' , emit: csv
+ path "versions.yml", emit: versions
script: // This script is bundled with the pipeline, in nf-core/hic/bin/
"""
check_samplesheet.py \\
$samplesheet \\
samplesheet.valid.csv
+
+ cat <<-END_VERSIONS > versions.yml
+ "${task.process}":
+ python: \$(python --version | sed 's/Python //g')
+ END_VERSIONS
"""
}
diff --git a/modules/nf-core/modules/custom/dumpsoftwareversions/main.nf b/modules/nf-core/modules/custom/dumpsoftwareversions/main.nf
new file mode 100644
index 0000000000000000000000000000000000000000..934bb467238fdabfe168c24b0aec18e92bf20deb
--- /dev/null
+++ b/modules/nf-core/modules/custom/dumpsoftwareversions/main.nf
@@ -0,0 +1,21 @@
+process CUSTOM_DUMPSOFTWAREVERSIONS {
+ label 'process_low'
+
+ // Requires `pyyaml` which does not have a dedicated container but is in the MultiQC container
+ conda (params.enable_conda ? "bioconda::multiqc=1.11" : null)
+ container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
+ 'https://depot.galaxyproject.org/singularity/multiqc:1.11--pyhdfd78af_0' :
+ 'quay.io/biocontainers/multiqc:1.11--pyhdfd78af_0' }"
+
+ input:
+ path versions
+
+ output:
+ path "software_versions.yml" , emit: yml
+ path "software_versions_mqc.yml", emit: mqc_yml
+ path "versions.yml" , emit: versions
+
+ script:
+ def args = task.ext.args ?: ''
+ template 'dumpsoftwareversions.py'
+}
diff --git a/modules/nf-core/modules/custom/dumpsoftwareversions/meta.yml b/modules/nf-core/modules/custom/dumpsoftwareversions/meta.yml
new file mode 100644
index 0000000000000000000000000000000000000000..5b5b8a6026302a3241359f09409f59529622574e
--- /dev/null
+++ b/modules/nf-core/modules/custom/dumpsoftwareversions/meta.yml
@@ -0,0 +1,34 @@
+name: custom_dumpsoftwareversions
+description: Custom module used to dump software versions within the nf-core pipeline template
+keywords:
+ - custom
+ - version
+tools:
+ - custom:
+ description: Custom module used to dump software versions within the nf-core pipeline template
+ homepage: https://github.com/nf-core/tools
+ documentation: https://github.com/nf-core/tools
+ licence: ['MIT']
+input:
+ - versions:
+ type: file
+ description: YML file containing software versions
+ pattern: "*.yml"
+
+output:
+ - yml:
+ type: file
+ description: Standard YML file containing software versions
+ pattern: "software_versions.yml"
+ - mqc_yml:
+ type: file
+ description: MultiQC custom content YML file containing software versions
+ pattern: "software_versions_mqc.yml"
+ - versions:
+ type: file
+ description: File containing software versions
+ pattern: "versions.yml"
+
+authors:
+ - "@drpatelh"
+ - "@grst"
diff --git a/modules/nf-core/modules/custom/dumpsoftwareversions/templates/dumpsoftwareversions.py b/modules/nf-core/modules/custom/dumpsoftwareversions/templates/dumpsoftwareversions.py
new file mode 100644
index 0000000000000000000000000000000000000000..d13903925467e97e353f0a4e6bcf9f6cdb8a3664
--- /dev/null
+++ b/modules/nf-core/modules/custom/dumpsoftwareversions/templates/dumpsoftwareversions.py
@@ -0,0 +1,89 @@
+#!/usr/bin/env python
+
+import yaml
+import platform
+from textwrap import dedent
+
+
+def _make_versions_html(versions):
+ html = [
+ dedent(
+ """\\
+ <style>
+ #nf-core-versions tbody:nth-child(even) {
+ background-color: #f2f2f2;
+ }
+ </style>
+ <table class="table" style="width:100%" id="nf-core-versions">
+ <thead>
+ <tr>
+ <th> Process Name </th>
+ <th> Software </th>
+ <th> Version </th>
+ </tr>
+ </thead>
+ """
+ )
+ ]
+ for process, tmp_versions in sorted(versions.items()):
+ html.append("<tbody>")
+ for i, (tool, version) in enumerate(sorted(tmp_versions.items())):
+ html.append(
+ dedent(
+ f"""\\
+ <tr>
+ <td><samp>{process if (i == 0) else ''}</samp></td>
+ <td><samp>{tool}</samp></td>
+ <td><samp>{version}</samp></td>
+ </tr>
+ """
+ )
+ )
+ html.append("</tbody>")
+ html.append("</table>")
+ return "\\n".join(html)
+
+
+versions_this_module = {}
+versions_this_module["${task.process}"] = {
+ "python": platform.python_version(),
+ "yaml": yaml.__version__,
+}
+
+with open("$versions") as f:
+ versions_by_process = yaml.load(f, Loader=yaml.BaseLoader) | versions_this_module
+
+# aggregate versions by the module name (derived from fully-qualified process name)
+versions_by_module = {}
+for process, process_versions in versions_by_process.items():
+ module = process.split(":")[-1]
+ try:
+ assert versions_by_module[module] == process_versions, (
+ "We assume that software versions are the same between all modules. "
+ "If you see this error-message it means you discovered an edge-case "
+ "and should open an issue in nf-core/tools. "
+ )
+ except KeyError:
+ versions_by_module[module] = process_versions
+
+versions_by_module["Workflow"] = {
+ "Nextflow": "$workflow.nextflow.version",
+ "$workflow.manifest.name": "$workflow.manifest.version",
+}
+
+versions_mqc = {
+ "id": "software_versions",
+ "section_name": "${workflow.manifest.name} Software Versions",
+ "section_href": "https://github.com/${workflow.manifest.name}",
+ "plot_type": "html",
+ "description": "are collected at run time from the software output.",
+ "data": _make_versions_html(versions_by_module),
+}
+
+with open("software_versions.yml", "w") as f:
+ yaml.dump(versions_by_module, f, default_flow_style=False)
+with open("software_versions_mqc.yml", "w") as f:
+ yaml.dump(versions_mqc, f, default_flow_style=False)
+
+with open("versions.yml", "w") as f:
+ yaml.dump(versions_this_module, f, default_flow_style=False)
diff --git a/modules/nf-core/modules/fastqc/functions.nf b/modules/nf-core/modules/fastqc/functions.nf
deleted file mode 100644
index da9da093d3f6025e328759a12adc2c1c9ede0d03..0000000000000000000000000000000000000000
--- a/modules/nf-core/modules/fastqc/functions.nf
+++ /dev/null
@@ -1,68 +0,0 @@
-//
-// Utility functions used in nf-core DSL2 module files
-//
-
-//
-// Extract name of software tool from process name using $task.process
-//
-def getSoftwareName(task_process) {
- return task_process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()
-}
-
-//
-// Function to initialise default values and to generate a Groovy Map of available options for nf-core modules
-//
-def initOptions(Map args) {
- def Map options = [:]
- options.args = args.args ?: ''
- options.args2 = args.args2 ?: ''
- options.args3 = args.args3 ?: ''
- options.publish_by_meta = args.publish_by_meta ?: []
- options.publish_dir = args.publish_dir ?: ''
- options.publish_files = args.publish_files
- options.suffix = args.suffix ?: ''
- return options
-}
-
-//
-// Tidy up and join elements of a list to return a path string
-//
-def getPathFromList(path_list) {
- def paths = path_list.findAll { item -> !item?.trim().isEmpty() } // Remove empty entries
- paths = paths.collect { it.trim().replaceAll("^[/]+|[/]+\$", "") } // Trim whitespace and trailing slashes
- return paths.join('/')
-}
-
-//
-// Function to save/publish module results
-//
-def saveFiles(Map args) {
- if (!args.filename.endsWith('.version.txt')) {
- def ioptions = initOptions(args.options)
- def path_list = [ ioptions.publish_dir ?: args.publish_dir ]
- if (ioptions.publish_by_meta) {
- def key_list = ioptions.publish_by_meta instanceof List ? ioptions.publish_by_meta : args.publish_by_meta
- for (key in key_list) {
- if (args.meta && key instanceof String) {
- def path = key
- if (args.meta.containsKey(key)) {
- path = args.meta[key] instanceof Boolean ? "${key}_${args.meta[key]}".toString() : args.meta[key]
- }
- path = path instanceof String ? path : ''
- path_list.add(path)
- }
- }
- }
- if (ioptions.publish_files instanceof Map) {
- for (ext in ioptions.publish_files) {
- if (args.filename.endsWith(ext.key)) {
- def ext_list = path_list.collect()
- ext_list.add(ext.value)
- return "${getPathFromList(ext_list)}/$args.filename"
- }
- }
- } else if (ioptions.publish_files == null) {
- return "${getPathFromList(path_list)}/$args.filename"
- }
- }
-}
diff --git a/modules/nf-core/modules/fastqc/main.nf b/modules/nf-core/modules/fastqc/main.nf
index 39c327b261f7a590f48f64b6dd0eaf24f44ef926..d250eca0758673a560fbcee6b4636fc783e05b57 100644
--- a/modules/nf-core/modules/fastqc/main.nf
+++ b/modules/nf-core/modules/fastqc/main.nf
@@ -1,22 +1,11 @@
-// Import generic module functions
-include { initOptions; saveFiles; getSoftwareName } from './functions'
-
-params.options = [:]
-options = initOptions(params.options)
-
process FASTQC {
tag "$meta.id"
label 'process_medium'
- publishDir "${params.outdir}",
- mode: params.publish_dir_mode,
- saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:getSoftwareName(task.process), meta:meta, publish_by_meta:['id']) }
conda (params.enable_conda ? "bioconda::fastqc=0.11.9" : null)
- if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
- container "https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0"
- } else {
- container "quay.io/biocontainers/fastqc:0.11.9--0"
- }
+ container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
+ 'https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0' :
+ 'quay.io/biocontainers/fastqc:0.11.9--0' }"
input:
tuple val(meta), path(reads)
@@ -24,24 +13,32 @@ process FASTQC {
output:
tuple val(meta), path("*.html"), emit: html
tuple val(meta), path("*.zip") , emit: zip
- path "*.version.txt" , emit: version
+ path "versions.yml" , emit: versions
script:
+ def args = task.ext.args ?: ''
// Add soft-links to original FastQs for consistent naming in pipeline
- def software = getSoftwareName(task.process)
- def prefix = options.suffix ? "${meta.id}${options.suffix}" : "${meta.id}"
+ def prefix = task.ext.prefix ?: "${meta.id}"
if (meta.single_end) {
"""
[ ! -f ${prefix}.fastq.gz ] && ln -s $reads ${prefix}.fastq.gz
- fastqc $options.args --threads $task.cpus ${prefix}.fastq.gz
- fastqc --version | sed -e "s/FastQC v//g" > ${software}.version.txt
+ fastqc $args --threads $task.cpus ${prefix}.fastq.gz
+
+ cat <<-END_VERSIONS > versions.yml
+ "${task.process}":
+ fastqc: \$( fastqc --version | sed -e "s/FastQC v//g" )
+ END_VERSIONS
"""
} else {
"""
[ ! -f ${prefix}_1.fastq.gz ] && ln -s ${reads[0]} ${prefix}_1.fastq.gz
[ ! -f ${prefix}_2.fastq.gz ] && ln -s ${reads[1]} ${prefix}_2.fastq.gz
- fastqc $options.args --threads $task.cpus ${prefix}_1.fastq.gz ${prefix}_2.fastq.gz
- fastqc --version | sed -e "s/FastQC v//g" > ${software}.version.txt
+ fastqc $args --threads $task.cpus ${prefix}_1.fastq.gz ${prefix}_2.fastq.gz
+
+ cat <<-END_VERSIONS > versions.yml
+ "${task.process}":
+ fastqc: \$( fastqc --version | sed -e "s/FastQC v//g" )
+ END_VERSIONS
"""
}
}
diff --git a/modules/nf-core/modules/fastqc/meta.yml b/modules/nf-core/modules/fastqc/meta.yml
index 8eb9953dce50e37c6ead461861bc39a1436308f2..b09553a3c3ce8519d6586542204408b8055043b0 100644
--- a/modules/nf-core/modules/fastqc/meta.yml
+++ b/modules/nf-core/modules/fastqc/meta.yml
@@ -15,6 +15,7 @@ tools:
overrepresented sequences.
homepage: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/
documentation: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/
+ licence: ['GPL-2.0-only']
input:
- meta:
type: map
@@ -40,10 +41,10 @@ output:
type: file
description: FastQC report archive
pattern: "*_{fastqc.zip}"
- - version:
+ - versions:
type: file
- description: File containing software version
- pattern: "*.{version.txt}"
+ description: File containing software versions
+ pattern: "versions.yml"
authors:
- "@drpatelh"
- "@grst"
diff --git a/modules/nf-core/modules/multiqc/functions.nf b/modules/nf-core/modules/multiqc/functions.nf
deleted file mode 100644
index da9da093d3f6025e328759a12adc2c1c9ede0d03..0000000000000000000000000000000000000000
--- a/modules/nf-core/modules/multiqc/functions.nf
+++ /dev/null
@@ -1,68 +0,0 @@
-//
-// Utility functions used in nf-core DSL2 module files
-//
-
-//
-// Extract name of software tool from process name using $task.process
-//
-def getSoftwareName(task_process) {
- return task_process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()
-}
-
-//
-// Function to initialise default values and to generate a Groovy Map of available options for nf-core modules
-//
-def initOptions(Map args) {
- def Map options = [:]
- options.args = args.args ?: ''
- options.args2 = args.args2 ?: ''
- options.args3 = args.args3 ?: ''
- options.publish_by_meta = args.publish_by_meta ?: []
- options.publish_dir = args.publish_dir ?: ''
- options.publish_files = args.publish_files
- options.suffix = args.suffix ?: ''
- return options
-}
-
-//
-// Tidy up and join elements of a list to return a path string
-//
-def getPathFromList(path_list) {
- def paths = path_list.findAll { item -> !item?.trim().isEmpty() } // Remove empty entries
- paths = paths.collect { it.trim().replaceAll("^[/]+|[/]+\$", "") } // Trim whitespace and trailing slashes
- return paths.join('/')
-}
-
-//
-// Function to save/publish module results
-//
-def saveFiles(Map args) {
- if (!args.filename.endsWith('.version.txt')) {
- def ioptions = initOptions(args.options)
- def path_list = [ ioptions.publish_dir ?: args.publish_dir ]
- if (ioptions.publish_by_meta) {
- def key_list = ioptions.publish_by_meta instanceof List ? ioptions.publish_by_meta : args.publish_by_meta
- for (key in key_list) {
- if (args.meta && key instanceof String) {
- def path = key
- if (args.meta.containsKey(key)) {
- path = args.meta[key] instanceof Boolean ? "${key}_${args.meta[key]}".toString() : args.meta[key]
- }
- path = path instanceof String ? path : ''
- path_list.add(path)
- }
- }
- }
- if (ioptions.publish_files instanceof Map) {
- for (ext in ioptions.publish_files) {
- if (args.filename.endsWith(ext.key)) {
- def ext_list = path_list.collect()
- ext_list.add(ext.value)
- return "${getPathFromList(ext_list)}/$args.filename"
- }
- }
- } else if (ioptions.publish_files == null) {
- return "${getPathFromList(path_list)}/$args.filename"
- }
- }
-}
diff --git a/modules/nf-core/modules/multiqc/main.nf b/modules/nf-core/modules/multiqc/main.nf
index da78080024721aee1d944f954150fa9352352aca..3dceb162a38e3763398f1a80efb24aaf12b829c0 100644
--- a/modules/nf-core/modules/multiqc/main.nf
+++ b/modules/nf-core/modules/multiqc/main.nf
@@ -1,21 +1,10 @@
-// Import generic module functions
-include { initOptions; saveFiles; getSoftwareName } from './functions'
-
-params.options = [:]
-options = initOptions(params.options)
-
process MULTIQC {
label 'process_medium'
- publishDir "${params.outdir}",
- mode: params.publish_dir_mode,
- saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:getSoftwareName(task.process), meta:[:], publish_by_meta:[]) }
- conda (params.enable_conda ? "bioconda::multiqc=1.10.1" : null)
- if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
- container "https://depot.galaxyproject.org/singularity/multiqc:1.10.1--py_0"
- } else {
- container "quay.io/biocontainers/multiqc:1.10.1--py_0"
- }
+ conda (params.enable_conda ? 'bioconda::multiqc=1.11' : null)
+ container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
+ 'https://depot.galaxyproject.org/singularity/multiqc:1.11--pyhdfd78af_0' :
+ 'quay.io/biocontainers/multiqc:1.11--pyhdfd78af_0' }"
input:
path multiqc_files
@@ -24,12 +13,16 @@ process MULTIQC {
path "*multiqc_report.html", emit: report
path "*_data" , emit: data
path "*_plots" , optional:true, emit: plots
- path "*.version.txt" , emit: version
+ path "versions.yml" , emit: versions
script:
- def software = getSoftwareName(task.process)
+ def args = task.ext.args ?: ''
"""
- multiqc -f $options.args .
- multiqc --version | sed -e "s/multiqc, version //g" > ${software}.version.txt
+ multiqc -f $args .
+
+ cat <<-END_VERSIONS > versions.yml
+ "${task.process}":
+ multiqc: \$( multiqc --version | sed -e "s/multiqc, version //g" )
+ END_VERSIONS
"""
}
diff --git a/modules/nf-core/modules/multiqc/meta.yml b/modules/nf-core/modules/multiqc/meta.yml
index 532a8bb1eff7f38e4f38dc1e36f2ce1f6c6657d1..63c75a450a47bfe3b26ea516a1fa44f955e99f77 100644
--- a/modules/nf-core/modules/multiqc/meta.yml
+++ b/modules/nf-core/modules/multiqc/meta.yml
@@ -11,6 +11,7 @@ tools:
It's a general use tool, perfect for summarising the output from numerous bioinformatics tools.
homepage: https://multiqc.info/
documentation: https://multiqc.info/docs/
+ licence: ['GPL-3.0-or-later']
input:
- multiqc_files:
type: file
@@ -29,10 +30,10 @@ output:
type: file
description: Plots created by MultiQC
pattern: "*_data"
- - version:
+ - versions:
type: file
- description: File containing software version
- pattern: "*.{version.txt}"
+ description: File containing software versions
+ pattern: "versions.yml"
authors:
- "@abhi18av"
- "@bunop"
diff --git a/nextflow.config b/nextflow.config
index 8c5d7ac6beffe20a5979f989d71d6737a5c87c55..d3ba2463c6e5f4f4e16233e6ac3299281c29ccfc 100644
--- a/nextflow.config
+++ b/nextflow.config
@@ -26,7 +26,6 @@ params {
// Boilerplate options
outdir = './results'
tracedir = "${params.outdir}/pipeline_info"
- publish_dir_mode = 'copy'
email = null
email_on_fail = null
plaintext_email = false
@@ -34,14 +33,12 @@ params {
help = false
validate_params = true
show_hidden_params = false
- schema_ignore_params = 'genomes,modules'
+ schema_ignore_params = 'genomes'
enable_conda = false
- singularity_pull_docker_container = false
// Config options
custom_config_version = 'master'
custom_config_base = "https://raw.githubusercontent.com/nf-core/configs/${params.custom_config_version}"
- hostnames = [:]
config_profile_description = null
config_profile_contact = null
config_profile_url = null
@@ -58,9 +55,6 @@ params {
// Load base.config by default for all pipelines
includeConfig 'conf/base.config'
-// Load modules.config for DSL2 module specific options
-includeConfig 'conf/modules.config'
-
// Load nf-core custom profiles from different Institutions
try {
includeConfig "${params.custom_config_base}/nfcore_custom.config"
@@ -68,13 +62,6 @@ try {
System.err.println("WARNING: Could not load nf-core/config profiles: ${params.custom_config_base}/nfcore_custom.config")
}
-// Load igenomes.config if required
-if (!params.igenomes_ignore) {
- includeConfig 'conf/igenomes.config'
-} else {
- params.genomes = [:]
-}
-
profiles {
debug { process.beforeScript = 'echo $HOSTNAME' }
conda {
@@ -126,11 +113,22 @@ profiles {
test_full { includeConfig 'conf/test_full.config' }
}
+// Load igenomes.config if required
+if (!params.igenomes_ignore) {
+ includeConfig 'conf/igenomes.config'
+} else {
+ params.genomes = [:]
+}
+
// Export these variables to prevent local Python/R libraries from conflicting with those in the container
+// The JULIA depot path has been adjusted to a fixed path `/usr/local/share/julia` that needs to be used for packages in the container.
+// See https://apeltzer.github.io/post/03-julia-lang-nextflow/ for details on that. Once we have a common agreement on where to keep Julia packages, this is adjustable.
+
env {
PYTHONNOUSERSITE = 1
R_PROFILE_USER = "/.Rprofile"
R_ENVIRON_USER = "/.Renviron"
+ JULIA_DEPOT_PATH = "/usr/local/share/julia"
}
// Capture exit codes from upstream processes when piping
@@ -160,10 +158,13 @@ manifest {
homePage = 'https://github.com/nf-core/hic'
description = 'Analysis of Chromosome Conformation Capture data (Hi-C)'
mainScript = 'main.nf'
- nextflowVersion = '!>=21.04.0'
- version = '1.3.0'
+ nextflowVersion = '!>=21.10.3'
+ version = '1.3.1'
}
+// Load modules.config for DSL2 module specific options
+includeConfig 'conf/modules.config'
+
// Function to ensure that resource requirements don't go beyond
// a maximum limit
def check_max(obj, type) {
diff --git a/nextflow_schema.json b/nextflow_schema.json
index f8154bdd36c9f9269222395f1cbd70b83c140cff..e3bebcefce4238fb5106f411c2abc3e8882f48a4 100644
--- a/nextflow_schema.json
+++ b/nextflow_schema.json
@@ -104,12 +104,6 @@
"help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.",
"fa_icon": "fas fa-users-cog"
},
- "hostnames": {
- "type": "string",
- "description": "Institutional configs hostname.",
- "hidden": true,
- "fa_icon": "fas fa-users-cog"
- },
"config_profile_name": {
"type": "string",
"description": "Institutional config name.",
@@ -184,22 +178,6 @@
"fa_icon": "fas fa-question-circle",
"hidden": true
},
- "publish_dir_mode": {
- "type": "string",
- "default": "copy",
- "description": "Method used to save pipeline results to output directory.",
- "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.",
- "fa_icon": "fas fa-copy",
- "enum": [
- "symlink",
- "rellink",
- "link",
- "copy",
- "copyNoFollow",
- "move"
- ],
- "hidden": true
- },
"email_on_fail": {
"type": "string",
"description": "Email address for completion summary, only when pipeline fails.",
@@ -260,13 +238,6 @@
"description": "Run this workflow with Conda. You can also use '-profile conda' instead of providing this parameter.",
"hidden": true,
"fa_icon": "fas fa-bacon"
- },
- "singularity_pull_docker_container": {
- "type": "boolean",
- "description": "Instead of directly downloading Singularity images for use with Singularity, force the workflow to pull and convert Docker containers instead.",
- "hidden": true,
- "fa_icon": "fas fa-toolbox",
- "help_text": "This may be useful for example if you are unable to directly pull Singularity containers to run the pipeline due to http/https proxy issues."
}
}
}
diff --git a/subworkflows/local/input_check.nf b/subworkflows/local/input_check.nf
index b664bc8caf10c1ee8b810f3108e0eedefc6b398b..cddcbb3ce0c909c6e951784ebf44c260b3389a7b 100644
--- a/subworkflows/local/input_check.nf
+++ b/subworkflows/local/input_check.nf
@@ -2,9 +2,7 @@
// Check input samplesheet and get read channels
//
-params.options = [:]
-
-include { SAMPLESHEET_CHECK } from '../../modules/local/samplesheet_check' addParams( options: params.options )
+include { SAMPLESHEET_CHECK } from '../../modules/local/samplesheet_check'
workflow INPUT_CHECK {
take:
@@ -12,12 +10,14 @@ workflow INPUT_CHECK {
main:
SAMPLESHEET_CHECK ( samplesheet )
+ .csv
.splitCsv ( header:true, sep:',' )
.map { create_fastq_channels(it) }
.set { reads }
emit:
- reads // channel: [ val(meta), [ reads ] ]
+ reads // channel: [ val(meta), [ reads ] ]
+ versions = SAMPLESHEET_CHECK.out.versions // channel: [ versions.yml ]
}
// Function to get list of [ meta, [ fastq_1, fastq_2 ] ]
diff --git a/workflows/hic.nf b/workflows/hic.nf
index bf53d9a547e27c726413efaf5fa51568ca6abcce..f29d026f221e3657b49a84dba3a7e6f9ee95b931 100644
--- a/workflows/hic.nf
+++ b/workflows/hic.nf
@@ -32,18 +32,10 @@ ch_multiqc_custom_config = params.multiqc_config ? Channel.fromPath(params.multi
========================================================================================
*/
-// Don't overwrite global params.modules, create a copy instead and use that within the main script.
-def modules = params.modules.clone()
-
-//
-// MODULE: Local to the pipeline
-//
-include { GET_SOFTWARE_VERSIONS } from '../modules/local/get_software_versions' addParams( options: [publish_files : ['tsv':'']] )
-
//
// SUBWORKFLOW: Consisting of a mix of local and nf-core/modules
//
-include { INPUT_CHECK } from '../subworkflows/local/input_check' addParams( options: [:] )
+include { INPUT_CHECK } from '../subworkflows/local/input_check'
/*
========================================================================================
@@ -51,14 +43,12 @@ include { INPUT_CHECK } from '../subworkflows/local/input_check' addParams( opti
========================================================================================
*/
-def multiqc_options = modules['multiqc']
-multiqc_options.args += params.multiqc_title ? Utils.joinModuleArgs(["--title \"$params.multiqc_title\""]) : ''
-
//
// MODULE: Installed directly from nf-core/modules
//
-include { FASTQC } from '../modules/nf-core/modules/fastqc/main' addParams( options: modules['fastqc'] )
-include { MULTIQC } from '../modules/nf-core/modules/multiqc/main' addParams( options: multiqc_options )
+include { FASTQC } from '../modules/nf-core/modules/fastqc/main'
+include { MULTIQC } from '../modules/nf-core/modules/multiqc/main'
+include { CUSTOM_DUMPSOFTWAREVERSIONS } from '../modules/nf-core/modules/custom/dumpsoftwareversions/main'
/*
========================================================================================
@@ -71,7 +61,7 @@ def multiqc_report = []
workflow HIC {
- ch_software_versions = Channel.empty()
+ ch_versions = Channel.empty()
//
// SUBWORKFLOW: Read in samplesheet, validate and stage input files
@@ -79,6 +69,7 @@ workflow HIC {
INPUT_CHECK (
ch_input
)
+ ch_versions = ch_versions.mix(INPUT_CHECK.out.versions)
//
// MODULE: Run FastQC
@@ -86,21 +77,10 @@ workflow HIC {
FASTQC (
INPUT_CHECK.out.reads
)
- ch_software_versions = ch_software_versions.mix(FASTQC.out.version.first().ifEmpty(null))
+ ch_versions = ch_versions.mix(FASTQC.out.versions.first())
- //
- // MODULE: Pipeline reporting
- //
- ch_software_versions
- .map { it -> if (it) [ it.baseName, it ] }
- .groupTuple()
- .map { it[1][0] }
- .flatten()
- .collect()
- .set { ch_software_versions }
-
- GET_SOFTWARE_VERSIONS (
- ch_software_versions.map { it }.collect()
+ CUSTOM_DUMPSOFTWAREVERSIONS (
+ ch_versions.unique().collectFile(name: 'collated_versions.yml')
)
//
@@ -113,14 +93,14 @@ workflow HIC {
ch_multiqc_files = ch_multiqc_files.mix(Channel.from(ch_multiqc_config))
ch_multiqc_files = ch_multiqc_files.mix(ch_multiqc_custom_config.collect().ifEmpty([]))
ch_multiqc_files = ch_multiqc_files.mix(ch_workflow_summary.collectFile(name: 'workflow_summary_mqc.yaml'))
- ch_multiqc_files = ch_multiqc_files.mix(GET_SOFTWARE_VERSIONS.out.yaml.collect())
+ ch_multiqc_files = ch_multiqc_files.mix(CUSTOM_DUMPSOFTWAREVERSIONS.out.mqc_yml.collect())
ch_multiqc_files = ch_multiqc_files.mix(FASTQC.out.zip.collect{it[1]}.ifEmpty([]))
MULTIQC (
ch_multiqc_files.collect()
)
- multiqc_report = MULTIQC.out.report.toList()
- ch_software_versions = ch_software_versions.mix(MULTIQC.out.version.ifEmpty(null))
+ multiqc_report = MULTIQC.out.report.toList()
+ ch_versions = ch_versions.mix(MULTIQC.out.versions)
}
/*