GATK no longer works when converting charliecloud image from `directory` to `squashFS` format
Context
I'm running a nextflow pipeline that includes a step using gatk, so use charliecloud container from gatk
(Note I've used charliecloud images for many other tools, and works fine)
Steps
- Downloaded gatk image from dockerhub as
ch-image pull -s /Xnfs/abc/charliecloud broadinstitute/gatk:4.3.0.0
- Convert to squashfs format (and create /scratch and /Xnfs dirs) using
/Xnfs/abc/nextflow_bin/update_ch_image.sh
- Run a specific
gatk
command (/gatk/gatk MarkDuplicates -I input.bam -O 4000_JU2817_to_H2_TOL_final_bowtie2.bam -M 4000_JU2817_to_H2_TOL_final_bowtie2_dup_metrics.txt --TMP_DIR /scratch/Bio/bletcher/tmp --CREATE_INDEX
)
Issue
Step 3 fine when ran directly on a compute node (i.e. in nextflow local
exec mode)
But when ran using slurm
executor mode, fails with:
Error executing process > 'mark_duplicates (4)'
Caused by:
Process `mark_duplicates (4)` terminated with an error exit status (247)
Command executed:
/gatk/gatk MarkDuplicates -I input.bam -O 4000_JU2817_to_H2_TOL_final_bowtie2.bam -M 4000_JU2817_to_H2_TOL_final_bowtie2_dup_metrics.txt --TMP_DIR /scratch/Bio/bletcher/tmp --CREATE_INDEX
Command exit status:
247
Command output:
(empty)
Command error:
17:37:51.663 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.3.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
[Wed Feb 08 17:37:51 GMT 2023] MarkDuplicates --INPUT input.bam --OUTPUT 4000_JU2817_to_H2_TOL_final_bowtie2.bam --METRICS_FILE 4000_JU2817_to_H2_TOL_final_bowtie2_dup_metrics.txt --TMP_DIR /scratch/Bio/bletcher/tmp --CREATE_INDEX true --MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP 50000 --MAX_FILE_HANDLES_FOR_READ_ENDS_MAP 8000 --SORTING_COLLECTION_SIZE_RATIO 0.25 --TAG_DUPLICATE_SET_MEMBERS false --REMOVE_SEQUENCING_DUPLICATES false --TAGGING_POLICY DontTag --CLEAR_DT true --DUPLEX_UMI false --FLOW_MODE false --FLOW_QUALITY_SUM_STRATEGY false --USE_END_IN_UNPAIRED_READS false --USE_UNPAIRED_CLIPPED_END false --UNPAIRED_END_UNCERTAINTY 0 --FLOW_SKIP_FIRST_N_FLOWS 0 --FLOW_Q_IS_KNOWN_END false --FLOW_EFFECTIVE_QUALITY_THRESHOLD 15 --ADD_PG_TAG_TO_READS true --REMOVE_DUPLICATES false --ASSUME_SORTED false --DUPLICATE_SCORING_STRATEGY SUM_OF_BASE_QUALITIES --PROGRAM_RECORD_ID MarkDuplicates --PROGRAM_GROUP_NAME MarkDuplicates --READ_NAME_REGEX <optimized capture of last three ':' separated fields as numeric values> --OPTICAL_DUPLICATE_PIXEL_DISTANCE 100 --MAX_OPTICAL_DUPLICATE_SET_SIZE 300000 --VERBOSITY INFO --QUIET false --VALIDATION_STRINGENCY STRICT --COMPRESSION_LEVEL 2 --MAX_RECORDS_IN_RAM 500000 --CREATE_MD5_FILE false --GA4GH_CLIENT_SECRETS client_secrets.json --help false --version false --showHidden false --USE_JDK_DEFLATER false --USE_JDK_INFLATER false
[Wed Feb 08 17:37:52 GMT 2023] Executing as ?@c6420node001 on Linux 5.10.0-19-amd64 amd64; OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: Version:4.3.0.0
INFOja2023-02-08 17:37:52nc_ioMarkDuplicates=fStart of doWork freeMemory: 2021975088; totalMemory: 2076049408; maxMemory: 31136546816
INFOJU2023-02-08 17:37:52l_bowMarkDuplicates00Reading input file and constructing read end information.
INFOer2023-02-08 17:37:52haredMarkDuplicatesblWill retain up to 112813575 data points before spilling to disk.
INFO 2023-02-08 17:37:54 MarkDuplicates Read 1502 records. 0 pairs never matched.
INFOir2023-02-08 17:37:55 MarkDuplicates After buildSortedReadEndLists freeMemory: 1278427360; totalMemory: 2201161728; maxMemory: 31136546816
INFOat2023-02-08 17:37:55tflowMarkDuplicates3aWill retain up to 973017088 duplicate indices before spilling to disk.
Using GATK jar /gatk/gatk-package-4.3.0.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /gatk/gatk-package-4.3.0.0-local.jar MarkDuplicates -I input.bam -O 4000_JU2817_to_H2_TOL_final_bowtie2.bam -M 4000_JU2817_to_H2_TOL_final_bowtie2_dup_metrics.txt --TMP_DIR /scratch/Bio/bletcher/tmp --CREATE_INDEX
ps: error while loading shared libraries: liblzma.so.5: cannot open shared object file: Error 107
Work dir:
/scratch/Bio/bletcher/nextflow_workdir/5d/153adfc1d4b8f8b5c7e64747e3c3c0
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
Further, when running the same workflow, in slurm
exec mode, but using a non-squashed image instead, it completes fine (very weird??)
Images are here: /Xnfs/abc/charliecloud/img/broadinstitute%gatk+4.3.0.0
and /Xnfs/abc/charliecloud/img/broadinstitute%gatk+4.3.0.0_squashed
Ever seen this @lmodolo ? Anything that ch-convert
is doing that might somehow explain this, like removing liblzma.so.5
in the image?