Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
1 result
CLUET David's avatar
Add Marine comments
dcluet authored
600343e4
History
CLUET David's avatar 600343e4
History
Name Last commit Last update
src
.gitignore
README.md
README.md

IJ_docker

This repository allows the creation of Ubuntu:20.04 based docker virtual machine encapsuling your version of ImageJ/FIJI. This virtual machine allows mutiple parallel analyses without graphical environment such as Virtual Machine (VM) available on High Performance Clusters (HPC) like IFB.

Table of contents

Overview

In order to display properly the mermaid graphs of this file please use Visual Studio Code with the Markdown Preview Mermaid Support Extension. Then type in: SHIFT + CTRL + V

Once installed, you will end up with a single directory into your home, containing all key sripts and files: ~/Input_Image_Analysis

Once the installation is done this is the only folder and files you have to deal with.

graph TD
    subgraph Legend
        bash[Bash]
        docker[Docker]
        python[Python]
        imagej[ImageJ]
        files[Files]
        folders[Folders]
    end

    subgraph macros
        demo[demo.java]
        empty[empty.java]
    end 

    subgraph IO[Input_Image_Analysis]
        run_parallel_ij[run_parallel_ij.sh]
        config_HPC[config_HPC.sh]
        transfer_to_HPC[transfer_to_HPC.sh]
        harvest_from_HPC[harvest_from_HPC.sh]
        macros
        config[parallel_ij.yml]
        parallel_ij_1[parallel_ij_1.tar]
    end

    style bash fill: #ccffcc
    style run_parallel_ij fill: #ccffcc
    style config_HPC fill: #ccffcc
    style transfer_to_HPC fill: #ccffcc
    
    style docker fill: #d6f5f5
    
    style python fill: #ffcc00
    
    style imagej fill: #ff9900
    style demo fill: #ff9900
    style empty fill: #ff9900
    
    style files fill: #c4c3d0
    style config fill:  #c4c3d0

    style folders fill: #e0e0e6
    style macros fill: #e0e0e6
    style IO fill: #e0e0e6

Here is a brief description of the files and their purpose

File Type Description
run_parallel_ij.sh bash script Run the parallel_ij docker with auto mounting of ~/Input_Image_Analysis inside of it.
config_HPC.sh bash script Configures and install dependencies on a distant VM.
transfer_to_HPC bash script Enables to easily transfer your docker and files to a distant VM.
harvest_from_HPC.sh bash script Enables to transfert output generated from a distant VM to your local machinee.
demo.java ImageJ macro Convert an RGB image into a 8-bit one.
test.java ImageJ macro An empty macro only able to read arguments and write into a log file, ready to receive your own code.
parallel_ij.yml yaml file Configuration file to read by the docker Python 3.8 engine.
parallel_ij_1.tar image file This image file of your docker will be used to be depoyed on VM or other machines.

Example of usage

First we need to create an input directory (test in this example) within the ~/Input_Image_Analysis folder. You can do it classically with your explorer. Store 4 images in this test folder (Cell1.jpg, ..., Cell4.jpg)

The whole analysis will be controled by the parameters set in the configuration parallel_ij.yml file that can be modify with any text editor (be sure to preserve the .yml extension):

# Name of the macro to use (in the ~/Input_Image_Analysis/macros directory)
macro: demo.java
# Name of the input folder within the ~/Input_Image_Analysis directory
input directory: test
# Image format
extension: .jpg
# Name of the output folder to be created (automatic UID or custom name)
output directory: auto
# Number of parallel jobs that will be executed
parallel processes: 4

With this specific configuration the analysis will be performed using the demo.java (present in the ~/Input_Image_Analysis/macros directory created, when the ij_docker image has been build), on all .jpg images contained in the test directory (that has to be in the ~/Input_Image_Analysis directory), and 4 images will be analyzed at the same time.

The output directory is set on auto meaning it will be named with the input directory and Unique ID (UID) combining date, time and the macro name.

Now the commandlines... Open a terminal (CTRL + T on linux) and type the following commands:

cd ~/Input_Image_Analysis
bash run_parallel_ij.sh

You should obtain a similar output:

--------------------------------------------------
Starting parallel_IJ_docker!
--------------------------------------------------

2023/03/24 11:18:19 PYTHON Initiating demo.java analysis
2023/03/24 11:18:23 PYTHON Setting ImageJ memory to 6666mB
Cell_3.jpg
Cell_4.jpg
Cell_2.jpg
Cell_1.jpg
2023/03/24 11:18:23 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_3.jpg*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo.log*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo
2023/03/24 11:18:23 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_4.jpg*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo.log*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo
2023/03/24 11:18:23 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_2.jpg*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo.log*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo
2023/03/24 11:18:23 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_1.jpg*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo.log*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo
OpenJDK 64-Bit Server VM warning: OpenJDK 64-Bit Server VM warning: OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0ignoring option PermSize=128m; support was removed in 8.0ignoring option PermSize=128m; support was removed in 8.0


OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
2023/03/24 11:18:24 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_2.jpg*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo.log*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo
2023/03/24 11:18:24 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_1.jpg*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo.log*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo
2023/03/24 11:18:24 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_3.jpg*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo.log*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo
2023/03/24 11:18:24 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_4.jpg*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo.log*/Input_Image_Analysis/2023-03-24_11-18-19_test_demo
2023/03/24 11:18:24 PYTHON Removing root rights on generated output

--------------------------------------------------
Terminating parallel_IJ_docker!
--------------------------------------------------

The analysis has created a new directory and a log file into ~/Input_Image_Analysis:

  • Year-Month-Day_Hour-Minute-Second_test_demo
  • Year-Month-Day_Hour-Minute-Second_test_demo.log

Meaning every time you execute athe docker the output folder (in auto mode) and the log file are fingerprinted with the date, time, input directory, and macro used.

Thus you will never overwrite an output when changing some parameters within your macro.

Detailled process

So what happened during thes few seconds? First the the `` script executes the following command:

docker run --rm -v ~/Input_Image_Analysis:/Input_Image_Analysis -it parallel_ij:1 bash /usr/bin/parallel_image_analysis.sh

The docker run parallel_ij:1 command initiates a docker container with the parallel_ij:1 image that you will build following the wiki.

By default an image is associated with default command CMD. In the present case it is parallel_image_analysis.sh located in /usr/bin/.

Nevertheless docker run parallel_ij:1 is not enough informative for someone who does not know this project. So to make it explicit we add the -it option that specify that a specific command has to be executed. This later is bash /usr/bin/parallel_image_analysis.sh

So docker run parallel_ij:1 has the same behaviour that docker run -it parallel_ij:1 bash /usr/bin/parallel_image_analysis.sh. The second one is just more explicit.

Moreover we want the docker to be able to access to ~/Input_Image_Analysis to read, manipulate and save output files. So the -v ~/Input_Image_Analysis:/Input_Image_Analysis option mounts the ~/Input_Image_Analysis local directory of your machine in /Input_Image_Analysis of the container. This specific folder is the one that the later programs will inspect and manipulate

So now our command line becomes: docker run -v ~/Input_Image_Analysis:/Input_Image_Analysis -it parallel_ij:1 bash /usr/bin/parallel_image_analysis.sh

Finally, as docker does not destroy a containerat the end of a calculation, we ask him to do it with the --rm (remove) option, generated the functional command:

docker run --rm -v ~/Input_Image_Analysis:/Input_Image_Analysis -it parallel_ij:1 bash /usr/bin/parallel_image_analysis.sh

So at this step we have:

graph TD
    subgraph Legend
        bash[Bash]
        docker[Docker]
        python[Python]
        imagej[ImageJ]
        files[Files]
        folders[Folders]
    end

    subgraph local[Local Machine]
        IO[Input_Image_Analysis]
        run_parallel_ij[run_parallel_ij.sh]
        DOCKER
    end

    container[parallel_ij:1 container]


    run_parallel_ij --> |run| DOCKER
    DOCKER --> |start/kill| container
    run_parallel_ij --> |associate| IO
    IO --> DOCKER

    style bash fill: #ccffcc
    style run_parallel_ij fill: #ccffcc
    style docker fill: #d6f5f5
    style DOCKER fill: #d6f5f5
    style container fill: #d6f5f5
    style python fill: #ffcc00
    style imagej fill: #ff9900 
    style files fill: #c4c3d0
    style folders fill: #e0e0e6
    style IO fill: #e0e0e6

Resulting in the following working container

graph TD

    subgraph macros
        demo[demo.java]
        empty[empty.java]
    end

    subgraph IOdocker[Input_Image_Analysis]
        macros
        config[parallel_ij.yml]
        input[Input Directory]
    end

    subgraph container[parallel_ij:1 container]
        IOdocker
        parall_image_analysis[parall_image_analysis.sh]
        ImageJ
        parallel_ij[parallel_ij.py]

    subgraph local[Local Machine]
        IO[Input_Image_Analysis]
        run_parallel_ij[run_parallel_ij.sh]
        DOCKER
    end

    container[parallel_ij:1 container]


    run_parallel_ij --> |run| DOCKER
    DOCKER --> |start/kill| container
    run_parallel_ij --> |associate| IO
    IO --> DOCKER

    style bash fill: #ccffcc
    style run_parallel_ij fill: #ccffcc
    style docker fill: #d6f5f5
    style DOCKER fill: #d6f5f5
    style container fill: #d6f5f5
    style python fill: #ffcc00
    style imagej fill: #ff9900 
    style files fill: #c4c3d0
    style folders fill: #e0e0e6
    style IO fill: #e0e0e6

Resulting in the following working container

graph TD

    subgraph macros
        demo[demo.java]
        empty[empty.java]
    end

    subgraph IOdocker[Input_Image_Analysis]
        macros
        config[parallel_ij.yml]
        input[Input Directory]
    end

    subgraph container[parallel_ij:1 container]
        IOdocker
        parall_image_analysis[parall_image_analysis.sh]
        ImageJ
        parallel_ij[parallel_ij.py]
    end

    style parall_image_analysis fill: #ccffcc
    style container fill: #d6f5f5
    style parallel_ij fill: #ffcc00
    style demo fill: #ff9900
    style empty fill: #ff9900
    style ImageJ fill: #ff9900
    style config fill:  #c4c3d0
    style input fill: #e0e0e6
    style IOdocker fill: #e0e0e6
    style macros fill: #e0e0e6

The next program to be executed, as explained above, is parallel_image_analysis.sh. Its first task is to generate a xvfb virual display so ImageJ can "display" its windows and particularly the ROI Manager.

graph TD

    style parall_image_analysis fill: #ccffcc
    style container fill: #d6f5f5
    style parallel_ij fill: #ffcc00
    style demo fill: #ff9900
    style empty fill: #ff9900
    style ImageJ fill: #ff9900
    style config fill:  #c4c3d0
    style input fill: #e0e0e6
    style IOdocker fill: #e0e0e6
    style macros fill: #e0e0e6

The next program to be executed, as explained above, is parallel_image_analysis.sh. Its first task is to generate a xvfb virual display so ImageJ can "display" its windows and particularly the ROI Manager.

graph TD

    subgraph macros
        demo[demo.java]
        empty[empty.java]
        demo[demo.java]
        empty[empty.java]
    end

    subgraph IOdocker[Input_Image_Analysis]
    subgraph IOdocker[Input_Image_Analysis]
        macros
        config[parallel_ij.yml]
        input[Input Directory]
        output[Output Directory]
        log
    end

    subgraph xvfb[Virtual Display XVFB]
        Windows
    end

    subgraph container[parallel_ij:1 container]
        IOdocker
    subgraph xvfb[Virtual Display XVFB]
        Windows
    end

    subgraph container[parallel_ij:1 container]
        IOdocker
        parall_image_analysis[parall_image_analysis.sh]
        ImageJ
        parallel_ij[parallel_ij.py]
        parallel_ij[parallel_ij.py]
        xvfb
    end

    parall_image_analysis --> |initiate| xvfb
    parall_image_analysis --> |run| parallel_ij
    config --> |input| parallel_ij
    demo --> |input| ImageJ
    input --> |input| parallel_ij
    parallel_ij --> |create| log
    parallel_ij --> |create| output
    parallel_ij --> |feed| log
    parallel_ij --> |allocate memory| ImageJ
    ImageJ --> |feed| log
    ImageJ --> |feed| output
    parallel_ij --> |execute| ImageJ
    ImageJ --> |manipulations| Windows
    parall_image_analysis --> |initiate| xvfb
    parall_image_analysis --> |run| parallel_ij
    config --> |input| parallel_ij
    demo --> |input| ImageJ
    input --> |input| parallel_ij
    parallel_ij --> |create| log
    parallel_ij --> |create| output
    parallel_ij --> |feed| log
    parallel_ij --> |allocate memory| ImageJ
    ImageJ --> |feed| log
    ImageJ --> |feed| output
    parallel_ij --> |execute| ImageJ
    ImageJ --> |manipulations| Windows

    style parall_image_analysis fill: #ccffcc
    style container fill: #d6f5f5
    style parallel_ij fill: #ffcc00
    style demo fill: #ff9900
    style empty fill: #ff9900
    style parallel_ij fill: #ffcc00
    style demo fill: #ff9900
    style empty fill: #ff9900
    style ImageJ fill: #ff9900
    style config fill:  #c4c3d0
    style input fill: #e0e0e6
    style IOdocker fill: #e0e0e6
    style macros fill: #e0e0e6
    style Windows fill: #ff9900
    style IOdocker fill: #e0e0e6
    style macros fill: #e0e0e6
    style Windows fill: #ff9900
    style output fill: #e0e0e6
    style log fill:  #c4c3d0

Then parallel_image_analysis.sh runs the python program parallel_ij.py. Its first task is read the parallel_ij.yml file. With the collected informations it immediately create the log file and start to feed it. Then the Output Directory is created.

The next critical step is to allocate the memory to all ImageJ process. This step corresponds to the following line of the above output.

2023/03/24 11:18:23 PYTHON Setting ImageJ memory to 6666mB
    style log fill:  #c4c3d0

Then parallel_image_analysis.sh runs the python program parallel_ij.py. Its first task is read the parallel_ij.yml file. With the collected informations it immediately create the log file and start to feed it. Then the Output Directory is created.

The next critical step is to allocate the memory to all ImageJ process. This step corresponds to the following line of the above output.

2023/03/24 11:18:23 PYTHON Setting ImageJ memory to 6666mB

The python script then explore the Input Directory to obtain a list of files with correct extension. Using this list it creates a bunch of commandlines that will be execute the macro (demo.java) with ImageJ for each file. These commands are executed as parallel processes (4 in the example).

When ImageJ perfeomrs calculation it generate files in the Output Directory and feed le log file. So, in this file you will have python and ImageJ entries allowing to know what was done.

The program stops when all the files have been treated, and the container is terminated.

The python script then explore the Input Directory to obtain a list of files with correct extension. Using this list it creates a bunch of commandlines that will be execute the macro (demo.java) with ImageJ for each file. These commands are executed as parallel processes (4 in the example).

When ImageJ perfeomrs calculation it generate files in the Output Directory and feed le log file. So, in this file you will have python and ImageJ entries allowing to know what was done.

The program stops when all the files have been treated, and the container is terminated.

Docker Installation

In order to install and use Docker you need administration rights on you machine.

Ubuntu 20.04

In order to install and configure Docker on an Ubuntu distribution follow the steps:

sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
sudo apt update
apt-cache policy docker-ce
sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
sudo apt update
apt-cache policy docker-ce

At this stage you should have a similar result:

apt-cache policy docker-ce
apt-cache policy docker-ce
docker-ce:
  Installed: (none)
  Candidate: 5:19.03.9~3-0~ubuntu-focal
  Version table:
     5:19.03.9~3-0~ubuntu-focal 500
        500 https://download.docker.com/linux/ubuntu focal/stable amd64 Packages

Note that docker-ce is not installed but that the candidate is provided by Docker repository for Ubuntu 20.04.

Then proceed to the installation and check the status of the Docker daemon

sudo apt install docker-ce
sudo systemctl status docker
sudo apt install docker-ce
sudo systemctl status docker
Output
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2020-05-19 17:00:41 UTC; 17s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 24321 (dockerd)
      Tasks: 8
     Memory: 46.4M
     CGroup: /system.slice/docker.service
             └─24321 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

By default the docker command can only be executed by the root user or by a member of the docker group that is automatically generated when installing Docker. In order to avoid using sudo every time you can follow the next steps.

Add the current user to the docker group and check that is properly done.

sudo usermod -aG docker ${USER}
id -nG
sudo usermod -aG docker ${USER}
id -nG
admindavid sudo lpadmin docker

To add an other user:

sudo usermod -aG docker username
sudo usermod -aG docker username

Windows 10 and 11

Docker seems pretty tricky to install on these OS. Please refer to the following guidelines

Mac OS

In order to install Docker on a Mac machine please follow these guidelines

WSL2 + Ubuntu 20.04

Instruction will be completed later.

Docker Image construction

Preparing for IJ_DOCKER building

Download or clone this repository into your HOME directory Download or clone this repository into your HOME directory

You should obtain the following arborescence

ij_docker
│   README.md
│   .gitignore    
│
└───src
    │   build_parallel_ij.sh
    |   config_HPC.sh
    |   config_HPC.sh
    │   parallel_ij.yml
    |   run_parallel_ij.sh
    |   transfer_to_HPC.sh
    │
    └───macros
    |   │   demo.java
    |   |   empty.java
    |   |   empty.java
    |
    └───parallel_ij
        |   config.sh
        |   Dockerfile
        |   exit.sh
        |   parallel_image_analyis.sh
        |
        └───code
            |   parallel_ij.py
            |
            └───python3
                |   system_stats.py

REQUIREMENT: ImageJ

Due to size limitations for gitlab repository ImageJ cannot be included here. Please download an Linux ImageJ version

Unzip the archive and move the ImageJ subfolder into the code one at the same level than python3 (e.g. in the subdiretory /path/to/ij_docker/src/parallel_ij/code). Make sure you have a file ImageJ.cfg next to the file ij.jar

Now you are ready to build the ij_docker image called parallel_ij.

Building the IJ_DOCKER image

Go to the ij_docker/src directory and run the build_parallel_ij.sh script to Go to the ij_docker/src directory and run the build_parallel_ij.sh script to build the image.

WARNING The path presented here is the one present on my computer. Use your own one

WARNING The path presented here is the one present on my computer. Use your own one

conda install -c conda-forge ncurses
cd ~/ij_docker/src
bash build_parallel_ij.sh

You should obtain a similar output:

--------------------------------------------------
Start building parallel_IJ_docker!
--------------------------------------------------

mkdir: impossible de créer le répertoire «/home/admindavid/Input_Image_Analysis»: Le fichier existe
sending incremental file list

sent 78 bytes  received 12 bytes  180.00 bytes/sec
total size is 382  speedup is 4.24
sending incremental file list

sent 129 bytes  received 17 bytes  292.00 bytes/sec
total size is 1,820  speedup is 12.47
sending incremental file list

sent 81 bytes  received 12 bytes  186.00 bytes/sec
total size is 696  speedup is 7.48
sending incremental file list

sent 76 bytes  received 12 bytes  176.00 bytes/sec
total size is 123  speedup is 1.40
sending incremental file list

sent 81 bytes  received 12 bytes  186.00 bytes/sec
total size is 291  speedup is 3.13
sending incremental file list

sent 82 bytes  received 12 bytes  188.00 bytes/sec
total size is 314  speedup is 3.34
[+] Building 103.8s (13/13) FINISHED                                                                                                                                                                         
sent 82 bytes  received 12 bytes  188.00 bytes/sec
total size is 314  speedup is 3.34
[+] Building 103.8s (13/13) FINISHED                                                                                                                                                                         
 => [internal] load .dockerignore                                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                                         0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                    0.0s
 => => transferring dockerfile: 693B                                                                                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                         7.8s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                                           0.0s
 => [internal] load build context                                                                                                                                                                       0.6s
 => => transferring context: 165.58MB                                                                                                                                                                   0.6s
 => [1/7] FROM docker.io/library/ubuntu:20.04@sha256:24a0df437301598d1a4b62ddf59fa0ed2969150d70d748c84225e6501e9c36b9                                                                                   0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                         7.8s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                                           0.0s
 => [internal] load build context                                                                                                                                                                       0.6s
 => => transferring context: 165.58MB                                                                                                                                                                   0.6s
 => [1/7] FROM docker.io/library/ubuntu:20.04@sha256:24a0df437301598d1a4b62ddf59fa0ed2969150d70d748c84225e6501e9c36b9                                                                                   0.0s
 => CACHED [2/7] RUN mkdir -p /ij_parallel                                                                                                                                                              0.0s
 => [3/7] COPY code* /ij_parallel/                                                                                                                                                                      0.3s
 => [4/7] RUN  chmod -R 777 /ij_parallel                                                                                                                                                                1.4s
 => [5/7] COPY config.sh /                                                                                                                                                                              0.0s
 => [6/7] COPY parallel_image_analysis.sh exit.sh /usr/bin/                                                                                                                                             0.0s
 => [7/7] RUN bash /config.sh                                                                                                                                                                          89.1s
 => exporting to image                                                                                                                                                                                  4.5s
 => => exporting layers                                                                                                                                                                                 4.5s
 => => writing image sha256:be5435cc8e68ab9c347fe34e13f2a36d6f0df30e2c97a9742a3522350a0f17c4                                                                                                            0.0s 
                                                                                                                                                                                                             
--------------------------------------------------                                                                                                                                                           
Build parallel_IJ_docker over!                                                                                                                                                                               
 => [3/7] COPY code* /ij_parallel/                                                                                                                                                                      0.3s
 => [4/7] RUN  chmod -R 777 /ij_parallel                                                                                                                                                                1.4s
 => [5/7] COPY config.sh /                                                                                                                                                                              0.0s
 => [6/7] COPY parallel_image_analysis.sh exit.sh /usr/bin/                                                                                                                                             0.0s
 => [7/7] RUN bash /config.sh                                                                                                                                                                          89.1s
 => exporting to image                                                                                                                                                                                  4.5s
 => => exporting layers                                                                                                                                                                                 4.5s
 => => writing image sha256:be5435cc8e68ab9c347fe34e13f2a36d6f0df30e2c97a9742a3522350a0f17c4                                                                                                            0.0s 
                                                                                                                                                                                                             
--------------------------------------------------                                                                                                                                                           
Build parallel_IJ_docker over!                                                                                                                                                                               
--------------------------------------------------


--------------------------------------------------
Transfering the build_report.log file in your home
--------------------------------------------------

CONTAINER ID   IMAGE           COMMAND                  CREATED        STATUS                              PORTS     NAMES
e6bc83ab4865   parallel_ij:1   "bash /usr/bin/exit.…"   1 second ago   Exited (0) Less than a second ago             nice_thompson
e6bc83ab4865   parallel_ij:1   "bash /usr/bin/exit.…"   1 second ago   Exited (0) Less than a second ago             nice_thompson
Preparing to copy...
Copying from container - 512B
Copying from container - 32.77kB
Copying from container - 65.54kB
Copying from container - 98.3kB
Copying from container - 123.8kB
Copying from container - 123.9kB
Copying from container - 124.4kB
Copying from container - 123.8kB
Copying from container - 123.9kB
Copying from container - 124.4kB
Copying from container - 124.9kB
Successfully copied 124.9kB to /home/admindavid/build_report.log
Successfully copied 124.9kB to /home/admindavid/build_report.log

--------------------------------------------------
Suppressing container!
--------------------------------------------------

e6bc83ab4865

--------------------------------------------------
Creating a tar archive of parallel_ij!
--------------------------------------------------
e6bc83ab4865

--------------------------------------------------
Creating a tar archive of parallel_ij!
--------------------------------------------------

All along the building of the parallel_ij:1 docker image, the Input_Image_Analysis directory is creating and key files and folders are automatically transfered into it.

Let's check that all is in order. First, is the image present in docker

admindavid@RMI2-3571-DC:~/Bureau/scripts/ij_docker/src$ docker images
REPOSITORY    TAG       IMAGE ID       CREATED       SIZE
parallel_ij   1         0b92abe87a32   2 days ago    1.31GB
ubuntu        20.04     61c45d0e9798   12 days ago   72.8MB

The parallel_ij:1 docker image should be present. Your image can be <1.31GB, as my custom ImageJ version contains the bioformat plugin...

Now let's see if the ~/Input_Image_Analysis directory has been correctly created.

admindavid@RMI2-3571-DC:~/Bureau/scripts/ij_docker/src$ cd ~/Input_Image_Analysis/
admindavid@RMI2-3571-DC:~/Input_Image_Analysis$ ls
macros  parallel_ij.yml

Testing the IJ_DOCKER image

Now create a test directory (all lower case) within the ~/Input_Image_Analysis directory. And add at least four (4) .jpg files (it will not work with .jpeg) in it. You can do it with your classical explorer. Just check that everything is fine.

cd ~/Input_Image_Analysis/test
ls
cd ~/Input_Image_Analysis/test
ls
Cell_1.jpg  Cell_2.jpg  Cell_3.jpg  Cell_4.jpg

Now we are ready to test the installation with pre-installed demo.java macro, which convert the images to 8bit ones...

First ensure that your system has at least 5 cores.

admindavid@RMI2-3571-DC:~/Input_Image_Analysis/test$ nproc
16

If so move to the ij_docker/src directory and start the demo analysis using the run_parallel_ij.sh script.

(*) You can edit the BASH script by changing the name of the docker image.

admindavid@RMI2-3571-DC:~/Input_Image_Analysis/test$ cd ~/Bureau/scripts/ij_docker/src
admindavid@RMI2-3571-DC:~/Bureau/scripts/ij_docker/src$ bash run_parallel_ij.sh 

--------------------------------------------------
Starting parallel_IJ_docker!
--------------------------------------------------

2023/03/13 12:05:25 PYTHON Initiating demo.java analysis
2023/03/13 12:05:29 PYTHON Setting ImageJ memory to 6666mB
Cell_3.jpg
Cell_4.jpg
Cell_2.jpg
Cell_1.jpg
2023/03/13 12:05:29 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_3.jpg*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo.log*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo
2023/03/13 12:05:29 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_4.jpg*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo.log*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo
2023/03/13 12:05:29 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_2.jpg*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo.log*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo
2023/03/13 12:05:29 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_1.jpg*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo.log*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0OpenJDK 64-Bit Server VM warning: OpenJDK 64-Bit Server VM warning: 
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0ignoring option PermSize=128m; support was removed in 8.0ignoring option PermSize=128m; support was removed in 8.0


OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
2023/03/13 12:05:31 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_2.jpg*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo.log*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo
2023/03/13 12:05:31 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_4.jpg*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo.log*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo
2023/03/13 12:05:31 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_1.jpg*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo.log*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo
2023/03/13 12:05:31 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_3.jpg*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo.log*/Input_Image_Analysis/2023-03-13_12-05-25_test_demo
2023/03/13 12:05:31 PYTHON Removing root rights on generated output

--------------------------------------------------
Terminating parallel_IJ_docker!
--------------------------------------------------

CONTAINER ID   IMAGE           COMMAND                  CREATED         STATUS                              PORTS     NAMES
cdb22efd56ff   parallel_ij:1   "bash /usr/bin/paral…"   6 seconds ago   Exited (0) Less than a second ago             frosty_nobel
cdb22efd56ff

As docker does not automatically destroy the container at the end of the process, the run_parallel_ij.sh script does it for you.

Now let's check what append into the ~/Input_Image_Analysis directory

admindavid@RMI2-3571-DC:~/Bureau/scripts/ij_docker/src$ cd ~/Input_Image_Analysis/
admindavid@RMI2-3571-DC:~/Input_Image_Analysis$ ls
2023-03-10_14-00-29_test_demo  2023-03-10_14-00-29_test_demo.log  macros  parallel_ij.yml  test
admindavid@RMI2-3571-DC:~/Input_Image_Analysis$ cd 2023-03-10_14-00-29_test_demo
admindavid@RMI2-3571-DC:~/Input_Image_Analysis/2023-03-10_14-00-29_test_demo$ ls
Cell_1.jpg  Cell_2.jpg  Cell_3.jpg  Cell_4.jpg

The analysis has automatically generated the 2023-03-10_14-00-29_test_demo directory, which name contains, the date, time, input folder and macro names, allowing tracking of the output. All 4 input images have been processed and their daughter files have the same name.

An other file has been generated: 2023-03-10_14-00-29_test_demo.log

This files contains all informations passed by the main Python script but also the demo.java macro.

It should contain the following informations:

admindavid@RMI2-3571-DC:~/Input_Image_Analysis$ nano 2023-03-10_14-00-29_test_demo.log

2023/03/10 14:00:29 PYTHON Initiating demo.java analysis
2023/03/10 14:00:33 PYTHON Setting ImageJ memory to 6666mB
2023/03/10 14:00:33 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_3.jpg*/Input_Image_Analysis/2023-03-10_14-00-29_test_demo.log*/>
2023/03/10 14:00:33 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_4.jpg*/Input_Image_Analysis/2023-03-10_14-00-29_test_demo.log*/>
2023/03/10 14:00:33 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_2.jpg*/Input_Image_Analysis/2023-03-10_14-00-29_test_demo.log*/>
2023/03/10 14:00:33 PYTHON Starting /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_1.jpg*/Input_Image_Analysis/2023-03-10_14-00-29_test_demo.log*/>
2023/3/10 14:0:33 IMAGEJ Treating: /Input_Image_Analysis/test/Cell_1.jpg
2023/3/10 14:0:33 IMAGEJ Treating: /Input_Image_Analysis/test/Cell_4.jpg
2023/3/10 14:0:33 IMAGEJ Treating: /Input_Image_Analysis/test/Cell_2.jpg
2023/3/10 14:0:33 IMAGEJ Treating: /Input_Image_Analysis/test/Cell_3.jpg
2023/03/10 14:00:34 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_1.jpg*/Input_Image_Analysis/2023-03-10_14-00-29_test_demo.lo>
2023/03/10 14:00:34 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_3.jpg*/Input_Image_Analysis/2023-03-10_14-00-29_test_demo.lo>
2023/03/10 14:00:34 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_2.jpg*/Input_Image_Analysis/2023-03-10_14-00-29_test_demo.lo>
2023/03/10 14:00:34 PYTHON Terminating /ij_parallel/ImageJ/ImageJ -macro /Input_Image_Analysis/macros/demo.java /Input_Image_Analysis/test/Cell_4.jpg*/Input_Image_Analysis/2023-03-10_14-00-29_test_demo.lo>
2023/03/10 14:00:34 PYTHON Removing root rights on generated output

You can see, as advertised, that the 4 analysis started simultaneously at 2023/3/10 14:0:33 and finished altogether at 2023/03/10 14:00:34.

How to use my own macro?

First ensure that your macro is stable and doesn't require any graphical input from the user (dialog box, waitForUser, ...).

Then make your macro able to work with the three following arguments in this specific order:

  1. absolute image path
  2. absolute log file path
  3. absolute path to the output directory

As only one argument can be passed by commandline from Python to ImageJ, the 3 arguments are concatenated with * as separator.

You can use the following structure within your macrop to retrieve them:

macro 'demo'{
    // Get all arguments contcatenated as one
    argument = getArgument();
    arguments = split(argument, '*');
    path_image = arguments[0];
    path_log = arguments[1];
    output_dir = arguments[2];
    ...

If you desire to feed the log file from your macro, just plug in it the following function:

    function append_log(message){
        // Append a formated message to log file.
        getDateAndTime(year, month, dayOfWeek, dayOfMonth,
                       hour, minute, second, msec);
        string = '' + year + '/' + (month + 1) + '/' + dayOfMonth;
        string += ' ' + hour + ':' + minute + ':' + second;
        string += ' IMAGEJ ' + message;
        File.append(string, path_log)
    }

Call it using the command: append_log(string_message);

By default ImageJ doesn't terminate it self at the end of execution of a macro. Thus it is necessary to add the following line at the end of your main code: run('Quit');

Now, just save it into the ~/Input_Image_Analysis/macros directory, and change accordingly the macro entry in the ~/Input_Image_Analysis/parallel_ij.yml config file.

There is no need to rebuild the image. Except if you have added and used a plugin in your local version.