CyVerse logo

Home_Icon Learning Center Home

OSG Tool Integration in DE


This quickstart contains the complete instructions for integrating osg tools in DE which is mainly intended for high-throughput processing


What is osg tool?

Any software/tool that can be run on the command line without the need to open any ports


Downloads, access, and services

In order to complete this tutorial you will need access to the following services/software

Prerequisite Preparation/Notes Link/Download
CyVerse account You will need a CyVerse account to complete this exercise CyVerse User Portal


We will use the following CyVerse platform(s):

Platform Interface Link Platform Documentation Quick Start
Discovery Environment Web/Point-and-click Discovery Environment DE Manual Discovery Environment Guide

Input and example data

In order to complete this quickstart you will need to have the following inputs prepared

Input File(s) Format Preparation/Notes Example Data
Read1.fastq fastq Make sure it is a fastq file and not fasta file Read1.fastq

Get started

  • Create a Docker image for your tool/software of interest
  • Build the Docker image for your tool/software
  • Test Docker image
  • Submit a pull request to OSG github repo
  • Integrate DE tool using “Add Tools” option in DE

1. Create a Docker image for your tool

This is the first step in the process of making OSG tool integration in DE. The minimum requirements for creating a Docker image include the following dependencies (apart from the dependencies that are needed for your tool of interest)

1.1 Ubuntu Operating system (preferred 16.04 and beyond)

1.2 Directories named cvmfs work

1.3 iRODS icommands version 4.0 or above.

1.4 An executable wrapper script

1.5 An upload file upload-files at /usr/bin in the container

We will use fastq-sample for integrating as OSG-tool in DE. Let’s first create a Dockerfile using your favorite editor which satisfies the above requirements

$ mkdir fastq-sample-osg && cd fastq-sample-osg

$ wget

$ wget
$ vi Dockerfile

FROM ubuntu:xenial
MAINTAINER Upendra Devisetty <>

RUN mkdir /cvmfs /work

RUN apt-get update \
    && apt-get install -y lsb curl apt-transport-https python3 python-requests libfuse2 wget gcc make libpcre3-dev libz-dev

# Install fastq-tools
RUN wget
RUN tar xvf fastq-tools-0.8.tar.gz
WORKDIR fastq-tools-0.8
RUN ./configure
RUN  make install fastq==0.8


# Define the iRODS package.
ENV ICMD_PKG="irods-icommands-4.1.10-ubuntu14-x86_64.deb"

# Install icommands.
        && dpkg -i "$ICMD_PKG" \
        && rm -f "$ICMD_PKG"

# Install the wrapper script and the script to upload the output files.
ADD wrapper /usr/bin/wrapper
ADD upload-files /usr/bin/upload-files

# Make the wrapper script the default command.
CMD ["wrapper"]


The Dockerfile and wrapper files are specific for fastq-sample tool. If you want to create OSG tool for your tool of interest, replace the specific parts of the scripts

2. Build and push the Docker image to Dockerhub

Once you create the Dockerfile, next step is to build the Docker image and push it to Dockerhub manually (you can also do an automated build)

$ docker build -t upendradevisetty/fastq-sample-osg:0.8 .

$ docker push upendradevisetty/fastq-sample-osg:0.8

3. Test Docker image

Testing of OSG-fastq-sample docker image can be done in two ways: Locally using Singularity and on Open Science Grid (OSG). Since many users don’t have access to OSG, we recommed that you test it locally.


This is very important step as it is very hard to troubleshoot after you integrate the OSG tool in DE

3.1 Create a folder with input file(s) and output folder on CyVerse Datastore

For this example, the only input file is Read1.fastq. I have this input file in this path on Datastore /iplant/home/upendra_35/fastq-sample-osg/Read1.fastq and output folder in this path on Datastore /iplant/home/upendra_35/fastq-sample-osg/output

3.2 Create input and output path files

The next step is to create an input and output path files that contains the paths to the input and output respectively.

$ cat input-paths.txt

$ cat output-paths.txt

3.3 Create input and output tickets from input and output paths files

Using <>, create tickets for both inputs and outputs

$ wget

$ mkdir sample_data

$ bash -r input-paths.txt > sample_data/input_ticket.list
$ cat sample_data/input_ticket.list
  # application/; version=1

$ bash -w output-paths.txt > sample_data/output_ticket.list
$ cat sample_data/output_ticket.list
  # application/; version=1

3.4 Create a config.json file in the sample_data folder

Here is an example of config.json for the fastq-sample-osg tool

$ vi sample_data/config.json
     "arguments": [
     "irods_host": "",
     "irods_port": 1247,
     "irods_job_user": "upendra_35",
     "irods_user_name": "job",
     "irods_zone_name": "",
     "input_ticket_list": "input_ticket.list",
     "output_ticket_list": "output_ticket.list",
     "status_update_url": "",
     "stdout": "out.txt",
     "stderr": "err.txt"

This is similar to running on the commandline like this..

$ fastq-sample -n 10 Read1.fastq

3.5 Pull the Docker image as singularity file (.sif)


You need to have Singularity installed first inorder to run this

$ singularity pull docker://upendradevisetty/fastq-sample-osg:0.8

This will create fastq-sample-osg_0.8.sif singularity image in your working directory

3.6 Test the singularity image

Once you have the input, output tickets and config files created, you are ready for the test with Singularity image


Before you run this, make sure that you remove the irods password in your system by running rm ~/.irods/.irodsA

$ cd sample_data

$ singularity exec ../fastq-sample-osg_0.8.sif ../wrapper
  running: configuration successfully loaded
  running: initializing the iRODS connection
  running: downloading the input files
  Enter your current iRODS password:
  running: processing the input files
  running: uploading the output files
  Enter your current iRODS password:
  Enter your current iRODS password:Enter your current iRODS password:

  Enter your current iRODS password:
  Enter your current iRODS password:
  Enter your current iRODS password:Enter your current iRODS password:
  Enter your current iRODS password:Enter your current iRODS password:completed: job completed successfully


It will prompt you to enter your irods passwords several times, if so, then keep pressing the ENTER until the job is successfully finished. The output files will be uploaded to your output folder in datastore.

Once your job has finished, you should expect to see the input (Read1.fastq) and output (sample.fastq) files in the current working directory and also in the in the output directory.

4. Submit a pull request to OSG github repo for fastq-sample-osg tool

Once the Singularity run works, add your Docker image in here. For this particular example, we will add upendradevisetty/fastq-sample-osg:0.8 in there.


You will have to fork and do a PR for this to work

Here is a screenshot of fastq-sample-osg:0.8 pull request to OSG github repo


After the PR is merged, it takes few hours for the image to be available on CVMFS.

5. Integrate DE tool using “Add Tools” option in DE

After the image is available on OSG, it is now ready to be integrated into DE.

5.1 Log-in to CyVerse Discovery Environment and click on the “Apps” window

5.2 Click “Manage Tools” -> “Tools” -> “Add Tool” and fill the details about your Docker image


Procede here to create an app interface in the CyVerse Discovery Environment

Additional information, help

Search for an answer: CyVerse Learning Center or CyVerse Wiki

Post your question using the intercom button on the bottom right of this page:

Fix or improve this documentation

Home_Icon Learning Center Home