Getting Started

Download

$ git clone https://git.astron.nl/RD/pulp2-cwl-folding

Building docker container to run the pipeline

To run the pipeline within a Docker container, use the Dockerfile located in docker/workflow directory of the repository to build the image:

$ cd pulp2-cwl-folding
$ docker build . -f docker/workflow/Dockerfile -t pulp2-cwl-folding:latest

One can specify their own name and tag for the docker image by providing different value for the -t option. If your containerization tool of choice is Apptainer, please refer to online documentation of how to use the Dockerfile to build the Apptainer container.

Manual installation

Note

Skip this section if you want to run LOFAR2 Beamformed Pulsar Folding Pipeline using the docker container.

For installing LOFAR2 Beamformed Pulsar Folding Pipeline a Debian-based operating system is recommended (e.g. Ubuntu 20.04). In order to run the pipeline you need to get the following packages onto your system:

$ apt-get -y update && apt-get -y upgrade && \
$ apt-get install -y  python-is-python3 && \
$ apt-get autoremove --purge && \
$ apt-get install -y aptitude apt-utils sudo python3 python3-pip python3-setuptools \
    python3-wheel python3-tk gcc gfortran 2to3 bc pgplot5 libcpgplot0 tcsh wget git \
    vim libhdf5-103 libhdf5-dev hdf5-tools python3-h5py swig cmake imagemagick ghostscript \
    a2ps gv mc bcal autoconf automake libtool libgsl-dev git-svn tmux screen terminator \
    libxml2-dev libxml2-utils libxslt-dev texlive texlive-extra-utils dialog latex2html \
    valgrind doxygen python3-sphinx bison libreadline-dev libx11-dev libxt-dev libx11-6 libxt6 \
    libyaml-cpp-dev pkg-config libglib2.0-dev libpng-dev libpng++-dev openmpi-bin libopenmpi-dev \
    mpich libmpich-dev libhdf5-openmpi-dev libhdf5-mpich-dev locales graphviz time dos2unix

To run the pipeline, you need a CWL runner, such as cwltool or toil. It is also necessary to install NodeJS required by CWL step of the workflow. For the Ubuntu 20.04 we recommend to use docker/requirements.txt file to install these and few other packages. It’s recommended to install these in a virtual environment:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install --no-cache-dir -r requirements.txt

In order for the pipeline to create Postscript files using ImageMagick’s convert tool the PS/PDF file types need to be enabled in the ImageMagick’s policy file, e.g. like this:

$ sed -i -e '/<policy [^>]*\"PDF\"/ {s//<!--\n\t&/; s~<*\"PDF\" />~&\n\t-->~;}' /etc/ImageMagick-*/policy.xml && \
  sed -i -e '/<policy [^>]*\"XPS\"/ {s//<!--\n\t&/; s~<*\"XPS\" />~&\n\t-->~;}' /etc/ImageMagick-*/policy.xml && \
  sed -i -e '/<policy [^>]*\"PS\"/ {s//<!--\n\t&/; s~<*\"PS\" />~&\n\t-->~;}' /etc/ImageMagick-*/policy.xml && \
  sed -i -e '/<policy [^>]*\"PS2\"/ {s//<!--\n\t&/; s~<*\"PS2\" />~&\n\t-->~;}' /etc/ImageMagick-*/policy.xml && \
  sed -i -e '/<policy [^>]*\"PS3\"/ {s//<!--\n\t&/; s~<*\"PS3\" />~&\n\t-->~;}' /etc/ImageMagick-*/policy.xml && \
  sed -i -e '/<policy [^>]*\"EPS\"/ {s//<!--\n\t&/; s~<*\"EPS\" />~&\n\t-->~;}' /etc/ImageMagick-*/policy.xml

Extra packages

Then you need to install the following software:

  • FFTW (v3.3.10 or newer)

  • CFITSIO (v4.5.0 or newer)

  • DAL2 library

    • python3-support branch: git clone -b python3-support https://git.astron.nl/ro/dal2.git

    • version: git reset --hard cb2bedb1abc7b8d574ec1d4b7f202eeaa16c84f5

  • ARMADILLO (v14.4.1 or newer)

  • SPLINTER

  • EIGEN

    • version: git reset --hard 3147391d

  • QD

  • SOFA

    • version: 20231011

    • both ANSI C and Fortran 77 releases

  • PSRDADA

  • CALCEPH (v3.5.5 or newer)

  • PSRXML

  • PSRCAT

  • TEMPO

    • version: git reset --hard f58695ff60a03485056a2428d4afbe1867eda6af

  • TEMPO2

    • version: git reset --hard d56c8856986ef1cc3618e0fbaa1de788d7708ad4

    • TEMPO2 and TEMPO2_PLUG_PATH env variables need to be installed before installing the package (see below)

  • EPSIC

    • version: git reset --hard 069cdb35a3618ce6b959ddf53d244858f847fe7a

  • PSRCHIVE

    • version: git reset --hard 2058bbb91c1b4275f017c1f93568cbaf5f965970

  • PSRFITS_UTILS

  • CLFD

    • version: git reset --hard 2500ee7264f69acb39053bd895006680711d4cc4

  • DSPSR

    • version: git reset --hard 8ac40cdf0262e119847d22243ac92c21deafaed6

  • LOFAR-SCRIPTS-REPO

Configuring extra packages

To ensure the proper setup of the installed software we recommend the following configuration setup:

$ export PGPLOT_DIR="/usr/lib/pgplot5"
$ export PGPLOT_DEV="/xwin"
$ export PGPLOT_TYPE="cps"
$ export PGPLOT_FONT="${PGPLOT_DIR}/grfont.dat"
$ export PGPLOT_BACKGROUND=white
$ export PGPLOT_FOREGROUND=black
$ export PGPLOT_INCLUDES="/usr/include"
$ export PSRCAT_FILE="${INSTALL_DIR}/bin/psrcat.db"
$ export TEMPO="${INSTALL_DIR}/src/tempo"
$ export TEMPO2="${INSTALL_DIR}/share/tempo2"
$ export TEMPO2_PLUG_PATH=${TEMPO2}/plugins
$ export PSRCHIVE_CONFIG=${INSTALL_DIR}/etc/psrchive.cfg

where ${INSTALL_DIR} is the top-level directory for the installed software, usually /usr/local.

Also copy the script scripts/make-psr-pav-diag.sh to the ${INSTALL_DIR}/bin directory.

Running the pipeline

The LOFAR2 Beamformed Pulsar Folding Pipeline can be run from the command line using a CWL runner, e.g., cwltool or toil, such as:

$ cwltool ${INSTALL_DIR}/pulp2-cwl-folding/workflows/pulp2-dspsr-folding.cwl input.json

or

$ toil-cwl-runner ${INSTALL_DIR}/pulp2-cwl-folding/workflows/pulp2-dspsr-folding.cwl input.json

where ${INSTALL_DIR} refers to the directory where pipeline repository pulp2-cwl-folding has been installed, and input.json file is the JSON file with the input parameters. You can see the example of input JSON file(s) in the tests/ directory of the repository. The pipeline parameters are described in the Pipeline overview page. Additionally, cwltool and toil come with a number of useful command line arguments. To name a few which could come handy, are:

  • --outdir: specifies the (relative) path to the directory containing the output of the pipeline (make sure to mount this directory when running the pipeline in a container);

  • --log-dir: specifies the location of the log files produces by the standard output and standard error (make sure to mount this directory when running the pipeline in a container);

  • --preserve-entire-environment: use your system’s environment variables when manually installing the dependencies;

  • --parallel: run jobs in parallel.

Please refer to their respective documentation for a full overview (cwltool and toil).

Running the pipeline from within a container

If you followed the Docker installation instructions in the corresponding section above, you can run the container using Docker as follows:

$ docker run --rm -v <source_directory>:<mount_point> -w <mount_point> pulp2-cwl-folding:latest cwltool /usr/local/src/pulp2-cwl-folding/workflows/pulp2-dspsr-folding.cwl input.json

There are a number of command-line options you might want consider adding when running cwltool and toil. For an overview of those we refer you to the corresponding sections of documentation for LOFAR Pre-Processing Pipeline.