Conda Installs¶

Presented by: Jordan Hayes (HPCC)

Summary¶

  • Why Conda
    • Software Challenges
    • Solutions
  • Configure
    • Virtual Environments
  • Install Examples
    • Pandas
    • FastQC
    • Tensorflow

Why Conda: Software Challenges¶

  • Non-Root access
  • Dependency conflicts
  • Sharable
  • Easy to execute

Why Conda: Solutions¶

  • Use non-root user
  • Installs in separate environments
  • Installs are modular and easy to share
  • Easy to load/unload environments

Configure: Locations¶

Create the file ~/.condarc in your home, with the following content:

channels:
  - defaults
pkgs_dirs:
  - ~/bigdata/.conda/pkgs
envs_dirs:
  - ~/bigdata/.conda/envs
auto_activate_base: false

https://hpcc.ucr.edu/manuals/hpc_cluster/package_manage/#virtual-environments


Configure: Virtual Environment¶

First, lets request a compute node, since running on the head nodes would be slow:

srun -p short -c 2 --mem=10g --pty bash -l

Create conda virtual environment for pandasEnv, since pandas is a python pacakge, we will pre install Python:

In [ ]:
conda create -y -n pandasEnv python=3.6.4

Activate pandasEnv virtual environment:

In [ ]:
conda activate pandasEnv

Install Examples: Pandas¶

Now that we have a virtual environment, we can install packages within it. Visit anaconda website to search for a pagckage:

https://anaconda.org/

After you find the package, click on the name of the package to see install command.

In [ ]:
#conda install -y -c anaconda pandas
conda install -y -n pandasEnv -c anaconda pandas

Install Examples: FastQC¶

Same proceedure as before:

  1. Create fastqcEnv virtual environment
  2. Activate fastqcEnv virtual environment
  3. Look up fastqc on Anaconda
  4. Install fastqc package
  5. Deactivate fastqcEnv virtual environment
In [ ]:
# Create virutal environment
conda create -y -n fastqcEnv

# Activate virutal environment
conda activate fastqcEnv

# Install fastqc
conda install -y -c bioconda fastqc

# Deactivate virtual environment
conda deactivate

Install Examples: Tensorflow¶

Same proceedure as before:

  1. Create tensorflowEnv virtual environment
  2. Activate tensorflowEnv virtual environment
  3. Look up tensorflow on Anaconda
  4. Install tensorflow package
  5. Deactivate tensorflowEnv virtual environment
In [ ]:
# Create virtual environment
conda create -y -n tensorflowEnv

# Activate virtual environment
conda activate tensorflow

# Install tensorflow
conda install -y -n tensorflowEnv -c conda-forge tensorflow

# Deactivate virtual environment
conda deactivate

Install Examples: Script¶

After the installs have completed, you may want to use 1 (or more) conda virtual environments within a job/script:

In [ ]:
#!/bin/bash -l

#SBATCH -p short
#SBATCH -c 2
#SBATCH --mem=10gb
#SBATCH --time=10:00
#SBATCH -J "Conda Demo"

# Already loaded by default
module load miniconda2

#### pandasEnv ####
# Activate virtual environment
conda activate pandasEnv
# Show file path to where pandas is installed
python -c 'import pandas; print(pandas.__file__)'
# Deactivate virtual environment
conda deactivate

#### fastqcEnv ####
# Activate virtual environment
conda activate fastqcEnv
# Run help
fastqc --help
# Deactivate virtual environment
conda deactivate

#### tensorflowEnv ####
# Activate virtual environment
conda activate tensorflowEnv
# Add a scalar and a list
python -c 'import tensorflow as tf; x = [1, 2, 3, 4, 5]; y = 1; print(tf.add(x, y))'
# Deactivate virtual environment
conda deactivate