ResearchComputing/singularity-jupyter-pyspark:latest

$ singularity pull shub://ResearchComputing/singularity-jupyter-pyspark:latest

Singularity Recipe

Bootstrap: shub
From: ResearchComputing/singularity-slurm-base

%labels
  AUTHOR sampedro@colorado.edu

%environment
  export SPARK_VERSION=2.2.1
  export SPARK_HOME=/opt/spark/spark-${SPARK_VERSION}-bin-hadoop2.7
  export PATH=$SPARK_HOME/bin:$PATH
  export PYTHONPATH=$SPARK_HOME/python/pyspark:$PYTHONPATH
  export PYSPARK_DRIVER_PYTHON=jupyter
  export PYSPARK_DRIVER_PYTHON_OPTS='notebook'
  export JUPYTER_DATA_DIR=$HOME/.singularity-jupyter

%post
  SPARK_VERSION=2.2.1
  SPARK_HOME=/opt/spark
  yum -y update
  yum -y install epel-release
  yum -y groupinstall 'Development Tools'
  yum -y install wget python34-devel python34-pip
  # Install Java JDK 8
  mkdir --parents /opt/java && cd /opt/java
  wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.rpm"
  yum -y localinstall *.rpm
  # Install Jupyter
  pip3 install --upgrade pip
  pip3 install jupyterhub==0.7.2
  pip3 install --upgrade notebook
  pip3 install findspark
  # Install spark
  mkdir --parents $SPARK_HOME && cd $SPARK_HOME
  wget http://apache.claz.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop2.7.tgz
  tar -xzf spark-${SPARK_VERSION}-bin-hadoop2.7.tgz

Collection


View on Datalad

Metrics

key value
id /containers/ResearchComputing-singularity-jupyter-pyspark-latest
collection name ResearchComputing/singularity-jupyter-pyspark
branch master
tag latest
commit 395caae383f63294530a747ec17b295a74bbe20f
version (container hash) daea24d6b714a25e3a5cd69b3e89764b
build date 2020-05-22T10:18:37.349Z
size (MB) 2207
size (bytes) 1195200543
SIF Download URL (please use pull with shub://)
Datalad URL View on Datalad
Singularity Recipe Singularity Recipe on Datalad
We cannot guarantee that all containers will still exist on GitHub.