ResearchComputing/singularity-jupyter-pyspark:latest
$ singularity pull shub://ResearchComputing/singularity-jupyter-pyspark:latest
Singularity Recipe
Bootstrap: shub
From: ResearchComputing/singularity-slurm-base
%labels
AUTHOR sampedro@colorado.edu
%environment
export SPARK_VERSION=2.2.1
export SPARK_HOME=/opt/spark/spark-${SPARK_VERSION}-bin-hadoop2.7
export PATH=$SPARK_HOME/bin:$PATH
export PYTHONPATH=$SPARK_HOME/python/pyspark:$PYTHONPATH
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'
export JUPYTER_DATA_DIR=$HOME/.singularity-jupyter
%post
SPARK_VERSION=2.2.1
SPARK_HOME=/opt/spark
yum -y update
yum -y install epel-release
yum -y groupinstall 'Development Tools'
yum -y install wget python34-devel python34-pip
# Install Java JDK 8
mkdir --parents /opt/java && cd /opt/java
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.rpm"
yum -y localinstall *.rpm
# Install Jupyter
pip3 install --upgrade pip
pip3 install jupyterhub==0.7.2
pip3 install --upgrade notebook
pip3 install findspark
# Install spark
mkdir --parents $SPARK_HOME && cd $SPARK_HOME
wget http://apache.claz.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop2.7.tgz
tar -xzf spark-${SPARK_VERSION}-bin-hadoop2.7.tgz
Collection
- Name: ResearchComputing/singularity-jupyter-pyspark
- License: MIT License
View on Datalad
Metrics
key | value |
---|---|
id | /containers/ResearchComputing-singularity-jupyter-pyspark-latest |
collection name | ResearchComputing/singularity-jupyter-pyspark |
branch | master |
tag | latest |
commit | 395caae383f63294530a747ec17b295a74bbe20f |
version (container hash) | daea24d6b714a25e3a5cd69b3e89764b |
build date | 2020-05-22T10:18:37.349Z |
size (MB) | 2207 |
size (bytes) | 1195200543 |
SIF | Download URL (please use pull with shub://) |
Datalad URL | View on Datalad |
Singularity Recipe | Singularity Recipe on Datalad |
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.