Python, R, Packages, and the Grid

Contents

 

Introduction

PLEASE NOTE that installation of conda and miniconda in home directories is no longer necessary for Grid 2.5. Please contact RCS if you have any questions about the versions of R and Python that are installed on Grid 2.5. In addition, if you have used a conda or miniconda installation out of your home directory, we counsel you to transition to using the central installations of R, Python2, and Python3.

Compute Grid 2.5 is using Anaconda, the full distribution, and miniConda, a slimmed down version, to provide feature-rich environments for Python and R, respectively. This page briefly discusses some need-to-knows for each environment. 

For the so-inclined or for those wishing more advanced capabilities, see Conda's Overview guide for general information, and their Managing... pages if you wish to use more of the advanced features. Please note that our central installations are read-only, so one would need to make any changes locally to home or project folders.

R

Compute Grid 2.5 offers both R and RStudio via the miniConda distribution. This provides CPU-optimized versions of R and its supporting libraries from CRAN.

Installing custom R packages

No special instructions are needed for this. Using the install.packages() command from within R or RStudio will download and install the specified packages in your home folder by default.

Python

Compute Grid 2.5 offers both Python2 (v2.7.x) and Python3 (v3.6.x) via the Anaconda distribution. This provides CPU-optimized versions of Python and its supporting libraries, including numpy, scipy, matplotlib, pandas, & scikitlearn, among others.

Python2 is still our default, so the commands python or python2 will both run python 2.7.x. For Python3, please use the command python3.

Installing custom Python modules

If you require a Python module that is not installed with the central Anaconda installation, you can install this yourself, and the module will be placed in a directory in your home folder. Due to the wrapper scripts that are installed on login nodes, this action must be performed via the back-end, compute nodes.

1. Set up an alias so it's easy to submit interactive jobs to back-end nodes:

alias my_run="bsub -app python-5g -q short_int -Ip"

2. Install your Python module by prefixing the install command with my_run:

my_run python -m pip install --user SomeModule

OR, If you are using Python3, use the python3 command instead:

my_run python3 -m pip3 install --user SomeModule

The modules will be placed in the directory $HOME/.local for use by your scripts and programs. If you are upgrading modules that are already installed centrally, insert --upgrade before the --user option.

Environments (Nota Bene!)

One of Anaconda's most useful features is the ability to create virtual environments. This is particularly helpful if you have multiple projects that depend on different versions of packages. With a virtual environment you can update the packages for one project without disturbing the packages of your other projects. Conda's documentation on managing environments is a good place to learn about this feature.

In addition, we'd like to bring a few important items to your attention:

  • Our version of Anaconda requires that you set the execution path (export PATH…) to point at the central install in order for all the components to work correctly. This is especially true for installing modules that may require compiling from source code.
  • By default, new enivornments are placed in your home directory. Once you source activate your environment, the execution path should point towards your local (home directory) install.
  • Environments cannot be used with the wrapper scripts in NoMachine and from the command line, as the execution path points at and prioritizes the central Anaconda installations. We hope this will be fixed in the next couple of months are we roll out better ways to manage the software versions and environments on the HBSGrid.

Installed Packages & Modules

Our base installations of R, Python2, and Python3 have the packages and modules listed below as a part of the central installation. You can use the instructions above to install additional or updates ones in either your home folder (the default location) or in another location that you must manually specify.

Updated 11/7/2019

Installed R3.5 Packages

abind         "1.4-5"      NA           
assertthat    "0.2.0"      NA           
backports     "1.1.2"      NA           
base          "3.5.1"      "base"       
base64enc     "0.1-3"      NA           
BH            "1.66.0-1"   NA           
bindr         "0.1.1"      NA           
bindrcpp      "0.2.2"      NA           
bit           "1.1-14"     NA           
bit64         "0.9-7"      NA           
bit64.bak     "0.9-7"      NA           
bitops        "1.0-6"      NA           
blob          "1.1.1"      NA           
boot          "1.3-20"     "recommended"
broom         "0.5.0"      NA           
callr         "2.0.4"      NA           
caret         "6.0-80"     NA           
caTools       "1.17.1"     NA           
cellranger    "1.1.0"      NA           
checkpoint    "0.4.4"      NA           
class         "7.3-14"     NA           
cli           "1.0.0"      NA           
clipr         "0.4.1"      NA           
cluster       "2.0.7-1"    NA           
codetools     "0.2-15"     NA           
colorspace    "1.3-2"      NA           
compiler      "3.5.1"      "base"       
config        "0.3"        NA           
crayon        "1.3.4"      NA           
curl          "3.2"        NA           
CVST          "0.2-2"      NA           
data.table    "1.11.4"     NA           
datasets      "3.5.1"      "base"       
DBI           "1.0.0"      NA           
dbplyr        "1.2.2"      NA           
ddalpha       "1.3.4"      NA           
DEoptimR      "1.0-8"      NA           
deployrRserve "9.0.0"      NA           
dichromat     "2.0-0"      NA           
digest        "0.6.15"     NA           
dimRed        "0.1.0"      NA           
doParallel    "1.0.13"     NA           
dplyr         "0.7.6"      NA           
DRR           "0.0.3"      NA           
evaluate      "0.11"       NA           
fansi         "0.2.3"      NA           
forcats       "0.3.0"      NA           
foreach       "1.5.0"      NA           
foreign       "0.8-71"     "recommended"
formatR       "1.5"        NA           
geometry      "0.3-6"      NA           
ggplot2       "3.0.0"      NA           
glmnet        "2.0-16"     NA           
glue          "1.3.0"      NA           
gower         "0.1.2"      NA           
graphics      "3.5.1"      "base"       
grDevices     "3.5.1"      "base"       
grid          "3.5.1"      "base"       
gtable        "0.2.0"      NA           
haven         "1.1.2"      NA           
hexbin        "1.27.2"     NA           
highr         "0.7"        NA           
hms           "0.4.2"      NA           
htmltools     "0.3.6"      NA           
htmlwidgets   "0.9"        NA           
httpuv        "1.4.5"      NA           
httr          "1.3.1"      NA           
ipred         "0.9-6"      NA           
IRdisplay     "0.5.0"      NA           
IRkernel      "0.8.11"     NA           
iterators     "1.0.10"     NA           
jsonlite      "1.5"        NA           
kernlab       "0.9-26"     NA           
KernSmooth    "2.23-15"    NA           
knitr         "1.20"       NA           
labeling      "0.3"        NA           
later         "0.7.3"      NA           
lattice       "0.20-35"    NA           
lava          "1.6.2"      NA           
lazyeval      "0.2.1"      NA           
lubridate     "1.7.4"      NA           
magic         "1.5-8"      NA           
magrittr      "1.5"        NA           
maps          "3.3.0"      NA           
markdown      "0.8"        NA           
MASS          "7.3-50"     "recommended"
Matrix        "1.2-14"     NA           
methods       "3.5.1"      "base"       
mgcv          "1.8-24"     "recommended"
MicrosoftR    "3.5.1.3"    NA           
mime          "0.5"        NA           
miniUI        "0.1.1"      NA           
mnormt        "1.5-5"      NA           
ModelMetrics  "1.1.0"      NA           
modelr        "0.1.2"      NA           
mongolite     "1.6"        NA           
munsell       "0.4.3"      NA           
nlme          "3.1-137"    NA           
nnet          "7.3-12"     NA           
numDeriv      "2016.8-1"   NA           
odbc          "1.1.3"      NA           
openssl       "1.0.2"      NA           
packrat       "0.4.9-3"    NA           
parallel      "3.5.1"      "base"       
pbdZMQ        "0.3-3"      NA           
pillar        "1.3.0"      NA           
pkgconfig     "2.0.1"      NA           
PKI           "0.1-5.1"    NA           
plogr         "0.2.0"      NA           
pls           "2.6-0"      NA           
plyr          "1.8.4"      NA           
png           "0.1-7"      NA           
praise        "1.0.0"      NA           
prettyunits   "1.0.2"      NA           
processx      "3.1.0"      NA           
prodlim       "2018.04.18" NA           
profvis       "0.3.4"      NA           
promises      "1.0.1"      NA           
psych         "1.7.8"      NA           
purrr         "0.2.5"      NA           
quantmod      "0.4-13"     NA           
R6            "2.2.2"      NA           
randomForest  "4.6-14"     NA           
rappdirs      "0.3.1"      NA           
rbokeh        "0.6.3"      NA           
RColorBrewer  "1.1-2"      NA           
Rcpp          "0.12.18"    NA           
RcppRoll      "0.3.0"      NA           
RCurl         "1.95-4.9"   NA           
readr         "1.1.1"      NA           
readxl        "1.1.0"      NA           
recipes       "0.1.1"      NA           
rematch       "1.0.1"      NA           
repr          "0.12.0"     NA           
reprex        "0.1.1"      NA           
reshape2      "1.4.3"      NA           
RevoIOQ       "10.0.0"     NA           
RevoMods      "11.0.1"     NA           
RevoUtils     "11.0.1"     NA           
RevoUtilsMath "11.0.0"     NA           
rJava         "0.9-9"      NA           
RJDBC         "0.2-5"      NA           
RJSONIO       "1.3-0"      NA           
rlang         "0.2.1"      NA           
RMariaDB      "1.0.6"      NA           
rmarkdown     "1.10"       NA           
robustbase    "0.93-2"     NA           
rpart         "4.1-13"     NA           
rprojroot     "1.3-1"      NA           
rsconnect     "0.8.8"      NA           
rstudioapi    "0.7"        NA           
RUnit         "0.4.32"     NA           
rvest         "0.3.2"      NA           
scales        "0.5.0"      NA           
selectr       "0.4-1"      NA           
sfsmisc       "1.1-2"      NA           
shiny         "1.1.0"      NA           
sourcetools   "0.1.7"      NA           
sparklyr      "0.6.4"      NA           
spatial       "7.3-11"     "recommended"
splines       "3.5.1"      "base"       
SQUAREM       "2017.10-1"  NA           
stats         "3.5.1"      "base"       
stats4        "3.5.1"      "base"       
stringi       "1.2.4"      NA           
stringr       "1.3.1"      NA           
survival      "2.42-6"     "recommended"
tcltk         "3.5.1"      "base"       
testthat      "2.0.0"      NA           
tibble        "1.4.2"      NA           
tidyr         "0.8.1"      NA           
tidyselect    "0.2.4"      NA           
tidyverse     "1.2.1"      NA           
timeDate      "3043.102"   NA           
tinytex       "0.6"        NA           
tools         "3.5.1"      "base"       
TTR           "0.23-3"     NA           
utf8          "1.1.4"      NA           
utils         "3.5.1"      "base"       
uuid          "0.1-2"      NA           
viridisLite   "0.3.0"      NA           
whisker       "0.3-2"      NA           
withr         "2.1.2"      NA           
xfun          "0.3"        NA           
xml2          "1.2.0"      NA           
xtable        "1.8-2"      NA           
xts           "0.11-0"     NA           
yaml          "2.2.0"      NA           
zoo           "1.8-3"      NA   

Installed Python2.7 Modules


alabaster==0.7.10
anaconda-client==1.6.14
anaconda-navigator==1.8.3
anaconda-project==0.8.2
asn1crypto==0.24.0
astroid==1.6.3
astropy==2.0.6
attrs==17.4.0
Babel==2.5.3
backports-abc==0.5
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
backports.ssl-match-hostname==3.5.0.1
beautifulsoup4==4.6.0
bitarray==0.8.1
bkcharts==0.2
blaze==0.11.3
bleach==2.1.3
bokeh==0.12.15
boto==2.48.0
Bottleneck==1.2.1
cdecimal==2.3
certifi==2018.10.15
cffi==1.11.5
chardet==3.0.4
click==6.7
cloudpickle==0.5.2
clyent==1.2.2
colorama==0.3.9
conda==4.5.11
conda-build==3.9.1
conda-verify==2.0.0
configparser==3.5.0
contextlib2==0.5.5
cryptography==2.2.2
cycler==0.10.0
Cython==0.28.2
cytoolz==0.9.0.1
dask==0.17.2
datashape==0.5.4
decorator==4.3.0
distributed==1.21.6
docutils==0.14
entrypoints==0.2.3
enum34==1.1.6
et-xmlfile==1.0.1
fastcache==1.0.2
filelock==3.0.4
Flask==0.12.2
Flask-Cors==3.0.3
funcsigs==1.0.2
functools32==3.2.3.post2
futures==3.2.0
gevent==1.2.2
glob2==0.6
gmpy2==2.0.8
greenlet==0.4.13
grin==1.2.1
h5py==2.7.1
heapdict==1.0.0
html5lib==1.0.1
idna==2.6
imageio==2.3.0
imagesize==1.0.0
ipaddress==1.0.22
ipykernel==4.8.2
ipython==5.6.0
ipython-genutils==0.2.0
ipywidgets==7.2.1
isort==4.3.4
itsdangerous==0.24
jdcal==1.4
jedi==0.12.0
Jinja2==2.10
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
jupyterlab==0.32.0
jupyterlab-launcher==0.10.5
keyring==13.2.1
kiwisolver==1.0.1
lazy-object-proxy==1.3.1
llvmlite==0.22.0
locket==0.2.0
log0==1.0.1
lxml==4.2.1
MarkupSafe==1.0
matplotlib==2.2.2
mccabe==0.6.1
mistune==0.8.3
mkl-fft==1.0.0
mkl-random==1.0.1
more-itertools==4.1.0
mpmath==1.0.0
msgpack-python==0.5.6
multipledispatch==0.5.0
mysql-connector-python==8.0.12
navigator-updater==0.1.0
nbconvert==5.3.1
nbformat==4.4.0
networkx==2.1
nltk==3.2.5
nose==1.3.7
notebook==5.4.1
numba==0.37.0+0.gb6b965afa.dirty
numexpr==2.6.4
numpy==1.16.4
numpydoc==0.8.0
odo==0.5.1
olefile==0.45.1
openpyxl==2.5.3
packaging==17.1
pandas==0.22.0
pandocfilters==1.4.2
parso==0.2.0
partd==0.3.8
path.py==11.0.1
pathlib2==2.3.2
patsy==0.5.0
pep8==1.7.1
pexpect==4.5.0
pickleshare==0.7.4
Pillow==5.1.0
pkginfo==1.4.2
pluggy==0.6.0
ply==3.11
prompt-toolkit==1.0.15
protobuf==3.6.0
psutil==5.4.5
ptyprocess==0.5.2
py==1.5.3
pycairo==1.15.4
pycodestyle==2.4.0
pycosat==0.6.3
pycparser==2.18
pycrypto==2.6.1
pycurl==7.43.0.1
pyflakes==1.6.0
Pygments==2.2.0
pylint==1.8.4
PyMySQL==0.9.2
pyodbc==4.0.23
pyOpenSSL==17.5.0
pyparsing==2.2.0
PySocks==1.6.8
pytest==3.5.0
python-dateutil==2.7.2
pytz==2018.4
PyWavelets==0.5.2
PyYAML==3.12
pyzmq==17.0.0
QtAwesome==0.4.4
qtconsole==4.3.1
QtPy==1.4.0
requests==2.18.4
rope==0.10.7
ruamel-yaml==0.15.35
scandir==1.7
scikit-image==0.13.1
scikit-learn==0.19.1
scipy==1.2.2
seaborn==0.8.1
Send2Trash==1.5.0
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.11.0
snowballstemmer==1.2.1
sortedcollections==0.6.1
sortedcontainers==1.5.10
Sphinx==1.7.3
sphinxcontrib-websupport==1.0.1
spyder==3.3.0
spyder-kernels==0.2.4
SQLAlchemy==1.2.7
statsmodels==0.8.0
subprocess32==3.2.7
sympy==1.1.1
tables==3.4.3
tblib==1.3.2
terminado==0.8.1
testpath==0.3.1
toolz==0.9.0
tornado==5.0.2
traitlets==4.3.2
typing==3.6.4
unicodecsv==0.14.1
urllib3==1.22
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.14.1
widgetsnbextension==3.2.1
wrapt==1.10.11
xlrd==1.1.0
XlsxWriter==1.0.4
xlwt==1.3.0
zict==0.1.3

Installed Python3.6 Modules


alabaster==0.7.10
anaconda-client==1.6.14
anaconda-navigator==1.8.3
anaconda-project==0.8.2
asn1crypto==0.24.0
astroid==1.6.3
astropy==3.0.2
attrs==17.4.0
Babel==2.5.3
backcall==0.1.0
backports.shutil-get-terminal-size==1.0.0
beautifulsoup4==4.6.0
bitarray==0.8.1
bkcharts==0.2
blaze==0.11.3
bleach==2.1.3
bokeh==0.12.15
boto==2.48.0
Bottleneck==1.2.1
certifi==2018.10.15
cffi==1.11.5
chardet==3.0.4
click==6.7
cloudpickle==0.5.2
clyent==1.2.2
colorama==0.3.9
conda==4.5.11
conda-build==3.9.1
conda-verify==2.0.0
contextlib2==0.5.5
cryptography==2.2.2
cycler==0.10.0
Cython==0.28.2
cytoolz==0.9.0.1
dask==0.17.2
datashape==0.5.4
decorator==4.3.0
distributed==1.21.6
docutils==0.14
entrypoints==0.2.3
et-xmlfile==1.0.1
fastcache==1.0.2
filelock==3.0.4
Flask==0.12.2
Flask-Cors==3.0.3
gevent==1.2.2
glob2==0.6
gmpy2==2.0.8
greenlet==0.4.13
h5py==2.7.1
heapdict==1.0.0
html5lib==1.0.1
idna==2.6
imageio==2.3.0
imagesize==1.0.0
ipykernel==4.8.2
ipython==6.3.1
ipython-genutils==0.2.0
ipywidgets==7.2.1
isort==4.3.4
itsdangerous==0.24
jdcal==1.4
jedi==0.12.0
jeepney==0.3.1
Jinja2==2.10
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
jupyterlab==0.32.0
jupyterlab-launcher==0.10.5
keyring==13.2.1
kiwisolver==1.0.1
lazy-object-proxy==1.3.1
llvmlite==0.22.0
locket==0.2.0
lxml==4.2.1
MarkupSafe==1.0
matplotlib==2.2.2
mccabe==0.6.1
mistune==0.8.3
mkl-fft==1.0.0
mkl-random==1.0.1
more-itertools==4.1.0
mpmath==1.0.0
msgpack-python==0.5.6
multipledispatch==0.5.0
mysql-connector-python==8.0.12
navigator-updater==0.1.0
nbconvert==5.3.1
nbformat==4.4.0
networkx==2.1
nltk==3.2.5
nose==1.3.7
notebook==5.4.1
numba==0.37.0
numexpr==2.6.4
numpy==1.14.2
numpydoc==0.8.0
odo==0.5.1
olefile==0.45.1
openpyxl==2.5.3
packaging==17.1
pandas==0.22.0
pandocfilters==1.4.2
parso==0.2.0
partd==0.3.8
path.py==11.0.1
pathlib2==2.3.2
patsy==0.5.0
pep8==1.7.1
pexpect==4.5.0
pickleshare==0.7.4
Pillow==5.1.0
pkginfo==1.4.2
pluggy==0.6.0
ply==3.11
prompt-toolkit==1.0.15
protobuf==3.6.0
psutil==5.4.5
ptyprocess==0.5.2
py==1.5.3
pycodestyle==2.4.0
pycosat==0.6.3
pycparser==2.18
pycrypto==2.6.1
pycurl==7.43.0.1
pyflakes==1.6.0
Pygments==2.2.0
pylint==1.8.4
PyMySQL==0.9.2
pyodbc==4.0.23
pyOpenSSL==17.5.0
pyparsing==2.2.0
PySocks==1.6.8
pytest==3.5.0
pytest-arraydiff==0.2
pytest-astropy==0.3.0
pytest-doctestplus==0.1.3
pytest-openfiles==0.3.0
pytest-remotedata==0.2.1
python-dateutil==2.7.2
pytz==2018.4
PyWavelets==0.5.2
PyYAML==3.12
pyzmq==17.0.0
QtAwesome==0.4.4
qtconsole==4.3.1
QtPy==1.4.0
requests==2.18.4
rope==0.10.7
ruamel-yaml==0.15.35
scikit-image==0.13.1
scikit-learn==0.19.1
scipy==1.3.0
seaborn==0.8.1
SecretStorage==3.0.1
Send2Trash==1.5.0
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.11.0
snowballstemmer==1.2.1
sortedcollections==0.6.1
sortedcontainers==1.5.10
Sphinx==1.7.3
sphinxcontrib-websupport==1.0.1
spyder==3.3.0
spyder-kernels==0.2.4
SQLAlchemy==1.2.7
statsmodels==0.8.0
sympy==1.1.1
tables==3.4.3
tblib==1.3.2
terminado==0.8.1
testpath==0.3.1
toolz==0.9.0
tornado==5.0.2
traitlets==4.3.2
typing==3.6.4
unicodecsv==0.14.1
urllib3==1.22
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.14.1
widgetsnbextension==3.2.1
wrapt==1.10.11
xlrd==1.1.0
XlsxWriter==1.0.4
xlwt==1.3.0
zict==0.1.3