Python, R, Packages, and the Grid

Contents

 

Intro

PLEASE NOTE that installation of conda and miniconda in home directories is no longer necessary for Grid 2.5. Please contact RCS if you have any questions about the versions of R and Python that are installed on Grid 2.5. In addition, if you have used a conda or miniconda installation out of your home directory, we counsel you to transition to using the central installations of R, Python2, and Python3.

Compute Grid 2.5 is using Anaconda, the full distribution, and miniConda, a slimmed down version, to provide feature-rich environments for Python and R, respectively. This page briefly discusses some need-to-knows.

For the so-inclined or for those wishing more advanced capabilities, see Conda's Overview guide for general information, and their Managing... pages if you wish to use more of the advanced features. Please note that our central installations are read-only, so one would need to make any changes locally to home or project folders.

Python

Compute Grid 2.5 offers both Python2 (v2.7.x) and Python3 (v3.6.x) via the Anaconda distribution. This provides CPU-optimized versions of Python and its supporting libraries, including numpy, scipy, matplotlib, pandas, & scikitlearn, among others.

Python2 is still our default, so the commands python or python2 will both run python 2.7.x. For Python3, please use the command python3.

Installing custom Python modules

If you require a Python module that is not installed with the central Anaconda installation, you can install this yourself, and the module will be placed in a directory in your home folder. Due to the wrapper scripts that are installed on login nodes, this action must be performed via the back-end, compute nodes.

1. Set up an alias so it's easy to submit interactive jobs to back-end nodes:

alias my_run="bsub -app python-5g -q short_int -Ip"

2. Install your Python module by prefixing the install command with my_run:

my_run python -m pip install --user SomeModule

OR, If you are using Python3, use the python3 command instead:

my_run python3 -m pip3 install --user SomeModule

The modules will be placed in the directory $HOME/.local for use by your scripts and programs. If you are upgrading modules that are already installed centrally, insert --upgrade before the --user option.

R

Compute Grid 2.5 offers both R and RStudio via the miniConda distribution. This provides CPU-optimized versions of R and its supporting libraries from CRAN.

Installing custom R packages

No special instructions are needed for this. Using the install.packages() command from within R or RStudio will download and install the specified packages in your home folder by default.

Environments

One of Anaconda's most useful features is the ability to create virtual environments. This is particularly helpful if you have multiple projects that depend on different versions of packages. With a virtual environment you can update the packages for one project without disturbing the packages of your other projects. Conda's documentation on managing environments is a good place to learn about this feature.

Installed Packages & Modules

Our base installations of R, Python2, and Python3 have the packages and modules listed below as a part of the central installation. You can use the instructions above to install additional or updates ones in either your home folder (the default location) or in another location that you must manually specify.

Installed R3.5 Packages

abind         "1.4-5"      NA           
assertthat    "0.2.0"      NA           
backports     "1.1.2"      NA           
base          "3.5.1"      "base"       
base64enc     "0.1-3"      NA           
BH            "1.66.0-1"   NA           
bindr         "0.1.1"      NA           
bindrcpp      "0.2.2"      NA           
bit           "1.1-14"     NA           
bit64         "0.9-7"      NA           
bit64.bak     "0.9-7"      NA           
bitops        "1.0-6"      NA           
blob          "1.1.1"      NA           
boot          "1.3-20"     "recommended"
broom         "0.5.0"      NA           
callr         "2.0.4"      NA           
caret         "6.0-80"     NA           
caTools       "1.17.1"     NA           
cellranger    "1.1.0"      NA           
checkpoint    "0.4.4"      NA           
class         "7.3-14"     NA           
cli           "1.0.0"      NA           
clipr         "0.4.1"      NA           
cluster       "2.0.7-1"    NA           
codetools     "0.2-15"     NA           
colorspace    "1.3-2"      NA           
compiler      "3.5.1"      "base"       
config        "0.3"        NA           
crayon        "1.3.4"      NA           
curl          "3.2"        NA           
CVST          "0.2-2"      NA           
data.table    "1.11.4"     NA           
datasets      "3.5.1"      "base"       
DBI           "1.0.0"      NA           
dbplyr        "1.2.2"      NA           
ddalpha       "1.3.4"      NA           
DEoptimR      "1.0-8"      NA           
deployrRserve "9.0.0"      NA           
dichromat     "2.0-0"      NA           
digest        "0.6.15"     NA           
dimRed        "0.1.0"      NA           
doParallel    "1.0.13"     NA           
dplyr         "0.7.6"      NA           
DRR           "0.0.3"      NA           
evaluate      "0.11"       NA           
fansi         "0.2.3"      NA           
forcats       "0.3.0"      NA           
foreach       "1.5.0"      NA           
foreign       "0.8-71"     "recommended"
formatR       "1.5"        NA           
geometry      "0.3-6"      NA           
ggplot2       "3.0.0"      NA           
glmnet        "2.0-16"     NA           
glue          "1.3.0"      NA           
gower         "0.1.2"      NA           
graphics      "3.5.1"      "base"       
grDevices     "3.5.1"      "base"       
grid          "3.5.1"      "base"       
gtable        "0.2.0"      NA           
haven         "1.1.2"      NA           
hexbin        "1.27.2"     NA           
highr         "0.7"        NA           
hms           "0.4.2"      NA           
htmltools     "0.3.6"      NA           
htmlwidgets   "0.9"        NA           
httpuv        "1.4.5"      NA           
httr          "1.3.1"      NA           
ipred         "0.9-6"      NA           
IRdisplay     "0.5.0"      NA           
IRkernel      "0.8.11"     NA           
iterators     "1.0.10"     NA           
jsonlite      "1.5"        NA           
kernlab       "0.9-26"     NA           
KernSmooth    "2.23-15"    NA           
knitr         "1.20"       NA           
labeling      "0.3"        NA           
later         "0.7.3"      NA           
lattice       "0.20-35"    NA           
lava          "1.6.2"      NA           
lazyeval      "0.2.1"      NA           
lubridate     "1.7.4"      NA           
magic         "1.5-8"      NA           
magrittr      "1.5"        NA           
maps          "3.3.0"      NA           
markdown      "0.8"        NA           
MASS          "7.3-50"     "recommended"
Matrix        "1.2-14"     NA           
methods       "3.5.1"      "base"       
mgcv          "1.8-24"     "recommended"
MicrosoftR    "3.5.1.3"    NA           
mime          "0.5"        NA           
miniUI        "0.1.1"      NA           
mnormt        "1.5-5"      NA           
ModelMetrics  "1.1.0"      NA           
modelr        "0.1.2"      NA           
mongolite     "1.6"        NA           
munsell       "0.4.3"      NA           
nlme          "3.1-137"    NA           
nnet          "7.3-12"     NA           
numDeriv      "2016.8-1"   NA           
odbc          "1.1.3"      NA           
openssl       "1.0.2"      NA           
packrat       "0.4.9-3"    NA           
parallel      "3.5.1"      "base"       
pbdZMQ        "0.3-3"      NA           
pillar        "1.3.0"      NA           
pkgconfig     "2.0.1"      NA           
PKI           "0.1-5.1"    NA           
plogr         "0.2.0"      NA           
pls           "2.6-0"      NA           
plyr          "1.8.4"      NA           
png           "0.1-7"      NA           
praise        "1.0.0"      NA           
prettyunits   "1.0.2"      NA           
processx      "3.1.0"      NA           
prodlim       "2018.04.18" NA           
profvis       "0.3.4"      NA           
promises      "1.0.1"      NA           
psych         "1.7.8"      NA           
purrr         "0.2.5"      NA           
quantmod      "0.4-13"     NA           
R6            "2.2.2"      NA           
randomForest  "4.6-14"     NA           
rappdirs      "0.3.1"      NA           
rbokeh        "0.6.3"      NA           
RColorBrewer  "1.1-2"      NA           
Rcpp          "0.12.18"    NA           
RcppRoll      "0.3.0"      NA           
RCurl         "1.95-4.9"   NA           
readr         "1.1.1"      NA           
readxl        "1.1.0"      NA           
recipes       "0.1.1"      NA           
rematch       "1.0.1"      NA           
repr          "0.12.0"     NA           
reprex        "0.1.1"      NA           
reshape2      "1.4.3"      NA           
RevoIOQ       "10.0.0"     NA           
RevoMods      "11.0.1"     NA           
RevoUtils     "11.0.1"     NA           
RevoUtilsMath "11.0.0"     NA           
rJava         "0.9-9"      NA           
RJDBC         "0.2-5"      NA           
RJSONIO       "1.3-0"      NA           
rlang         "0.2.1"      NA           
RMariaDB      "1.0.6"      NA           
rmarkdown     "1.10"       NA           
robustbase    "0.93-2"     NA           
rpart         "4.1-13"     NA           
rprojroot     "1.3-1"      NA           
rsconnect     "0.8.8"      NA           
rstudioapi    "0.7"        NA           
RUnit         "0.4.32"     NA           
rvest         "0.3.2"      NA           
scales        "0.5.0"      NA           
selectr       "0.4-1"      NA           
sfsmisc       "1.1-2"      NA           
shiny         "1.1.0"      NA           
sourcetools   "0.1.7"      NA           
sparklyr      "0.6.4"      NA           
spatial       "7.3-11"     "recommended"
splines       "3.5.1"      "base"       
SQUAREM       "2017.10-1"  NA           
stats         "3.5.1"      "base"       
stats4        "3.5.1"      "base"       
stringi       "1.2.4"      NA           
stringr       "1.3.1"      NA           
survival      "2.42-6"     "recommended"
tcltk         "3.5.1"      "base"       
testthat      "2.0.0"      NA           
tibble        "1.4.2"      NA           
tidyr         "0.8.1"      NA           
tidyselect    "0.2.4"      NA           
tidyverse     "1.2.1"      NA           
timeDate      "3043.102"   NA           
tinytex       "0.6"        NA           
tools         "3.5.1"      "base"       
TTR           "0.23-3"     NA           
utf8          "1.1.4"      NA           
utils         "3.5.1"      "base"       
uuid          "0.1-2"      NA           
viridisLite   "0.3.0"      NA           
whisker       "0.3-2"      NA           
withr         "2.1.2"      NA           
xfun          "0.3"        NA           
xml2          "1.2.0"      NA           
xtable        "1.8-2"      NA           
xts           "0.11-0"     NA           
yaml          "2.2.0"      NA           
zoo           "1.8-3"      NA   

Installed Python2.7 Modules


alabaster==0.7.10
anaconda-client==1.6.14
anaconda-navigator==1.8.3
anaconda-project==0.8.2
asn1crypto==0.24.0
astroid==1.6.3
astropy==2.0.6
attrs==17.4.0
Babel==2.5.3
backports-abc==0.5
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
backports.ssl-match-hostname==3.5.0.1
beautifulsoup4==4.6.0
bitarray==0.8.1
bkcharts==0.2
blaze==0.11.3
bleach==2.1.3
bokeh==0.12.15
boto==2.48.0
Bottleneck==1.2.1
cdecimal==2.3
certifi==2018.10.15
cffi==1.11.5
chardet==3.0.4
click==6.7
cloudpickle==0.5.2
clyent==1.2.2
colorama==0.3.9
conda==4.5.11
conda-build==3.9.1
conda-verify==2.0.0
configparser==3.5.0
contextlib2==0.5.5
cryptography==2.2.2
cycler==0.10.0
Cython==0.28.2
cytoolz==0.9.0.1
dask==0.17.2
datashape==0.5.4
decorator==4.3.0
distributed==1.21.6
docutils==0.14
entrypoints==0.2.3
enum34==1.1.6
et-xmlfile==1.0.1
fastcache==1.0.2
filelock==3.0.4
Flask==0.12.2
Flask-Cors==3.0.3
funcsigs==1.0.2
functools32==3.2.3.post2
futures==3.2.0
gevent==1.2.2
glob2==0.6
gmpy2==2.0.8
greenlet==0.4.13
grin==1.2.1
h5py==2.7.1
heapdict==1.0.0
html5lib==1.0.1
idna==2.6
imageio==2.3.0
imagesize==1.0.0
ipaddress==1.0.22
ipykernel==4.8.2
ipython==5.6.0
ipython-genutils==0.2.0
ipywidgets==7.2.1
isort==4.3.4
itsdangerous==0.24
jdcal==1.4
jedi==0.12.0
Jinja2==2.10
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
jupyterlab==0.32.0
jupyterlab-launcher==0.10.5
keyring==13.2.1
kiwisolver==1.0.1
lazy-object-proxy==1.3.1
llvmlite==0.22.0
locket==0.2.0
log0==1.0.1
lxml==4.2.1
MarkupSafe==1.0
matplotlib==2.2.2
mccabe==0.6.1
mistune==0.8.3
mkl-fft==1.0.0
mkl-random==1.0.1
more-itertools==4.1.0
mpmath==1.0.0
msgpack-python==0.5.6
multipledispatch==0.5.0
mysql-connector-python==8.0.12
navigator-updater==0.1.0
nbconvert==5.3.1
nbformat==4.4.0
networkx==2.1
nltk==3.2.5
nose==1.3.7
notebook==5.4.1
numba==0.37.0+0.gb6b965afa.dirty
numexpr==2.6.4
numpy==1.16.4
numpydoc==0.8.0
odo==0.5.1
olefile==0.45.1
openpyxl==2.5.3
packaging==17.1
pandas==0.22.0
pandocfilters==1.4.2
parso==0.2.0
partd==0.3.8
path.py==11.0.1
pathlib2==2.3.2
patsy==0.5.0
pep8==1.7.1
pexpect==4.5.0
pickleshare==0.7.4
Pillow==5.1.0
pkginfo==1.4.2
pluggy==0.6.0
ply==3.11
prompt-toolkit==1.0.15
protobuf==3.6.0
psutil==5.4.5
ptyprocess==0.5.2
py==1.5.3
pycairo==1.15.4
pycodestyle==2.4.0
pycosat==0.6.3
pycparser==2.18
pycrypto==2.6.1
pycurl==7.43.0.1
pyflakes==1.6.0
Pygments==2.2.0
pylint==1.8.4
PyMySQL==0.9.2
pyodbc==4.0.23
pyOpenSSL==17.5.0
pyparsing==2.2.0
PySocks==1.6.8
pytest==3.5.0
python-dateutil==2.7.2
pytz==2018.4
PyWavelets==0.5.2
PyYAML==3.12
pyzmq==17.0.0
QtAwesome==0.4.4
qtconsole==4.3.1
QtPy==1.4.0
requests==2.18.4
rope==0.10.7
ruamel-yaml==0.15.35
scandir==1.7
scikit-image==0.13.1
scikit-learn==0.19.1
scipy==1.2.2
seaborn==0.8.1
Send2Trash==1.5.0
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.11.0
snowballstemmer==1.2.1
sortedcollections==0.6.1
sortedcontainers==1.5.10
Sphinx==1.7.3
sphinxcontrib-websupport==1.0.1
spyder==3.3.0
spyder-kernels==0.2.4
SQLAlchemy==1.2.7
statsmodels==0.8.0
subprocess32==3.2.7
sympy==1.1.1
tables==3.4.3
tblib==1.3.2
terminado==0.8.1
testpath==0.3.1
toolz==0.9.0
tornado==5.0.2
traitlets==4.3.2
typing==3.6.4
unicodecsv==0.14.1
urllib3==1.22
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.14.1
widgetsnbextension==3.2.1
wrapt==1.10.11
xlrd==1.1.0
XlsxWriter==1.0.4
xlwt==1.3.0
zict==0.1.3

Installed Python3.6 Modules


alabaster==0.7.10
anaconda-client==1.6.14
anaconda-navigator==1.8.3
anaconda-project==0.8.2
asn1crypto==0.24.0
astroid==1.6.3
astropy==3.0.2
attrs==17.4.0
Babel==2.5.3
backcall==0.1.0
backports.shutil-get-terminal-size==1.0.0
beautifulsoup4==4.6.0
bitarray==0.8.1
bkcharts==0.2
blaze==0.11.3
bleach==2.1.3
bokeh==0.12.15
boto==2.48.0
Bottleneck==1.2.1
certifi==2018.10.15
cffi==1.11.5
chardet==3.0.4
click==6.7
cloudpickle==0.5.2
clyent==1.2.2
colorama==0.3.9
conda==4.5.11
conda-build==3.9.1
conda-verify==2.0.0
contextlib2==0.5.5
cryptography==2.2.2
cycler==0.10.0
Cython==0.28.2
cytoolz==0.9.0.1
dask==0.17.2
datashape==0.5.4
decorator==4.3.0
distributed==1.21.6
docutils==0.14
entrypoints==0.2.3
et-xmlfile==1.0.1
fastcache==1.0.2
filelock==3.0.4
Flask==0.12.2
Flask-Cors==3.0.3
gevent==1.2.2
glob2==0.6
gmpy2==2.0.8
greenlet==0.4.13
h5py==2.7.1
heapdict==1.0.0
html5lib==1.0.1
idna==2.6
imageio==2.3.0
imagesize==1.0.0
ipykernel==4.8.2
ipython==6.3.1
ipython-genutils==0.2.0
ipywidgets==7.2.1
isort==4.3.4
itsdangerous==0.24
jdcal==1.4
jedi==0.12.0
jeepney==0.3.1
Jinja2==2.10
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
jupyterlab==0.32.0
jupyterlab-launcher==0.10.5
keyring==13.2.1
kiwisolver==1.0.1
lazy-object-proxy==1.3.1
llvmlite==0.22.0
locket==0.2.0
lxml==4.2.1
MarkupSafe==1.0
matplotlib==2.2.2
mccabe==0.6.1
mistune==0.8.3
mkl-fft==1.0.0
mkl-random==1.0.1
more-itertools==4.1.0
mpmath==1.0.0
msgpack-python==0.5.6
multipledispatch==0.5.0
mysql-connector-python==8.0.12
navigator-updater==0.1.0
nbconvert==5.3.1
nbformat==4.4.0
networkx==2.1
nltk==3.2.5
nose==1.3.7
notebook==5.4.1
numba==0.37.0
numexpr==2.6.4
numpy==1.14.2
numpydoc==0.8.0
odo==0.5.1
olefile==0.45.1
openpyxl==2.5.3
packaging==17.1
pandas==0.22.0
pandocfilters==1.4.2
parso==0.2.0
partd==0.3.8
path.py==11.0.1
pathlib2==2.3.2
patsy==0.5.0
pep8==1.7.1
pexpect==4.5.0
pickleshare==0.7.4
Pillow==5.1.0
pkginfo==1.4.2
pluggy==0.6.0
ply==3.11
prompt-toolkit==1.0.15
protobuf==3.6.0
psutil==5.4.5
ptyprocess==0.5.2
py==1.5.3
pycodestyle==2.4.0
pycosat==0.6.3
pycparser==2.18
pycrypto==2.6.1
pycurl==7.43.0.1
pyflakes==1.6.0
Pygments==2.2.0
pylint==1.8.4
PyMySQL==0.9.2
pyodbc==4.0.23
pyOpenSSL==17.5.0
pyparsing==2.2.0
PySocks==1.6.8
pytest==3.5.0
pytest-arraydiff==0.2
pytest-astropy==0.3.0
pytest-doctestplus==0.1.3
pytest-openfiles==0.3.0
pytest-remotedata==0.2.1
python-dateutil==2.7.2
pytz==2018.4
PyWavelets==0.5.2
PyYAML==3.12
pyzmq==17.0.0
QtAwesome==0.4.4
qtconsole==4.3.1
QtPy==1.4.0
requests==2.18.4
rope==0.10.7
ruamel-yaml==0.15.35
scikit-image==0.13.1
scikit-learn==0.19.1
scipy==1.3.0
seaborn==0.8.1
SecretStorage==3.0.1
Send2Trash==1.5.0
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.11.0
snowballstemmer==1.2.1
sortedcollections==0.6.1
sortedcontainers==1.5.10
Sphinx==1.7.3
sphinxcontrib-websupport==1.0.1
spyder==3.3.0
spyder-kernels==0.2.4
SQLAlchemy==1.2.7
statsmodels==0.8.0
sympy==1.1.1
tables==3.4.3
tblib==1.3.2
terminado==0.8.1
testpath==0.3.1
toolz==0.9.0
tornado==5.0.2
traitlets==4.3.2
typing==3.6.4
unicodecsv==0.14.1
urllib3==1.22
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.14.1
widgetsnbextension==3.2.1
wrapt==1.10.11
xlrd==1.1.0
XlsxWriter==1.0.4
xlwt==1.3.0
zict==0.1.3