'UNSW - Science

UNSW - Science - HPC

Python

The system version of Python on Katana (version 2.6.6) is installed in /usr/bin and will work for simple Python programs.

For more complicated Python programs including those requiring specific Python modules such as BioPython, NumPy and SciPy we have installed several different versions of Python which are available via the module command and are listed at the bottom of the page.

You can request a Python module to be installed by sending a request to the IT Service Centre but you can install it for yourself using the Virtual Environment and PIP commands as described in the section below.

Note:If you wish to use one of the Python modules for your own code (or for code that you have obtained) you should make sure that the first line is

#!/usr/bin/env python

rather than

#!/usr/bin/python

otherwise it will use the system version rather than the version you want.

If you wish to use the Anaconda Python implementation then the best option is to install it yourself by following the detailed instructions on the installing software for yourself page.

Installing Python Libraries For Yourself using Virtualenv using Python 2

You can install additional Python libraries for yourself by using the virtualenv command to creates a new environment
in a directory you specify and then using the pip command as well as the standard installation methods. The full process is as follows:

Creating a Virtual Environment

Set up the directory where we will be keeping all of our Python modules, in this case mypythonenv.

[z1234567@katana ~]$ mkdir mypythonenv

We now load the python module and set up the virtual environment called mypythonenv.

[z1234567@katana ~]$ module load python/2.7.9
[z1234567@katana ~]$ virtualenv --system-site-packages mypythonenv

Using a Previously Created Virtual Environment

Once you have created your Virtual Environment you must be activate it prior to use. This is done by sourcing the activate file in the following way.

[z1234567@katana ~]$ . mypythonenv/bin/activate
(mypythonenv)[z1234567@katana ~]$ 

Once the environment is loaded the environment name in parenthesis will be at the start of your command prompt and you have access to your own version of Python.

In particular you can then use

(mypythonenv)[z1234567@katana ~]$ pip install

or standard Python methods like typing

(mypythonenv)[z1234567@katana ~]$ python setup.py install

to install Python modules
that are not already installed on Katana without asking for an administrator to do it for you.

For example to install the Python module pymongodb you would first check to see what Python modules are already installed

(mypythonenv)[z1234567@katana ~]$ pip freeze

and after confirming that pymongodb was not on the list you would type

(mypythonenv)[z1234567@katana ~]$ pip install pymongodb

to install it. Then whenever you loaded the environment using

[z1234567@katana ~]$ . mypythonenv/bin/activate
(mypythonenv)[z1234567@katana ~]$ 

pip freeze would tell you that pymongodb is available.

Leaving the Virtual Environment

Once you have finished using your personal version of Python you exit the virtual environment using the deactivate command.

(mypythonenv)[z1234567@katana ~]$ deactivate
[z1234567@katana ~]$

More Information on the PIP Command

PIP allows you to install a python package for yourself by using the command

pip install some-package-name

You can also remove the python module by typing:

pip uninstall some-package-name

A full user guide for PIP is available at http://www.pip-installer.org/en/latest/user_guide.html.

A Full Example

Here is the full collection of commands to install an updated version of a packages that is already installed.

[z1234567@katana ~]$ which checkm 
/usr/bin/which: no checkm in (...)
[z1234567@katana ~]$ module add python/2.7.10
[z1234567@katana ~]$ which checkm 
/share/apps/python/2.7.10/bin/checkm
[z1234567@katana ~]$ pip freeze | grep checkm
checkm-genome==1.0.3

At this point if you are happy with that version of CheckM there is no need to set up and use a Virtual Environment you don't need to do anything else. If that version is not recent enough then you can install an updated version as described below.

[z1234567@katana ~]$ mkdir mypythonenv
[z1234567@katana ~]$ virtualenv --system-site-packages mypythonenv
New python executable in mypythonenv/bin/python
Installing setuptools, pip, wheel...done.
[z1234567@katana ~]$ . mypythonenv/bin/activate
(mypythonenv)[z1234567@katana ~]$ pip freeze | grep checkm
checkm-genome==1.0.3
(mypythonenv)[z1234567@katana ~]$ which checkm
/share/apps/python/2.7.10/bin/checkm
(mypythonenv)[z1234567@katana ~]$ pip install --upgrade --no-deps checkm-genome
Collecting checkm-genome
  Using cached checkm-genome-1.0.5.tar.gz
Building wheels for collected packages: checkm-genome
  Running setup.py bdist_wheel for checkm-genome
  Stored in directory: /home/z1234567/.cache/pip/wheels/5c/af/88/a63a69a33e60edf894f2664823153c5aa54578
Successfully built checkm-genome
Installing collected packages: checkm-genome
  Found existing installation: checkm-genome 1.0.3
    Not uninstalling checkm-genome at /share/apps/python/2.7.10/lib/python2.7/site-packages, 
    outside environment /home/z1234567/mypythonenv
Successfully installed checkm-genome-1.0.5
(mypythonenv)[z1234567@katana ~]$ which checkm
~/mypythonenv/bin/checkm
(mypythonenv)[z1234567@katana ~]$ pip freeze | grep checkm
checkm-genome==1.0.5

Installing Python Libraries For Yourself using Virtualenv using Python 3

In Python 3 the virtualenv command has been replaced by pyvenv which uses the Python 3 venv module which works in the same way. As always with Python 3 you will need to use python3, pip3, etc. rather than the Python 2 versions of the commands.

The full process is as follows:

Creating a Virtual Environment

Set up the directory where we will be keeping all of our Python modules, in this case mypython3env.

[z1234567@katana ~]$ mkdir mypython3env

We now load the python module and set up the virtual environment called mypython3env.

[z1234567@katana ~]$ module load python/3.4.3
[z1234567@katana ~]$ pyvenv mypython3env

Using a Previously Created Virtual Environment

Once you have created your Virtual Environment you must be activate it prior to use. This is done by sourcing the activate file in the following way.

[z1234567@katana ~]$ . mypython3env/bin/activate
(mypython3env)[z1234567@katana ~]$ 

Once the environment is loaded the environment name in parenthesis will be at the start of your command prompt and you have access to your own version of Python.

In particular you can then use

(mypython3env)[z1234567@katana ~]$ pip3 install

or standard Python methods like typing

(mypython3env)[z1234567@katana ~]$ python3 setup.py install

to install Python modules
that are not already installed on Katana without asking for an administrator to do it for you.

For example to install the Python 3 module pymongodb you would first check to see what Python modules are already installed

(mypython3env)[z1234567@katana ~]$ pip3 freeze

and after confirming that pymongodb was not on the list you would type

(mypython3env)[z1234567@katana ~]$ pip3 install pymongodb

to install it. Then whenever you loaded the environment using

[z1234567@katana ~]$ . mypython3env/bin/activate
(mypython3env)[z1234567@katana ~]$ 

pip freeze would tell you that pymongodb is available.

Leaving the Virtual Environment

Once you have finished using your personal version of Python you exit the virtual environment using the deactivate command.

(mypython3env)[z1234567@katana ~]$ deactivate
[z1234567@katana ~]$

More Information on the PIP3 Command

PIP3 (the version of PIP for Python 3) allows you to install a python package for yourself by using the command

pip3 install some-package-name

You can also remove the python module by typing:

pip3 uninstall some-package-name

A full user guide for PIP is available at http://www.pip-installer.org/en/latest/user_guide.html.

A Full Example

Here is the full collection of commands to install a package called TensorFlow using PIP3.

[z1234567@katana ~]$ module load python/3.4.3
[z1234567@katana ~]$ pyvenv mypython3env
[z1234567@katana ~]$ . mypython3env/bin/activate
(mypython3env) [z1234567@katana ~]$  export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.9.0-cp34-cp34m-linux_x86_64.whl
(mypython3env) [z1234567@katana ~]$ pip3 install $TF_BINARY_URL
Collecting tensorflow==0.9.0 from https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.9.0-cp34-cp34m-linux_x86_64.whl
  Using cached https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.9.0-cp34-cp34m-linux_x86_64.whl
Collecting protobuf==3.0.0b2 (from tensorflow==0.9.0)
  Using cached protobuf-3.0.0b2-py2.py3-none-any.whl
Collecting six>=1.10.0 (from tensorflow==0.9.0)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting wheel>=0.26 (from tensorflow==0.9.0)
  Using cached wheel-0.29.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.8.2 in /share/apps/python/3.4.3/lib/python3.4/site-packages (from tensorflow==0.9.0)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /share/apps/python/3.4.3/lib/python3.4/site-packages (from protobuf==3.0.0b2->tensorflow==0.9.0)
Installing collected packages: six, protobuf, wheel, tensorflow
  Found existing installation: six 1.9.0
    Not uninstalling six at /share/apps/python/3.4.3/lib/python3.4/site-packages, outside environment /home/z1234567/mypython3env
Successfully installed protobuf-3.0.0b2 six-1.9.0 tensorflow-0.9.0 wheel-0.29.0                                                                                                                              
You are using pip version 7.1.2, however version 8.1.2 is available.                                                                                                                                         
You should consider upgrading via the 'pip install --upgrade pip' command.                                                                                                                                   

As we didn't specify that we wanted to upgrade already installed Python modules we aren't using the latest possible version of everything. So we can remove the directory we just created and try again.

(mypython3env) [z1234567@katana ~]$ deactivate                
[z1234567@katana ~]$ rm -fr mypython3env
 
[z1234567@katana ~]$ pyvenv mypython3env
[z1234567@katana ~]$ . mypython3env/bin/activate
(mypython3env) [z1234567@katana ~]$ export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.9.0-cp34-cp34m-linux_x86_64.whl
(mypython3env) [z1234567@katana ~]$ pip3 install --upgrade $TF_BINARY_URL
Collecting tensorflow==0.9.0 from https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.9.0-cp34-cp34m-linux_x86_64.whl
  Using cached https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.9.0-cp34-cp34m-linux_x86_64.whl (27.6MB)
Collecting numpy>=1.8.2 (from tensorflow==0.9.0)
  Using cached numpy-1.11.1.zip (4.7MB)
Collecting six>=1.10.0 (from tensorflow==0.9.0)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting wheel>=0.26 (from tensorflow==0.9.0)
  Using cached wheel-0.29.0-py2.py3-none-any.whl (66kB)
Collecting protobuf==3.0.0b2 (from tensorflow==0.9.0)
  Using cached protobuf-3.0.0b2-py2.py3-none-any.whl (326kB)
Collecting setuptools (from protobuf==3.0.0b2->tensorflow==0.9.0)
  Using cached setuptools-24.0.2-py2.py3-none-any.whl (441kB)
Installing collected packages: numpy, six, wheel, setuptools, protobuf, tensorflow
  Found existing installation: numpy 1.9.2
    DEPRECATION: Uninstalling a distutils installed project (numpy) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Not uninstalling numpy at /share/apps/python/3.4.3/lib/python3.4/site-packages, outside environment /home/z1234567/mypython3env
  Running setup.py install for numpy
  Found existing installation: six 1.9.0
    Not uninstalling six at /share/apps/python/3.4.3/lib/python3.4/site-packages, outside environment /home/z1234567/mypython3env
  Found existing installation: setuptools 12.0.5
    Not uninstalling setuptools at /share/apps/python/3.4.3/lib/python3.4/site-packages, outside environment /home/z1234567/mypython3env
Successfully installed numpy-1.9.2 protobuf-3.0.0b2 setuptools-12.0.5 six-1.9.0 tensorflow-0.9.0 wheel-0.29.0
You are using pip version 7.1.2, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(mypython3env) [z1234567@katana ~]$ 

Python Versions Available via the Module Command

Software Name Version
BioPython 1.61 BioPython 1.61
BioPython 1.62 BioPython 1.62
BioPython 1.64 BioPython 1.64
BioPython 1.65 BioPython 1.65
BioPython 1.65-python-2.7.10 BioPython 1.65-python-2.7.10
BioPython 1.68 BioPython 1.68
BioPython 2.7.12 BioPython 2.7.12
Python 2.7.10 Python 2.7.10
Python 2.7.12 Python 2.7.12
Python 2.7.12-gcc6 Python 2.7.12-gcc6
Python 2.7.3 Python 2.7.3
Python 2.7.6 Python 2.7.6
Python 2.7.8 Python 2.7.8
Python 2.7.9 Python 2.7.9
Python 3.4.3 Python 3.4.3
Python 3.5.2 Python 3.5.2