What factors determine how fast my job runs?
The speed of a HPC job is determined by a number of different factors. The following list contains the factors that will have the biggest impact on how fast you job will run as well as the best place for your job to run.
- Is my job very I/O intensive? If so you should copy your datasets down to local scratch at the start of your job so that it can take advantage of local disk.
- How much memory does your job need? If your peak memory requirement is higher than the available memory then the job will slow to a crawl whilst memory is written to disk.
- How many cores does your job require? A job requiring 12 cores on a cluster with nodes of 8 cores will not work as efficiently as on a node with 12 cores.
Systems in the Faculty of Science
Each HPC system is different and has different capabilities. Depending on the attributes of the types of jobs that you would like to run you will find that some clusters are better than others for your purposes.
Condor is a distributed computing system that is used by the School of Mathematics and Statistics and the Climate Change Research Centre. Jobs are submitted from a user desktop and are then farmed out to take advantage of the spare processor cycles of desktop machines when they are idle.
Because the job is transferred to a desktop machine which may be required at any time Condor is best suited for people who want to run large numbers of jobs which fit within the capabilities of the member desktops and are either checkpointed or can be restarted without significant data loss. Ideally Condor jobs should write their output periodically to a directory that can then be used for input data for a new job. Jobs that have checkpointing may run on different machines over the life of the job which is one reason that Condor is perfect for extremely long running jobs.
Condor jobs can take advantage of any of the software on the machines in the pool or be compiled to take advantage of software that may not be available.
Katana is a blade based Dell cluster available to staff and postgraduates in schools and research groups that contribute funds to expand the cluster based on their contribution to the cluster. Currently this includes the School of Mathematics and Statistics, Climate Change Research Centre, Australian School of Business, BABS, Australian Centre for Astrobiology, TARS and
Katana is designed for users who wish to run computationally intensive jobs who may require a large amount of memory per core or who just want to run a large number of jobs at the same time.