• DocumentCode
    668174
  • Title

    The scaling of many-task computing approaches in python on cluster supercomputers

  • Author

    Lunacek, Monte ; Braden, Jazcek ; Hauser, Thomas

  • Author_Institution
    Res. Comput., Univ. of Colorado Boulder, Boulder, CO, USA
  • fYear
    2013
  • fDate
    23-27 Sept. 2013
  • Firstpage
    1
  • Lastpage
    8
  • Abstract
    We compare two packages for performing manytask computing (MTC) in Python: IPython Parallel and Celery. We describe these packages in detail and compare their features as applied to many-task computing on a cluster, including a scaling study using over 12,000 cores and several thousand tasks. We use mpi4py as a baseline for our comparisons. Our results suggest that Python is an excellent way to manage many-task computing and that no single technique is the obvious choice in every situation.
  • Keywords
    parallel processing; software packages; Celery package; IPython Parallel package; Python; cluster supercomputers; many-task computing approach; mpi4py;
  • fLanguage
    English
  • Publisher
    ieee
  • Conference_Titel
    Cluster Computing (CLUSTER), 2013 IEEE International Conference on
  • Conference_Location
    Indianapolis, IN
  • Type

    conf

  • DOI
    10.1109/CLUSTER.2013.6702678
  • Filename
    6702678
    • Link To Document
    https://search.isc.ac/dl/search/defaultta.aspx?DTC=49&DC=668174