The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop:PyHPC 2020: 9th Workshop on Python for High-Performance and Scientific Computing
Authors: Steven R. Brandt, Bita Hasheminezhad, and Nanmiao Wu (Louisiana State University); Sayef A. Sakin, Alex R. Bigelow, and Katherine E. Isaacs (University of Arizona); Kevin Huck (University of Oregon); and Hartmut Kaiser (Louisiana State University)
Abstract: We describe JetLag, a Python-based environment that provides access to a distributed, interactive, asynchronous many-task (AMT) computing framework called Phylanx. This environment encompasses the entire computing process, from a Jupyter front-end for managing code and results to the collection and visualization of performance data.
We use a Python decorator to access the abstract syntax tree of Python functions and transpile them into a set of C++ data structures which are then executed by the HPX runtime. The environment includes services for sending functions and their arguments to run as jobs on remote resources.
A set of Docker and Singularity containers is used to simplify the setup of the JetLag environment. The JetLag system is suitable for a variety of array computational tasks, including machine learning and exploratory data analysis.