AarXiv2022

Automatic Parallelization of Python Programs for Distributed Heterogeneous Computing

Jun Shirako, Akihiro Hayashi, Sri Raj Paul, Alexey Tumanov, Vivek Sarkar

Abstract

'Vivek Sarkar'] Abstract. This paper introduces a novel approach to automatic aheadof-time (AOT) parallelization and optimization of sequential Python programs for execution on distributed heterogeneous platforms. Our approach enables AOT source-to-source transformation of Python programs, driven by the inclusion of type hints for function parameters and return values. These hints can be supplied by the programmer or obtained by dynamic profiler tools; multi-version code generation guarantees the correctness of our AOT transformation in all cases.

§ The Valyu brief

The paper, in five minutes

Read from the full text via Valyu Contents

The paper presents an ahead-of-time source-to-source compiler that automatically parallelizes Python code for distributed heterogeneous clusters, achieving up to 20,000× speedup on Summit.

What’s new

i.Ahead-of-time source-to-source transformation of Python using type hints and dynamic profiling
ii.Multi-version code generation guaranteeing correctness of parallelized code
iii.Extensions to the polyhedral framework that unify user-written and implicit loops and generate CPU/GPU code variants
iv.Integration with the Ray runtime for scheduling tasks across heterogeneous nodes
v.Empirical evaluation demonstrating massive speedups on a real-world radar application

How they did it

The system extracts type hints (provided by the programmer or inferred by a profiler) and applies source-to-source transformations guided by a polyhedral optimizer that unifies explicit and implicit loops. It generates multiple code versions for CPU and GPU, ensuring correctness via multi-versioning, and deploys the optimized code using Ray to schedule distributed tasks across nodes.

What they found

Over 20,000× speedup on the Space-Time Adaptive Processing radar application using 24 nodes and 144 GPUs on the OLCF Summit supercomputer

§ The Valyu brief

Reading the full paper and taking notes. This takes a few seconds…

§ Ask this paper

Ask a question about this paper

Valyu reads the full text and answers from what the paper actually says.

§ Related work in other archives

Similar papers in all four archives

The same idea searched in arXiv, PubMed, bioRxiv and chemRxiv at once, so a method can lead you to where it is being applied.

AarXiv3 found

PPubMed4 found

BbioRxiv4 found

CchemRxiv2 found

Searching the other archives…