Powerful and Productive Parallel Programming With Chapel: Building an Open-Source Language That Scales From Laptops to Supercomputers

Building an Open-Source Language That Scales From Laptops to Supercomputers

TL; DR: A parallel programming language developed to increase computing productivity, Chapel accelerates development by permitting distributed systems to be programmed using a scalable, portable, and natural syntax. Creators and developers of the open-source language aspire for the language to bridge the gap between traditional Fortran, C, and C++ coders and programmers who prefer more modern languages such as Java, Python, or MATLAB. Language Manager Tom MacDonald told us about the advantages, backstory, and future of the language.

As applications of data analytics, machine learning, and scientific computing continue to explode, researchers are quickly adapting to new platforms and tools and seizing opportunities for bigger simulations, more data, and higher levels of computational productivity.

Instead of retraining themselves as engineers, however, researchers can instead turn toward Chapel, an emerging open-source language that simplifies parallel computing by integrating features and capabilities crucial to scalable computing into a single programming language.

“A scientist wants to be able to express science,” Language Manager Tom MacDonald said. “They don’t want to have to learn the details of a complicated computer architecture and how to program to that particular setup because there’s just going to be something new in the not-too-distant future. That’s why we feel Chapel is a better approach to parallel programming than what currently exists today.”

Creating a Language to Simplify Parallel Programming

In parallel computing, calculations and processes are carried out simultaneously. Complex problems are broken into smaller tasks that are executable on separate processors.

Cray Inc., a company specializing in supercomputing, data storage, and big data analytics, began developing Chapel as part of a 2003 high-productivity computing systems program sponsored by the advanced projects research agency of the US Department of Defense. The program challenged vendors like Cray to improve the productivity of supercomputers.

Image of Tom MacDonald, Language Manager of Chapel

Language Manager Tom MacDonald oversees a team of 13 developers working on the open-source language.

Cray called the project Cascade after the mountain range east of its corporate headquarters in Seattle, Washington. Chapel, an approximate acronym for Cascade High Productivity Language, garnered encouraging user feedback with its proof-of-concept implementation under the program. When the research program ended in 2012, Cray continued building the language and doubled the size of the team. With core processing speeds effectively maxed out, manufacturers are now looking for performance boosts through parallel computing and multi-core processors.

Traditional parallel computing, according to Tom, started with legacy languages such as Fortran, C, and C++. Developers would then insert directives and libraries to coordinate communication between the different processors.

“What Chapel is attempting to do is to integrate all of that into one programming language so that you don’t have to worry about all the different idiosyncrasies of the various pieces you might have to use to write a traditional parallel program,” he said.

For example, a simple parallel loop in Chapel looks like:

Programmers can compile and run Chapel programs online without installing anything, or look at documentation for Hello World examples or language primers. The loop’s iterations would run in parallel, using all the processors on your workstation or compute node. With some minor changes, the same loop could distribute the work across all the nodes of a cluster or supercomputer.

Three Chapel Advantages for Productivity and Portability

From the outset, Tom said, Chapel was designed specifically for parallel computing on a large range of scales and systems.

“It’s intended to not only address the high-end computers but the parallelism available on every consumer desktop computer, as well,” he said. “In our opinion, that doesn’t really exist, and Chapel is the first language to really attempt to do this in an integrated fashion.”

1. Cross-Platform Compatibility Ensures Scalability

To guarantee portability, Tom said the Chapel team isolates architecture-specific components into what they call locales, which can be swapped out to target different configurations. The programming language runs on macOS, various flavors of Linux, and Cygwin or the Bash PowerShell in Windows 10, as well as Amazon Web Services and the cloud.

“If you get too specific in how you target a particular architecture, it’s going to be a lot of work to port that over to a new architecture,” he said. “The intent is to have a language you can run on your multi-core laptop, take that same program, and in a very natural way, scale it up to a much larger machine.”

2. Efficient Coding Yields Rapid Prototyping

Chapel frequently draws comparisons to Python for being a high-productivity language with easily write-able and readable code that enables developers to quickly create usable and testable programs. Unlike several previous parallel languages, however, Chapel supports what the team calls a multi-resolution philosophy that allows programmers to write very abstract code and incrementally add more detail.

“You’re able to prototype something very quickly,” Tom said. “With very few lines of code, you’ll be able to get something that gives you an idea of whether you’re going to be successful. Then, you can add additional details and specification for how to map the computation to the system in order to improve scalability and performance.”

3. Natural Syntax Saves Developers Time

In creating the parallel programming language, developers loosely modeled Chapel’s syntax after C because of the language’s pervasive popularity. Chapel also includes many features from sequential languages to boost productivity, such as Modula’s left-to-right declaration style.

Type inference, iterator functions, object-oriented programming, and a rich set of array types combine the productivity of Python with the performance of C, according to Tom. “You can naturally express very large data structures that might spread across a distributed computer system in a natural way,” he said.

Principal Engineer Brad Chamberlain said that programmers from a variety of backgrounds find attributes they like in Chapel. For example, Fortran programmers appreciate Chapel’s multidimensional arrays, while Python coders enjoy the inclusion of iterators.

“People sometimes give us a hard time for developing a new language, rather than extending an existing one, but this has worked out well for us,” he said. “We haven’t been beholden to baggage in existing languages that can hinder parallel computing while cherry-picking attractive features.”

A Supercomputer Company’s Dedication to Open-Source Programming

During the initial research program, Cray established the language as a publicly available, open-source project by moving Chapel to a SourceForge-based repository.

“We were never something that was intended to be proprietary,” Tom said. “We intend to be broad-reaching and ubiquitous in the various architectures we support.”

Beyond lowering the barriers to adoption, establishing Chapel as an open-source technology attracted external users and contributors. The Chapel Implementers and Users Workshop (CHIUW) convenes annually to explore new ways to advance the language, while the Chapel Users Group (CHUG) fittingly holds a social event or happy hour each year for the community of users, enthusiasts, and developers interested in Chapel.

The project is now hosted on GitHub and is distributed under Apache License, version 2.0. At Cray, Tom oversees a team of 13 developers dedicated to hardening Chapel for widespread use in production environments.

“All our code is available, and we strive to involve the community,” Tom said. “We get lots of good ideas and even some development help from the broader community.”

Patience and a 5-Year Plan Leading to a Mature, Upcoming Version 2.0

As can be expected with any project aiming to accomplish something that has never been done before, the Chapel team has encountered and overcome all sorts of challenges, Tom said.

Following the initial project, the Chapel team embarked on a five-year program to take the language from proof-of-concept to production. Within a year, Tom said he and the team expect to have a mature version of Chapel ready for users to deploy across various projects and systems.

“It’s a matter of being able to take the time to do it right,” Tom said. “We started out with a prototype many years ago, but we’ve worked hard to create a much more mature version of the Chapel language and Chapel compiler.”

Laura Stamey

Questions or Comments? Ask Laura!

Ask a question and Laura will respond to you. We strive to provide the best advice on the net and we are here to help you in any way we can.