The Art of Computing Power: How YellowDog Masters Hybrid and Multicloud Scheduling and Orchestration

Yellowdog Masters Hybrid And Multicloud Scheduling And Orchestration

TL; DR: YellowDog is a multicloud management platform that intelligently orchestrates computing power, enabling businesses worldwide to accelerate and optimize complex data processes. The company provides a single interface for finding the ideal source of compute in terms of cost, performance, sustainability, and various other factors. With the introduction of YellowDog Index, a database for computing power, the company is delivering new levels of transparency, prediction, and efficiency to customers across a range of industries.

From 1999 until 2020, millions of people donated the spare resources of their internet-connected home computers to the Search for Extraterrestrial Intelligence (SETI).

The project, run out of the University of California, Berkeley, aimed to leverage the world’s idle computing power to analyze scientific data and, hopefully, find signs of alien life.

YellowDog logo

YellowDog unleashes optimal computing power.

Researchers shuttered the operation on March 31 of this year after more than two decades of work. The effort may have failed to reveal ET, but it had lasting implications in the hearts and minds of curious individuals everywhere.

For one, it inspired Gareth Williams, the founder of YellowDog, to create a powerful multicloud platform that intelligently orchestrates computing power.

“Inspired by SETI, he started thinking about how we could make use of underutilized power — whether on mobile devices, servers, et cetera — and essentially be able to make those resources appear as a single unit of compute,” said Simon Ponsford, CTO of YellowDog. “That’s where we started.”

Simon, who has vast experience in the startup arena, was the first employee to join the YellowDog team. He told us that over the years, the company’s vision shifted from accumulating and selling excess power to automating the best sources of compute for every workload, be it from datacenters or cloud providers.

The ultimate goal is to help businesses reach their full potential by optimizing complex data processes via predictive scheduling and orchestration. Today, organizations in industries, such as financial services, aerospace, and CGI, among others, turn to YellowDog for IT solutions that promote operational agility.

Helping Scale, Control, and Optimize Computing Power

Simon told us his customers aren’t always your average business looking to host an exchange server. Many need massive computing power and come to YellowDog in a state of desperation after failing to find what they need elsewhere.

“We all think of the cloud as being this unlimited compute resource, and it’s just not,” he said. “For many companies, like those in visual effects, bursty, complex and resource-heavy applications are commonplace. We come across customers who need anything between 1,000 and 10,000 servers for a month.”

When computing resources are not available for enterprises with such dynamic workloads, the result is often missed deadlines and exceeded budgets. But with YellowDog’s workload management technology, those concerns become irrelevant.

“No one has ever had a job too big for YellowDog,” Simon said. “We have some quite clever machine learning technology that allows you to work out, based on some job history, how long jobs will take to run. And then we guarantee that we will deliver by your deadline.”

The platform is packed with features, from the provisioning and management of computing environments to optimization of workload performance, visibility, and coordination.

It allows users to scale resources across hybrid and multicloud solutions, control computing sources to satisfy business and policy requirements, and optimize resource use via advanced machine learning.

A Single Interface for Finding the Best Source of Compute

Every business has its specific parameters when it comes to defining the best source of compute. For those concerned about price, YellowDog can help lower costs by increasing the efficiency of existing on-premises computing power and enabling intelligent use of cloud resources.

Simon said others are concerned about the environmental impact of datacenters that consume immense amounts of electricity and emit carbon dioxide. The number of datacenters worldwide has grown from 500,000 in 2012 to more than 8 million in 2019, due, in part, to a growing demand for storing, managing, and analyzing big data.

A collage of YellowDog internal photographs

The UK-based team, led by Founder Gareth Williams, works to keep environmental impact to a minimum.

According to YellowDog, average server usage levels are below 50%, and of all the machines installed in datacenters across the globe, approximately 30% are turned on but never used.

YellowDog’s philosophy is that there’s already enough computing power available in the world — we just need to manage it more efficiently. To that end, the company’s carbon impact is 57% less than that of typical datacenter solutions for workload management.

In addition to cost and environmental impact, YellowDog customers can tailor their resource use around factors like availability, latency, time to provision, and performance between node types. Best of all, the YellowDog Object Store service uses APIs to seamlessly integrate with object storage facilities of cloud providers.

“With YellowDog Object Store, you can distribute and even stream your data across multiple providers without having to think, ‘I need to copy this data elsewhere,’” Simon told us. “It manages that data flow both in and outside of the organization.”

Use Cases for Enterprises, Financial Services, Animation Studios, and Beyond

YellowDog customers run the gamut from enterprises and financial organizations to production agencies.

For example, the company helped the independent investment agency Fulcrum Asset Management transform its use of cloud and on-premises computing power, resulting in record performance and cost-savings.

The 3D animation studio nGenious, on the other hand, used YellowDog’s GPU Cloud Rendering in conjunction with Maya modeling software and the Redshift render engine to speed production on its first-ever feature animation.

YellowDog also helped the UK architectural practice Powell Dobson meet a strict three-week deadline to accelerate CPU rendering for an upcoming production that would be shown to potential investors.

Simon told us that YellowDog operates under an agile model of development guided by customer feedback and the company’s carefully planned roadmap.

“Every single development cycle is focused on what customers want as well as our longer-term vision,” he said. “So it all comes through, but essentially, we put out releases every day.”

And, because YellowDog’s infrastructure is containerized, the company can dynamically update individual components as needed without subjecting customers to downtime.

Introducing the YellowDog Index Compute Database

This summer, YellowDog will release YellowDog Index, a comprehensive database of price and performance across various infrastructure resources.

“YellowDog automatically makes all the decisions about what the best source of compute is for your job, but we’ve had a lot of people question if the system actually made the right decision,” Simon told us. “YellowDog Index, which will be completely open to the public, will allow users to view every single instance type, every price, in every region, and across every potential infrastructure. I’m excited that people will have this insight.”

The index will benchmark performance down to the node.

“You might find a four-core node in AWS, but you might not realize there’s a two-core node available from Azure with a higher overall CPU performance,” Simon said. “YellowDog Index makes that information transparent for anyone who is interested.”

Of course, the YellowDog platform uses this information to automate compute scheduling and orchestration, saving users time. “You don’t have to spend time looking at the prices, availability, et cetera, the platform will do all of that dynamically,” Simon said. “But the information is there if you need it.”