Operational Analytics in the Cloud: Rockset is a Serverless Data Management Engine Helping Users Build APIs and Real-Time Dashboards

TL: DR: Rockset‘s search and analytics engine enables devs to build serverless microservices, data APIs, and data-driven applications with ease. The cloud-based solution’s data management system helps modernize cloud stacks while streamlining the move from batch to real-time analytics. With an agile, customer-centric approach to development that includes weekly releases, Rockset is furthering its commitment to building a data-driven future.

Venkat Venkataramani’s more than eight years as an engineering director at Facebook headquarters in Menlo Park, California, provided him with a wealth of experience, to say the least.

There, he managed the engineers who built Facebook’s online data systems — the same systems that store all user data and power its web and mobile products. He helped conceive, build, and scale technologies to run Facebook’s cache infrastructure, database infrastructure, client abstractions, and streaming data services. And he recruited a strong team to scale its online data serving systems.

Today, as Co-Founder and CEO of Rockset, he leverages those skills in making it easier for developers to build data-driven services and products in the cloud.

“Facebook was able to be extremely creative and introduce lots of new products quickly because of its online data infrastructure, and that’s exactly what we’re bringing to the market with Rockset,” Venkat said.

The company, based in San Mateo and founded in September 2016, is also backed by a team of industry veterans from Yahoo, Google, Oracle, and VMware who have vast experience in web-scale data management and distributed systems. In three short years, Rockset has attracted significant interest from developers looking to power their data-driven applications and interactive dashboards with SQL queries directly on raw data — all without the need to manage custom pipelines, servers, or databases.

Financial backing from Greylock Partners and Sequoia Capital means Rockset is poised for the rapid growth the company needs to further its mission: to provide operational analytics at warp speed.

In some ways, Venkat said that data management is still in its infancy. Early transactional systems are best suited for retail sales, CRM tasks, financial transactions, and order entry, but they’re not recommended for complex queries.

To determine how much a business sold in a specific region one week compared with another, businesses used to turn to analytical systems, such as data warehouses. But, although data warehouses can handle complex queries, they’re also quite slow, requiring analyzation in batches.

Enter operational analytics, a type of analytics focused on driving insights at the same rate that information is received, and thus maximizing business potential. This new approach is Rockset’s specialty.

“Now, as the world begins to function in real-time to increase operational efficiency, companies are beginning to adopt an approach known as operational analytics, which allows for real-time data access and high query complexity,” Venkat said. “Rockset is driving the market in that capacity.”

With the serverless approach inherent in operational analytics, information from various parts of a business is processed in real-time, creating a continuous feedback loop that fuels the decision-making process. And Rockset streamlines that process, extracting value from data without the time, preparation, tools, and expense typically incurred.

“Rockset automatically turns datasets into SQL tables without data engineering or custom extracting, transforming, and loading (ETL),” Venkat said. ‘We also index the data so you can get fast SQL without managing databases or doing any database administration. End to end, the whole process takes place in just a couple of minutes.”

If the fact that Rockset empowers developers to search and analyze NoSQL data using SQL in real-time weren’t impressive enough, the company recently released the first real-time SQL solution for Amazon DynamoDB.

“Until now, SQL on DynamoDB was just not possible,” Venkat said. “There’s been a lot of movement and migrations from traditional SQL-based systems to NoSQL systems like DynamoDB, but one thing you often have to give up is operational reporting on that data — something you used to take for granted using PostgreSQL, MySQL, or Oracle.”

This fast and flexible NoSQL database service supports some of the most extensive applications around the globe at scale through single-digit multisecond performance. Today, numerous AWS customers use DynamoDB to support mobile, web, gaming, ad tech, and Internet of Things applications that demand access to low-latency data at any scale.

With this release, Rockset is bringing operational analytics and real-time business reporting to DynamoDB users. Because the operational analytics engine is serverless, it does not require provisioning, capacity planning, or server administration in the cloud, allowing developers and data engineers to run highly complex queries with the power of SQL.

Users may also visualize DynamoDB data in live SQL-based dashboards, such as Tableau, and build custom dashboards and applications with real-time SQL, among other actions.

“Once it is given read access to a DynamoDB table, Rockset reflects changes as they occur in DynamoDB by making use of changelogs in DynamoDB streams,” Venkat said in an August 20, 2019, press release. “This gives users an up-to-date (within a few seconds) indexed version of their DynamoDB table in Rockset. Each SQL query against this table is distributed and executed in parallel to ensure that query results return in milliseconds.”

Venkat said that with Rockset, it’s as easy to get started as it is to build data-driven services and products in the cloud. “Once you get an account, just point us to your data,” he said. “You will get fast SQL tables, and you can go build your applications, APIs, and dashboards directly on top of Rockset.”

And it’s a swift process. Without Rockset, Venkat said it could take two people and a few weeks to go from a DynamoDB table to running fast SQL on data. With Rockset, the process takes one or two minutes.

We’ve all heard rumors that the skilled talent gap is a significant problem in the tech industry, but research from 2018 shows that it’s about to get worse. In a recent report, “Future of Work: The Global Talent Crunch,” consulting firm Korn Ferry predicted a talent deficiency could cost the U.S. more than $162 billion by 2030, jeopardizing its current position as the world’s leading technology market.

As the industry shift from batch to real-time analytics continues, Rockset can serve as an extension of your team, saving everyone from sales, marketing, and business operations to security and dev ops teams significant time and energy. In addition, when you modernize your cloud stack with serverless data management, there’s no provisioning required, and you only pay for what you use.

“When you’re trying to move from a batch to real-time solution and give your operations team superpowers, you need a system like Rockset,” Venkat said. “We’re not just optimized for hardware — we’re optimized for people. At the end of the day, the biggest ROI comes from how quickly your team can build things without having to increase your staff.”

In August, Rockset created a plugin to export data from Apache Kafka to a collection of documents, allowing users to build real-time dashboards or APIs on top of Rockset data. Rockset is also partnering with Confluent, the company behind Kafka, which currently lists the Kafka Connect plugins in its Confluent Hub, and integrates them into its Confluent Platform.

In terms of upcoming news, Venkat said the company abides by a weekly release schedule to ensure it is always up-to-date with consumer needs.

“We are very nimble in terms of our stack,” he said. “We can work on a new feature and get it out to everyone before we go home that day.”