It’s another hot day in Gainesville, Florida, as PJ and I enter Volta Coffee and Tea. In the corner is a friendly, unassuming genius, named Christian von Kleist.
I have had the chance to meet Christian before, but never had the chance to converse in the depth that unfolded in the two hours following our handshake at Volta. I wasn’t prepared for the onslaught of technical ability, organization, and overall command of technology Christian was soon to unveil. It’s no wonder companies like Google (Project Tango), NASA, and Autodesk have chosen to partner with Paracosm after seeing their early development and promise.
Christian is a CTO Paracosmonaut and co-founder of Paracosm.io (definition of the word Paracosm), and in the following post, I am going to try and do Christian and Paracosm some justice, paraphrasing everything we discussed.
The Paracosm.io Mission Statement
At its heart, Paracosm is dedicated to building a cheap, reliable platform for 3D modeling the world with a Kinect sensor or a phone.
“Paracosm makes it possible to pull your phone (or a dedicated 2.5D camera) out of your pocket, point it around a room or space you are inside of, and then produce an accurate 3D model you can run around in like a video game, or use to take accurate measurements,” Christian said.
Their core technology allows anyone to scan virtually any location themselves, upload the data, and receive in return an accurate, to-scale, 3D model of the space.
Paracosm’s Collaboration with Google: Project Tango
Applications of Paracosm’s Technology
The list of potential applications for this technology seems virtually endless and industries (not just companies), such as the movie, gaming, construction, and forensic industries, are knocking on Paracosm’s door to get their hands on it.
“The applications are so numerous that we can’t possibly come up with all of the use-cases, and our big focus right now is to capture one industry and get some use-cases out there and eventually tackle other verticals as well,” Christian said.
“Another awesome idea is for the movie industry. Apparently, it is really attractive to be able to quickly scan a set at a relatively low quality, but one still representative of the space, and get a rough preview of what the actors will look like moving around the space and set up good camera angles,” he told us. “These are the weird, niche ideas that we would never think of, but people are really excited about it.”
The cost-savings over using traditional 3D artists are considerable. To accurately model a real space is a very difficult and time-consuming task for any 3D artist. What Paracosm can do in tens of minutes is weeks’ worth of work for a human.
In addition, Paracosm isn’t looking to replace 3D artists. They are just able to give these workers a base to start from, with real measurements, textures, and colors they can fill in with the details.
The Challenges of Producing 3D Models: Cheap Cameras & Large Spaces
Christian went on to describe how difficult it is to use handheld devices to produce 3D models.
“These cameras are incredibly noisy,” he explained. “They can tell the rough distance to the surface, but there is a lot of magic and difficulties we had to work out just to use this data when trying to scan a very large space.”
“We are down at the level of bits and messing with drivers, all the way up to recognizing what is a wall, what is a floor, and taking all of the 2D photographs and stitching them together.”
“What we try to do is have the cleanest division of the layers of abstraction possible, so we have really well-defined inputs and outputs at each level,” Christian said.
This allows Paracosm to switch out software components and to evaluate the results scientifically. They ship new versions of their software — not based simply on humans deciding if a small set of observations looks better, but on automated testing against multiple, real-world scenes — to decide if the software has improved.
This scientific approach has allowed Paracosm to steadily out-perform the rest of the industry.
The Technology Stack
Paracosm’s technology stack is divided into three main parts.
- The Hardware: This includes the camera and Microsoft Surface tablet.
- Linode Instances: This includes their main website, Customer Lobby, and Redis job queue.
- AWS: The hardcore processing units are on AWS and utilize S3, EC2, IAM, among others.
As of late, Paracosm’s Starter Kit contains a 2.5D Camera (RGB+D), a Microsoft Surface tablet, a hard-shell case, and two months’ worth of access to their Cloud Computing platform, all for $2,999.
The main engineering done on the hardware side of the project seemed to be a whole lot of driver work done in order to get the performance they required for the processing end of things.
During our visit with Christian, we didn’t focus too much of our talk on the hardware aspects of the project, as I was much too curious about their backend and processing implementation.
The Customer Lobby & Job Queue (Linode)
Once a 2.5D video is captured, users have a Customer Lobby that runs on a Linode instance to interface with and upload their work. This Linode instance transfers the files to Amazon S3 and creates the necessary jobs that need to be run to change the video into a true 3D model.
The job stack is stored in Redis, which also runs on Linux. One of the beautiful design decisions Paracosm made was parallelizing the entire process. With their algorithm, the video does not need to be processed from beginning to end. Instead, much like bit-torrent, it is processed in chunks, which can run on separate processes.
This allows Paracosm to process video much faster than their competition and to scale easily. For example, for a higher workload and/or faster results, all they need to do is spin up additional workers in their AWS Worker Cloud.
The EC2 Workers & S3 Storage
Ah, now we are on to where the hardcore magic happens.
Spawning EC2 Workers
Paracosm utilizes the AWS API extensively in order to coordinate spinning up new EC2 instances (with GPU support) when they are needed, in order to meet demand. They call each new EC2 instance a “worker.”
These workers, when spawned into the world, immediately try to connect to the Redis job queue and check for any work to be done. If there is, they mark the work as started and process away until the work is completed, save to S3 if necessary, then mark the work as completed.
Since Redis transactions are atomic, this guarantees that no job will accidentally be started twice.
“We have, as a company, taken advantage of a lot of smart components out there,” Christian said, using Redis as an example. “We didn’t have to write so much stuff, because Redis exists.”
The algorithms needed to process the Kinect output into a full 3D model are, as you would expect, computationally complex and on the cutting edge of even the largest companies’ research.
“Our algorithms are CUDA and C++ that are glued together with Ruby, and each worker has the ability to process data from the early components, all the way to the final output,” Christian said.
Storage & Security
Paracosm’s team is fanatic about saving every piece of data that they capture. This allows them to continually improve their testing data sets and also provide their customers with the option of recreating their 3D models with every improved version of their software.
As you might expect, this video data and its intermediate steps become very large storage burdens very quickly. Currently, the team is utilizing S3 to store all of this long-term data, along with the final 3D models themselves.
In the future, Christian and his team plan on utilizing Glacier to store the long-term data, with the short-term 3D model results being held on S3.
“One of the great things about S3 is it inherently both supports time-sensitive access control and CDN-ifies the files for our users,” Christian said.
Paracosm runs everything inside their own VPC, with strict access control via IAM. The workers are only granted the access they need and their AWS system is not accessible via the Internet.
We talked more about their security implementations and how incredibly careful they are about their users’ data, but it is probably best if I leave that portion out.
Wrapping It Up
I have tried my best to describe Paracosm’s product, its well-thought-out hardware and software stack, and its brilliant engineers, but I feel I have barely scratched the surface.
According to Christian, as soon as you enter their office, you see nothing but smiling faces and people who are passionate and believe in what they are doing. For me, I couldn’t be more pleased to catch a glimpse of anything else.
In Christian’s case, you can feel his passion for seeking out new challenges, learning new things, and taking the risk of being on the forefront of a technology. It is inspiring to be around such drive, even if only for a few hours.
“It doesn’t matter where you go or what you do, as long as you love it.” – Christian von Kleist