TL; DR: SketchAR is a mobile drawing app leveraging augmented reality, artificial intelligence, and neural networks to guide users through step-by-step art lessons. The tool projects virtual images on paper, making it easy to trace each drawing with the help of an AI-powered virtual assistant. By merging art, technology, and education, SketchAR is bringing new opportunities to a whole new community of creatives.
As every artist knows, there is nothing more intimidating than a blank white canvas. It takes confidence to confront the problem of where to start and what technique to use — and for budding artists who lack the education to inform such decisions, it’s often easier to walk away than make a mistake.
But fear of failure doesn’t have to be a roadblock to success. Today, technological advances are helping people who have always wanted to experiment with art do so without hesitation. One of the most groundbreaking examples of innovation in this area is SketchAR, a mobile drawing app that allows users to dive into the world of art with the assistance of augmented reality.
The app, available for iOS, Android, and Microsoft’s HoloLens headset, projects virtual images on top of physical paper, enabling even the most novice of artists to trace sketches onto surfaces.
Andrey Drobitko, CEO of SketchAR, leveraged his dual backgrounds in art and technology to launch the app on iOS in 2016. “The app is a great example of how augmented reality, computer vision, machine learning, and neural networks can help people learn new life skills,” he said. “Combining art and tech makes it easy for those without any background in art to experience the feeling of being an artist.”
Approximately 90% of SketchAR subscribers are adults ages 25 to 35 who wish to learn to draw. The app makes it easy for users to do just that with the guidance of an AI-powered virtual assistant that offers personalized advice based on user activity. Ultimately, by helping users pick up skills through step-by-step lessons, SketchAR hopes to empower them to achieve artistic goals they never thought were possible.
Step-by-Step Art Lessons Featuring a Quirky Virtual Assistant
Andrey told us he originally conceived of SketchAR as a tool professional artists could use to trace sketches onto various surfaces. But he soon suspected the tool would be especially appealing for novice and practicing artists.
To test that theory, the SketchAR team compared search queries for “learn to draw” with “learn to sing,” “learn to dance,” “learn to program,” and “learn to fight.” Results indicated that “learn to draw” was the most frequent query in China, France, and Russia. In Spain, the search term was second only to “learn to dance.”
“For those who are looking to gain new life skills, learning to draw is one of the most frequent search queries, and that’s why we thought it would be a good idea to use augmented reality as a tool to make the process easier,” Andrey said.
Because it’s always easier to take on something new with the help of a mentor, SketchAR’s virtual assistant is available from the first lesson to help users become comfortable with the learning process. Unlike most teachers, this one takes the form of a feathered creature with wayward eyes — but it isn’t nearly as bird-brained as it seems.
The genius of this virtual assistant lies in its ability to offer personalized advice through the power of machine learning. After observing activity for a few lessons, the bot will gain an understanding of the user’s skill level and provide feedback accordingly.
“The virtual assistant is an extra special feature because it is very sensitive to the users’ actions and can analyze everything that the camera sees to recommend the next step,” Andrey said. “For example, the assistant can see the lines that you draw incorrectly and instruct you to correct them, whether that’s making lines darker or whatever else.”
Merging Computer Vision, AI, and Neural Networks
The SketchAR staff, made up of developers, artists, and highly skilled experts in computer vision, places a significant emphasis on research. The team is laser-focused on advancing the augmented reality industry as a whole to make current tools increasingly stable, improving usability. In fact, Andrey said the crew spends almost 90% of its time on R&D.
In May, the company announced that it has developed a method enabling cameras to detect flat services without the typical anchor points. Using traditional methods, the camera of a smartphone can recognize boundaries of a piece of paper within shades of white, but if an object — such as the artist’s hand — blocks those points, the image may disappear. SketchAR’s new algorithm, known as Tracking Method Without Markers and included in SketchAR 5.0, improves accuracy and keeps virtual images steady.
“Computer vision typically works well with faces — as you can see using Snapchat or Instagram — because it’s easy to teach the camera to recognize and detect them,” Andrey said. “But our product features much more than faces, which is a pretty unusual use case for a smartphone camera. Since there wasn’t any solution on the market, we had to solve the problem ourselves.”
The company is also known for cutting-edge technology such as the use of a neural network approach, which leverages data from the company’s data from real-life users of SketchAR. The data is used to further improve the stability of the algorithms that power the application.
Brand Integrations and Community Building
SketchAR also has applications for brands looking for a creative and effective way to promote products through augmented reality. In November 2018, the company hosted a live drawing event where attendees drew cartoons using Microsoft HoloLens to celebrate the premiere of “The Grinch” movie in Paris.
SketchAR also partners with organizations in the art, technology, education, and entertainment communities in an effort to create synergy between them. For example, the company joined forces with smartphone provider Huawei late last year at the eco-Connect Europe 2018 event in Rome, where they set up a collaborative live drawing area using the app in conjunction with the Huawei Mate20Pro. The company also provided an opportunity for users to test the app on paper.
Events like these help brands promote themselves in a creative way while furthering SketchAR’s mission to build a community of tech-based artists. In addition to helping users expand their skill set, Andrey said the app has a distinct therapeutic effect.
“In our modern world, people are very busy, and they’re dealing with a lot of stress,” he said. “Art is like meditation, and our product makes it easier for people to achieve those benefits through augmented reality.”
In terms of community building, Andrey said the company is “in the process of developing something huge.” For now, the project is being kept under tight wraps, but it will likely fall in line with SketchAR’s motto: “Earth without art is just eh.”
“This project will unite a lot of people from around the globe in terms of augmented reality,” Andrey said.