Luminati’s Proxy Network: Enabling Businesses to Extract Publicly Available Web Data at Scale Without Being Blocked

TL; DR: In a world where data-driven insights are vital to competition, Luminati is on a mission to help businesses gather information from the web without being blocked or mislead. The company’s P2P network leverages millions of idle devices to bring transparency to the internet. With a continuous update cycle and a customer-centric approach, Luminati is helping businesses worldwide tap into the power of publicly available data.

Today’s headlines proclaim that data has usurped oil as the world’s most valuable resource. It’s easy to brush off these reports as sensational, but companies that are trying to gain a competitive edge should pay close attention.

The data economy — a global system in which large volumes of information are gathered, organized, and analyzed to create value — is undoubtedly shaping the future of business. By leveraging a data-driven approach, companies can make well-informed decisions and identify new opportunities, leading to increased efficiency, innovation, and revenue.

But collecting information from the internet isn’t as easy as it once was. Many websites now block or display misleading information when too many requests are sent from the same IP, indicating that the user is leveraging a proxy service to gather data.

To ensure reliable results, savvy business owners are turning to advanced proxy services like Luminati. The company’s P2P network uses the residential IP addresses of millions of idle devices to extract data efficiently and without detection.

“The residential proxy, which we invented, is a new type of proxy,” said Or Lencher, CEO at Luminati, which was founded in 2014. “We created a win-win situation where the device owner provides consent to contribute his IP address in exchange for a valuable benefit.”

Device owners who opt in to the network and share their idle resources receive compensation in various forms, such as ad-free or no-cost applications from a partner developer. Resources are only used when the device is connected to wifi, not in use, and has sufficient battery power.

Ultimately, Luminati’s goal is to help businesses worldwide tap into the power of publicly available data by bringing transparency back to the web in the most ethical way possible.

Or told us that the most important assets in the Luminati system are the device owners — and the company pulls out all the stops to ensure their protection. Luminati only uses the resources of device owners who have given their full consent to become a peer, and all peers may opt-out at any time.

Other than the device’s IP address, the Luminati SDK does not access or use any personal information, and resources will never be used for purposes such as crypto mining or storage. The company’s services are fully compliant with data protection laws, such as the GDPR, and Luminati monitors usage via automated and manual systems to ensure all networks are free from abuse.

App developers who receive consent to route traffic through their users’ devices via the Luminati SDK are able to monetize their IPs. This allows them to earn revenue — up to $5,000 for every 100,000 users — and offer the app to users without ads or for free. “The app and software developers that embed our SDK basically receive a new monetization channel,” Or said.

Based on statistics from partner developers (remember, Luminati does not collect personal data), Or told us the process improves the user experience.

“Users who opt in to the Luminati network generally spend 10% to 15% more time with the app, either because they receive premium features or can bypass annoying ads,” he said. “Today, we have more than 40 million unique IPs every month in every single point on the globe — even in Antarctica.”

Luminati screens customers to verify that they are using the platform for approved purposes, such as brand protection, business intelligence, website testing, travel data aggregation, or actions of a similar nature. No customer is permitted to use the Luminati network for DDoS attacks, ad fraud, spam, fraudulent engagement, impersonation for fraudulent purposes, or illegal activity.

“Every potential customer that wants to try the network out is put through a rigorous compliance procedure,” Or said. “We need to know who they are and what their use case is, and our compliance officers are there to personally handle any reports of abuse.”

Today, more than 10,000 customers, including many Fortune 500 companies, use Luminati’s proxy solutions for various reasons — the most common being competitive intelligence.

“Ecommerce is a significant vertical for us,” Or said. “One of the main reasons to collect publicly available data is to aggregate and compare prices for different products and services, as well as understand the trends around those prices.”

Customers from the travel industry, such as agencies, airlines, hotels, and rental car companies, frequently use Luminati’s residential IPs to aggregate and compare prices. For these customers, price comparison is a part of daily operations, helping companies gain a competitive edge.

Users also commonly leverage the network for brand protection and to combat online advertising fraud.

“They’re two different verticals, but they’re very similar,” Or said. “In terms of brand protection, companies need to detect fake products being sold online by fraudsters, and they need to collect massive amounts of data to do that.”

To detect fraudulent and malicious ad activity, customers can mimic a real user in a particular county or city to verify the accuracy of campaign data for ads and affiliate links across the globe.

“Because we have so many IPs in every place around the world, our customers can use the network not just to collect data, but to collect data as it is viewed from a specific customer’s eyes.”

To keep up with customer demand, Or said Luminati releases 30 to 50 software updates daily. The company has invested significant time and money in its DevOps team, so products evolve exceptionally quickly. The rapid development cycle is also due to carefully defined performance metrics and constant interaction with customers.

“We don’t wait for our customers to talk to us — we come to them,” Or said. “A large portion of my time is spent traveling around the world meeting customers from businesses small and large. And it’s not just me — all of our product managers are on the phone talking to customers.”

After receiving feedback, the team will typically work to incorporate suggested changes into the platform at record speed. Fast-paced innovation is part of the company’s DNA, and Or said the goal is to keep it that way. After all, Luminati is providing a vital service.

“Our value proposition is that you can’t really extract data at scale without using Luminati,” he said. “We are like an electricity company — you can create a refrigerator, but you need electricity to run it. Well, you can analyze data, but only if you have a way to extract it.”

Or said in that, in the past few years, even the least technically inclined organizations have started to accept the fact that without data-driven decision making, they won’t likely survive.

“Because we’re one of the only platforms that help to bring better online transparency, we consider it our responsibility to ensure customers won’t get blocked,” he said. “They need to be able to focus on the data itself.”

Keren Pakes, Head of Communications at Luminati, said it’s also the company’s role as leaders of an extensive IP proxy network to set standards for ethical data collection. “We know that data, with all its power, needs to be kept secure and safe,” she said. “We’re planning to hold the first compliance forum on the topic at the end of February, and we hope it becomes a tradition.”

Or said he recently hosted an education session on ethical proxy use and data collection at the Amsterdam University of Applied Sciences. “It’s important that the next generation of students will graduate with the knowledge to ensure a secure and sustainable industry,” he said.