Basis Technologies Cuts Costs, Slashes Query Time with Ocient

Massive datasets and tight turnarounds

In 2017, Basis Technologies launched its workflow automation, optimization, and data reconciliation platform for digital AdTech. With Basis, customers can run all aspects of their multi-channel digital ad campaigns — from bidding to billing and data reconciliation — on a single platform. According to Michael Freiman, Chief Technology Officer at Basis, AdTech is all about data volume and speed.

“Not only are we dealing with a huge volume of data, but we’re also talking about a very short response time to bid requests.” —Michael Freiman, CTO  

Every day, Basis reviews up to 300 billion auction bid requests, bidding on up to 2 billion of them. The company must respond to those requests on behalf of its customers in about 50 milliseconds. “If we take longer than that, we’re out of the game,” he says. Their 700 bidder and event servers manage all these bidding events and track post-win events such as clicks, conversions, and video processing.  

It’s no exaggeration when Freiman states, “The amount of data, and the various types of data, that our servers produce is absolutely massive.” Their systems ingest 60,000 – 80,000 enormous JSON documents per second, all of which must be parsed, structured, and stored for future use. And the data flows 24/7/365.  

Basis queries this data constantly, running campaign forecasting, inventory compilation, bid filtration, bid shading, and other ad hoc queries simultaneously. To meet a 3 – 5 second service-level objective (SLO) for campaign forecasting, Basis was running other important queries like inventory compilation and bid shading one at time, 24 hours a day. Clearly, Basis’s business had reached a point where running these concurrent, orthogonal workloads demanded a higher level of workload management and processing power than their Elastic-based storage cluster — or even a relational database management system — could deliver without a massive infrastructure investment.  

Basis needed a single-platform solution that could handle massive, mixed service-class workloads efficiently so they could meet current SLOs without increasing infrastructure management costs.  

The recipe for success

Basis worked with Ocient to consolidate 10 workloads on the Ocient Hyperscale Data Warehouse to deliver real-time analysis for its always-on workloads at scale. The end-to-end solution enables Basis to ingest, load, and parse its 24/7 streaming data into arrays and multi-dimensional arrays, perform sophisticated joins and correlated sub-queries on trillion-row tables, and run complex queries — including Spark queries — in seconds.  

With Ocient’s Compute Adjacent Storage Architecture (CASA), Basis can transform, stream, and load data for its high concurrency workloads directly, resulting in minimal latency. Ocient’s support for a comprehensive set of indexes and complex table joins further speeds up queries, while the Ocient SQL Query Optimizer identifies the ideal execution path for up to 100 parallel queries. And with Ocient’s sophisticated workload management capabilities, Basis can prioritize urgent queries without compromising performance on other important jobs. 

Dedicated to helping Freiman consolidate and run Basis’s hyperscale workloads on a single platform, the Ocient Customer Solutions and Workload Services team worked closely with his team to build a solution tailored to Basis’s unique needs. By the end of the pilot phase, the platform was fully operational and running in production.

“It was incredibly challenging and complicated, but the Ocient engineers and the members of the Basis DSP platform team put a lot of elbow grease and sweat equity into it. That was the recipe for success.” —Michael Freiman, CTO  

Driving innovation at hyperscale

By migrating 10 workloads onto the Ocient Hyperscale Data Warehouse platform, Basis can meet its 3 – 5 second SLO for campaign forecasting while delivering results in minutes instead of hours for even the most complex queries running against trillions of data points. The Ocient solution also reduces inherent ETL latency while delivering query results nearly 100x faster than competing data warehouse solutions.   

In full production, Basis sends a petabyte of uncompressed data to the Ocient system every week. Freiman notes that without Ocient’s advanced resource management capabilities, he could not manage that amount of critical data on a single platform. “Not only do a lot of software solutions rely on it, but there are also people from inside the company that need access to that data set for debugging, business analysis, and campaign analysis on behalf of customers both internal and external,” he says. With typical resource management, a few errant queries could bring the entire workflow to a halt, costing Basis money with every second of processing time lost. With Ocient, errant queries are no longer a concern.  

Not only was Ocient able to deliver the solution without increasing infrastructure management costs, Freiman estimates that Basis has reduced hosting costs by 30% by running on Ocient. Given that the company’s already massive data continues to grow, cost savings will multiply in the future. And, as Basis looks to move to the cloud, Ocient can make the journey alongside it, helping Basis achieve the scale and efficiency it needs to drive innovation in AdTech for the future.  

READY TO GO HYPERSCALE?

Watch Michael Freiman’s chat with the Ocient team or book a demo to find out how Ocient can help your organization do more at hyperscale.