By Steve Sarsfield, Developer Advocate at Ocient
There was a time when everything was simple. The world we live in today is full of electronics, hardware, and devices that generate data. Because of our connected devices, we now live in a world with a tremendous amount of data generated automatically. We can specifically point to IoT data as one of the main culprits for the insanity. Everything from telephones to toll booths to personal fitness devices generates data.
Companies increasingly see the importance of a shift toward a complete analytics platform that can handle both the workload of data lakehouses and faster-moving data, like IoT data. Companies seek a solution where they don’t have to wait for ETL processing or long data loads to get up-to-date information. Real-time analytics is the next generation of analytical engines.
This blog will explore how this wave of innovation will help lower costs and improve the way organizations deploy analytics.
What is Real-Time Analytics?
Pure Real-Time
At its strictest definition, real-time applies to closed systems where data is processed and results are delivered within a fixed time window. Think of a fighter jet’s heads-up display: the jet generates a steady stream of data, and that data must hit the display at sub-second speed. Any delay could mean the difference between good and bad decisions. That’s true real-time, and some scientists insist on this exacting definition.
Business Real-Time
Businesses, on the other hand, don’t operate in closed systems. Data variety and velocity are unpredictable, making a strict definition of real-time unrealistic. Having sub-second information, although entirely possible, might be very expensive to achieve in worst-case scenarios. For example, if you’re a retailer and need to achieve sub-second web reporting, the system must be built to do so even during the holiday rush. Sub-second latency isn’t always necessary. A 5-minute lag might still count as “real-time” if the system delivers insights quickly enough to keep up with the pace of business.
Businesses define the windows within which they should achieve success. The window might be measured in seconds, minutes, or even hours. Real-time is often about delivering continuous analytics as data arrives, not waiting for batch processes to load overnight. It’s about being fast enough to meet your needs, even if it’s not fighter-jet fast.
Corporate Use Cases
The following industries rely on real-time analytics:
- Telecommunications – Managing network traffic or ensuring quality of service.
- AdTech – Dealing with massive real-time bidding data for advertisements.
- Finance – Monitoring stock prices or detecting fraud.
- Gaming – Seeing how a change in the app impacts usage and in-app purchases
Public Sector Use Cases
In the public sector, real-time analytics offers significant benefits for government and military applications by enabling rapid decision-making in critical operations.
- Cybersecurity – Enabling quick responses to cyber attacks on national infrastructure.
- Situational Awareness – Providing situational awareness by processing data from drones, sensors, and satellite feeds and allowing for timely tactical decisions.
- Logistics and Supply Chain Management – Ensuring efficient resource distribution so that goods can get to their destinations without delay.
- Disaster Response – Allowing agencies to react quickly to emergencies and potential security threats.
These capabilities give government and military organizations peace of mind by enabling data-driven, real-time insights in high-pressure situations.
A Brief History of Analytics in Real Time
Before purpose-built real-time analytics solutions existed, companies often created their own by piecing together multiple technologies. They would integrate databases, data streaming tools, in-memory processing, and custom-built analytics layers to achieve near real-time performance. However, this approach came with significant challenges:
- Companies found themselves creating complex architectures to deal with real-time data that required a diverse team with a diverse set of skills. They might pair streaming technologies like SPARK with data warehouse technologies like Vertica or Redshift. They built custom pipelines to connect the various software.
- Without a unified platform, synchronization, data latency, and manual intervention were constant problems. Troubleshooting was particularly difficult, since there was no single log file to examine for errors. It took legwork to track them down.
- Using multiple solutions also increased licensing costs and required specialized teams to maintain the systems, making it expensive and time-consuming.
The New Way – A Unified Platform
Some of today’s analytics providers offer a unified platform for handling high-speed data ingestion, processing, and analysis. With these solutions, you don’t have to integrate multiple systems. They simplify the process, eliminating the need for experts on individual solutions.
There’s also a licensing benefit, since these platforms require fewer licenses. Organizations can act on live data instantly without the delays, complexities, and costs associated with older patchwork solutions.
Key Players
Ocient, Apache Druid, Apache Pinot (StarTree) and ClickHouse are four prominent vendors in this space.
- Ocient offers unique hardware acceleration. With its Compute Adjacent Storage Architecture® (CASA) and use of AMD EPYC processors and SSDs, it is ideal for industries like finance and telecommunications, as well as the public sector.
- Apache Druid is an open-source platform optimized for real-time ingestion and interactive queries. Ease of scalability is a significant advantage, though operational complexity and rising storage costs can present challenges. Click here to learn more about how Druid compares to Ocient.
- ClickHouse offers a fast, cost-effective solution for ad-hoc analytics. It faces limitations in real-time ingestion and concurrency under heavy workloads.
- Apache Pinot, often deployed with the company StarTree, is another powerful analytics platform designed for low-latency, high-throughput queries on large-scale event-driven data.
Choosing the perfect platform depends on business needs. Ocient has performed several benchmarks to determine which platform is best for any given situation. In fact, we’ve just released a new benchmark comparison using the TPC–H Flattened Star-Schema Benchmark (SSB). Benchmarks of this type commonly test at a scale factor of 100 (600 million rows) or 1,000 (6 billion rows). We tested at scale factor 90,000 (536 billion rows), and our results eclipsed the competitors on single queries, concurrency queries, and disk space needed for storage.
Key Capabilities
Key to real-time analytics is the integration of data streams, powerful analytics engines, and a scalable infrastructure that can handle the high-speed data flow required for immediate processing. The following capabilities are also fundamental:
- Data Ingestion and Integration: The system must be able to ingest data from multiple, diverse sources (structured, semi-structured, and unstructured) in real-time.
- Low-latency Data Processing: A crucial feature is processing data with minimal delay.
- Scalability: The system should scale horizontally and vertically to handle increasing data volumes and more concurrent users or queries.
- Advanced Analytics and Machine Learning: Systems may integrate advanced analytics such as predictive modeling, geospatial, anomaly detection, and machine learning.
- Event-driven Architecture: the system can orchestrate data events. It generates alerts as soon as relevant data conditions are met, triggering analysis or actions.
- Real-time Visualization and Dashboards: Users need immediate access to insights, often through dashboards and visualization tools that refresh in real-time. This helps business leaders or operational teams make fast, data-driven decisions.
- Automation and Alerts: Systems can automate actions based on predefined rules or conditions, such as sending alerts to users or initiating workflows in response to specific triggers (e.g., fraud detection, operational issues).
- Data Storage and Management: Systems must efficiently manage and store historical and real-time data, enabling fast retrieval for ongoing analysis. They use the latest processors, memory and storage to speed up the task.
- Fault Tolerance and Reliability: To ensure continuous operation, systems must have built-in fault tolerance, allowing them to recover quickly from failures or data processing disruptions.
How can Ocient Help?
Ocient’s analytics platform is designed to address modern analytics needs by incorporating several key features:
- Massive Data Ingestion: Ocient is optimized for ingesting and processing large-scale datasets in real-time. Its architecture supports high-speed data streaming, enabling real-time data collection from multiple sources, such as sensors, IoT devices, and transactional systems.
- Extreme Scalability: Ocient’s platform is built to scale horizontally, allowing it to handle petabytes of data and trillions of records with low-latency performance. This capability is essential for modern analytics, as data volumes can grow rapidly, and the system must maintain performance without delays.
- Compute Adjacent Storage Architecture® (CASA): Ocient’s CASA places NVMe drives directly adjacent to processors, enabling the platform to execute hundreds of millions of parallel tasks as soon as data comes off the drive. With 12 NVMe drives per chassis, data is scanned at over 1.2 terabits per second, achieving 25 million 4KB random reads per second. This architecture, combined with techniques like kernel bypass, minimizes bottlenecks and maximizes response times, ensuring faster and more efficient data processing. This design helps users extract greater value from their data at unprecedented speed.
- High-Performance Query Processing: Ocient uses a highly optimized query engine to execute complex SQL queries at high speeds. It supports real-time query execution over massive datasets, critical for delivering immediate insights without compromising accuracy.
- Advanced Indexing and Compression: Ocient uses advanced indexing techniques (such as n-gram indexing) and data compression algorithms to minimize storage and retrieval times. This ensures that even large datasets can be queried and processed in real-time without performance bottlenecks.
Ocient can deliver the high performance, low latency, and scalability required for real-time analytics. Its solution works across the telecommunications, finance, and defense industries. Click here for a real world example of how Ocient helped one customer unlike new real-time analytics capabilities.