George Kondiles, Chief Architect and Co-Founder at Ocient explains how Ocient loads and transforms hyperscale data.
Kondiles discusses the loading and transformation infrastructure for the Ocient Hyperscale Data Warhouse. This infrastructure is crucial for analyzing and querying extremely large datasets.
Ocient’s loading and transformation technology and infrastructure were designed with three key principles in mind: linear scalability, low time to queryability, and exactly once semantics.
The “firehose” of information, arriving from sources such as S3 (where there may be large numbers of CSV files) or a queuing system like Kafka (which may contain JSON documents), presents a significant challenge. But Ocient’s technology is capable of receiving and handling the high volume and large amount of information and storing it in our foundation layer.
