By Dylan Murphy, Director of Product Management at Ocient
I remember watching Knight Rider growing up and dreaming about the future of connected vehicles. Looking back, I’m not sure what K.I.T.T, the car from the show, was connected to besides Michael Knight. The ten minutes I spent watching clips online didn’t provide any more insight, but it was a fun journey down memory lane nonetheless. While its crime-fighting capabilities were undoubtedly impressive, I’m certain that the boring minivan sitting in my driveway puts that car to shame from a technical perspective.
Even outside of autonomous driving, today’s vehicles generate a tremendous amount of data. Manufacturers are interested in part performance, safety improvements, personalizing the driving experience, and marketing data. Like most initiatives involving hyperscale data, collecting the data is a good start. Analyzing it in interactive time in a cost-effective way has many vehicle manufacturers looking for help. We know because we’ve spoken to them, and while it was difficult, I had to break it to them that Michael Knight and K.I.T.T were not coming to the rescue.
Data analysis requirements for connected vehicles are growing faster than ever
According to Morgan Stanley, vehicles will soon be generating as much as 40TB of data per hour from cameras, radar, and other sensors. As we partner with vehicle manufacturers, we are learning more about their ambitious plans to get ahead of this rapid growth by analyzing petabytes of data generated by their growing fleets of connected vehicles. These vehicles include the ones that we ourselves drive, and they also include commercial vehicles used for shipping, agriculture, etc. The use cases range in their analytics response times and data retention requirements. The requirement is to stream their data into an environment for analytics, but many use cases also require historical data that goes back many years. Many of the use cases have structured data that includes geospatial data, data that represents locations, trips, and areas of land. We’ve heard the concern about striking the right balance between cost, performance, and how much data they analyze.
Another consistent theme that we’ve heard dealing with connected vehicles is that companies are looking for partners to help solve these problems. They don’t simply want another data warehouse, reporting tool, or data movement tool. They want a solution provided by a group of experts who will help them tackle this multi-year challenge. They want to work with companies who will work together on the roadmap of the solution together. They want to go at this alone as much as Michael Knight wants to fight crime with a Model T.
Three challenges stalling hyperscale data analytics for connected vehicle manufacturers
From a technical perspective, there are three major challenges when it comes to analyzing data for connected vehicles – data ingestion at scale, geospatial analytics at scale, and usability. Collecting data and streaming it into a data warehouse at petabyte scale requires an infrastructure that can support tens of millions of records streaming per second. These records may or may not be structured and likely require some level of transformation. Once the records are transformed to the layout of the tables in the warehouse, they need to be available for queries with very little latency. The requirements for this data volume come close to pushing the possibilities of physics. They require horizontal scaling of your loading, streaming, and transformation infrastructure so that you can add more hardware and scale linearly. Without the data in the warehouse, you can’t analyze it. In many cases, if it’s there too late, it basically won’t work for the use case.
Geospatial analytics at scale is very difficult. When you’re joining on data that is a simple comparison, like whether one string matches another, massive joins can be done quickly through hardware. When you’re joining tables based on geospatial evaluations, like whether a point exists within a polygon, things get more complex. You’re dealing with the curvature of the earth and complex geospatial algorithms that need resources to compute efficiently. We’ve spoken to customers who are analyzing millions of points, and it brings their system to a standstill. The prospect of analyzing billions, or even trillions, of geographies is daunting.
Vehicle manufacturers want to develop analytics solutions at this scale, and they need to make sure that the system gets used by their people. Their analysts have a wide range of skills from SQL experts to people who can only drag and drop objects in a BI tool. They’re looking for tools that use standards like SQL, JDBC, ODBC, etc. so that the hyperscale analytics solution can serve all their end users. They also need a system that is usable from an internal-management perspective. They don’t want an army of people to manage and support the system. Big data environments that need to be built out piecemeal add to the challenge, cost, and uncertainty.
The opportunity for vehicle manufacturers to embrace hyperscale data analytics is no longer a “nice to have.” It’s now essential to their ability to maintain a competitive advantage not only in the realm of building better vehicles, but also key to developing entirely new streams of services from hands-free driving technology to the latest touchscreen infotainment apps.
Cue the Knight Rider Theme Song
Ocient is uniquely positioned to work with vehicle manufacturers on their hyperscale connected vehicle analytics solutions. We are focused on the same scale, working closely on building solutions as partners, and delivering solutions that can be used with standards. There is no easy way to build out a solution for these types of analytics. Ocient has the vision, technology, and the people to make this possible.
Two years ago, before our product was in production with enterprise customers, we were talking to a massive, well-known car company about our vision around geospatial analytics at scale. I asked them why they were talking to such a new company when they could work with the big established players. They reiterated my points above, saying that solving this problem was going to take a unique set of people, vision, and technology. Two years later into our geospatial journey, we’re demonstrating the vision laid out then.
I want to believe that K.I.T.T would feel confident with today’s connected vehicles. His ability to fight crime and speak naturally with his driver was impressive. It was almost as if the car was backed up by a human actor. Unfortunately, my minivan would probably treat him much like I treat my parents when trying to help them get on a zoom session with their grandchildren. The good news is that there is a solution that can come to the rescue for these analytics problems. It’s Ocient.