By Tom Hofman, VP of Solutions, Ocient National Security Solutions
Threat intelligence has a timing problem that everyone in security operations knows and almost nobody’s data layer accommodates: indicators arrive after the activity they describe. A C2 domain published today may have been active for a year. The operationally important question is not whether the indicator will appear tomorrow — your detection stack will handle that within minutes of loading it. It is whether the indicator already appeared in your environment before anyone knew to look for it.
What the practitioner literature says
This gap is well documented in established security research. SANS Institute’s analysis of retroactive and proactive threat intelligence hunting lays out the same problem this post is built around: most organizations receive a list of indicators of compromise and are left to conduct ad hoc retroactive hunts for them across whatever historical data happens to still be available, with the completeness of that hunt bounded entirely by retention depth rather than analyst skill (SANS Institute, “On the Hunt: The Retroactive and Proactive Hunt for CTI Indicators”). Verizon’s 2025 Data Breach Investigations Report, drawn from more than 22,000 analyzed incidents, independently reinforces why the timing problem is real: third-party involvement in breaches doubled year over year to 30%, and a meaningful share of the incident dataset involves compromises that were active well before the organization (or a threat intelligence feed) had any indicator to search for (Verizon 2025 DBIR).
Even where historical data nominally exists, the mechanics of most architectures work against retroactive search. Splunk’s own documentation for federated search against Amazon S3, one of the more common patterns for keeping long-tail data cheaply, states plainly that the feature “is not intended for high frequency and real-time searches” and that customers attempting to use it that way “will likely incur higher costs than natively ingesting and searching in the Splunk platform” (Splunk documentation). That is not a criticism of Splunk. It is Splunk being honest, in its own documentation, about what the architecture is and isn’t for. The pattern it describes is the industry-wide one: hot, warm, and cold tiers are typically different systems, and moving a large intelligence sweep from one to the other is exactly the kind of workload the vendors themselves flag as a poor fit.
In our own conversations with financial-institution security teams, the most common phrasing is some version of: “we can answer for the last 60 days; beyond that we give the honest non-answer.” For a systemically important institution responding to a regulator or a board, the honest non-answer is expensive.
The sweep as a query
With full-resolution flow and DNS history that stays indexed and queryable, a retrospective sweep is a join, not a project:
SELECT f.src_ip, f.dst_ip, f.flow_start_time,
f.bytes_total, i.indicator_value, i.source
FROM flows f
JOIN threat_intel_indicators i
ON f.dst_ip = i.ip_value
WHERE i.batch_date = CURRENT_DATE -- today's new indicators
AND f.flow_start_time >= CURRENT_DATE - INTERVAL '24' MONTH
ORDER BY f.flow_start_time;The same pattern covers DNS (join on query name), TLS (join on certificate hash or JA4 fingerprint), and combinations. In production this runs as a standing job: every new intelligence batch sweeps the full history automatically, surfacing matches as normalized alerts in the SIEM or SOAR queue analysts already monitor. The sweep runs against the deep history this layer holds; the finding surfaces where the SOC already looks. Nothing about this asks anyone to leave the tools they have.
Two properties make this real rather than aspirational. First, completeness: a sweep over sampled data returns “no matches found in the 3% we kept,” which is the honest non-answer again. Second, interactivity at scale: the join above touches years of records; if it takes a weekend, analysts will not iterate on it, and if it requires rehydration tickets, it will not run at all. Index lookups against multi-year history (IP quality, domain matching, fingerprint equality) are precisely the query shapes a purpose-built analytics layer should serve in seconds to minutes, and the shapes we optimize OcientAIQ™ deployments around.
Why current architectures make sweeping expensive by design
Notice the shape of the obstacles in those accounts: cold tiers that require rehydration projects, federated search modes that the platforms themselves scope to occasional use, per-search scan caps, and metering by data scanned. None of these are bugs. They are the natural consequences of architectures that separate “affordable retention” from “queryable data”, and of pricing models that charge for looking. A standing job that sweeps every new intelligence batch against two years of full-resolution history is precisely the workload those designs discourage: high-frequency, large-scan, exploratory.
The OcientAIQ answer inverts the design assumptions. History stays in one tier (indexed at load, resident on the platform, queryable in ANSI SQL), so a sweep is an indexed join, not a scan of a cold bucket. Specifically, the join above needs an indexed lookup on dst_ip and a time filter on flow_start_time at the same time, across the table, which is a two-column index problem, not a one-column one. Building multiple indexes on the same columns at load time, rather than picking a single clustering key and living with a scan whenever a query needs a different angle on the data, is what keeps that join a lookup instead of a table scan as the history grows into the trillions of rows. And under fixed-capacity economics the thousandth sweep costs the same as the first: nothing marginal. That combination is what turns retrospective intelligence from an occasional project into a scheduled control, and it holds at scales (trillions of flow records, years of history) where the conventional wisdom said interactive lookup was not achievable.
The capability framing
In our experience, institutions evaluating this tend to frame it not as a cost question but as a capability threshold: either the organization can answer “were we ever exposed?” completely, or it cannot. Teams that can answer it completely change their intelligence workflows: indicators stop being only forward-looking watch entries and become questions immediately asked of the past. That is a different security posture, not a faster version of the old one.
Ready to learn more? https://ocient.com/contact-us.