By Matt McCraith, Principal Solutions Architect, Ocient National Security Solutions
Command-and-control implants that check in on long intervals (daily, or slower with jitter) are indistinguishable from background noise at small sample sizes. Thirty daily check-ins spread across a month of enterprise flow data do not stand out on any single day. The signal is statistical and cumulative: consistent source/destination pairing, low byte counts, and above all regularity of interval. That aggregate-over-time property is exactly what peer-reviewed detection research measures when it evaluates interval-based and behavioral approaches to command-and-control identification (Computer Networks, 2023).
It is worth meeting an obvious objection head-on: sophisticated adversaries know this too, and the most capable have moved command-and-control onto legitimate cloud services. Recent reporting on APT28’s toolkit documents the shift directly, with custom implants using consumer file-storage APIs like Icedrive and Filen for C2 over encrypted HTTPS, so the destination itself looks benign and the traffic blends with normal SaaS use (ESET, Sednit reloaded, 2026). That evolution changes where the signal lives, but it does not remove the reason to keep full history. Regular or semi-regular check-in behavior is still pervasive across commodity malware and the broad middle of intrusion activity, and even the advanced tooling still polls its cloud destination on a cadence. What changes is that when the destination is a trusted, encrypted cloud endpoint, reputation and signature controls have nothing to flag. The only thing separating a malicious poll from a benign one is the pattern over time (regularity, volume against the host’s own baseline, timing), which is precisely what full-fidelity history lets you compute, and sampled, short-retention data does not. The query below still applies; it just means the interesting dst_ip may belong to a service everyone uses, and the verdict comes from the pattern rather than the address.
Why the data usually defeats the analysis
Two data-layer decisions kill this detection before any logic runs.
Sampling. Capturing every packet on a fast link is often impractical, so flow monitoring is built to sample: process one packet in every N and infer the rest. Cisco’s NetFlow/sFlow documentation shows how wide that dial is, configurable from 1:1 (every packet) all the way to as sparse as one in 262,144 (Cisco). There is no single industry-standard rate; in practice it varies widely with link speed and platform. But a comparatively conservative 1:1000, a common working point rather than the sparse extreme, is enough to make the problem concrete: a flow that occurs 30 times a month is captured, on average, about 3% of the time, and sparser rates on higher-throughput links make that worse, not better. Peer-reviewed work confirms the intuition: packet sampling measurably degrades machine-learning-based detection of malicious traffic on flow data (Computer Networks, 2023). Interval-regularity analysis is even more fragile than classification: you cannot compute the variance of intervals you did not record.
Retention. The confidence of a beacon verdict grows with observation time. Thirty days of history gives you 30 observations of a daily beacon, which is suggestive at best. Twelve months gives you 365, which is conclusive, and enough to see the operator’s working hours, the infrastructure rotations, and the dwell period start. Teams we speak with consistently keep 30-90 days of searchable flow data; the analysis they want to run needs quarters, not weeks.
The detection, in SQL
Given unsampled flow records with multi-year retention, beacon candidates reduce to a grouped interval-statistics query: group by source/destination pair and test the regularity of the intervals between contacts. Conceptually (simplified for clarity; production versions add allow-listing, byte-symmetry filters, and jitter tolerance):
SELECT src_ip, dst_ip, contacts, avg_gap, gap_stddev
FROM (
SELECT
src_ip,
dst_ip,
COUNT(gap_seconds) AS contacts,
AVG(gap_seconds) AS avg_gap,
STDDEV(gap_seconds) AS gap_stddev
FROM (
SELECT
src_ip,
dst_ip,
EPOCH(start_time) - EPOCH(LAG(start_time) OVER (
PARTITION BY src_ip, dst_ip
ORDER BY start_time
)) AS gap_seconds
FROM flows
WHERE start_time >= now() - months(18)
) gaps
GROUP BY src_ip, dst_ip
) stats
WHERE contacts >= 20
AND gap_stddev < 0.1 * avg_gap;
Nothing here is exotic. The outer WHERE clause is the entire detection: many contacts, metronomic spacing. The inner query computes the gap between each contact and the one before it; the middle layer turns those gaps into per-pair statistics; the outer filter keeps only the pairs whose spacing is too regular to be organic. This runs as written on the OcientAIQ™ SQL dialect.
Jitter doesn’t defeat this. Adversaries who randomize check-in intervals to avoid looking mechanical are still bounded by a plausible operational range, and the gap_stddev < 0.1 * avg_gap threshold can be loosened to accommodate that variance while still excluding the much wider randomness of organic traffic.
Separating a real beacon from benign periodic traffic like software update checks and heartbeat pings comes down to two filters production versions add on top: allow-listing known vendor update endpoints, and checking how many other hosts in the environment contact the same destination. A beacon’s destination is typically unique to the compromised host or a small set of them; a software update server is contacted by every machine running that software.
What makes this feasible or infeasible is the FROM clause: an 18-month window over unsampled flow records is billions to trillions of rows at enterprise-to-carrier scale, and the query has to return interactively enough that an analyst can iterate on thresholds.
The capability point
That last requirement is where platform architecture matters more than detection logic, and where most current architectures quietly fail. If the long history lives on unindexed object storage behind a filtering pipeline, the 18-month query above is a scan job measured in hours, and no analyst iterates on thresholds at that speed. If it lives in a consumption-priced cloud platform, every iteration has a visible cost attached, and threshold tuning (dozens of exploratory runs against the full window) becomes a line item someone questions. Either way the detection exists on paper and not in practice.
In our experience, teams rarely run a scan like this proactively. The hours-long job and the metered iteration tend to get greenlit only once something has already gone wrong, in the middle of an incident-response engagement, when cost is no longer the objection. That is precisely the moment slowness is most expensive. A scan job measured in hours is not a line item at that point; it is scope that keeps growing while the query runs, and threshold tuning against a live incident does not get the dozens of exploratory passes it needs. Teams we have supported through an active IR describe this as one of the more painful parts of the response: not a hypothetical edge case, but the moment the data layer is asked to perform and cannot.
Three design choices make it practical instead: full-resolution capture (every flow record, since the statistics are only as good as the completeness); indexes built at load time, so the interactive path over multi-year history is a lookup, not a scan; and a rollup-plus-raw layout where long-horizon aggregates are computed as data loads while raw records remain available for drill-down. Serious network operators already run retention this way, with multi-year daily rollups over shorter raw windows, and it is the pattern we build into OcientAIQ deployments, at ingest rates and row counts (hundreds of millions of source IPs an hour, trillions of retained rows) that this category of analysis was widely assumed not to reach.
The indexing point is easy to skim past as an implementation detail, but it is the reason any of this works. This is TimeKey® indexing, described in more depth in our previous post: rather than committing to a single clustering key, OcientAIQ builds multiple indexes across a table’s columns at load time. A single index gets you a fast lookup on one column; the beaconing query above groups and filters on src_ip, dst_ip, and time in the same pass, and a rollup this wide only stays interactive if the engine can use more than one index at once instead of falling back to a scan the moment the query touches a second column. That is what lets an analyst iterate on the detection thresholds in real time rather than submitting a query and going to lunch. It is also tied to the other half of the same design problem: an engine that can index at load time but cannot load fast enough to keep up with the flow export off a large network never gets the chance to prove the query is fast, because the data queue backs up first. Both halves have to hold at the same time for the SQL above to be a scheduled job instead of a demo.
The same architectural principle applies here as in the fuller scaling piece: ingest rate, storage depth, and query concurrency scale independently on the OcientAIQ Unified Data Platform rather than sharing one resource pool. A team whose bottleneck is ingest can add ingest capacity without paying for query capacity it doesn’t need, and vice versa. That independence is what keeps an 18-month interval-statistics query interactive instead of a scan job. It is also a meaningful part of why running this detection at petabyte-to-exabyte scale tends to cost less than what most organizations already spend for a narrower, samples-only version of the same visibility.
To be clear about where this sits: a detection like this runs against the deep flow history this layer holds, and its output is an alert into the SIEM or SOAR your analysts already use. The point is not to move the SOC off the tools it runs on; it is to give those tools a memory long enough and complete enough that a query like the one above returns an answer instead of a shrug.
The teams we work with describe the outcome simply: detections they had classified as “not possible with our data” became scheduled queries. That reframing, from tooling problem to data problem to solved problem, is the recurring theme across this series.
Ready to learn more? Get in touch.