Incremental computation for data center monitoring

How to update rack and room telemetry from changed sensor readings and measure whether the approach saves work on a real operations trace.

NVIDIA and Wistron thermal visualization showing heat above a row of server racks

Start with a rack or room rollup

A monitoring service may keep the latest power reading from every rack and publish a room total. When one rack reports a new value and the other racks stay as they were, a service that rescans every rack repeats almost all of the work of its previous scan.

The alternative is to keep each rack’s previous reading beside the room total. When a rack reports, subtract its previous value, add the new one, and store the new one. The same difference updates a facility view whenever that view totals the same rack readings. The NVIDIA and Wistron thermal visualization above is a separate example, from NVIDIA’s PhysicsNeMo data center design study, which models temperature and airflow in a hot aisle.

A room total is the simplest case of a general idea. Differential dataflow reports only the output differences caused by input changes and does no work elsewhere, and the DBSP paper describes incremental maintenance of grouping, aggregation and recursion in rich query languages. The rack sum needs neither framework to show the principle.

Illustrative. A room of 48 racks in which one rack reports 5.1 kW instead of 4.8 kW. The room total changes by 5.1 − 4.8 = 0.3 kW.

Handle the events that break a simple sum

In an illustrative room, a rack drawing 4.8 kW reports 5.1 kW, and the room total rises by 0.3 kW. A retry of the same event must add nothing the second time. Apache Kafka’s idempotent producer discards retried sends by tracking sequence numbers per producer and partition, and the producer configuration documents that with idempotence off a retry may write duplicates. That covers one hop between a producer and a broker, so the service still needs its own check for a repeated reading.

A reading that arrives late needs a timestamp check so it cannot replace a newer one. Apache Flink’s documentation separates event time, the time a reading occurred at its source, from processing time, and uses watermarks to mark progress through event time. Events that arrive after the watermark has passed their timestamp are late. If a rack moves rooms, subtract its contribution from the old room and add it to the new one.

The aggregate determines how much state to keep. An average can be maintained from its sum and count. A maximum needs another way to find the next hottest reading when the current maximum drops. A sliding window also changes when a reading expires, even if no device sends a new one. Test each of those transitions against the existing service before replacing a refresh.

Replay the bursts and measure the service

Replay timestamped readings through the current service and the incremental version. Include normal operation, delayed reports, missing sensors, synchronized polls, and the first refresh after a restart. Compare the outputs after each batch. Measure update latency, CPU time, and energy per fixed amount of telemetry at the same ingestion rate. Include the cost of keeping the previous readings and rebuilding state after a restart.

Use the same measurement boundary for both runs. The U.S. Department of Energy’s best practices guide defines power usage effectiveness as the ratio of the total annual energy to run the data center facility to the total annual energy to run the IT equipment, and notes that it does not define the overall efficiency of a data center. ISO/IEC 30134-2 standardizes the same indicator. It relates two energy totals. To learn whether a software change did less work, measure the monitoring workload directly. A facility-level claim calls for its own metering and operating data.

Illustrative. The same 48 racks and the same single change. The delta update subtracts the stored 4.8 kW and adds 5.1 kW to the room total. It needs the stored value for every rack, which is a state cost to measure.

Start with a rack or room rollup

Handle the events that break a simple sum

Replay the bursts and measure the service

Sources