Operational Envelopes

Boundary helps distributed systems operators by providing insights into their infrastructure using network flow data. Of the questions an infrastructure operator might ask, one of the most important is:

Is the system in an expected or normal state?

To help our users answer this question, we explored the problem within the larger context of providing richer alerting and fault detection capabilities from Boundary. Some of the use cases we imagined are:

Notify a user when a system moves from a perceived normal state to a new perceived normal state.

Use perceived normal states to track multiple normal state paths a system can take before a partial or complete failure.

Use the observed normal state paths to help find how close the system is to observed faults/failure scenarios.

IT operators and managers establish SLAs with Boundary to report/alert when a system approaches failure scenarios — e.g. notifying when system state matches observed failures with 90% confidence.

In this post, we provide an answer to the above question by building normal behavior profiles from network data on Boundary's internal infrastructure.

Specifically we describe our prototype experiments, observations, and their results. As we continue this series of articles, we'll begin looking toward:

• Description of tracking of multiple normal states attained by a distributed system.
• Failure paths derived from the captured multiple normal and outlier states.
• Using failure paths to provide further insights as described in the use cases above.

Experiment Setup

For our experiments, we subscribe to streaming data from our analytics engine using Boundary's "Conversations" API — specifically, the volume_1s_port_protocol query.

This enables us to look at traffic data between two services on a single port. This approach can be easily extended to other streaming queries provided by Boundary.

Operational Envelope

We use computational geometry to build a normal behavior profile that we call an "Operational Envelope". Operational Envelopes are created by taking a configurable window of data points, and drawing an "envelope" over these data points.

We use the Graham scan algorithm to create an Operational Envelope. For data collected on a volume_1s_port_protocol query between all ZooKeeper nodes on port 2888 (used for internal peer-to-peer communication, e.g. leader election):

• X-axis: Bits Received
• Y-axis: Bits Sent
• Ratio = Outlier Count / Normal Count

Operational Envelope Construction

The X axis shows bits received and the Y axis shows bits sent.

We use ingress and egress traffic volume as dimensions — the ingress and egress traffic volume of a service will often exhibit meaningful correlations during the normal operation of a distributed system.

We exploit this property to define the normal behavior profile using traffic volume correlations for distributed systems.

The same principle can be applied for creating Operational Envelopes on other metrics provided by Boundary, such as packet retransmits or TCP handshake latency.

Outlier Detection

The primary objective of creating an Operational Envelope is to help detect outliers and fault states.

Any traffic that falls inside an Operational Envelope represents normal behavior, and any traffic that falls outside is an outlier.

For example, 5% of all observed points were tagged as outliers. This ratio changed from 1% to 5% by approximately 700 data points.

The outlier points that fall farther from an Operational Envelope represent sudden spikes: breaks from normal behavior.

In the context of internal ZooKeeper traffic on port 2888, this signifies that the peer-to-peer communication among ZooKeeper nodes spiked — potentially due to state communication between nodes, e.g. leader election.

Metrics

Outlier Ratio = Outlier observations count / Normal observations count.

Results for four experiments on ZooKeeper traffic flow on port 2888, each experiment (except the last) running for approximately 5 minutes:

Outlier Count	Normal Count	Outlier Ratio %
31	648	5
70	1039	7
88	1302	7
149	2000	7

We see that outlier ratio achieves a constant value and some outlier points are on the edge of an Operational Envelope. This indicates:

• For a number of points that fall inside normal state, there are corresponding number of points in outlier state to keep the outlier ratio constant.
• The Operational Envelope created using the initial window size is not large enough to cover the spread of normal behavior for the given query.

A constant outlier ratio also potentially signifies a new "normal behavior".

There are two ways to handle this:

• Use the above information to change the window size to a larger value and start fresh to create a new Operational Envelope.
• Use these metrics as a trigger to dynamically evolve the Operational Envelope.

Challenges and Next Steps

With this approach for normal behavior and outlier detection defined, we must now move to make it more useful. A few approaches we investigated:

• Transforming and displaying an Operational Envelope as an evolving time series.
• Enabling users to tag and calculate false positive rate for outlier detection.
• Providing configurable tolerances to tag/reassign some outlier points as normal state.
• Methods to capture evolving normal states that a distributed system can attain due to workload, internal or external changes.

Conclusion

This post introduced Operational Envelopes and explored the ways they can improve one's understanding and debugging of distributed system behavior using network flow data. We also described the construction of Operational Envelopes, and their use for alerting and fault diagnosis capabilities.

There are potentially other methods and techniques one might use to answer this question. For details on our path to this particular approach, see: Tech Talk: Finger Pointing in Distributed Systems

References

[1] Edward Stehle, Kevin Lynch, Maxim Shevertalov et al. On the use of Computational Geometry to Detect Software Faults. In Proceedings of the 7th IEEE/ACM International Conference on Autonomic Computing and Communications, June 2010.