The Pathologies of Big "Messy" Data in Telco • Jay Fenton • YOW! 2015
Jay Fenton discusses the challenges of "big messy data" in the telecommunications industry, highlighting how legacy systems fail to handle the scale and complexity. He presents a suite of modern tools, including Graphistry, IPython, and D3.js, for large-scale data visualization and analysis, advocating for a "human-computer symbiosis" to empower engineers in managing complex networks.
The telecommunications industry, while a progenitor of big data, suffers from unique pathologies related to what can be called “big messy data.” Unlike the problems of scale addressed by companies like Facebook or Google, the challenge in telecoms lies in data that is often ill-formed, error-prone, and processed by legacy systems. For decades, these latent errors went undetected because the tools to inspect the data at scale simply didn’t exist.
The Ailments of a Legacy-Driven Industry
The telecom sector is insular, slow to adopt the innovations common in the wider tech world. Its infrastructure often relies on:
- Archaic Tools: Core operations still run on software like Nagios (1999) and Cacti (2001), which are incapable of visualizing or managing networks at the scale of tens of thousands of devices and millions of customers.
- Prohibitively Expensive Databases: Vertically-scaled Oracle databases are common, but the multi-million dollar cost to scale them means that companies choose not to collect the vast majority of available data. For instance, a network device might offer over 24,000 monitorable metrics, of which 10,000 are counters, yet a typical ISP monitors only about 12 of them at a slow 15-minute cadence.
- Inefficient Data Handling: Complex data pipelines rely on batched file transfers, often using voluminous XML for counter data, introducing significant latency. A single report might depend on a pipeline processing 40,000 files per hour, making real-time fault management nearly impossible.
- No Single Source of Truth: Different departments maintain their own overlapping but incomplete inventory datasets, leading to inconsistencies and “revenue leakage”—services provided but not billed, or vice-versa.
These issues are compounded by the fact that the data itself is multi-layered and complex, reflecting the underlying network protocols. A problem at the TCP layer might originate from a fault at the physical transmission layer, but the tools to correlate these events are lacking.
A Modern Toolkit for Messy Data
To combat these pathologies, a modern approach is required, focusing on empowering human operators through advanced data processing and visualization.
Processing and Querying with IPython and Splunk
IPython (Jupyter) notebooks, combined with libraries like Pandas and Numpy, provide a powerful, iterative environment for cleaning, analyzing, and questioning data. It’s an indispensable tool for exploring datasets, comparing sources of truth (e.g., Cacti vs. Splunk data), and even generating simple but effective visualizations like a bitmap of circuit latency over 30 days, which can immediately reveal systemic problems across millions of data points.
For heavy lifting and dealing with messy data formats, Splunk is invaluable. Its query language (SPL) is particularly effective for the kind of regex extractions and data transformations required to make sense of telco data.
Visualizing Network Topology with Graphistry
To understand the complex relationships within a network, graph visualization is key. While older tools like Gephi exist, modern platforms like Graphistry offer a transformative experience. By leveraging server-side GPUs for massive-scale graph layout and client-side WebGL for rendering, Graphistry allows engineers to interactively explore a network of tens of thousands of nodes in a web browser. This “living diagram” makes it easy to spot structural anomalies, such as disconnected islands of equipment that represent configuration errors. The data for these visualizations is often prepared in an IPython notebook and plotted with a single command.
Visualizing Time-Series Data with D3.js and WebGL
For the massive volumes of time-series data generated by networks, new visualization paradigms are needed.
- D3.js: This JavaScript library is perfect for creating custom, data-driven visualizations. One powerful application is a “waterfall” diagram to debug complex data pipelines, showing the state of thousands of files over time and instantly highlighting processing delays or failures that would otherwise be lost in logs.
- WebGL and 3JS: For visualizing the raw performance data of thousands of circuits simultaneously, WebGL can be used to render a 3D surface where the x-axis is time, the y-axis represents each circuit, and the z-axis shows a metric like traffic volume. This creates an intuitive view of the network’s “ebb and flow,” making patterns like link saturation, data loss, and the behavior of synthetic test circuits immediately apparent. With custom vertex and pixel shaders, it’s possible to render millions of data points smoothly in a browser.
The Philosophy: Human-Computer Symbiosis
For years, the industry attempted to solve network management with purely algorithmic, expert-system-based root cause analysis, with little success. The crucial missing element was the human operator. The most powerful pattern-matching machine on the planet is the human brain. The goal of these modern tools is not to replace the human, but to achieve a “human-computer symbiosis.” By collecting more data and using technology to visualize it in dense, intuitive ways, we empower engineers to use their innate pattern-recognition abilities to manage complexity at a scale that was previously impossible. This approach is not limited to telecoms; it will be essential for managing the highly distributed, large-scale microservice architectures of the future.