Implementation of a Large Scale Monitoring System

Imagine you are setting up a backend system that is providing services to over one million autonomous devices that are in constant, real-time communication with the backend – Internet of Things at scale.

Data from the devices is passed through a chain of systems and even routed to 3rd party systems for further computation. Results are then passed back to the devices in near real-time so that they can make autonomous decisions.

The Challenge

The challenge is to keep an overview of what is going in in this huge, distributed system. There is no lack of data about the backend system and the devices connected to it: on the contrary, you need to analyze huge amounts of it, present aggregated facts to decision makers and even make autonomous control decisions, all in near real-time.

The Solution: Elasticsearch to The Rescue

We designed and implemented a monitoring system based on the Elastic stack: using Logstash to filter and extract the relevant data, Elasticsearch to store it and provide fast search capabilities and Kibana to display it.

The system is running worldwide and consists of dozens of Elasticsearch clusters, the largest of which has over 100 nodes. Over 8 terabytes of data are ingested daily.

Our Expertise

Data strategies and architectures

Target Group

Data Evangelists

Upcoming Events
Training AI to Play The Beer Distribution Game
2 hours - 2021-10-27
Mastering The Complexity of Digital Transformation
2 hours - 2021-10-28
Modeling Growth Strategies For Professional Service Firms
2 hours - 2021-11-18
Go to event overview

Recent Blogposts
Meetup: Agile Transformation Tools
Olga Medvedeva - 2021-09-27
Meetup: Understanding The Beer Distribution Game
Olga Medvedeva - 2021-09-01
Introduction to The Business Prototyping Toolkit
Oliver Grasl - 2021-08-17
Go to blog overview

Email newsletter

Subscribe to our newsletter and stay up to date about the latest trends and news.

Name

Email