I always enjoy to visit telcos’ monitoring centres. One feels like to be in NASA. It’s a pity that mostly it is not allowed to take any pictures.
It is interesting that rarely there is just one vendor of this monitoring. Usually it is a mix of various applications, which inform about the most important KPIs and events. Often we can see open applications to be used. If you are working in support you probably know at least some of them – cacti, zabbix, nagios, observium, MRTG, munin, zenoss, etc. One of them is grafana.
I’ve recently seen a T1-operator monitoring based on Grafana. The nice thing about it is that it allows to include various data sources, analytics where we can easily combine charts with related events and it supports multi-tenancy. Just before Christmas we have got a new version 4.1 with a new alerting feature. The alert rules are easy to configure using existing graph panels and threshold levels can be set simply by dragging handles. The rules will continually be evaluated by grafana-server and notifications will be sent out when the rule conditions are met. Besides the altering there are many other improvements and enhancements.
Anyway whatever tool you use, happy NYE monitoring!