THE CHALLENGE
The operational needs necessitated the rerouting of crucial service alerts to a specialised incident management team. This directive specifically mandated the transmission of automated alerts from monitoring systems to a designated destination.
However, the internal event management team exclusively facilitated notification capabilities through only by SMTP (email), an internally accessible HTTP API, or an external API that mandated custom headers for authentication credentials to be passed.
Unfortunately, Google Cloud Monitoring lacked the functionality to dispatch notifications to internal API endpoints. Additionally, it lacked the capability to incorporate the essential custom headers necessary for authentication when interfacing with the external API. Also, while the SMTP option still existed for external data transmission, by nature SMTP is unreliable and has other security vulnerabilities so it also became an unsuitable choice.
Given these constraints, a superior solution was imperative to ensure seamless and secure transmission of critical alerts.
THE SOLUTION
Source: Forbes
In response to the critical need for secure cloud-based system functionality, we devised and executed a comprehensive strategy:
- At the core of this initiative was the establishment of a Pub/Sub topic, serving as the central conduit for Cloud Monitoring to send alerts to (a native feature of the service).
- A Managed Instance Group comprising Elastic Logstash compute instances was intricately configured to subscribe to this topic whose primary role was to meticulously parse the incoming data from messages and seamlessly transmit it to the internal HTTP endpoint of the event management team.
- To ensure the secure transmission of this data, we leveraged an existing Direct Interconnect, which had been previously established for general on-premise to Cloud connectivity, thereby ensuring a robust and reliable pathway for data transfer.
- To streamline the alerting process, access permissions were meticulously granted to the topic. This enabled the monitoring of service accounts, facilitating the setup of Cloud Monitoring Notification Channels. These channels were instrumental in forwarding critical alerts directly to the designated topic.
- A crucial element in this setup was the utilisation of the Alert Policies Documentation Field. This field was employed to embed custom data essential for the internal API within the alerts themselves, ensuring that all requisite information was encapsulated within the alerts.
- Logstash, adept at extracting and manipulating data, extracted the required information from the designated field within the message with very little configuration. It then reformatted this data to align precisely with the specifications required by the API, ensuring seamless compatibility between the alerts and the backend systems.
- This strategic design not only facilitated the secure transmission of critical alerts but also introduced a layer of adaptability and flexibility. Changes or updates to the backend event management system could be effortlessly accommodated without necessitating extensive modifications to thousands of alert policies, streamlining the maintenance process significantly.
THE RESULTS
![Logstash Introduction | Logstash Reference [7.17] | Elastic](https://www.elastic.co/guide/en/logstash/7.17/static/images/logstash.png)
Source: Elastic
Enabling Compute Metadata within the Instance Template was a pivotal step in empowering Logstash to make informed environmental decisions, conduct automated acceptance testing, and ensure uniformity across different environments.
The decision to utilise Logstash was strategic on multiple fronts. Its adaptability to function within a network with internal routing was instrumental. Additionally, the existence of a pre-existing Logstash solution significantly expedited the deployment process for this use case, requiring only minimal adjustments to the pipeline code.
The adoption of Pub/Sub as the message bus service proved to be a game-changer. Its inherent guarantee of at-least-once delivery not only ensured the reliability of message transmission but also facilitated effective management of Logstash subscriber scaling. Simplicity in implementation was another significant advantage, as Pub/Sub boasted minimal management requirements, streamlining the deployment process.
A key optimisation came from the augmentation of user-provided data in Alert Policies with the essential information required by the API. This strategic integration significantly simplified and accelerated user access to the centralised alerting system, enhancing the overall user experience.
By leveraging this suite of tools and services, the organisation successfully transitioned away from external routing of alerts. Instead, it embraced robust, reliable, and secure protocols and services, reinforcing the integrity and safety of its alerting infrastructure.
The culmination of these efforts resulted in a substantial enhancement of the client's ability to effectively monitor and manage its cloud-based systems. This transition ensured seamless and uninterrupted operations, bolstering the organisation's resilience in handling critical system alerts...
...And we were able to deliver this whole service which could be used by all users of the organisation in a controlled and agile manner in only a few weeks!
As IT specialists, Mesoform can help your business overcome similar challenges and provide efficient solutions in comparison to competitors.
To find out how Mesoform can help your business become more secure, stable and efficient contact us at This email address is being protected from spambots. You need JavaScript enabled to view it.
https://www.mesoform.com/contact
If you would like to discuss any of these topics in more detail, please feel free to get in touch
