Monitoring Slow Queries on MongoDB

Overview

Database performance issues can be difficult to identify without sufficient observability. To improve visibility into MongoDB performance, we introduced slow query monitoring within an existing Kubernetes-based deployment.

The objective was to identify long-running database operations, capture them through the existing logging infrastructure and provide actionable alerts to the engineering team.

What initially appeared to be a simple configuration change required a deeper understanding of the existing MongoDB deployment, including the container entry point, startup behaviour, configuration management and database initialisation process.

Challenge

The existing MongoDB deployment was already running successfully in Kubernetes, so slow query monitoring needed to be introduced without disrupting the established deployment.

The main challenge was that MongoDB was using the official container image, which included its own entry point and startup process. Changing how the container was launched or replacing the entry point could unintentionally bypass existing functionality, including the database initialisation process.

The MongoDB version also needed to be considered, as available configuration options and profiling behaviour can vary between versions.

The implementation therefore needed to:

• Enable slow query monitoring without disrupting the existing deployment
• Preserve the official MongoDB container entry point
• Introduce the required configuration through the existing Kubernetes deployment
• Enable database profiling with an appropriate slow query threshold
• Capture slow query events through the existing logging infrastructure
• Convert those events into CloudWatch metrics and alerts
• Validate the complete solution before production rollout

Solution

The solution was to enable MongoDB slow query profiling while preserving the existing container startup behaviour.

A Kubernetes ConfigMap was created to manage two components:

• A mongod.conf configuration file containing the MongoDB profiling settings
• An initialisation script used to enable profiling for the relevant databases

MongoDB was configured to identify slow operations using a threshold of 100ms.

Rather than replacing the official MongoDB container entry point, the configuration file was passed to the container through its startup arguments. This allowed the existing entry point to continue managing the normal MongoDB startup process.

The initialisation script was mounted into the container's /docker-entrypoint-initdb.d/ directory, allowing the official MongoDB entry point to execute it as part of the database initialisation process.

The monitoring flow was then configured as:

MongoDB → Slow Query Logs → CloudWatch → Metric → Alert → Slack

A CloudWatch log metric filter was created to identify MongoDB slow query events. These events could then be converted into metrics and used by the existing alerting infrastructure to notify the engineering team through Slack.

Testing and Validation

Before introducing the changes to the main environment, the MongoDB configuration was tested locally using the same MongoDB version as the target deployment.

This confirmed that:

• The MongoDB configuration loaded correctly
• The existing container entry point continued to function
• The initialisation script executed successfully
• Database profiling was enabled
• Slow queries were recorded in the system.profile collection
• Slow query events appeared in the MongoDB container logs

The CloudWatch monitoring component was also tested independently using Terraform. A test log group and metric filter were created, followed by a simulated MongoDB slow query event.

This confirmed that the expected log pattern could be detected and converted into a CloudWatch metric before being integrated into the wider monitoring and alerting workflow.

The complete configuration could then be validated in an environment representative of production before rollout.

Results

The implementation improved visibility into MongoDB database performance and provided a repeatable approach for detecting slow queries.

Slow database operations could now be:

• Identified through MongoDB profiling
• Captured through MongoDB logs
• Detected using CloudWatch log metric filters
• Converted into monitoring metrics
• Used to trigger alerts through the existing Slack notification workflow

The implementation also demonstrated the importance of understanding existing infrastructure before making configuration changes. In this case, preserving the official MongoDB container entry point was essential to introducing the new monitoring capability without disrupting the established deployment.

By testing the MongoDB configuration, logging and CloudWatch components separately before rollout, potential issues could be identified and resolved incrementally.

What began as a requirement to detect slow database queries ultimately improved the platform's wider observability capabilities, providing the engineering team with greater visibility into database performance and potential bottlenecks.

Improve Your Database Observability

Slow database queries can be difficult to identify before they begin affecting application performance. At Mesoform, we help organisations improve observability across their cloud infrastructure, applications and data platforms.

Looking to improve the visibility, monitoring and performance of your cloud environment?

Get in touch with Mesoform to see how we can help.