11 million installations can't be wrong

MySQL Journal

Subscribe to MySQL Journal: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get MySQL Journal: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


MySQL Journal Authors: Sematext Blog, OnPage Blog, Dynatrace Blog, Greg Schulz, Jyoti Bansal

Related Topics: MySQL Journal

Blog Feed Post

Docker Log Management & Enrichment

Over the last several months we’ve made all kinds of improvements to Sematext Docker Agent (SDA).  If you’re not familiar with SDA yet, here it is in a nutshell

Sematext Docker Agent is a modern, open-source, Docker-native monitoring and log collection agent. It runs as a tiny container on each Docker host and provides automatic collection and processing of Docker Metrics, Events and Logs for all cluster nodes and all auto-discovered containers. It works with Kubernetes, Mesos, Docker Swarm, Docker Datacenter, Docker Cloud, as well as Amazon EC2, RancherOS, and CoreOS.

By default SDA collects all logs from all containers, but we’ve recently added LOGSENE_ENABLED_DEFAULT, a new flat that lets you control if log collection should be enabled by default or not. When set to false,  SDA will collect logs only from containers labelled with a LOGSENE_TOKEN=<Logsene app token> that, in addition to enabling log collection, also specifies where logs should be shipped. This is really useful because it:

  • Gives each team in an organization full control over log collection for their containers and the routing to their Logsene Apps or Elasticsearch indices. As you can imagine, this is very handy for larger organisations where e.g. one Swarm or Kubernetes cluster is shared by several teams, and each team needs to enable/disable log collection however they see fit and ship logs to different destinations.
  • Lets one exclude collection of logs from infrastructure containers whose logs may not contain enough value to be worth collecting.

The figure below illustrates this.  There are 3 containers running Nginx, Rancher agent, and MySQL.  We want to have explicit control over which containers logs are collected and where they are shipped.  We want to collect only Nginx logs, and we want to ship it to a specific Logsene app, not the default one set in SDA. We can accomplish that by doing the following

  • Set LOGSENE_ENABLED_DEFAULT=false flag in SDA config
  • Set Docker label LOGSENE_ENABLED=true for the Nginx container to enable its log collection
  • Set Docker label LOGSENE_TOKEN=…. for the Nginx container to specify which Logsene app we want to ship logs

https://sematext.com/wp-content/uploads/2017/05/LOGSENE_ENABLED_DEFAULT_... 300w, https://sematext.com/wp-content/uploads/2017/05/LOGSENE_ENABLED_DEFAULT_... 768w" sizes="(max-width: 960px) 100vw, 960px" />

Sematext Docker Agent log routing via container labels – disable log collection by default and enable it only for the Nginx container

 

We can also inverse the setup and enable log collection for all containers by default, as illustrated in the following figure.

https://sematext.com/wp-content/uploads/2017/05/LOGSENE_ENABLED_DEFAULT_... 300w, https://sematext.com/wp-content/uploads/2017/05/LOGSENE_ENABLED_DEFAULT_... 768w" sizes="(max-width: 960px) 100vw, 960px" />

Sematext Docker Agent log routing via container labels (default settings) – collect and ship all logs, either to default Logsene app or to container specific apps.

We’ve also introduced TAGGING_LABELS.  This flag lets you enrich your container logs with data extracted from your existing Docker environment variables or labels.  Just specify patterns for values to extract, e.g. TAGGING_LABELS=”com.docker.,com.myorg.,role*”. This will add extracted values to your log events as additional fields (you can also think of this as meta-data) and let you easily slice and dice your logs by using values from labels or environment variables.  You can, of course, also build custom reports using this data, thus extracting more value and operational insight from your existing data in Sematext Cloud.

https://sematext.com/wp-content/uploads/2017/05/LOGS_WITH_DOCKER_LABELS-... 300w, https://sematext.com/wp-content/uploads/2017/05/LOGS_WITH_DOCKER_LABELS-... 768w, https://sematext.com/wp-content/uploads/2017/05/LOGS_WITH_DOCKER_LABELS-... 1024w" sizes="(max-width: 2419px) 100vw, 2419px" />Enriched logs with Docker labels

Need a tool that collects your containers Metrics + Events + Logs?  Try Sematext Docker Agent, it’s all open-source and can send your data to Sematext Cloud so you don’t have to manage or build out the backend for storing all the monitoring data, alerting, etc.  For feature requests, bugs, or PRs, see sematext/sematext-agent-docker.

 

Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.