Kubernetes is an open source platform to manage containerized applications. it enables you to manage, scale and deploy your containerized apps in a clustered environment. We can orchestrate our containers across various hosts with Kubernetes, scale the containerized apps with all resources on the fly, and have a centralized container management environment. In effect, while Kubernetes solves old problems, it can also create new ones. Specifically, adopting containers and container orchestration requires teams to rethink and adapt their monitoring strategies to account for the new infrastructure layers introduced in a distributed Kubernetes environment.

Centralized logging is an instrumental component of running and managing Kubernetes clusters at scale. Developers need access to logs for debugging and monitoring applications, operations teams need access for monitoring applications, and security needs access for monitoring. However, tasks such as log collection and metrics aggregation can be challenging within Kubernetes and Docker deployments due to the stateless natures of pods and containers. Open source tools like Elasticsearch, Kibana and Fluent Bit also known as EFK, provide effective methods to collect and store a wide variety of operational data to simplify these and similar problems.

In this webinar we will deploy an Elasticsearch cluster in Kubernetes and use Fluent bit to collect, transform, and ship log data to the Elasticsearch backend. Fluent Bit is an open source and multi-platform Log Processor and Forwarder which allows you to collect data/logs from different sources, unify and send them to multiple destinations. Its fully compatible with Docker and Kubernetes environments.

We will set up on our Kubernetes nodes to tail container log files, filter and transform the log data, and deliver it to the Elasticsearch cluster, where it will be indexed and stored. We will then use Kibana to Visualise the logs and create Interactive dashboards.

We will discuss:

Kubernetes monitoring Challenges

Why Opendistro for Elasticsearch for monitoring Kubernetes

Deploy Elasticsearch and Kibana on Kubernetes

Deploy Fluent Bit

Live Demo