The Sidecar is a popular cloud/distributed system design pattern. The name of the pattern is inspired by the motorcycle sidecar. While the main application includes the core capabilities, the sidecar augments or enhances the application with additional features but at the same time provides the necessary isolation to prevent single point failures.
Logging Constraints:
In Kubernetes, the logs written by the application to stdout/stderr can be collected with the default configurations. However the default design does not support collecting logs from files and segregation of log source when the application writes to a single file or multiple files. This logging limitation can be overcome by following a Sidecar approach.
Components:
Node: Physical/virtual worker machine in the Kubernetes cluster on which the workloads is scheduled.
Application Pod: The pod created via Deployment workload, includes your main application.
Application Container: The container which runs the core application capabilities.
Sidecar Container: The container which runs the necessary capabilities to collect the application logs.
Logging Agent Pod: The pod created via DaemonSet workload, responsible for forwarding logs to the external logging service.
Logging Service: The service used to collect and analyze your application logs. ex: splunk, elastic search
Design:
The Application Container could write to multiple log streams possibly based on the different feature set it supports.
One Sidecar Container per log stream is created and this has its lifecycle bound to the Application Container.
The Sidecar Container will tail logs from the associated log file (App_Log_File1 for Sidecar Container1) and emit to stdout/stderr.
Logs from the stdout/stderr of all pods can be accessed either via kubectl or directly from the designated log folder (var/log) on the Node.
Based on the implementation, the Logging Agent Pod will collect the various log streams from either of the two approaches and forward it to the Logging Service.
Alternative:
The Sidecar Container can include the Logging Agent instead of a separate pod running the agent. This would mean the sidecar also forwards the logs to the Logging Service.
This is a resource intensive approach since based on the number of log streams (assume n) in the Application Container, ‘n’ Sidecar Container per Application Container is created and each would include the Logging Agent. Ex: If we had 5 Application Container, each had 2 log streams then 2 Sidecar Container is created per Application Container, a total of 10 Logging Agent instances
The other point of consideration is the reliability, Logging Agent Pod created via the DaemonSet workload ensures this pod always runs on every Node.
Conclusion:
This design will not be necessary if your application writes to a single stream however in case of multiple log files on the main application container, the sidecar approach is currently the only way to collect logs while maintaining the source.
