Tips and Tricks for managing ELK configuration



A few months ago I published “Demystifying ELK stack” article that summarizes my knowledge about setting up and configuring the system for collecting, processing and presenting logs, based on Filebeat, Logstash, Kibana, and Elasticsearch. Since then I’ve learned a few new DevOps things which help me and my teammates to work more effectively with ELK. I think they’re worth sharing.

Shrinking Filebeat configuration 🔗︎

I use Filebeat to collect data from log files and send them to Logstash for further processing and analyzing. Filebeat configuration is in YAML format and the most important part of it is the section filebeat.prospectors which is responsible for configuring harvesting data. A sample configuration looks as follows:

filebeat.prospectors:
- input_type: log
  paths:
    - c:\inetpub\wwwroot\MyApp\logs\Client.Web.log
  scan_frequency: 10
  encoding: utf-8
  multiline.pattern: '^(\d{4}-\d{2}-\d{2}\s)'
  multiline.negate: true 
  multiline.match: after  
  fields_under_root: true
  fields:
    app_env: test
    app_name: client
    type: web

Currently, I have 10 internal environments for different purposes (manual testing, automated UI testing, load testing, presentations for different customers etc.). Each environment consists of two web applications and each of them produces 2 log files (diagnostic and performance data). That gives me 10 * 2 * 2 = 40 log files and for every single one of them, I have to configure a separate prospector. I can’t use a single prospector with a wildcard in the path attribute because there is a need to add additional metadata, such as environment name, app name and log type (attributes defined in the fields node). However, some of the attributes are the same for every prospector which causes a massive configuration duplication and makes it harder to modify those common values. I was even thinking about preparing some kind of template for prospector configuration with custom PowerShell script that could facilitate creating a config for the entire environment. Instead of rushing to develop an in-house solution though, I started from browsing YAML specification and I found Merge Key Language-Independent Type which seemed to be a solution to my problem. The !!merge feature together with Anchors and Aliases allows to define and reuse keys of the map - simply speaking it’s a kind of variables in YAML files. In the following example, I’ve defined a common configuration with anchor &PROSPECTOR_COMMON_OPTIONS and merge it into any prospector configuration with << : operator.

PROSPECTOR_COMMON_OPTIONS : &PROSPECTOR_COMMON_OPTIONS
    scan_frequency: 10
    encoding: utf-8
    multiline.pattern: '^(\d{4}-\d{2}-\d{2}\s)'
    multiline.negate: true
    multiline.match: after  
    fields_under_root: true

filebeat.prospectors:
- input_type: log
  paths:
    - C:\logs\manualtest\Client.Web.log
  << : *PROSPECTOR_COMMON_OPTIONS
  fields:
    app_env: manualtest
    app_name: client
    type: web

- input_type: log
  paths:
    - C:\logs\automatedui\Office.Web.log
  << : *PROSPECTOR_COMMON_OPTIONS
  fields:
    app_env: automatedui
    app_name: office
    type: web

Thanks to this little trick I was able to reduce the number of entries in my Filebeat configuration and improve its maintainability. A lot of people criticize YAML for being hard to read and edit - in comparizon to JSON format - but it has some less known features which give it more possibilities than JSON has.

Temporary variables in Logstash configuration 🔗︎

In the Be the first to know of the bug article I described how we can easily integrate Logstash with Microsoft Teams to create some kind of early-warning system. In the proposed solution I used mutate filter to create extra fields which hold additional data consumed only by the output section, for example URL for Kibana filter, Jira create issue link or Microsft Teams webhook. After a while, I’ve realized that this additional data is unnecessarily stored in ElasticSearch index and consumes a lot of space. Thankfully, the authors of Filebeat foresaw the need for temporal variables and introduced Logstash Metadata. Now, instead of adding fields to events only for processing purpose, we can store them in the dedicated @metadata field:

 mutate
 {
    add_field =>
    {
        "[@metadata][webhookUrl]" => "https://outlook.office.com/webhook/0c744aca-7d19-4556/IncomingWebhook/ceb6ba15106147a57e14e03d662de6/86aafacf-4c13-9780-5d9063b10fb6"
    }
}

Automating the configuration update 🔗︎

Every time I changed Logstash or Filebeat configuration I had to log in to the appropriate server, replace the old config with the new one, restart the service and examine the service log file if the whole operation was successful. If something failed, I needed to correct the config file and repeat the whole routine. It was a very tedious process and nobody in the team besides me knew to how to do it. I even wrote the whole instruction down but the number of steps or the need to log into Linux server repealed others. A better solution than creating manual instruction is to automate the process. The easiest part was to create the script that updates Filebeat configuration because it resides on the Windows server:

function Update-FilebeatConfig
{
    [CmdletBinding()]
    param(
        $ComputerName, 
        $Credentials, 
        $FilebeatSrcFile
     )
    $session = New-PSSession -ComputerName $ComputerName -Credential $Credentials
    Copy-Item $FilebeatSrcFile -Destination "C:\Tools\Filebeat\" -ToSession $session -Force
    Invoke-Command -Session $session -ScriptBlock {
        Write-Verbose "Restarting filebeat..."
        Restart-Service filebeat
        Get-Service filebeat
        Write-Verbose "Filebeat restart finished."
    }
    Remove-PSSession -Session $session
}

Now everybody can easily update Filebeat configuration by invoking this function as follows:

Update-FilebeatConfig -ComputerName "app.server.lan" -Credentials (Get-Credential) -FilebeatSrcFile "./filebeat.yml" -Verbose

PowerShell Core on Linux 🔗︎

However, the real challenge for me was to automate the same process for Logstash configuration that lives on the Linux server. I started by installing PowerShell Core on my Linux server with the following commands:

sudo apt-get install libunwind8 libicu55 liblttng-ust0
wget https://github.com/PowerShell/PowerShell/releases/download/v6.1.0/powershell_6.1.0-1.ubuntu.16.04_amd64.deb
sudo dpkg -i powershell_6.1.0-1.ubuntu.16.04_amd64.deb
sudo apt-get install -f

Depending on your Linux distribution and version you might need to use a different package of PowerShell Core If you are working on Ubuntu, you can check your current version with lsb_release -a command. If everything went well, you should be able to enter the PowerShell console with pwsh command.

Beside installing PowerShell I also needed to enable PowerShell Remoting. This can be accomplished by installing OMI PSRP packages:

wget https://github.com/PowerShell/psl-omi-provider/releases/download/v1.4.1-28/psrp-1.4.1-28.universal.x64.deb
wget https://github.com/Microsoft/omi/releases/download/v1.4.2-5/omi-1.4.2-5.ssl_100.ulinux.x64.deb
sudo dpkg -i omi-1.4.2-5.ssl_100.ulinux.x64.deb
sudo dpkg -i psrp-1.4.1-28.universal.x64.deb

Despise all my concerns the installation went pretty smoothly (I only needed to adjust the version of package responsible for SSL) and I was able to remotely invoke command on the Linux server from my Windows workstation with Invoke-Command cmdlet. Then I was able to easily automate the process of updating Logstash config:

$sharedContext = {
    function Watch-File {
        [CmdletBinding()]
        param (
            [string] $FileName,
            [string] $StopContentPositive,
            [string] $StopContentNegative
        )
        $ErrorActionPreference = "Stop"    
        $stream =  New-Object  System.IO.FileStream $FileName, ([System.IO.FileMode]::Open), ([System.IO.FileAccess]::Read), ([System.IO.FileShare]::ReadWrite)
        $tries = 0
        $maxTries = 100
        
        try{
            try{
                $stream.Seek(0,  [IO.SeekOrigin]::End)
                $streanReader = New-Object System.IO.StreamReader $stream, ([System.Text.Encoding]::UTF8)
                do{
                    $line = $streanReader.ReadLine() 
                    $tries = $tries +1
                    if($null -eq $line)
                    {                    
                        Start-Sleep -Seconds 2
                    }else{
                        Write-Verbose $line
                    }
                    
                    if($line -like "*$StopContentNegative*")
                    {
                        Write-Error "Cannot restart logstash"
                        break;
                    }
                    
                }while ((-not ($line -like "*$StopContentPositive*")) -and ($tries -lt $maxTries)) 
    
                if($tries -eq $maxTries){
                    Write-Error "Cannot restart logstash"
                }               
            }finally{
                $streanReader.Dispose()
            }    
        }finally{
            $stream.Dispose()
        }
       
    }
}

function Update-LogstashConfig
{
    [CmdletBinding()]
    param(
            $ComputerName, 
            $Credentials, 
            $LogstashSrcFile
        )

    $sessionOptions = New-PSSessionOption -SkipCACheck -SkipRevocationCheck -SkipCNCheck
    $session = New-PSSession -ComputerName $ComputerName -Credential $Credentials -Authentication basic -UseSSL -SessionOption $sessionOptions
    Copy-Item $LogstashSrcFile -Destination /etc/logstash/conf.d/ -ToSession $session -Force
    Invoke-Command -Session $session -ScriptBlock {
        . ([scriptblock]::Create($using:sharedContext))   
        Write-Verbose "Restarting logstash..."
        systemctl restart logstash
        Watch-File -FileName "/var/log/logstash/logstash-plain.log" -StopContentPositive "Pipelines running" -StopContentNegative "Failed to execute action"
    } 
    Remove-PSSession -Session $session
}

I’ve enriched my script with Watch-File function that forwards Logstash logs and blocks the process until the restart is finished. Thanks to that we have a live stream of what is going on during the restart. The Logstash config update can be performed with the command:

Update-LogstashConfig -ComputerName 'elk.server.lan' -Credentials (Get-Credential) -LogstashSrcFile "./logstash/App.conf" -Verbose

I’ve put all of the above scripts together with the config files in the source control so everybody in the team can easily modify and deploy new ELK configuration.

TL;DR 🔗︎

Thanks to !!merge, Anchors and Aliases I can simulate variables in YAML and create reusable parts of Filebeat configuration. The Logstash @metadata field allows me to create variables needed only for processing logic without polluting ElasticSeach indices. With PowerShell Core I can easily manage Linux servers directly from my Windows workstation and automatically deploy ELK configuration.


If you find this blog post useful and you think it's worth to share this idea with others, please don't hesitate to use these buttons below:

See Also


comments powered by Disqus