Testing Logstash Pipelines

In my current role, I’ve become the ’logging guy’. Because of this, I’ve spent quite a lot of time creating and updating Logstash pipelines. One of my biggest bugbears with Logstash is the lack of built in testing functionality for filters.

My search for a solution has had me considering building something myself, migrating away from Logstash (though, not exactly feasible 🙁), and eventually discovering existing tools. Combining said existing tools has resulted, in what I believe to be, a pretty tidy solution.

Even though I’m writing about Logstash, I’d recommend looking into Vector (which has built in testing). That is, if migrating away from Logstash is feasible or if you’re building something from scratch and Logstash isn’t a hard requirement.

Testing Syntax Validity (Logstash Config Verifier)

If you take anything away from this post, let it be this. Logstash has a flag ,--config.test_and_exit, which, when used alongside the regular command and flags used to start Logstash e.g. logstash --config.test_and_exit, validates the config and then exits. Returning a non-zero exit code if any errors are found.

There are a few things, a couple that may be obvious, to be aware of when using this flag:

When using environment variables without defaults in the config e.g. ${PORT} vs ${PORT:8080}, you must ensure that the environment variables are set before running the command (as you would during deployment)
Plugins may validate the values of the config options passed into them, you must pass in values that satisfy the validation criteria if not using the actual deployed values (likely the case for local development or in CI)
If pipelines.yml is pointing to the wrong directory, one that doesn’t contain *.conf files, Logstash won’t return a non-zero exit code but it will log a message like ‘No config files found in path {path: "<path>/*"}’.

Testing Filter Behaviour (Logstash Filter Verifier)

To test the behaviour of filters, I’m using the aptly named Logstash Filter Verifier (LFV).

Logstash Filter Verifier is a tool that allows you to test the behaviour of Logstash filters by writing test cases with input events and assertions against the outputted events.

As of writing, the latest stable release is v1.6.3 (released April 2021), and the latest beta release is v2.0.0-beta.3 (released August 11 2024). The lack of development activity was an initial source of concern, however, initial testing proved that it is still capable.

My recommendation is to use the stable release unless you need to test setups using pipeline to pipeline communication, which are only supported from v2.

So far, I’ve only implemented this for real on single pipeline instances with v1, and I’ve done a proof of concept for multiple pipelines using v2.

I’ve included some example config files and testcases below. But if you’d prefer to click around some code, I’ve made a couple of examples available in GitHub: https://github.com/aadam-ali/examples/tree/main/logstash-filter-verifier.

Each example lists out the files that need to be created, and any useful information about them.

Example 1: Single Pipeline (using v1.6.3)

logstash.conf

Create a logstash.conf file with the following content.

input {
  http {
    port => 8080
  }
}

filter {
  if [service] == "foo" {
    mutate { add_field => { "some_additional_info" => "this service is great" } }
  }

  mutate { remove_field => ["event", "host"] }
}

output {
  stdout {}
}

These filters add a field if the event has a field [service] with the value foo, and removes the event and host fields.

testcases.yml

The test case file specification is simple and clearly documented.

Create a testcases.yml file with the following content.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
codec: json_lines
ignore:
  - '@timestamp'
testcases:
  - description: Foo Logs
    input:
      - >
        {
          "message": "hi",
          "service": "foo"
        }
    expected:
      - message: "hi"
        service: "foo"
        some_additional_info: "this service is great"

Dockerfile

We’ll run this test inside of a Docker container, so you don’t need to install anything directly on your machine (unless you don’t have Docker installed).

Create a Dockerfile with the following content.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
FROM docker.elastic.co/logstash/logstash-oss:9.4.4@sha256:627203a21474816e1d0f5f8a16442649c92138a0185cc18a790d3412e69b9f38
ARG LFV_VERSION=1.6.3

USER root
RUN microdnf install -y diffutils wget && microdnf clean all
RUN wget --progress=dot:giga https://github.com/magnusbaeck/logstash-filter-verifier/releases/download/${LFV_VERSION}/logstash-filter-verifier_${LFV_VERSION}_linux_amd64.tar.gz && \
  tar xzf logstash-filter-verifier_${LFV_VERSION}_linux_amd64.tar.gz -C /usr/bin && \
  rm logstash-filter-verifier_${LFV_VERSION}_linux_amd64.tar.gz

USER logstash
RUN mkdir --mode 755 --parents /usr/share/logstash/tests
COPY --chown=logstash:root --chmod=644 logstash.conf /usr/share/logstash/pipeline/logstash.conf
COPY --chown=logstash:root --chmod=644 testcases.yml /usr/share/logstash/tests/testcases.yml

With these three files created, you now have the components required to run a basic test. Run the following commands to build the Docker image and run the test.

docker build -t lfv1 .
docker run --rm --entrypoint /usr/bin/logstash-filter-verifier lfv1 \
    /usr/share/logstash/tests/testcases.yml /usr/share/logstash/pipeline

You should see something like:

Running tests in testcases.yml...
☑ Comparing message 1 of 1 (Foo Logs) from testcases.yml

Summary: ☑ All tests: 1/1
	 ☑ testcases.yml: 1/1

Example 2: Multiple Pipelines (using v2.0.0-beta.3)

Compared to the above example, there are a few differences in the testcases.yml file and *.conf files required to get these tests working.

These instructions have been written assuming you ran through the first example.

logstash.conf

Replace the contents of logstash.conf with the following.

input {
  http {
    id   => "one_input_http_0"
    port => 8080
  }

  # Only used for LFV
  stdin {
    id    => "one_input_stdin_0"
    codec => json
  }
}

filter {
  mutate {
    id           => "one_filter_mutate_0"
    remove_field => ["event", "host"]
  }
}

output {
  if [service] == "foo" {
    pipeline {
      id      => "one_output_pipeline_0"
      send_to => two
    }
  }
}

You’ve probably noticed that each plugin reference now has an id, this is so LFV can target specific inputs (to inject events through) and to target filters to mock - mocking is useful where you have filters that are dynamic e.g. making a HTTP request.

Also, the target input(s) must configure a codec, this is no longer defined in the testcases.yml file.

logstash-two.conf

Create logstash-two.conf with the following contents.

input {
  pipeline {
    id      => "two_input_pipeline_0"
    address => two
  }
}

filter {
  mutate {
    id        => "two_filter_mutate_0"
    add_field => { "some_additional_info" => "this service is great" }
  }
}

output {
  stdout { id => "two_output_stdout_0" }
}

This pipeline receives events from logstash.conf and is now handling the addition of the extra field.

pipelines.yml

Create pipelines.yml with the following contents.

- pipeline.id: one
  path.config: /usr/share/logstash/pipeline/logstash.conf
- pipeline.id: two
  path.config: /usr/share/logstash/pipeline/logstash-two.conf

testcases.yml

Replace line 1 of testcases.yml with:

input_plugin: one_input_stdin_0

As mentioned earlier:

.input_plugin is set to the ID of the input plugin which LFV will inject events through
.codec is no longer required, the codec must be configured on the input plugin referenced by .input_plugin

Dockerfile

Update line 2 to be:

ARG LFV_VERSION=v2.0.0-beta.3

Update line 12 of to be:

COPY --chown=logstash:root --chmod=644 *.conf /usr/share/logstash/pipeline

Add the following line to the end of the file.

COPY --chown=logstash:root --chmod=644 pipelines.yml /usr/share/logstash/config/pipelines.yml

With the files setup, you now have the components to run a basic multi-pipeline test. Run the following commands to:

Build the Docker image
Start the LFV daemon (this is what allows us to test setups with pipeline to pipeline communication)
Wait for the daemon to become ready
Run the tests

docker build -t lfv2 .
docker run --rm --interactive --entrypoint /bin/bash lfv2 << EOF
/usr/bin/logstash-filter-verifier daemon start --wait-for-state-timeout 120s &
sleep 5
/usr/bin/logstash-filter-verifier daemon run --pipeline /usr/share/logstash/config/pipelines.yml --testcase-dir /usr/share/logstash/tests
EOF

You should see something like:

Waiting for /tmp/lfv-3662630544/logstash-instance/4mZ7RJOt/logstash.log to appear...
Daemon listening on /tmp/logstash-filter-verifier.sock
Ready to process tests
☑ Comparing message 1 of 1 (Foo Logs) from testcases.yml

Summary: ☑ All tests: 1/1
	 ☑ testcases.yml: 1/1

Things to Be Aware Of

LFV does not respect the logstash.yml file (source code), instead any config options must be set using the --logstash-arg flag e.g. --logstash-arg=--pipeline.ecs_compatibility --logstash-arg disabled
LFV clears all environment variables, to forward environment variables use the --keep-env flag e.g. --keep-env PATH
LFV doesn’t work with some codecs e.g. cloudwatch_logs, however using json_lines or line as a replacement is sufficient if you know the shape of the logs that Logstash will be ingesting
LFV requires assertions to be made for each non-pipeline output and the assertions must be in order of when they would be emitted (it would be great to target outputs like we can with inputs in v2)
LFV v2 requires all inputs, outputs, and filters to have the id configuration option set for the reasons mentioned above (input targeting, and filter mocking)
LFV v2 (in daemon mode i.e. testing multi-pipeline setups) unlike v1 requires the input plugins since LFV will send events through the one that has an id of input_plugin’s value and use its configured codec

Final Thoughts

This solution has helped me solve a problem that I’ve been wanting to for a while, and I’m glad to have finally got there.

It’s made modifying Logstash configurations much less painful as the tests provide a quicker feedback loop than deploying the changes into a testing environment.

Because the tests run in CI, there’s a reduced risk of bad changes making their way into the main branch and ultimately being deployed into a live environment.

There’s increased confidence when making changes to Logstash configs. Meaning, that as the ’logging guy’, I don’t become a bottleneck for my team when Logstash changes are required.

#observability

Thank you for taking some time out of your day to read this post! Please feel free to share this with anyone who you think might benefit from it.

If you would like to stay up to date with future posts, subscribe to the RSS feed.