After experimenting
the other day, I continued thinking about better ways to solve my analytics problem.
I did not really want to run a separate fluentd instance in a container just to use out_exec
but I wanted to use same functionality.
I decided to take an afternoon and see how difficult it would be to create a go output plugin
and built fluent-bit-exec_out
.
graph LR
nginx ---> |ltsv.*| fluent-bit
fluent-bit --> |sematext.*| sematext
subgraph fluentbit
direction TB
fluent-bit --> exec_out
exec_out --> fluent-bit
end
Building a plugin
I started looking for examples https://github.com/fluent/fluent-bit-go/network/dependents
on how to build a plugin.
I’m not a Go developer by default, so most of my code is not particularly nice but it works.
Read More →
I’ve been using the self-hosted version plausible
with my blog and development site.
It generally works well, but there are times I’d like to have a little bit more customization on the reports.
I’m already collecting nginx logs using ltsv
and fluent-bit’s tail
plugin.
Since I have extra time to tinker over my winter break, I’ve been experimenting a bit.
graph LR
javascript ----> plausible
nginx -->|ltsv| fluent-bit
fluent-bit -->|forward| processing
processing -.-> firewall
processing --> sematext
processing -->|event api| plausible
fluent-bit -->|raw logs| sematext
Default Plausible Configuration
graph LR
javascript --> plausible
The default path with Plausible is easy.
You create a site in their admin and copy the Javascript snippet onto your site.
This works ok for simple cases but doesn’t work for most bots and such that do not run javascript.
Since I want to get information on the bots that are hitting my site we need to do something with our server logs.
Read More →
Fluent Bit Exec Plugin