maybit/identity-provider
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Feature: Add Vector Log Shipping to Loki for Keycloak #1

Merged
kstiehl merged 3 commits from feat/add-vector-log-shipper into main 2026-07-07 13:41:11 +00:00
Conversation 2 Commits 3 Files changed 9 +199 -7
patrick commented 2026-06-17 13:35:12 +00:00
Owner
Copy link

Summary

This PR introduces a new, reusable NixOS module for shipping systemd journal logs to a remote sink using Vector. It immediately implements this module on the keycloak-dev host to parse and forward Keycloak logs to Grafana Loki.

Key Changes

  • New Reusable Vector Module (modules/vector/default.nix)
    • Introduces the services.vectorJournaldToSink abstraction.
    • Sets up a journald source and a loki sink using Snappy compression and JSON encoding.
    • Implements automatic log enrichment (injecting .service and .host fields) and exposes an extraTransformSource option for custom Vector Remap Language (VRL) transformations.
    • Handles secure secret management for the Loki API token using Vector's exec secret backend to ensure the token never enters the Nix store.
    • Creates and configures the vector user, granting it systemd-journal access.
  • Keycloak-Dev Host Configuration (hosts/keycloak-dev/configuration.nix)
    • Refactors the Keycloak domain string into a reusable keycloakDomain variable.
    • Enables the Vector module, configuring it to watch keycloak.service and ship to the logs-prod-012 Grafana Loki instance.
    • Injects a custom VRL script to parse Keycloak's JSON log payload, extracting .level and .loggerName, and mapping the message to .keycloak_msg.
    • Appends static deployment labels (service_name, deployment_environment, domain, url) to the log streams.
    • Maps the vector/loki_api_token via SOPS with proper ownership assigned to the vector user.
  • Flake & Secrets (flake.nix, secrets.enc.yaml)
    • Exposes vectorJournaldToSink in nixosModules and wires it into the keycloak-dev host.
    • Updates the SOPS encrypted payload with the new Loki API token.
### **Summary** This PR introduces a new, reusable NixOS module for shipping `systemd` journal logs to a remote sink using Vector. It immediately implements this module on the `keycloak-dev` host to parse and forward Keycloak logs to Grafana Loki. ### **Key Changes** * **New Reusable Vector Module (`modules/vector/default.nix`)** * Introduces the `services.vectorJournaldToSink` abstraction. * Sets up a `journald` source and a `loki` sink using Snappy compression and JSON encoding. * Implements automatic log enrichment (injecting `.service` and `.host` fields) and exposes an `extraTransformSource` option for custom Vector Remap Language (VRL) transformations. * Handles secure secret management for the Loki API token using Vector's `exec` secret backend to ensure the token never enters the Nix store. * Creates and configures the `vector` user, granting it `systemd-journal` access. * **Keycloak-Dev Host Configuration (`hosts/keycloak-dev/configuration.nix`)** * Refactors the Keycloak domain string into a reusable `keycloakDomain` variable. * Enables the Vector module, configuring it to watch `keycloak.service` and ship to the `logs-prod-012` Grafana Loki instance. * Injects a custom VRL script to parse Keycloak's JSON log payload, extracting `.level` and `.loggerName`, and mapping the message to `.keycloak_msg`. * Appends static deployment labels (`service_name`, `deployment_environment`, `domain`, `url`) to the log streams. * Maps the `vector/loki_api_token` via SOPS with proper ownership assigned to the `vector` user. * **Flake & Secrets (`flake.nix`, `secrets.enc.yaml`)** * Exposes `vectorJournaldToSink` in `nixosModules` and wires it into the `keycloak-dev` host. * Updates the SOPS encrypted payload with the new Loki API token.
patrick self-assigned this 2026-06-17 13:35:24 +00:00
kstiehl requested changes 2026-06-18 18:41:48 +00:00
Dismissed
nix/modules/vector/default.nix
@ -0,0 +34,4 @@
type = lib.types.path;
description = ''
Path to a file containing only the loki API token.
Loaded via systemd EnvironmentFile, never ends up in the Nix store.
kstiehl commented 2026-06-18 18:23:16 +00:00
Owner
Copy link

Einfach gelogen 😄

Einfach gelogen 😄
patrick commented 2026-07-07 07:06:37 +00:00
Author
Owner
Copy link

Hab ich gelöscht.

Hab ich gelöscht.
nix/modules/vector/default.nix Outdated
@ -0,0 +86,4 @@
};
users.groups.vector = { };
environment.etc."vector/get-secrets.sh" = lib.mkIf (cfg.sinkType == "loki") {
kstiehl commented 2026-06-18 18:27:28 +00:00
Owner
Copy link

Lass mal keine komischen scripte unter /etc abglegen.

SystemD kann das alles schon.

LoadCredential=loki_token:"${cfg.loki.apiTokenFile}"
Lass mal keine komischen scripte unter /etc abglegen. SystemD kann das alles schon. ``` LoadCredential=loki_token:"${cfg.loki.apiTokenFile}" ```
patrick commented 2026-07-07 07:07:47 +00:00
Author
Owner
Copy link

Ist jetzt umgebaut auf systemd-LoadCredential.

Ist jetzt umgebaut auf systemd-LoadCredential.
nix/modules/vector/default.nix
@ -0,0 +114,4 @@
'';
};
secret.loki_token = lib.mkIf (cfg.sinkType == "loki") {
kstiehl commented 2026-06-18 18:37:41 +00:00
Owner
Copy link
type = "directory";
path = "$CREDENTIALS_DIRECTORY"
``` type = "directory"; path = "$CREDENTIALS_DIRECTORY" ```
patrick marked this conversation as resolved
kstiehl approved these changes 2026-07-07 13:41:07 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
kstiehl
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
patrick
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
maybit/identity-provider!1
No description provided.