A simple way to monitor shell activity on Linux

When administering Linux servers in exposed or multi-user environments, maintaining a reliable audit trail of shell activity is essential for security and incident response. Logging the commands executed by users provides valuable visibility for troubleshooting, forensic investigations, and accountability. This article explains how to configure shell command logging on Linux and forward these logs to Loki with Vector.

Configuring command logging in journald

The shell command logging will be done by a trap that collects the command before its execution, escapes it to avoid JSON breaking and logs it in journald.

On zsh

Add these lines to /etc/zsh/zshrc
This will run a hook on each command execution that will log the command in a JSON format

json_escape() {
    printf '%s' "$1" | sed \
        -e 's/\\/\\\\/g' \
        -e 's/"/\\"/g' \
        -e 's/\t/\\t/g' \
        -e 's/\r/\\r/g' \
        -e ':a;N;$!ba;s/\n/\\n/g'
}

preexec() {
    [[ "$1" == logger* ]] && return

    cmd=$(json_escape "$1")

    logger -p local0.notice -t zsh \
        "{\"user\":\"$USER\",\"command\":\"$cmd\"}"
}

On Bash

Add these lines to /etc/bash.bashrc

__last_command=""

json_escape() {
    printf '%s' "$1" | sed \
        -e 's/\\/\\\\/g' \
        -e 's/"/\\"/g' \
        -e 's/	/\\t/g' \
        -e 's/\r/\\r/g' \
        -e ':a;N;$!ba;s/\n/\\n/g'
}

preexec() {
    local command="$1"
    local escaped_command
    [[ "$command" == logger* ]] && return
    [[ "$command" == "$__last_command" ]] && return
    __last_command="$command"

    escaped_command=$(json_escape "$command")

    logger -p local0.notice -t bash -- \
        "{\"user\":\"$(json_escape "$USER")\",\"command\":\"$escaped_command\"}"
}

trap 'preexec "$BASH_COMMAND"' DEBUG

Forwarding logs to Grafana Loki using Vector

Installing vector

Add vector repository

curl -fsSL https://setup.vector.dev | sh

Install vector

apt update
apt install vector

Configuring Vector

The vector configuration is located in /etc/vector/vector.yaml.
It is composed of three main parts : sinks, transforms and sources.
Logs collection is defined in the sources section. Data present in the logs can then be filtered or transformed, these actions are specified in the transforms section. Finally, logs forwarding to centralized servers is defined in the sinks section.

To configure shell logs forwarding, we will need to add these blocs their corresponding sections.
sinks:

  loki_shell:
    encoding:
      codec: json
    endpoint: https://loki.example.com
    inputs:
    - session_logs
    type: loki

transforms:

  session_logs:
    condition: '.SYSLOG_IDENTIFIER == "zsh"'
    inputs:
    - journald
    type: filter

sources:

  journald:
    exclude_units:
    - vector
    labels:
      job: shell_logs
    type: journald

To apply the new configuration, we will have to restart vector

systemctl restart vector

Checking logs reception

To check logs reception on Loki, go to the drilldown section on Grafana, configure filter to only show shell_logs job, check the "message" field to only display JSON logs, type some commands and they should be displayed.