Howto – bash audit / command logger

Introduction

Having a complete history of all typed commands can be very helpful in many scenarios:

• when several administrators work together on the same server and need to know what was done previously
• when someone need to redo an older sequence of commands or to understand an undocumented maintenance process
• for troubleshooting or forensic analysis, by crosschecking the date of an event or of a file with the commands executed at that date

The standard ‘.bash_history’ file of the shell is unfortunately not written on disk in the case of a crash and it may be deleted by the user.
Another problem is that when many shell sessions are running concurrently, their logging will only occur when they are closed, therefore the commands of the history will not appear in their chronological order.
Furthermore, ‘.bash_history’ will not include essential information like the ‘working directory’ of the command; and by default the repetition or re-edition of commands will not be logged, too.

Some solutions exist to improve this, either by patching or installing binaries:

• ‘bash-BOFH’ patching and recompiling: works well but need a new patch for each release of the bash
• ‘snoopy’: is logging all commands except shell builtins
• ‘rootsh / sniffy / ttyrpld / ttysnoop’: logs everything, also output of commands, it may be useful but it generates very verbose logs
• ‘grsecurity’ patched kernels: powerful but it may be a not suitable solution if an official kernel is required (e.g. for Oracle DB)
• there is also an old ‘sshd’ patch (‘http://www.kdvelectronics.eu/ssh-logging/ssh-logging.html‘)
• ‘screen -x’ can also be useful for cooperation work, but it is not a command logger

In contrast to that, the presented method is very easy to deploy; it is just a shellscript that is running in bash (standard shell on most systems) and therefore it is architecture independent.
It will allow a complete audit of all commands/builtins executed interactively in the bash.
Note that a user can avoid calling this file by starting a shell with options like ‘–norc’; he also can unset or overwrite variables like ‘PROMPT_COMMAND’.
Therefore this script is useful for audit but an alternative solution with bash patching should be considered if the security requirements are the priority.

The result of the shellscript will be sent via syslog and will look like the following example:

2012-05-31T15:40:52.530657+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22] /root: id
2012-05-31T15:41:10.533963+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22] /root: id #again, up-key
2012-05-31T15:41:26.421925+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22] /root: id #again, ctrl-r
2012-05-31T15:41:44.223986+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22] /root: (echo a; echo b); { echo c; echo d; }; echo e | ( cat && echo f)
2012-05-31T15:41:49.687712+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22] /root: exit
2012-05-31T15:41:49.690234+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22]: #=== bash session ended. ===
2012-05-31T15:51:01.399710+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22]: #=== New bash session started. ===
2012-05-31T15:51:02.980606+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22] /root: id
2012-05-31T15:51:09.327614+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22] /root: ssh localhost
2012-05-31T15:51:14.768578+02:00 debian1 [audit root/. as root/31874 on pts/1/127.0.0.1:47702->127.0.0.1:22]: #=== New bash session started. ===
2012-05-31T15:51:48.031612+02:00 debian1 [audit root/. as root/31874 on pts/1/127.0.0.1:47702->127.0.0.1:22] /root: echo hi from ssh
2012-05-31T15:51:49.875367+02:00 debian1 [audit root/. as root/31874 on pts/1/127.0.0.1:47702->127.0.0.1:22] /root: exit
2012-05-31T15:51:49.877675+02:00 debian1 [audit root/. as root/31874 on pts/1/127.0.0.1:47702->127.0.0.1:22]: #=== bash session ended. ===
2012-05-31T15:51:51.075034+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22] /root: id
2012-05-31T15:51:53.027380+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22] /root: less /var/log/user.log
2012-05-31T16:00:06.676849+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22]: #=== New bash session started. ===
2012-05-31T16:00:12.690457+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22] /home/pointsoftware: ls
2012-05-31T16:00:16.114366+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22] /home/pointsoftware: cd /
2012-05-31T16:00:21.002682+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22] /: ps aux
2012-05-31T16:00:30.714988+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22] /root: less /var/log/user.log
2012-05-31T16:00:37.754175+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22] /: ps aux
2012-05-31T16:00:37.758383+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22]: #=== bash session ended. ===

Explanation of the method

First of all we tune up some bash settings related to history:

#'history' options
declare -rx HISTFILE="$HOME/.bash_history"
declare -rx HISTSIZE=500000                                 #nbr of cmds in memory
declare -rx HISTFILESIZE=500000                             #nbr of cmds on file
declare -rx HISTCONTROL=""                                  #does not ignore spaces or duplicates
declare -rx HISTIGNORE=""                                   #does not ignore patterns
declare -rx HISTCMD                                         #history line number
history -r                                                  #to reload history from file if a prior HISTSIZE has truncated it
if [ "${OSTYPE:0:7}" != "solaris" ] #following not working in solaris
then
  if groups | grep -q root
  then
    declare -x TMOUT=3600                                     #timeout for root's sessions
    chattr +a "$HISTFILE"                                     #set append-only
  fi
fi

#history substitution ask for a confirmation
shopt -s histverify

The bash history could also include timestamps with following options, but we will not need it, as our solution will create another log file (‘/var/log/userlog.info’) with all details like timestamp, working directory, PID, userid, etc..

#add timestamps in history - obsoleted with logger/syslog
#'http://www.thegeekstuff.com/2008/08/15-examples-to-master-linux-command-line-history/#more-130'
declare -rx HISTTIMEFORMAT='%F %T '

A nicer colored shell prompt is also very handy when we scroll back the terminal. BTW, it can be very useful to set your terminal to have a least 10’000 lines of scrollback.

#prompt & color
#'http://www.pixelbeat.org/docs/terminal_colours/#256'
#'http://www.frexx.de/xterm-256-notes/'
_backnone="\e[00m"
_backblack="\e[40m"
_backblue="\e[44m"
_frontred_b="\e[01;31m"
_frontgreen_b="\e[01;32m"
_frontgrey_b="\e[01;37m"
_frontgrey="\e[00;37m"
_frontblue_b="\e[01;34m"
PS1="\[${_backblue}${_frontgreen_b}\]\u@\h:\[${_backblack}${_frontblue_b}\]\w\\$\[${_backnone}${_frontgreen_b}\] "

By default ‘ctrl-s’ will block the shell until you press ‘ctrl-q’ (flow control). We disable this in order to be able to search forward and backward for commands (‘ctrl-s’/'ctrl-r’):

#enable forward search ('ctrl-s')
#'http://ruslanspivak.com/2010/11/25/bash-history-incremental-search-forward/'
if shopt -q login_shell
then
  stty -ixon
fi

Following shortcut keys are available in ‘bash’:

# bash shortcuts
#'http://www.techrepublic.com/article/master-the-linux-bash-command-line-with-these-10-shortcuts/5827311'
#'http://www.hypexr.org/bash_tutorial.php'
# ctrl-r                reverse search
# ctrl-s                forward search
# alt-. or esc-.        reuse 1st arg
# ctrl-a                Move cursor to beginning of line
# ctrl-e                Move cursor to end of line
# meta-b                Move cursor back one word
# meta-f                Move cursor forward one word
# ctrl-w                Cut the last word
# ctrl-u                Cut everything before the cursor 
# ctrl-k                Cut everything after the cursor
# ctrl-y                Paste the last thing to be cut
# ctrl-_                Undo
#bash-history-cheat-sheet.pdf 'http://www.catonmat.net/download/bash-history-cheat-sheet.pdf'
# Emacs Mode Shortcuts:
# CTRL-p                Fetch the previous command from the history list.
# CTRL-n                Fetch the next command from the history list.
# CTRL-r                Search history backward (incremental search).
# CTRL-s                Search history forward (incremental search).
# Meta-p                Search backward using non-incremental search.
# Meta-n                Search forward using non-incremental search.
# Meta-<                Move to the first line in the history.
# Meta->                Move to the end of the history list.
# Vi Mode Shortcuts:
# k                     Fetch the previous command from the history list.
# j                     Fetch the next command from the history list.
# /string or CTRL-r     Search history backward for a command matching string.
# ?string or CTRL-s     Search history forward for a command matching string.
# n                     Repeat search in the same direction as previous.
# N                     Repeat search in the opposite direction as previous.
# G                     Move to the N-th history line (for example, 15G).

To retrieve the original username (not the current username that may be changed with ‘su’), we tried this method using a loop to seek the parent-PID in ‘/proc/’:

#seek the oldest parent 'bash' process, to get $AUDIT_LOGINID and $AUDIT_LOGINUSER,
#which may be different from $USER after 'su' or 'sudo' commands
AUDIT_LOGINID=$$
AUDIT_LOGINID2=$AUDIT_LOGINID
while AUDIT_LOGINID2="$(awk '/PPid:/ {print $2}' /proc/$AUDIT_LOGINID2/status)" && [ "$AUDIT_LOGINID2" != "1" ]
do
  if [ "$(awk '/Name:/ {print $2}' /proc/$AUDIT_LOGINID2/status)" == "bash" ]
  then
    AUDIT_LOGINID="$AUDIT_LOGINID2"
  fi
done
AUDIT_LOGINUSER=$(awk "/^Uid:/ {print \$2}" /proc/$AUDIT_LOGINID/status)
AUDIT_LOGINUSER=$(awk -F":" "\$3 ~ /$AUDIT_LOGINUSER/ {print \$1}" /etc/passwd)
#old: AUDIT_LOGINUSER=$(awk -F":" "/^[^:]*:[^:]*:$AUDIT_LOGINUSER:/ {print \$1}" /etc/passwd)
#old: AUDIT_LOGINUSER=$(getent passwd $AUDIT_LOGINUSER | sed -e 's%:.*%%')"

We then used a simpler approach, using the ‘who -mu’ command:

declare -rx AUDIT_LOGINUSER="$(who -mu | awk '{print $1}')"
declare -rx AUDIT_LOGINPID="$(who -mu | awk '{print $6}')"
declare -rx AUDIT_USER="$USER"                              #defined by pam during su/sudo
declare -rx AUDIT_PID="$$"
declare -rx AUDIT_TTY="$(who -mu | awk '{print $2}')"
declare -rx AUDIT_SSH="$([ -n "$SSH_CONNECTION" ] && echo "$SSH_CONNECTION" | awk '{print $1":"$2"->"$3":"$4}')"
declare -rx AUDIT_STR="[audit $AUDIT_LOGINUSER/$AUDIT_LOGINPID as $AUDIT_USER/$AUDIT_PID on $AUDIT_TTY/$AUDIT_SSH]"
declare -x AUDIT_HISTLINE="0"                               #to avoid logging the same line twice
declare -rx AUDIT_SYSLOG="1"                                #to use a local syslogd

Now we need a way to perform the logging at each execution of command. The following solution using ‘PROMPT_COMMAND’ is working but the syslog messages are sent after the command execution,
this causes ‘su’ or ‘sudo’ commands to appear only after logouts, and ‘cd’ commands to display the wrong working directory:

#'http://jablonskis.org/2011/howto-log-bash-history-to-syslog/'
declare -rx PROMPT_COMMAND='history -a >(tee -a ~/.bash_history | logger -p user.info -t "$AUDIT_STR $PWD")' #no subshell is used here, it would else duplicate execution!

‘history -a >(tee -a ~/.bash_history | logger -p user.info -t “$AUDIT_STR $PWD”)’ will take the history commands that are still uncommited to disk (‘history -a’) and write them to a process substitution (‘>()’) that will be the standard input for the ‘tee -a’ command, which will append the result to ‘~/.bash_history’ and send it also to the ‘logger’ command via a pipe; logger will send it to the local syslog daemon.

Another solution is to use ‘trap’ DEBUG, which is executed before the command.
See ‘http://superuser.com/questions/175799/does-bash-have-a-hook-that-is-run-before-executing-a-command‘ and ‘http://www.davidpashley.com/articles/xterm-titles-with-bash.html‘.
Short example (does not work with piped command):

set -o functrace; trap 'echo -ne "===$BASH_COMMAND===${_backnone}${_frontgrey}\n"' DEBUG

Longer example with a subshell (works with piped command):

set +o functrace                                            #disable trap DEBUG inherited in functions, command substitutions or subshells, normally the default setting already
function AUDIT_DEBUG() {
  echo -ne "${_backnone}${_frontgrey}"                      #disable prompt colors for the command's output
  (history -a >(logger -p user.info -t "$AUDIT_STR $PWD" < <(tee -a ~/.bash_history))) && sync && history -c && history -r
}
set -o functrace                                            #enable trap DEBUG inherited for all subsequent functions; required to audit commands beginning with the char '(' for a subshell
#=> problem: auto-completion in commands avoids logging them
#launches AUDIT_DEBUG() and then stops the trap DEBUG, to avoid a useless rerun of AUDIT_DEBUG() during the execution of $PROMPT_COMMAND
declare -rx PROMPT_COMMAND="trap 'AUDIT_DEBUG; trap DEBUG' DEBUG; $PROMPT_COMMAND"

‘history -a >(logger -p user.info -t “$AUDIT_STR $PWD” < <(tee -a ~/.bash_history))' will take the history commands that are still uncommited to disk ('history -a') and write them to a process substitution ('>()’) that will be the standard input for the ‘tee -a’ command, which will append the result to ‘~/.bash_history’ and to another process substition (‘<()'), which finally get read by the 'logger' command.
'history -c && history -r' are here forcing a refresh of the history because 'history -a' was called within a subshell and therefore the new history commands that were just appent to file will keep their "new" status outside of the subshell, causing their logging to re-occur on every function call... (cf. 'http://stackoverflow.com/questions/103944/real-time-history-export-amongst-bash-terminal-windows‘)
Note that without the subshell, piped bash commands would hang… (it seems that the trap + process substitution interfer with stdin redirection)

The final solution is quicker and it is avoiding ‘sync’ and ‘history -r’ that are time consuming.
It turns out that this solution is much simpler and works well with piped commands, subshells, aborted commands with ‘ctrl-c’ and all test-cases that we could try.
The trick is again to use a trap DEBUG function, to have set the required history options (HISTCONTROL, HISTIGNORE) and to disable the trap in functions, command substitutions or subshells.

set +o functrace                                            #disable trap DEBUG inherited in functions, command substitutions or subshells, normally the default setting already
shopt -s extglob                                            #enable extended pattern matching operators
function AUDIT_DEBUG() {
  local AUDIT_LASTHISTLINE="${AUDIT_HISTLINE}"
  local AUDIT_CMD="$(history 1)"                            #current history command
  AUDIT_HISTLINE="${AUDIT_CMD%%+([^ 0-9])*}"
  if [ "${AUDIT_HISTLINE}" != "${AUDIT_LASTHISTLINE}" ]     #avoid logging unexecuted commands after 'ctrl-c', 'empty+enter', or after 'ctrl-d'
  then
    echo -ne "${_backnone}${_frontgrey}"                    #disable prompt colors for the command's output
    #remove in last history cmd its line number (if any) and send to syslog
    if [ -n "$AUDIT_SYSLOG" ]
    then
      if ! logger -p user.info -t "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}"
      then
        echo error "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}"
      fi
    else
      echo $( date +%F_%H:%M:%S ) "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}" >>/var/log/userlog.info
    fi
  fi
  #echo "===cmd:$BASH_COMMAND/subshell:$BASH_SUBSHELL/fc:$(fc -l -1)/history:$(history 1)/histline:${AUDIT_CMD%%+([^ 0-9])*}/last_histline:${AUDIT_LASTHISTLINE}===" #for debugging
}

#audit the session closing
function AUDIT_EXIT() {
  local AUDIT_STATUS="$?"
  if [ -n "$AUDIT_SYSLOG" ]
  then
    logger -p user.info -t "$AUDIT_STR" "#=== bash session ended. ==="
  else
    echo $( date +%F_%H:%M:%S ) "$AUDIT_STR" "#=== bash session ended. ===" >>/var/log/userlog.info
  fi
  exit "$AUDIT_STATUS"
}

#make audit trap functions readonly; disable trap DEBUG inherited (normally the default setting already)
declare -frx +t AUDIT_DEBUG
declare -frx +t AUDIT_EXIT
 #audit the session opening
if [ -n "$AUDIT_SYSLOG" ]
then
  logger -p user.info -t "$AUDIT_STR" "#=== New bash session started. ===" #audit the session openning
else
  echo $( date +%F_%H:%M:%S ) "$AUDIT_STR" "#=== New bash session started. ===" >>/var/log/userlog.info
fi

#enable the trap DEBUG (at every call of $PROMPT_COMMAND) and trap EXIT
declare -rx PROMPT_COMMAND="trap 'AUDIT_DEBUG; trap DEBUG' DEBUG"
declare -rx BASH_COMMAND                                    #current command executed by user or a trap
declare -rx SHELLOPT                                        #shell options, like functrace
trap AUDIT_EXIT EXIT                                        #audit the session closing

When a bash command is executed it first launches the AUDIT_DEBUG(),
then the trap DEBUG is disabled to avoid a useless rerun of AUDIT_DEBUG() during the execution of pipes-commands;
at the end, when the prompt is displayed, it re-enables the trap DEBUG.

Installation

Step 1 – Login as root and to create a file ‘/etc/bash_franzi’ (download bash_franzi) with following content:

#Pointsoftware AG, 2013-01-17
#created by francois scheurer
#filename: '/etc/bash_franzi'
#This file must be sourced by '~/.bashrc', which is the last runned startup script for bash invocation for login interactive, login non-interactive and non-login interactive shells.
#
#Having a complete history of all typed commands can be very helpful in many scenarios:
#  when several administrators work together on the same server and need to know what was done previously
#  when someone need to redo an older sequence of commands or to understand an undocumented maintenance process
#  for troubleshooting or forensic analysis, by crosschecking the date of an event or of a file with the commands executed at that date
#
#The standard '.bash_history' file of the shell is unfortunately not written on disk in the case of a crash and it may be deleted by the user.
#Another problem is that when many shell sessions are running concurrently, their logging will only occur when they are closed, therefore the commands of the history will not appear in their chronological order.
#Furthermore, '.bash_history' will not include essential information like the 'working directory' of the command; and by default the repetition or re-edition of commands will not be logged, too.
#
#Some solutions exist to improve this, either by patching or installing binaries:
#  'bash-BOFH' patching and recompiling: works well but need a new patch for each release of the bash
# 'snoopy': is logging all commands except shell builtins
#  'rootsh / sniffy / ttyrpld / ttysnoop': logs everything, also output of commands, it may be useful but it generates very verbose logs
#  'grsecurity' patched kernels: powerful but it may be a not suitable solution if an official kernel is required (e.g. for Oracle DB)
#  there is also an old 'sshd' patch ('http://www.kdvelectronics.eu/ssh-logging/ssh-logging.html')
#  'screen -x' can also be useful for cooperation work, but it is not a command logger
#
#In contrast to that, the presented method is very easy to deploy; it is just a shellscript that is running in bash (standard shell on most systems) and therefore it is architecture independent.
#It will allow a complete audit of all commands/builtins executed interactively in the bash.
#Note that a user can avoid calling this file by starting a shell with options like '--norc'; he also can unset or overwrite variables like 'PROMPT_COMMAND'.
#Therefore this script is useful for audit but an alternative solution with bash patching should be considered if the security requirements are the priority.
#
#Note on Solaris:
#       In Solaris please use ‘grep’ without the ‘-q’ option, like this:
#               if groups | grep root &>/dev/null
#       Please also remove the following line (chattr unsupported in Solaris:
#               chattr +a “$HISTFILE”
#       Then modify your /etc/syslog.conf to include this line:
#               user.info /var/adm/userlog.info
#       To assign ‘bash’ as the login shell in Solaris: passwd -e /bin/bash .
#       Make sure that the audit-script is sourced (=included) correctly during the bash invocation.
#       If your bash version is too old, $HISTCONTROL will not allow you to log duplicated commands correctly.
#       svcadm restart system/system-log
#       svcadm disable ssh
#       svcadm enable ssh


#to avoid sourcing this file more than once
if [ "${OSTYPE:0:7}" != "solaris" ] #following not working in solaris
then
  if [ "$AUDIT_INCLUDED" == "$$" ] || { [ -z "$SSH_ORIGINAL_COMMAND" ] && [ "$(cat /proc/$$/cmdline)" == 'bash-c"/etc/forcecommand.sh"' ]; }
  then
    return
  else
    declare -rx AUDIT_INCLUDED="$$"
  fi
fi

#prompt & color
#'http://www.pixelbeat.org/docs/terminal_colours/#256'
#'http://www.frexx.de/xterm-256-notes/'
_backnone="\e[00m"
_backblack="\e[40m"
_backblue="\e[44m"
_frontred_b="\e[01;31m"
_frontgreen_b="\e[01;32m"
_frontgrey_b="\e[01;37m"
_frontgrey="\e[00;37m"
_frontblue_b="\e[01;34m"
#PS1="\[${_backblue}${_frontgrey_b}\]\u@\h:\[${_backblack}${_frontblue_b}\]\w\\$\[${_backnone}${_frontgrey_b}\] " #grey
PS1="\[${_backblue}${_frontgreen_b}\]\u@\h:\[${_backblack}${_frontblue_b}\]\w\\$\[${_backnone}${_frontgreen_b}\] " #green
#PS1="\[${_backblue}${_frontred_b}\]\u@\h:\[${_backblack}${_frontblue_b}\]\w\\$\[${_backnone}${_frontred_b}\] " #red
declare -rx PS1

#'history' options
declare -rx HISTFILE="$HOME/.bash_history"
declare -rx HISTSIZE=500000                                 #nbr of cmds in memory
declare -rx HISTFILESIZE=500000                             #nbr of cmds on file
declare -rx HISTCONTROL=""                                  #does not ignore spaces or duplicates
declare -rx HISTIGNORE=""                                   #does not ignore patterns
declare -rx HISTCMD                                         #history line number
history -r                                                  #to reload history from file if a prior HISTSIZE has truncated it
if [ "${OSTYPE:0:7}" != "solaris" ] #following not working in solaris
then
  if groups | grep -q root
  then
    declare -x TMOUT=3600                                     #timeout for root's sessions
    chattr +a "$HISTFILE"                                     #set append-only
  fi
fi
shopt -s histappend
shopt -s cmdhist

#history substitution ask for a confirmation
shopt -s histverify

#add timestamps in history - obsoleted with logger/syslog
#'http://www.thegeekstuff.com/2008/08/15-examples-to-master-linux-command-line-history/#more-130'
#declare -rx HISTTIMEFORMAT='%F %T '

#enable forward search ('ctrl-s')
#'http://ruslanspivak.com/2010/11/25/bash-history-incremental-search-forward/'
if shopt -q login_shell && [ -t 0 ]
then
  stty -ixon
fi



#bash audit & traceability
#
#
#
declare -rx AUDIT_LOGINUSER="$(who -mu | awk '{print $1}')"
declare -rx AUDIT_LOGINPID="$(who -mu | awk '{print $6}')"
declare -rx AUDIT_USER="$USER"                              #defined by pam during su/sudo
declare -rx AUDIT_PID="$$"
declare -rx AUDIT_TTY="$(who -mu | awk '{print $2}')"
declare -rx AUDIT_SSH="$([ -n "$SSH_CONNECTION" ] && echo "$SSH_CONNECTION" | awk '{print $1":"$2"->"$3":"$4}')"
declare -rx AUDIT_STR="[audit $AUDIT_LOGINUSER/$AUDIT_LOGINPID as $AUDIT_USER/$AUDIT_PID on $AUDIT_TTY/$AUDIT_SSH]"
declare -x AUDIT_HISTLINE="0"                               #to avoid logging the same line twice
declare -rx AUDIT_SYSLOG="1"                                #to use a local syslogd
#
#
#
#the logging at each execution of command is performed with a trap DEBUG function
#and having set the required history options (HISTCONTROL, HISTIGNORE)
#and to disable the trap in functions, command substitutions or subshells.
#it turns out that this solution is simple and works well with piped commands, subshells, aborted commands with 'ctrl-c', etc..
set +o functrace                                            #disable trap DEBUG inherited in functions, command substitutions or subshells, normally the default setting already
shopt -s extglob                                            #enable extended pattern matching operators
function AUDIT_DEBUG() {
  local AUDIT_LASTHISTLINE="${AUDIT_HISTLINE}"
  local AUDIT_CMD="$(history 1)"                            #current history command
  AUDIT_HISTLINE="${AUDIT_CMD%%+([^ 0-9])*}"
  if [ "${AUDIT_HISTLINE}" != "${AUDIT_LASTHISTLINE}" ]     #avoid logging unexecuted commands after 'ctrl-c', 'empty+enter', or after 'ctrl-d'
  then
    echo -ne "${_backnone}${_frontgrey}"                    #disable prompt colors for the command's output
    #remove in last history cmd its line number (if any) and send to syslog
    if [ -n "$AUDIT_SYSLOG" ]
    then
      if ! logger -p user.info -t "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}"
      then
        echo error "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}"
      fi
    else
      echo $( date +%F_%H:%M:%S ) "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}" >>/var/log/userlog.info
    fi
    return 0
  else
    return 1
  fi
  #echo "===cmd:$BASH_COMMAND/subshell:$BASH_SUBSHELL/fc:$(fc -l -1)/history:$(history 1)/histline:${AUDIT_CMD%%+([^ 0-9])*}/last_histline:${AUDIT_LASTHISTLINE}===" #for debugging
}
#
#
#
#audit the session closing
function AUDIT_EXIT() {
  local AUDIT_STATUS="$?"
  if [ -n "$AUDIT_SYSLOG" ]
  then
    logger -p user.info -t "$AUDIT_STR" "#=== session closed ==="
  else
    echo $( date +%F_%H:%M:%S ) "$AUDIT_STR" "#=== session closed ===" >>/var/log/userlog.info
  fi
  exit "$AUDIT_STATUS"
}
#
#
#
#make audit trap functions readonly; disable trap DEBUG inherited (normally the default setting already)
declare -frx +t AUDIT_DEBUG
declare -frx +t AUDIT_EXIT
#
#
#
#audit the session opening
if [ -n "$AUDIT_SYSLOG" ]
then
  logger -p user.info -t "$AUDIT_STR" "#=== session opened ===" #audit the session openning
else
  echo $( date +%F_%H:%M:%S ) "$AUDIT_STR" "#=== session opened ===" >>/var/log/userlog.info
fi
#
#
#
#when a bash command is executed it launches first the AUDIT_DEBUG(),
#then the trap DEBUG is disabled to avoid a useless rerun of AUDIT_DEBUG() during the execution of pipes-commands;
#at the end, when the prompt is displayed, re-enable the trap DEBUG
        #declare -rx PROMPT_COMMAND="AUDIT_DONE=; trap 'AUDIT_DEBUG && AUDIT_DONE=1; trap DEBUG' DEBUG; [ -n \"\$AUDIT_DONE\" ] && echo '-----------------------------'"
        #NOK: declare -rx PROMPT_COMMAND="echo "-----------------------------"; trap 'AUDIT_DEBUG; trap DEBUG' DEBUG; echo '-----------------------------'"
        #OK:  declare -rx PROMPT_COMMAND="echo "-----------------------------"; trap 'AUDIT_DEBUG; trap DEBUG' DEBUG"
declare -rx PROMPT_COMMAND="[ -n \"\$AUDIT_DONE\" ] && echo '-----------------------------'; AUDIT_DONE=; trap 'AUDIT_DEBUG && AUDIT_DONE=1; trap DEBUG' DEBUG"
declare -rx BASH_COMMAND                                    #current command executed by user or a trap
declare -rx SHELLOPT                                        #shell options, like functrace
trap AUDIT_EXIT EXIT                                        #audit the session closing
#
#
#
#audit SSH commands bypassing the bash (ssh -c/SCP/SFTP)
if [ -n "${SSH_ORIGINAL_COMMAND}" ]
then
  if [ -n "$AUDIT_SYSLOG" ]
  then
    logger -p user.info -t "$AUDIT_STR $PWD" "${SSH_ORIGINAL_COMMAND}"
  else
    echo $( date +%F_%H:%M:%S ) "$AUDIT_STR $PWD" "${SSH_ORIGINAL_COMMAND}" >>/var/log/userlog.info
  fi
fi

#endof

 
Change its ownership/permissions:

chown root:root /etc/bash_franzi
chmod 644 /etc/bash_franzi

Step 2 – This file need to be sourced automatically at start by updating the init-file ‘.bashrc’:

for i in /etc/profile /etc/skel/.bashrc /root/.bashrc /home/*/.bashrc; do
  if ! grep -q "source /etc/bash_franzi" "$i"
  then
    echo "===updating $i==="
    cat >>"$i" <<EOF
#added by francois scheurer
if [ -f /etc/bash_franzi ]
then
  source /etc/bash_franzi
fi
#endof added
EOF
  fi
done

Step 3 – Configure ‘rsyslogd’:

Add following at the end of ‘/etc/rsyslog.conf’:

#added by francois scheurer
$ActionFileDefaultTemplate RSYSLOG_FileFormat
#endof

Create ‘/etc/rsyslog.d/45-franzi.conf’ with following:

#added by francois scheurer

# Filter duplicated messages
$RepeatedMsgReduction off

# Enable high precision timestamps
$ActionFileDefaultTemplate RSYSLOG_FileFormat

# Log bash audit generated log messages to file
if $syslogfacility-text == 'user' and $syslogseverity-text == 'info' and $syslogtag startswith '[audit' then /var/log/userlog.info

#then drop them
& ~

#'http://content.hccfl.edu/pollock/aunix2/logging.htm'
#'http://www.rsyslog.com/doc/rsyslog_conf_filter.html'

Restart 'rsyslogd':

/etc/init.d/rsyslog restart

Now do a logout and login and you should find the audit in '/var/log/userlog.info'. ^_^

Optional Step 4 - If you also want to audit SCP/SFTP, create also a file '/etc/forcecommand.sh' (download forcecommand.sh) with following content:

#!/bin/bash
#Pointsoftware AG, 2012-08-24
#filename: '/etc/forcecommand.sh'
#created by francois scheurer
#used for bash audit, see '/etc/bash_franzi'

if [ -n "${SSH_ORIGINAL_COMMAND}" ]
then
  exec bash -c "${SSH_ORIGINAL_COMMAND}"
else
  exec -l bash -li
fi
#endof

 
Change its ownership/permissions:

chown root:root /etc/forcecommand.sh
chmod 755 /etc/forcecommand.sh

Then add the following line to your ‘/etc/ssh/sshd_config’:

ForceCommand "/etc/forcecommand.sh"

 
And reload ‘sshd’:

/etc/init.d/ssh reload

 
(hint: test your sshd by starting a second session before closing the current session, just in case )