Using goaccess to generate reports on webserver access logs

Linux LXC containers

If you're running a public webserver, it can be useful to know more about your audience. You can use Google analytics on your site, or use a tool to process the access logs of your webserver. goaccess is a lightweight tool to go through your webserver log files and create a report on the entries.

You can use geoip to make the ip's processed more meaningful.

Install goaccess

We will install goaccess and the GeoIP packages. If you're not Debian, use the appropriate install tool for your distribution. On Debian:

apt-get install geoip-bin geoip-database geoip-database-extra goaccess

To get the latest goaccess, the goaccess website advises these additional steps.

Warning

You shouldn't do this on a production server but on a local machine. Get the logs over to the machine to process locally for instance via sftp.

To install a more recent goaccess:

echo "deb https://deb.goaccess.io/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list.d/goaccess.list
wget -O - https://deb.goaccess.io/gnugpg.key | sudo apt-key add -

apt-get update
apt-get install goaccess

If the first line doesn't work, this might be because lsb_release doesn't work. Usually this translates to the release name such as jessie, sid, ... The geoip data is installed in /usr/share/GeoIP. If you have it installed in another directory, keep that in mind for the goaccess configuration:

/usr/share/GeoIP/GeoIP.dat
/usr/share/GeoIP/GeoIPCity.dat

A more recent database can be downloaded from Maxmind. The goaccess.conf also hase more info on the topic:

wget -q https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz

Configuration

In the goaccess configuration file, set the appropriate options. Edit the config file with your favourite editor:

vi /etc/goaccessrc

Set the correct logfile for the webserver you are using such as Apache, nginx, ... If you are not sure, examine an access.log file. An example log entry from nginx:

192.168.x.y - - [10/Apr/2017:14:07:34 +0200] "GET /index.html HTTP/1.1" 200 16963 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; InfoPath.3; .NET4.0C; .NET4.0E; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Microsoft Outlook 14.0.7172; ms-office; MSOffice 14)"

The nginx standard logging format used, as specified in the docs:

$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

With the date, time and logformat specified, goaccess should be able to process the logs:

time-format %H:%M:%S
date-format %d/%b/%Y
log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"

Here's my goacces.conf file obtained via "grep ^[^#] /etc/goaccess.conf"

time-format %H:%M:%S
date-format %d/%b/%Y
log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"
config-dialog false
color-scheme 1
no-color false
static-file .css
static-file .CSS
static-file .dae
static-file .DAE
static-file .eot
static-file .EOT
static-file .gif
static-file .GIF
static-file .ico
static-file .ICO
static-file .jpeg
static-file .JPEG
static-file .jpg
static-file .JPG
static-file .js
static-file .JS
static-file .map
static-file .MAP
static-file .mp3
static-file .MP3
static-file .pdf
static-file .PDF
static-file .png
static-file .PNG
static-file .svg
static-file .SVG
static-file .swf
static-file .SWF
static-file .ttf
static-file .TTF
static-file .txt
static-file .TXT
static-file .woff
static-file .WOFF
agent-list false
http-method true
http-protocol true
no-query-string false
no-term-resolver false
real-os true
no-progress false
with-mouse false
with-output-resolver false
geoip-city-data /usr/share/GeoIP/GeoIPCity.dat

Run

To generate a report, preferably from a local machine:

zcat -f access.log* | goaccess -c -a -o report.html

This will generate a report.html file from the access logs. You might get an error from zcat because access.log isn't a gzipped file. The result is a html page, open it with a browser.

Go access example