NAME 

AWFFull - A Webalizer Fork, Full o' features

SYNOPSIS 

awffull [ option ... ] [ log-file ]

DESCRIPTION 

AWFFull is a web server log analysis program based on "The Webalizer". AWFFull produces usage statistics in HTML format for viewing with a browser. The results are presented in both columnar and graphical format, which facilitates interpretation. Yearly, monthly, daily and hourly usage statistics are presented, along with the ability to display usage by site, URL, referrer, user agent (browser), user name, search strings, entry/exit pages, and country (some information may not be available if not present in the log file being processed).

AWFFull supports CLF (common log format) log files, as well as Combined log formats as defined by NCSA and others, and variations of these which it attempts to handle intelligently. In addition, AWFFull also supports wu-ftpd xferlog formatted log files, allowing analysis of ftp servers, and squid proxy logs. Logs may also be compressed, via gzip. If a compressed log file is detected, it will be automatically uncompressed while it is read. Compressed logs must have the standard gzip extension of .gz.

This documentation applies to AWFFull Version 3.7.4

CHANGES FROM WEBALIZER 

AWFFull is based on the Webalizer code and has a number of large and small changes. These include:
o
GeoIP IP Address look-ups for more accurate country detection
o
Resizable graphs
o
Display more than 12 months of the site history on the front page
o
Additional page count tracking and sort by same
o
Some minor visual tweaks, including Geolizer's use of Kb, Mb etc for Volumes
o
Additional Pie Charts for URL counts, Entry and Exit Pages, and Sites
o
Horizontal lines on graphs that are more sensible and easier to read
o
User Agent and Referral tracking is now calculated via PAGES not HITS
o
User bookmarkings are estimated.
o
GNU style long command line options are now supported (eg --help)
o
Can choose what is a page by excluding "what isn't" vs the original "what is" method.
o
Requests to the server are displayed with the matching referring URL.
o
A Table of 404 Errors, and the referring URL can be generated.
o
An external CSS file can be used with the generated html.
o
Manual optimisation of the config file is now easier with a post analysis summary output.
o
Specified IP's & Addresses can be assigned to a given country.
o
Additional Dump options for post AWFFull, detailed analysis.
o
Lotus Domino v6 logs are now detected and processed.
Additional changes and improvements are planned and undergoing implementation.
See the TODO file for details.

RUNNING AWFFULL 

AWFFull is designed to be run from a Unix command line prompt or as a job. Once executed, the general flow of the program is:
o
A default configuration file is scanned for. A file named awffull.conf is searched for in the current directory, and if found, it's configuration data is parsed. If the file is not present in the current directory, the file /etc/awffull.conf is searched for and, if found, is used instead.
o
Any command line arguments given to the program are parsed. This may include the specification of a configuration file, which is processed at the time it is encountered.
o
If a log file was specified, it is opened and made ready for processing. If no log file was given, STDIN is used for input. If the log filename '-' is specified, STDIN will be forced.
o
If an output directory was specified, the program does a to that directory in preparation for generating output. If no output directory was given, the current directory is used.
o
If no hostname was given, the program attempts to get the hostname using a system call. If that fails, localhost is used.
o
A history file is searched for in the current directory (output directory) and read if found. This file keeps totals for previous months, which is used in the main index.html HTML document. Note: The file location can now be specified with the HistoryName configuration option.
o
If incremental processing was specified, a data file is searched for and loaded if found, containing the 'internal state' data of the program at the end of a previous run. Note: The file location can now be specified with the IncrementalName configuration option.
o
Main processing begins on the log file. If the log spans multiple months, a separate HTML document is created for each month.
o
After main processing, the main index.html page is created, which has totals by month and links to each months HTML document.
o
A new history file is saved to disk, which includes totals generated by AWFFull during the current run.
o
If incremental processing was specified, a data file is written that contains the 'internal state' data at the end of this run.

INCREMENTAL PROCESSING 

Version 1.2x of The Webalizer added incremental run capability. Simply put, this allows processing large log files by breaking them up into smaller pieces, and processing these pieces instead. What this means in real terms is that you can now rotate your log files as often as you want, and still be able to produce monthly usage statistics without the loss of any detail. Basically, AWFFull saves and restores all internal data in a file named awffull.current. This allows the program to 'start where it left off' so to speak, and allows the preservation of detail from one run to the next. The data file is placed in the current output directory, and is a plain ASCII text file that can be viewed with any standard text editor. It's location and name may be changed using the IncrementalName configuration keyword.

Some special precautions need to be taken when using the incremental run capability of AWFFull. Configuration options should not be changed between runs, as that could cause corruption of the internal data stored. For example, changing the MangleAgents level will cause different representations of user agents to be stored, producing invalid results in the user agents section of the report. If you need to change configuration options, do it at the end of the month after normal processing of the previous month and before processing the current month. You may also want to delete the awffull.current file as well.

AWFFull also attempts to prevent data duplication by keeping track of the timestamp of the last record processed. This timestamp is then compared to current records being processed, and any records that were logged previous to that timestamp are ignored. This, in theory, should allow you to re-process logs that have already been processed, or process logs that contain a mix of processed/not yet processed records, and not produce duplication of statistics. The only time this may break is if you have duplicate timestamps in two separate log files... any records in the second log file that do have the same timestamp as the last record in the previous log file processed, will be discarded as if they had already been processed. There are lots of ways to prevent this however, for example, stopping the web server before rotating logs will prevent this situation. This setup also necessitates that you always process logs in chronological order, otherwise data loss will occur as a result of the timestamp compare.

REVERSE DNS LOOKUPS 

AWFFull no longer supports DNS lookups. Please use an external program such as DNShistory or DNSTran instead.
o
http://www.summary.net/soft/dnstran.html
o

With version 3.7.1 of AWFFull, GeoIP capability can be used for more accurate country detection.

COMMAND LINE OPTIONS 

AWFFull supports many different configuration options that will alter the way the program behaves and generates output. Most of these can be specified on the command line, while some can only be specified in a configuration file. The command line options are listed below, with references to the corresponding configuration file keywords.

General Options

-h --help
Display all available command line options and exit program.
-V --version
Display program version and exit program.
-v --verbose
Verbosity. Display debugging information for errors and warnings. Multiple v's will increase the amount of information displayed.
--match_counts
Display optimisation useful information pertaining to the number of matches against various Group, Hide and Ignore options.
-i --ignore_history
IgnoreHist. Ignore history. USE WITH CAUTION. This will cause AWFFull to ignore any previous monthly history file only. Incremental data (if present) is still processed.
-p --preserve_state
Incremental. Preserve internal data between runs.
-q --quiet
Quiet. Suppress informational messages. Does not suppress warnings or errors.
-Q --vquiet
ReallyQuiet. Suppress all messages including warnings and errors.
-T --timing
TimeMe. Force display of timing information at end of processing.
-c --config=FILE
Use configuration file FILE.
-n NAME
HostName. Use the hostname NAME.
-o --output=DIR
OutputDir. Use output directory DIR.
-t NAME
ReportTitle. Use NAME for report title.
ReportTitle. Use NAME for report title.
-F --logtype=TYPE
LogType. Specify log type to be processed. Value can be either auto, clf, combined, domino, ftp or squid format. If not specified, will default to auto format. FTP logs must be in standard wu-ftpd xferlog format. In auto mode, the log format will, hopefully, be successfully ascertained.
-f
FoldSeqErr. Fold out of sequence log records back into analysis, by treating as if they were the same date/time as the last good record. Normally, out of sequence log records are simply ignored.
-Y
CountryGraph. Suppress country graph.
-G
HourlyGraph. Suppress hourly graph.
-x NAME
HTMLExtension. Defines HTML file extension to use. If not specified, defaults to html. Do not include the leading period.
-H
HourlyStats. Suppress hourly statistics.
-L
GraphLegend. Suppress color coded graph legends.
-l NUM
GraphLines. Specify number of background lines. Default is 2. Use zero ('0') to disable the lines.
-P NAME
PageType. Specify file extensions that are considered pages. Sometimes referred to as pageviews.
-m NUM
VisitTimeout. Specify the Visit timeout period. Specified in number of seconds. Default is 1800 seconds (30 minutes).
-I NAME
IndexAlias. Use the filename name as an additional alias for index..
-M NUM
MangleAgents. Mangle user agent names according to the mangle level specified by num. Mangle levels are:
5 Browser name and major version.
4 Browser name, major and minor version.
3 Browser name, major version, minor version to two decimal places.
2 Browser name, major and minor versions and sub-version.
1 Browser name, version and machine type if possible.
0 All information (left unchanged).
-g NUM
GroupDomains. Automatically group sites by domain. The grouping level specified by num can be thought of as 'the number of dots' to display in the grouping. The default value of 0 disables any domain grouping.

Hide Options

-a NAME
HideAgent. Hide user agents matching name.
-r NAME
HideReferrer. Hide referrer matching name.
-s NAME
HideSite. Hide site matching name.
-X NAME
HideAllSites. Hide all individual sites (only display groups).
-u NAME
HideURL. Hide URL matching name.

Table size options

-A NUM
TopAgents. Display the top num user agents table.
-R NUM
TopReferrers. Display the top num referrers table.
-S NUM
TopSites. Display the top num sites table.
-U NUM
TopURLs. Display the top num URL's table.
-C NUM
TopCountries. Display the top num countries table.
-e NUM
TopEntry. Display the top num entry pages table.
-E NUM
TopExit. Display the top num exit pages table.
--use_geoip
Enables the use of the Maxmind GeoIP capability for more accurate detection of countries.

NOTE! Do not enable GeoIP if you analyse files that have had the IP Address translated to a Fully Qualified Host Name. Use either raw IP Addresses and GeoIP, or Names and disable GeoIP. ie. Don't use GeoIP AND DNShistory.

--match_counts
Display the various Group/Hide etc Match Counts. This option is ideal for optimisation of the awffull.conf file. Just be careful with optimising Agents in particular, as the order is typically important.

CONFIGURATION FILES 

Configuration files are standard text files that may be created or edited using any standard editor. Blank lines and lines that begin with a pound sign ('#') are ignored. Any other lines are considered to be configuration lines, and have the form "Keyword Value", where the below, and 'Value' is the value to assign to that particular option. Any text found after the keyword up to the end of the line is considered the keyword's value, so you should not include anything after the actual value on the line that is not actually part of the value being assigned. The file sample.conf provided with the distribution contains lots of useful documentation and examples as well.

General Configuration Keywords

LogFile name
Use log file named name. If none specified, STDIN will be used.
LogType name
Specify log file type as name. Values can be either web, squid or ftp, with the default being web.
OutputDir dir
Create output in the directory dir. If none specified, the current directory will be used.
HistoryName name
Filename to use for history file. Relative to output directory unless absolute name is given (ie: starts with '/'). Defaults to awffull.hist' in the standard output directory.
IndexMonths num
Number of Months to display on the main index page. Defaults to 12. Max of 60. (5 years x 12 - can be changed in awffull.h MAXHISTLEN)
ReportTitle name
Use the title string name for the report title. If none
Use the title string name for the report title. If none specified, use the default of (in English) "Usage Statistics for ".
Hostname name
Set the hostname for the report as name. If none specified, an attempt will be made to gather the hostname via a system call. If that fails, localhost will be used.
UseHTTPS ( yes | no )
Use https:// on links to URLS, instead of the default http://, in the 'Top URL's' table.
Quiet ( yes | no )
Suppress informational messages. Warning and Error messages will not be suppressed.
ReallyQuiet ( yes | no )
Suppress all messages, including Warning and Error messages.
TimeMe ( yes | no )
Force timing information at end of processing.
GMTTime ( yes | no )
Use GMT (UTC) time instead of local timezone for reports.
IgnoreHist ( yes | no )
Ignore previous monthly history file. USE WITH CAUTION. Does not prevent Incremental file processing.
FoldSeqErr ( yes | no )
Fold out of sequence log records back into analysis by treating them as if they had the same date/time as the last good record. Normally, out of sequence log records are ignored.
CountryGraph ( yes | no )
Display Country Usage Graph in output report.
GeoIP ( yes | no )
GeoIP enables or disables the use of the Maxmind GeoIP capability for more accurate detection of countries. Default is 'no'.

NOTE! Do not enable GeoIP if you analyse files that have had the IP Address translated to a Fully Qualified Host Name. Use either raw IP Addresses and GeoIP, or Names and disable GeoIP. ie. Don't use GeoIP AND DNShistory.

GeoIPDatabase file
GeoIPDatabase is the location of the GeoIP database file. Default is '/usr/local/share/GeoIP/GeoIP.dat', which is where a default GeoIP install will put it. Note that the database is updated monthly. See Maxmind's website for details: http://www.maxmind.com/app/geoip_country
FlagsLocation dir
If set, (default is not) will create an Image link to a flag to display in the country table. Identified by TLD. The directory is for a web server, and could be an entire URL if appropriate. A trailing slash is not necessary.
DailyGraph ( yes | no )
Display Daily Graph in output report.
DailyStats ( yes | no )
Display Daily Statistics in output report.
HourlyGraph ( yes | no )
Display Hourly Graph in output report.
HourlyStats ( yes | no )
Display Hourly Statistics in output report.
TopURLsbyHITsGraph ( yes | no )
Display the top URLs by HITS pie chart
TopURLsbyVolGraph ( yes | no )
Display the top URLs by VOLUME pie chart
TopEntryPagesGraph ( no | hits | visits )
Display Top Entry Pages Pie Chart, either by HITS or by VISITS
TopExitPagesGraph ( no | hits | visits )
Display Top Exit Pages Pie Chart, either by HITS or by VISITS
TopSitesbyPagesGraph ( yes | no )
Display the top SITES by Page Impressions pie chart
TopSitesbyVolGraph ( yes | no )
Display the top SITES by VOLUME pie chart
TopAgentsGraph ( yes | no )
Display the top User Agents by Page Impressions pie chart
PageType name
Define the file extensions to consider as a page. If a file is found to have the same extension as name, it will be counted as a page (sometimes called a pageview).
NotPageType name
Define the file extensions to exclude from being considered as a page. This is the direct and incompatible opposite of PageType. You can use one set or the other, but not both. Do not add the "." or use any wildcards. As a general rule.
GraphLegend ( yes | no )
Allows the color coded graph legends to be enabled/disabled.
GraphLines num
Display background reference lines and Y Axis scales. Sizing/Number of is now auto generated. For backwards compatibility, any number > 0 enables. Disable by using zero ('0'). Default is 2 - ie on.
YearlySubtotals ( yes | no )
YearlySubtotals will display the subtotal for a given year in the main page. This is in addition to the Grand Total of all years.
VisitTimeout num
Specifies the visit timeout value. Default is 1800 seconds (30 minutes). A visit is determined by looking at the difference in time between the current and last request from a specific site. If the difference is greater or equal to the timeout value, the request is counted as a new visit. Specified in seconds.
IndexAlias name
Use name as an additional alias for index.*.
MangleAgents num
Mangle user agent names based on mangle level num. See the -M command line switch for mangle levels and their meaning. The default is 0, which doesn't mangle user agents at all.
SearchEngine name variable
Allows the specification of search engines and their query strings. The name is the name to match against the referrer string for a given search engine. The variable is the cgi variable that the search engine uses for queries. See the sample.conf file for example usage with common search engines.
Incremental ( yes | no )
Enable Incremental mode processing.
IncrementalName name
Filename to use for incremental data. Relative to output directory unless an absolute name is given (ie: starts with '/'). Defaults to 'webalizer.current' in the standard output directory.

Top Table Keywords

TopAgents num
Display the top num User Agents table. Use zero to disable.
AllAgents ( yes | no )
Create separate HTML page with All User Agents.
TopReferrers num
Display the top num Referrers table. Use zero to disable.
AllReferrers ( yes | no )
Create separate HTML page with All Referrers.
TopSites num
Display the top num Sites table. Use zero to disable.
TopKSites num
Display the top num Sites (by KByte) table. Use zero to disable.
AllSites ( yes | no )
Create separate HTML page with All Sites.
TopURLs num
Display the top num URLs table. Use zero to disable.
TopKURLs num
Display the top num URLs (by KByte) table. Use zero to disable.
AllURLs ( yes | no )
Create separate HTML page with All URLs.
TopCountries num
Display the top num Countries in the table. Use zero to disable.
TopEntry num
Display the top num Entry Pages in the table. Use zero to disable.
TopExit num
Display the top num Exit Pages in the table. Use zero to disable.
TopSearch num
Display the top num Search Strings in the table. Use zero to disable.
AllSearchStr ( yes | no )
Create separate HTML page with All Search Strings.
TopUsers num
Display the top num User names in the table. Use zero to disable. User names are only available if using HTTP based authentication.
Top404Errors num
Display the top num 404 Error requests and their referring URLS in the table. Use zero to disable.
All404Errors ( yes | no )
Create separate HTML page with All 404 Error requests.
AllUsers ( yes | no )
Create separate HTML page with All User names.

Hide/Ignore/Group/Include Keywords

Do be aware that name can be enclosed in double quotes to allow for spaces to match as well. This is a new feature in AWFFull.

GroupAndHideXXX
All the Hide and Group "name" options can be combined in a single config line. eg GroupAndHideURL. Saves on duplication of essentially duplicate entries.
HideAgent name
Hide User Agents that match name.
HideReferrer name
Hide Referrers that match name.
HideSite name
Hide Sites that match name.
HideAllSites ( yes | no )
Hide all individual sites. This causes only grouped sites to be displayed.
HideURL name
Hide URL's that match name.
HideUser name
Hide User names that match name.
IgnoreAgent name
Ignore User Agents that match name.
IgnoreReferrer name
Ignore Referrers that match name.
IgnoreSite name
Ignore Sites that match name.
IgnoreURL name
Ignore URL's that match name.
IgnoreUser name
Ignore User names that match name.
GroupAgent name [Label]
Group User Agents that match name. Display Label in 'Top Agent' table if given (instead of name).
GroupReferrer name [Label]
Group Referrers that match name. Display Label in 'Top Referrer' table if given (instead of name).
GroupSite name [Label]
Group Sites that match name. Display Label in 'Top Site' table if given (instead of name).
GroupDomains num
Automatically group sites by domain. The value num specifies the level of grouping, and can be thought of as the 'number of dots' to be displayed. The default value of 0 disables domain grouping.
GroupURL name [Label]
Group URL's that match name. Display Label in 'Top URL' table if given (instead of name).
GroupUser name [Label]
Group User names that match name. Display Label in 'Top User names' table if given (instead of name).
IncludeSite name
Force inclusion of sites that match name. Takes precedence over Ignore# keywords.
IncludeURL name
Force inclusion of URL's that match name. Takes precedence over Ignore# keywords.
IncludeReferrer name
Force inclusion of Referrers that match name. Takes precedence over Ignore# keywords.
IncludeAgent name
Force inclusion of User Agents that match name. Takes precedence over Ignore* keywords.
IncludeUser name
Force inclusion of User names that match name. Takes precedence over Ignore* keywords.
AssignToCountry address country_code
Assign a given address to a specified Country. eg. *.bigpond.com au. Useful when you get a lot of .com's etc that are actually specific to a given country.

HTML Generation Keywords

CSSFilename awffull.css
Use this file for the stylesheet to apply against the generated HTML output. An existing file is not overwritten, so modify to your hearts content!
HTMLExtension text
Defines the HTML file extension to use. Default is html. Do not include the leading period!
HTMLPre text
Insert text at the very beginning of the generated HTML file. Defaults to a standard html 3.2 DOCTYPE record.
HTMLHead text
Insert text within the <HEAD></HEAD> block of the HTML file.
HTMLBody text
Insert text in HTML page, starting with the <BODY> tag. If used, the first line must be a <BODY ...> tag. Multiple lines may be specified.
HTMLPost text
Insert text at top (before horizontal. rule) of HTML pages. Multiple lines may be specified.
HTMLTail text
Insert text at bottom of the HTML page. The text is top and right aligned within a table column at the end of the report.
HTMLEnd text
Insert text at the very end of the HTML page. If not specified, the default is to insert the ending </BODY> and </HTML> tags. If used, you must supply these tags yourself.
GraphIndexX num
GraphIndexY num
Sets the X & Y size of the main front page summary graph in pixels. Some size combinations will give silly results. Defaults to 512 x 256.
GraphMonthlyX num
GraphMonthlyY num
Sets the X & Y size of the Month summary graph in pixels. Some size combinations will give silly results. Defaults to 512 x 400.
GraphDailyX num
GraphDailyY num
Sets the X & Y size of the Daily summary graph in pixels. Some size combinations will give silly results. Defaults to 512 x 400.
GraphPieX num
GraphPieY num
Sets the X & Y size of all Pie charts in pixels. Some size combinations will give silly results. Defaults to 512 x 300.

Dump Object Keywords

AWFFull allows you to export processed data to other programs by using tab delimited text files. The Dump* commands specify which files are to be written, and where.

DumpPath name
Save dump files in directory name. If not specified, the default output directory will be used. Do not specify a trailing slash (/).
DumpExtension name
Use name as the filename extension for dump files. If not given, the default of tab will be used.
DumpHeader ( yes | no )
Print a column header as the first record of the file.
DumpSites ( yes | no )
Dump the sites data to a tab delimited file.
DumpURLs ( yes | no )
Dump the URL data to a tab delimited file.
DumpReferrers ( yes | no )
Dump the referrer data to a tab delimited file. This data is only available if using a log that contains referrer information (ie: a combined format web log).
DumpAgents ( yes | no )
Dump the user agent data to a tab delimited file. This data is only available if using a log that contains user agent information (ie: a combined format web log).
DumpUsers ( yes | no )
Dump the user name data to a tab delimited file. This data is only available if processing a wu-ftpd xferlog or a web log that contains HTTP authentication information.
DumpSearchStr ( yes | no )
Dump the search string data to a tab delimited file. This data is only available if processing a web log that contains referrer information and had search string information present.
DumpCountries ( yes | no )
Dump the country data to a tab delimited file. This data is modified by the AssignToCountry option.
DumpEntryPages ( yes | no )
Dump the Entry Pages data to a tab delimited file.
DumpExitPages ( yes | no )
Dump the Exit Pages data to a tab delimited file.
ColorHit ( rrggbb | 00805c )
Sets the graph's hit-color to the specified html color (no '#').
ColorFile ( rrggbb | 0000ff )
Sets the graph's file-color to the specified html color (no '#').
ColorSite ( rrggbb | ff8000 )
Sets the graph's site-color to the specified html color (no '#').
ColorKbyte ( rrggbb | ff0000 )
Sets the graph's kilobyte-color to the specified html color (no '#').
ColorPage ( rrggbb | 00c0ff )
Sets the graph's page-color to the specified html color (no '#').
ColorVisit ( rrggbb | ffff00 )
Sets the graph's visit-color to the specified html color (no '#').
ColorBookm ( rrggbb | ff00ff )
Sets the graph's bookmark-color to the specified html color (no '#').
PieColor1 ( rrggbb | 800080 )
Sets the pie's first optional color to the specified html color (no '#').
PieColor2 ( rrggbb | 80ffc0 )
Sets the pie's second optional color to the specified html color (no '#').
PieColor3 ( rrggbb | ff00ff )
Sets the pie's third optional color to the specified html color (no '#').
PieColor4 ( rrggbb | ffc480 )
Sets the pie's fourth optional color to the specified html color (no '#').

FILES 

awffull.conf
Default configuration file. Is searched for in the current directory and if not found, in the /etc/ directory.
awffull.hist
Monthly history file for previous 12 months. (can be changed)
awffull.current
Current state data file (Incremental processing). (can be changed)
xxxxx_YYYYMM.html
Various monthly HTML output files produced. (extension can be changed)
xxxxx_YYYYMM.png
Various monthly image files used in the reports.
xxxxx_YYYYMM.tab
Monthly tab delimited text files. (extension can be changed)

BUGS 

o
Please see:

Report bugs to .

COPYRIGHT 

Copyright (C) 2004, 2005, 2006, 2007 Stephen McInerney Copyright (C) 2006 Benoit Rouits Copyright (C) 1997-2000 by Bradford L. Barrett. Distributed under the GNU GPL. See the files "COPYING" and "Copyright", supplied with all distributions for additional information.

AUTHORS 

Steve McInerney <> Alexander Lazic <> Patrick Ben Koetter <p@state-of-mind.dei> Benoit Rouits <> Stanislaw Yurievich Pusep Bradford L. Barrett <>