Site Statistics

Introduction

Statistics of visits to pages in a web can be generated manually or automatically, on a per-web basis. Statistics are compiled as a running total for each month.They include totals for Topic Views, Topic Saves, Attachment Uploads, Most Popular Topics with number of views, and Top Contributors showing total of saves and attachment uploads. Statistics from previous months are saved, and a new row is written to the table at the beginning of each month.

• You can create a WebStatistics link using %STATISTICSTOPIC%

Statistics are compiled by a script that processes the Foswiki event log. The script can be run two ways:

• Manually, by an authorized user from a browser URL
• Automatically by a shell script, usually run as a cron job

Manually running statistics

The following form can be used to run statistics and create the WebStatistics topics for selected or all webs.

Note: The current user - WikiGuest must have access rights to read/write the WebStatistics topics, and if missing, must have authority to create the WebStatistics in the selected webs.

Run statistics on a list of one or more webs

Enter a comma separated list of webnames. Enter a subweb by specifying ParentWeb/SubWeb/ Web names must follow web naming rules and start with an upper case letter.

Additional Settings

Autocreate missing WebStatistics

Select this option if you want the Statistics script to auto-create the WebStatistics if it is missing. (Note: this operation must be enabled in the configuration).

Process sub-webs

Process any subwebs of the requested webs. If this is not selected, then sub-webs must be explicitly listed in the list of webs.

Specify date to process

Specify the date of the logs to process in YYYYMM format. Example: 201203 to request March 2012.

Controlling access to the statistics script

By default, the statistics script may be run by anyone, however the user must have CHANGE permission on the WebStatistics topic. Statistics generation can create a significant workload on the server. In order to further restrict statistics generation:

• Add statistics to the {AuthScripts} list (This is now the default on Foswiki 1.1.4). This is required to prevent unauthenticated guest users from running statistics.
  • If using Apache authentication, be sure to add it to the Apache configuration.
• If desired, create a Group for users that should be permitted to run the statistics script, or use the AdminGroup.
• Visit Configure: Logging and Statistics and set {Stats}{StatisticsGroup} to the group permitted to run statistics.
  • Note: Regardless of this setting, members of the AdminGroup can always run statistics.

Configuring for automatic operation

You can automatically generate usage statistics for all webs. To enable this:

• The WebStatistics topic must be present in all webs where you want to have statistics. See below for how to create the WebStatistics topic.
• Call the bin/statistics script from a cron job, once a day is recommended. This will update the WebStatistics topics in all top level webs. (Sub-webs are not processed by default).
• The script must run as the same user as the CGI scripts are running, which is user nobody on many systems.
• Example crontab entry:
  0 0 * * * (cd /path/to/bin; ./statistics >/dev/null 2>&1)

There is a workaround in case you can't run the script as the CGI user. Use the curl or wget tools available on most distributions. These two examples perform the same function using curl and wget in the crontab. You should only use one of them.

 0 0 * * * (curl -F webs=Sandbox -F subwebs=1 your.host.com/bin/statistics >/dev/null 2>&1)
 0 0 * * * (wget --post-data 'webs=Sandbox&subwebs=1' -O - your.host.com/bin/statistics >dev/null 2>&1)

Both curl and wget can perform http authentication, and can log in if you are using Apache login. See the platform documentation for wget or curl for more information. If using Template login, authentication will not be possible, and the CGI request will run as the WikiGuest user. If you use this workaround, the WebStatistics topics you are updating will have to be writable by WikiGuest, and you cannot add statistics to the {AuthScripts} list

If you do not have curl or wget available on your platform, Foswiki provides a very simple script that can post to or get from a web server. It has no authentication capabilities. Deprecated! This script is deprecated and may be removed in a future release.
Run the utility tools/geturl.pl in your cron job and specify the URL of the bin/statistics script as a parameter as shown here:

   0 0 * * * (cd /path/to/tools; ./geturl.pl POST mydomain.com /urlpath/to/bin/statistics?webs=Sandbox\;subwebs=1 >/dev/null 2>&1)

.

When running from the command line or a cron job, you can pass parameters to the script like this:

./statistics -logdate 200605 -webs Userweb,Sandbox -autocreate 1 -subwebs 1

Parameter	Usage	Default
-logdate	Specify date of logfile in YYYYDD format.	Current logfile
-webs	Comma separated list of webs	All top level webs
-autocreate	1 to enable inclusion of subwebs	disabled. Subwebs must be explicitly listed in -webs to be processed.

Creating the WebStatistics topics

As of release 1.1.4 of Foswiki, the WebStatistics topics are no longer shipped by default. The bin/statistics script can automatically create the required topics when enabled in the configuration, or when requested by URL Parameter autocreate=1.

• If you want the Statistics topic to be automatically created whenever needed, in every web
  • Visit https://nats-www.informatik.uni-hamburg.de/bin/configure#LoggingAndStatistics and set {Stats}{AutoCreateTopic} to Always

• If you want to manually add the WebStatistics topic to a subset of webs
  • Visit https://nats-www.informatik.uni-hamburg.de/bin/configure#LoggingAndStatistics and set {Stats}{AutoCreateTopic} to Permitted
  • Use the below form to run statistics on one or more webs.
  • or add -autocreate 1 to the statistics script as shown above.

• If yow want to prohibit autocreate of WebStatistics topics
  • Visit https://nats-www.informatik.uni-hamburg.de/bin/configure#LoggingAndStatistics and set {Stats}{AutoCreateTopic} to Prohibited
  • Note: This is the default behavior.

Generating statistics manually by URL

The bin/statistics script can also be executed as a CGI script, Use the form at the top of this topic to run statistics.

As of Foswiki 1.1.5, the statistics script must be run using the "post" method. Direct entry of a URL is no longer allowed.

Tailoring the WebStatistics layout

The WebStatistics topics are not shipped with Foswiki by default. The topics are created on demand by the statistics script when requested by configuration. The topics are created as follows:

1. The User web is searched for User.WebStatisticsTemplate - if this topic does not exist
   • The System web is searched for System.WebStatisticsTemplate. (This topic is shipped with Foswiki by default)
2. If {Stats}{AutoCreateTopic} is enabled in the configuration, or the autocreate parameter is passed to the statistics script, then a new WebStatistics is created using the WebStatisticsTemplate topic found in the first step.
   • If autocreate is not requested, and the WebStatistics topic does not exist, then statistics will not be computed for the web.

The template topic contains an %INCLUDE% for help text and table headings, and a critical marker line that defines the columns to be recorded by the statistics script. (This allows the help text or headings to be modified without needing to edit every WebStatistics topic.)

• The WebStatistics topic looks for User.DefaultWebStatistics - if this topic does not exist.
  • The System.DefaultWebStatistics is included. (This topic is shipped with Foswiki by default).

To tailor the help text or column heading, copy System.DefaultWebStatistics to User.DefaultWebStatistics and tailor as desired. This prevents your changes from being lost when Foswiki is updated.

To tailor the actual columns recorded and the order of the columns, copy System.WebStatisticsTemplate to User.WebStatisticsTemplate and tailor as desired. Then remove the previous WebStatistics and allow the statistics script to auto-create a new topic based upon the new template. Alternatively you can edit the WebStatistics in each web.

Statistics are written into the WebStatistics in the following order.

• If a marker line is found in the topic, statistics are inserted into the topic after the marker line.
• If no marker line is present, but the topic contains existing statistics, the current month is update, or a new month is written after the previous month.
• If no marker line is present and no previous statistics exist, a new line is added to the end of the topic.

Removal of the marker line causes statistics to be gathered in chronological order (oldest first). By default, statistics are reported in reverse chronological order.

The marker line defines the order of the columns collected by the statistics script. The order of the columns should correspond to the table heading in the DefaultWebStatistics. Default marker line is:

| <!--statDate--> | <!--statViews--> | <!--statSaves--> | <!--statUploads--> | <!--statTopViews--> | <!--statTopContributors--> |

Restricting access to statistics topics

WebStatistics topics can have access controls like other topics.

If you want to have these access settings copied to new WebStatistics topics, follow these steps:

1. Copy topic WebStatisticsTemplate from System web to User web
2. Edit the topic settings (via the "More" link) and write (if you want to restrict topics views to logged in users):

         * Set DENYTOPICVIEW = WikiGuest
   

For existing WebStatistics topics you will need to write these settings manually.

Managing the WebStatistics topics

Each WebStatistics topic will continue to grow, a new row added every month. In addition, every update of the statistics topic will create a new entry in the WebStatistics topic revision history. These can become extremely large over time and Foswiki does not provide any mechanism for automatically archiving statistics topics. For good ongoing performance, it is recommended to archive statistics annually:

• Enable {Stats}{AutoCreateTopic} or add -autocreate 1 to the cron job that runs the statistics script.
• Rename the WebStatistics to WebStatisticsYYYY at year end.
• Optional: Tailor the DefaultWebStatistics topic, adding a search for archived statistics, for example:

%SEARCH{"." type="regex" topic="%STATISTICSTOPIC%*" excludetopic="%BASETOPIC%" scope="topic" nonoise="on" header="*Other Statistics Topics:*" format="   * $web.$topic"}%

Following these steps will start a fresh statistics topic with a clean revision history.