Lemur Query Log Toolbar - Server Documentation
Version 2.3
[« Back to Query Log Toolbar Homepage]
The Lemur Query Log Server side components are the complement to the Lemur Query Log Toolbar. The server components are based on Java Servlets and a MySQL database and will allow a researcher to collect data as gathered from clients with the Lemur Query Log Toolbar installed.
Contents
- Prerequisites for installation
- Installation as a new Application under an existing Container
- Installation using the Stand-Alone Server
- The Query Log Server Database
Prerequisites for installation
To install and use the Query Log Server components, you must have a MySQL database installation (version 5+ recommended) and Java SE 5+ installed. If you want to just run the servlets, you must also have a servlet container configured (such as Apache's Tomcat). If you do not have a viable servlet container, you can always run the stand-alone server package.
Installation as a new Application under an existing Container (i.e. Tomcat)
- Download and untar and ungzip the Query Log Servlets package from http://www.lemurproject.org/querylogtoolbar/QueryLogServlets.tar.gz
-
Setup your MySQL Server Database
- log on to your MySQL server as the root user ($ mysql -u root -p)
- create a database named "querylogtoolbar" (mysql> create database querylogtoolbar;)
- create the user to connect to the database (mysql> GRANT ALL PRIVILEGES ON querylogtoolbar.* TO 'queryloguser'@'localhost' IDENTIFIED BY 'queryloguser' WITH GRANT OPTION; )
- exit the MySQL shell (mysql> quit; )
- Run the querylogtoolbar_schema.sql script ($ mysql -u queryloguser -p querylogtoolbar < querylogtoolbar_schema.sql )
-
Copy and configure the conf/querylogserver.conf file
- Copy the querylogserver.conf file to somewhere that the Tomcat / Container user can read from
-
Open with a text editor
- edit the querylogserver.db.connection to point at your MySQL database host, port and database name ( format: jdbc:mysql://$MYSQL_HOSTNAME:$MYSQL_PORT/$DATABASENAME?autoReconnect=true )
- edit the querylogserver.db.username and querylogserver.db.password with your values from (2)(c) above
- set the temp directory in querylogserver.server.tempdirectory to your temp directory ( /tmp )
- In your $CATALINA_HOME/webapps/ directory create a new directory for the QueryLogServlets (i.e. "QueryLogToolbar"). This will be your <your_webapp_directory> as noted in the items below
- Copy the contents of the web/* folder to the webapps folder you just created
- Open the file WEB-INF/web.xml in a text editor and modify the path to the querylogserver.conf file in the <param-value> for the configurationPath from the file in (3) above
- Restart your Tomcat (or other container) installation
- Test to see if your installation is working by opening a web browser and going to: http://<your_hostname>:<your_port>/<your_webapp_directory>/GetConfiguration
- The address that the clients (via the settings dialog) should set is: http://<your_hostname>:<your_port>/<your_webapp_directory>/
Installation using the Stand-Alone Server
Installation using the stand-alone server is somewhat similar to the installation instructions for using an existing container (in fact, the stand-alone server application actually creates its own Apache Tomcat container locally). If you do not have an existing container, the stand-alone server application will allow you to run a container and the Query Log Servlets manually.
- Download and untar / ungzip the Stand-alone Query Log Server package from http://www.lemurproject.org/querylogtoolbar/LemurQueryLogServer.tar.gz
-
Setup your MySQL Server Database
- log on to your MySQL server as the root user ($ mysql -u root -p)
- create a database named "querylogtoolbar" (mysql> create database querylogtoolbar;)
- create the user to connect to the database (mysql> GRANT ALL PRIVILEGES ON querylogtoolbar.* TO 'queryloguser'@'localhost' IDENTIFIED BY 'queryloguser' WITH GRANT OPTION; )
- exit the MySQL shell (mysql> quit; )
- Run the querylogtoolbar_schema.sql script ($ mysql -u queryloguser -p querylogtoolbar < querylogtoolbar_schema.sql )
-
Copy and configure the conf/querylogserver.conf file
- Copy the querylogserver.conf file to somewhere that the Tomcat / Container user can read from
-
Open with a text editor
- edit the querylogserver.db.connection to point at your MySQL database host, port and database name ( format: jdbc:mysql://$MYSQL_HOSTNAME:$MYSQL_PORT/$DATABASENAME?autoReconnect=true )
- edit the querylogserver.db.username and querylogserver.db.password with your values from (2)(c) above
- set the temp directory in querylogserver.server.tempdirectory to your temp directory ( /tmp )
- Open the file webapps/QueryLogToolbar/WEB-INF/web.xml in a text editor and modify the path to the querylogserver.conf file in the <param-value> for the configurationPath from the file in (3) above
- Run the file “run_server.sh” (or run_server.bat if on windows) from a command line. The server should display some information regarding it's startup to the console. Note that the default port is 4592, but this can be changed by adding the port to the run_server command (i.e. “run_server.sh 8099” will use port 8099 instead). If all went well, the server should be started and you can verify that it is started by browsing to http://<your_hostname>:<your_port>/QueryLogToolbar/GetConfiguration
- The address that the clients (via the settings dialog) should set is: http://<your_hostname>:<your_port>/QueryLogToolbar/
The Query Log Server Database
The incoming data to the Query Log Server eventually gets stored in the MySQL server database as configured above. Pictured below is the database schema (as set up in the querylogtoolbar_schema.sql script). Click on the image to see the full view.
Query Log Server Data Dictionary
| Table: tblClients | ||
| Holds the client information for log uploads | ||
| Field | Type | Description |
|---|---|---|
| ClientID | Autonumber(bigint) | The unique client ID |
| ClientIP | Varchar(64) | The IP address associated with this client |
| ClientSessionID | Varchar(128) | The client unique session ID for this IP (note: not shown in the above diagram) |
| ClientUserAgent | Varchar(255) | The last known user agent (web browser version information) for this client |
| Table: tblLogs | ||
| The actual individual log uploads by clients | ||
| Field | Type | Description |
| LogID | Autonumber(bigint) | The unique ID of this query log |
| ClientID | bigint (FK) | Foreign key of the client ID of this log (tblClients) |
| LogUploadTimestamp | datetime | The date/time this log was uploaded |
| Table: tblLogAction | ||
| The raw set of actions performed in a log | ||
| Field | Type | Description |
| ActionID | Autonumber(bigint) | The unique action ID |
| LogID | Bigint (FK) | Foreign key to the Log ID for this action (tblLogs) |
| ActionTypeID | Bigint (FK) | Foreign key to the action type ID for this action (tblLogActionType) |
| LogPageID | Bigint (FK) | (can be null) Foreign key to log page (tblLogPage) to associate URL of the page with this action |
| ActionTimestamp | datetime | The date/time of this action |
| Table: tblLogActionType | ||
| Defines the available log action types collected | ||
| Field | Type | Description |
| ActionTypeID | bigint | The unique action type ID |
| ActionType | Varchar(255) | The action type (as defined by list of available actions in the upload log widget) |
| Table: tblLogPages | ||
| Stores information about individual web pages visited in a log | ||
| Field | Type | Description |
| LogPageID | Autonumber(bigint) | The unique log page ID |
| LogID | bigint (FK) | The foreign key to the LogID (tblLogs) |
| PageLoadTimestamp | datetime | The date/time the page was loaded in the browser |
| TimeOnPage | Int | Time (in seconds) spent on the page before moving to a different page, tab, or closing the browser |
| NumScrollEvents Int The number of scroll events (with mouse or scrollbars) during viewing of the page | ||
| Table: tblPageCopyEvents | ||
| Stores information about items copied (Ctrl+C) from a web page | ||
| Field | Type | Description |
| CopyEventID | Autonumber(bigint) | The unique copy event ID |
| ActionID | bigint (FK) | The action ID this is associated with (tblLogAction) |
| CopyText | Text | The actual text copied |
| Table: tblActionPageClick | ||
| Stores information about mouse click events on pages | ||
| Field | Type | Description |
| ActionPageClickID | Autonumber(bigint) | The unique click event ID |
| ActionID | bigint (FK | The action ID this is associated with (tblLogAction) |
| WhichButton | Int | 0=left; 1=right; 2=middle |
| LinkText | Text | The link text clicked on (if any) |
| Table: tblSearchPages | ||
| Stores search page information (Google, MSN, etc.) | ||
| Field | Type | Description |
| SearchPageID | Autonumber(bigint) | Unique search page ID |
| LogI | Bigint (FK) | Foreign key to the log this search page is associated with (tblLogs) |
| ActionID | bigint (FK) | The action ID this is associated with this search event (tblLogAction) |
| SearchTimestamp | datetime | Date/time of the search |
| SearchURL | Varchar(2048) | The actual URL that performed the search |
| SearchQuery | Varchar(255) | The query of the search |
| NumResults | Int | Total number of results as returned by the search engine (could be actual, could be estimated…) |
| Table: tblSearchResults | ||
| Stores the results from a search page | ||
| Field | Type | Description |
| SearchResultID | Autonumber(bigint) | The unique search result ID |
| SearchPageID | Bigint (FK) | Foreign key to the search page ID (tblSearchPages) |
| ResultRank | Integer | The rank of the page (in the search results document) |
| ResultLink | Varchar(2048) | The URL of the result item |
| ResultTitle | Varchar(255) | The title of the result item |
| ResultSnippet | Text | The snippet of the result item (if available) |
| Table: tblLogPageSearches | ||
| Correlates individual log pages with a search page item | ||
| Field | Type | Description |
| LogPageID | Bigint (FK) | Foreign key to the log page this search page is associated with (tblLogPages) |
| SearchPageID | bigint (FK) | Foreign key to the search page this log page item is associated with (tblSearchPages) |
