Lemur Query Log Toolbar - Server Documentation

Version 2.3


[« Back to Query Log Toolbar Homepage]


The Lemur Query Log Server side components are the complement to the Lemur Query Log Toolbar. The server components are based on Java Servlets and a MySQL database and will allow a researcher to collect data as gathered from clients with the Lemur Query Log Toolbar installed.

Contents


Prerequisites for installation

To install and use the Query Log Server components, you must have a MySQL database installation (version 5+ recommended) and Java SE 5+ installed. If you want to just run the servlets, you must also have a servlet container configured (such as Apache's Tomcat). If you do not have a viable servlet container, you can always run the stand-alone server package.

[Back to Top]

Installation as a new Application under an existing Container (i.e. Tomcat)

  1. Download and untar and ungzip the Query Log Servlets package from http://www.lemurproject.org/querylogtoolbar/QueryLogServlets.tar.gz
  2. Setup your MySQL Server Database
    1. log on to your MySQL server as the root user ($ mysql -u root -p)
    2. create a database named "querylogtoolbar" (mysql> create database querylogtoolbar;)
    3. create the user to connect to the database (mysql> GRANT ALL PRIVILEGES ON querylogtoolbar.* TO 'queryloguser'@'localhost' IDENTIFIED BY 'queryloguser' WITH GRANT OPTION; )
    4. exit the MySQL shell (mysql> quit; )
    5. Run the querylogtoolbar_schema.sql script ($ mysql -u queryloguser -p querylogtoolbar < querylogtoolbar_schema.sql )
  3. Copy and configure the conf/querylogserver.conf file
    1. Copy the querylogserver.conf file to somewhere that the Tomcat / Container user can read from
    2. Open with a text editor
      1. edit the querylogserver.db.connection to point at your MySQL database host, port and database name ( format: jdbc:mysql://$MYSQL_HOSTNAME:$MYSQL_PORT/$DATABASENAME?autoReconnect=true )
      2. edit the querylogserver.db.username and querylogserver.db.password with your values from (2)(c) above
      3. set the temp directory in querylogserver.server.tempdirectory to your temp directory ( /tmp )
  4. In your $CATALINA_HOME/webapps/ directory create a new directory for the QueryLogServlets (i.e. "QueryLogToolbar"). This will be your <your_webapp_directory> as noted in the items below
  5. Copy the contents of the web/* folder to the webapps folder you just created
  6. Open the file WEB-INF/web.xml in a text editor and modify the path to the querylogserver.conf file in the <param-value> for the configurationPath from the file in (3) above
  7. Restart your Tomcat (or other container) installation
  8. Test to see if your installation is working by opening a web browser and going to: http://<your_hostname>:<your_port>/<your_webapp_directory>/GetConfiguration
  9. The address that the clients (via the settings dialog) should set is: http://<your_hostname>:<your_port>/<your_webapp_directory>/

[Back to Top]

Installation using the Stand-Alone Server

Installation using the stand-alone server is somewhat similar to the installation instructions for using an existing container (in fact, the stand-alone server application actually creates its own Apache Tomcat container locally). If you do not have an existing container, the stand-alone server application will allow you to run a container and the Query Log Servlets manually.

  1. Download and untar / ungzip the Stand-alone Query Log Server package from http://www.lemurproject.org/querylogtoolbar/LemurQueryLogServer.tar.gz
  2. Setup your MySQL Server Database
    1. log on to your MySQL server as the root user ($ mysql -u root -p)
    2. create a database named "querylogtoolbar" (mysql> create database querylogtoolbar;)
    3. create the user to connect to the database (mysql> GRANT ALL PRIVILEGES ON querylogtoolbar.* TO 'queryloguser'@'localhost' IDENTIFIED BY 'queryloguser' WITH GRANT OPTION; )
    4. exit the MySQL shell (mysql> quit; )
    5. Run the querylogtoolbar_schema.sql script ($ mysql -u queryloguser -p querylogtoolbar < querylogtoolbar_schema.sql )
  3. Copy and configure the conf/querylogserver.conf file
    1. Copy the querylogserver.conf file to somewhere that the Tomcat / Container user can read from
    2. Open with a text editor
      1. edit the querylogserver.db.connection to point at your MySQL database host, port and database name ( format: jdbc:mysql://$MYSQL_HOSTNAME:$MYSQL_PORT/$DATABASENAME?autoReconnect=true )
      2. edit the querylogserver.db.username and querylogserver.db.password with your values from (2)(c) above
      3. set the temp directory in querylogserver.server.tempdirectory to your temp directory ( /tmp )
  4. Open the file webapps/QueryLogToolbar/WEB-INF/web.xml in a text editor and modify the path to the querylogserver.conf file in the <param-value> for the configurationPath from the file in (3) above
  5. Run the file “run_server.sh” (or run_server.bat if on windows) from a command line. The server should display some information regarding it's startup to the console. Note that the default port is 4592, but this can be changed by adding the port to the run_server command (i.e. “run_server.sh 8099” will use port 8099 instead). If all went well, the server should be started and you can verify that it is started by browsing to http://<your_hostname>:<your_port>/QueryLogToolbar/GetConfiguration
  6. The address that the clients (via the settings dialog) should set is: http://<your_hostname>:<your_port>/QueryLogToolbar/

[Back to Top]

The Query Log Server Database

The incoming data to the Query Log Server eventually gets stored in the MySQL server database as configured above. Pictured below is the database schema (as set up in the querylogtoolbar_schema.sql script). Click on the image to see the full view.

Query Log Server Data Dictionary

Table: tblClients
Holds the client information for log uploads
FieldTypeDescription
ClientIDAutonumber(bigint)The unique client ID
ClientIPVarchar(64)The IP address associated with this client
ClientSessionIDVarchar(128)The client unique session ID for this IP (note: not shown in the above diagram)
ClientUserAgentVarchar(255)The last known user agent (web browser version information) for this client
 
Table: tblLogs
The actual individual log uploads by clients
FieldTypeDescription
LogIDAutonumber(bigint)The unique ID of this query log
ClientIDbigint (FK)Foreign key of the client ID of this log (tblClients)
LogUploadTimestampdatetimeThe date/time this log was uploaded
 
Table: tblLogAction
The raw set of actions performed in a log
FieldTypeDescription
ActionIDAutonumber(bigint)The unique action ID
LogIDBigint (FK)Foreign key to the Log ID for this action (tblLogs)
ActionTypeIDBigint (FK)Foreign key to the action type ID for this action (tblLogActionType)
LogPageIDBigint (FK)(can be null) Foreign key to log page (tblLogPage) to associate URL of the page with this action
ActionTimestampdatetimeThe date/time of this action
 
Table: tblLogActionType
Defines the available log action types collected
FieldTypeDescription
ActionTypeIDbigintThe unique action type ID
ActionTypeVarchar(255)The action type (as defined by list of available actions in the upload log widget)
 
Table: tblLogPages
Stores information about individual web pages visited in a log
FieldTypeDescription
LogPageIDAutonumber(bigint)The unique log page ID
LogIDbigint (FK)The foreign key to the LogID (tblLogs)
PageLoadTimestampdatetimeThe date/time the page was loaded in the browser
TimeOnPageIntTime (in seconds) spent on the page before moving to a different page, tab, or closing the browser
NumScrollEvents Int The number of scroll events (with mouse or scrollbars) during viewing of the page
 
Table: tblPageCopyEvents
Stores information about items copied (Ctrl+C) from a web page
FieldTypeDescription
CopyEventIDAutonumber(bigint)The unique copy event ID
ActionIDbigint (FK)The action ID this is associated with (tblLogAction)
CopyTextTextThe actual text copied
 
Table: tblActionPageClick
Stores information about mouse click events on pages
FieldTypeDescription
ActionPageClickIDAutonumber(bigint)The unique click event ID
ActionIDbigint (FKThe action ID this is associated with (tblLogAction)
WhichButtonInt0=left; 1=right; 2=middle
LinkTextTextThe link text clicked on (if any)
 
Table: tblSearchPages
Stores search page information (Google, MSN, etc.)
FieldTypeDescription
SearchPageIDAutonumber(bigint)Unique search page ID
LogIBigint (FK)Foreign key to the log this search page is associated with (tblLogs)
ActionIDbigint (FK)The action ID this is associated with this search event (tblLogAction)
SearchTimestampdatetimeDate/time of the search
SearchURLVarchar(2048)The actual URL that performed the search
SearchQueryVarchar(255)The query of the search
NumResultsIntTotal number of results as returned by the search engine (could be actual, could be estimated…)
 
Table: tblSearchResults
Stores the results from a search page
FieldTypeDescription
SearchResultIDAutonumber(bigint)The unique search result ID
SearchPageIDBigint (FK)Foreign key to the search page ID (tblSearchPages)
ResultRankIntegerThe rank of the page (in the search results document)
ResultLinkVarchar(2048)The URL of the result item
ResultTitleVarchar(255)The title of the result item
ResultSnippetTextThe snippet of the result item (if available)
 
Table: tblLogPageSearches
Correlates individual log pages with a search page item
FieldTypeDescription
LogPageIDBigint (FK)Foreign key to the log page this search page is associated with (tblLogPages)
SearchPageIDbigint (FK)Foreign key to the search page this log page item is associated with (tblSearchPages)

[Back to Top]