Tomcat WebVoyage Enhancer

Tags: webvoyage , intermediate , export , xslt , ims , vle , elympics , openurl , sfx , refworks , endnote , ris
Last Updated: Mar 08, 2011 14:37

---

• Description

  WebVoyáge Enhancer is a perl daemon that sits between Apache and Tomcat. It intercepts requests and responses and allows customisation of responses before they reach Apache and user's browser. WebVoyáge Enhancer has a modular design, and most of the built-in functionality is to provide a framework for plugins.

• Features:
  • String translations
  • Extensibility
  • Plugins (see below for further information)
    • Enhanced OpenURLs
    • Short holdings summary
    • Open links to external sites in a new window
    • Reference Management (extended print/export options)
    • Improved patron information screen
    • Fixes to navigation of component parts (related records)
    • Reordering of the "search within" drop-down options in Advanced Search
    • Enhancements to patron request screens (e.g. hiding the barcode field)
    • Links from 856 fields in search results lists
    • Show links to host records for component parts in results list
  • Plugins can be enabled or disabled as needed so that unnecessary functionality is not loaded or executed

The Enhancer is designed to be light on resource usage. In normal scenarios memory and CPU consumption should not be an issue. Some plugins have features that may have performance implications. E.g. if plugin_elinks may need to read all bib and mfhd records (depending on configuration options) of a search result page. However, at least on our systems even this has not affected performance noticeably.

---

• Author: Ere Maijala
• Additional author(s):
• Institution: None
• Year: 2010

  ---

• License: MPL 1.1 / GPL 2.0
• Short description: Use, modification and distribution of the code are permitted provided the copyright notice, list of conditions and disclaimer appear in all related material.
• Link to terms: MPL 1.1, GPL 2.0

  ---

• Skill required for using this code: intermediate

  ---

State

Stable (in production)

Programming language

Perl

Software requirements

Tomcat WebVoyáge. Tested in Voyager 7.2.1 (any feedback on problems in other versions welcome)

Screen captures

Download

Version 1.8

Working example

https://tyrni.amkit.fi/vwebv/searchBasic?sk=en_US
https://masto.amkit.fi/vwebv/searchBasic?sk=fi_EN

Using the following Ex Libris open interfaces

-

Changes

Version 1.8 - 3 March 2011

• Fixed opening references in new window
• Fix getting "subfield" from a fixed field
• plugin_component_parts: Improved handling of missing identifiers and database handles
• plugin_patron_data: Improved error checking and handling
• plugin_patron_data: Implemented support for external authentication
• plugin_patron_data: Fixed normalization of login id's (e.g. SSN)
• plugin_requests: Modify only HOLD requests to avoid messing with CALLSLIPs

Version 1.73 - 2 December 2010

• plugin_openurl: Improved OpenURL creation for title fields and page numbers
• plugin_component_parts: Improved normalization of linking fields so that trailing periods don't break linking

Version 1.72 - 6 October 2010 (The Bug Fix Day)

• Fixed several SQL statement leaks

Version 1.71 - 6 October 2010

• plugin_openurl: fixed wrong url_encode call

Version 1.7 - 6 October 2010

• plugin_component_parts and plugin_elinks: process also the last record in results lists displaying 25 records
• Improved performance by loading Oracle libraries upon startup (whether it makes any difference, I don't know)
• Reloading configuration is now done via exiting child processes, just in case loading modules in child processes has an effect on memory usage (child processes not being able to share static data)

Version 1.6 - 17 September 2010

• plugin_component_parts: added links to host records in results lists
• plugin_holdings: added sorting of locations to
• plugin_openurl: added customization options to OpenURL settings (e.g. open in a new dialog)
• Fixed line endings (Windows -> Unix) in wv_enhancer.sh

Version 1.5 - 15 July 2010

• Added plugin_elinks that allows displaying of links from 856 fields of bib and mfhd records in the search results lists
• plugin_patron_data: fixed compilation

Version 1.4 - 7 July 2010

• Added plugin_requests that allows enhancements to the patron request screens, e.g. hiding barcode entry
• plugin_patron_data: display the active barcode instead of any barcode

Version 1.3 - 2 July 2010

• Added plugin_search_order that allows reordering of the "search within" drop-down options in the Advanced Search tab

Version 1.21 - 28 June 2010

• Oracle environment variables are now set from voyager.env if they're not already set. This makes it easier to start WebVoyáge Enhancer e.g. from Solaris SMF.

Version 1.20 - 16 June 2010

• Changed renewal count display in plugin_patron_data to work properly with the markup in Voyager 7.2.1
• Fixed plugin_holdings_summary to show the summary also after a patron request (e.g. a hold request) has been made

Version 1.18 - 17 May 2010

• Fixed a typo from wv_enhancer.sh restart function (test -> then)
• Fixed the version number wv_enhancer reports on startup

Version 1.17 - 12 May 2010

• Added plugin_component_parts that fixes the navigation of component parts (WebVoyáge bug, hopefully will be resolved in a future version)
• Added send_request method to the utils class for sending requests to Tomcat
• Fixed plugin_openurl so that it works in Staff View too

Version 1.16 - 26 Apr 2010

• Added a safeguard counter to child reaper
• Fixed the URL check in plugin_references to be language-independent

Version 1.14 - 1 Apr 2010

• Fixed child process accounting so that "maximum children" setting works properly
• Fixed startup script to also work with an empty USER

Version 1.13 - 29 Mar 2010

• Fixed OpenURL and holdings summary plugins to work properly when the search results in a single record

Version 1.12 - 9 Mar 2010

• Improved handling of child processes in busy environments

Version 1.1 - 23 Feb 2010

• General: Added dynamic reloading of configuration and plugins when configuration file has been updated. Invalid configuration or plugins with errors are rejected without disrupting the service
• General: Added keepalive timeout that will close a connection after a specified timeout. This might help avoid too many Tomcat processes
• plugin_patron_data: Added possibility to display renewal count and limit in patron's charged items
• plugin_patron_data: Now exposes get_patron_id function that can be used to retrieve the ID of a logged-in patron

Version 1.0 - 8 Feb 2010

• Initial release

Installation instructions

Extract the package somewhere, e.g. /m1/voyager/xxxdb/local/wv_enhancer. This path is used throughout these instructions, change as necessary. Note that by default you will need root permissions to change Apache configuration and restart it.

1. Make wv_enhancer.sh and wv_enhancer.pl executable (chmod +x ...)
2. Edit wv_enhancer.sh and change paths in CONFIG, DIR, DAEMON, PIDFILE and USER accordingly.
3. Edit wv_enhancer.config. The most important settings are in general and tomcat stanzas.
   1. Settings are described in the table below.
   2. Tomcat address and port can be found in /m1/shared/apache2/conf/jk/workers.properties
4. Add wv_enhancer.sh to the system startup
   1. This can be accomplished in many ways depending on the OS. See possible ways in the beginning of wv_enhancer.sh
   2. wv_enhancer does not need to be run by root. Set USER to e.g. voyager in wv_enhancer.sh to run as a different user.
5. Start wv_enhancer and verify that the log shows it has started up.
6. Test that wv_enhancer can connect to Tomcat by using telnet on the server:

   If telnet reports that it is connected, everything is good so far. Exit telnet by entering the escape combination it says and entering command quit. If instead it spits out some garbage and HTML regarding the service being unavailable, it means the Tomcat address or port is wrong, or Tomcat is not running.

7. Redirect Apache to use WebVoyáge Enhancer instead of going directly to Tomcat by modifying /m1/shared/apache2/conf/jk/workers.properties. You will probably need root permissions to do this. Comment out the original setting for worker.ajp13_lb_xxxdb_vwebv.port and add the new definition. Example:
8. Restart Apache for the change to take effect (e.g. run as root the command '/m1/shared/apache2/bin/apachectl restart')
9. Test that everything works. If you encounter any trouble, consult wv_enhancer.log and /m1/shared/apache2/logs/mod_jk.log

   If you encounter trouble in WebVoyáge after implementing WebVoyáge Enhancer, try switching Apache back to using Tomcat directly to see if WebVoyáge Enhancer is the culprit.

Upgrading to a New Version

Upgrading to a new version is usually simple. You can just replace all but the config file and the startup script. Without any further changes the functionality should not be affected (apart from bug fixes) unless noted in the release notes. You can then modify the configuration file to enable new or additional features. See the documentation below and the distributed config file for examples of new configuration directives.

New Configuration Directives in Version 1.7

No new directives were introduced in this version.

New Configuration Directives in Version 1.6

Log Files

If debugging is enabled, wv_enhancer.log may grow quickly. Debug mode is not recommended for production systems except for possible troubleshooting. If debug=2, dump.dat will grow even faster.

If the daemon crashes, any output can be found in wv_enhancer.log.err. It could be useful to determine the reason.

As usual, it is a good idea to monitor the growth of the log file and/or implement a suitable log rotation scheme. The daemon does not keep the file open, so it can be archived or removed at any time. Log rotation can be done with e.g. logadm, logrotate or a simple crontab entry. Here is an example crontab entry that rotates the log every Sunday at midnight and keeps the last two log files:

Diagram of the Data Flow

Settings

WebVoyáge Enhancer reloads the configuration and plugins whenever the timestamp of the configuration file changes. You can use touch command to update the timestamp after modifying a plugin, avoiding the need to restart the daemon. Note that it can take up to 10 seconds for the reload to happen, and it won't affect the listening address and port or database connection settings. Invalid configuration or plugins with errors are rejected without disrupting the service. See the log file for possible errors if the changes don't take effect. Note also that changing a module does not trigger a reload, only the timestamp of the configuration file is checked.

Translation Files

Translation files are simple text files that can be used to change any content in the HTML responses. Translation files are read line by line. There are two different types of translation lines:

1. simple textual translations:

   original string is replaced with new string.

2. Regular expressions:

   regexp is applied to the HTML.

Translations are applied only if content type is 'text/html;charset=UTF-8'.
Be careful not to translate strings commonly found in bibliographic records or other content. This can be avoided e.g. by adding tags or parts of them to the translation:

Plugins and Configuration

Plugins are called in the index number order. If a plugin is used for both preprocessing and postprocessing, it has to be listed in both. Index numbers need not be same.

plugin_test

Provided as a very simple example or skeleton that can be used as a template for new plugins. By default it returns a simple custom HTML page if &test=1 is added to the URL, and rewrites page title "WebVoyáge Titles" to "WebVoyáge Results List".

plugin_links_in_new_window

Makes all links (<a href="...">) to external sites open in a new window. Does not require any configuration.

plugin_holdings_summary

Displays a condensed summary of holdings that replaces the holdings jump links.

plugin_patron_data

Displays extended patron information. Patron barcode is stored on login and used to fetch additional patron information for the patron display. This plugin is not yet complete. It can display user barcode and expiration date, but additional fields can be easily added.

Comment out any of the texts using a hash mark (#) to hide the field.

plugin_patron_data exposes get_patron_id function that can be used to retrieve the ID of a logged-in patron. See the module itself for an example of the usage. Note that there is a small possibility that a valid ID is returned even though the user is not properly logged in, so use this only on pages a logged-in user can access.

plugin_references

Replaces the built-in Export function with an enhanced print/export function. The following features are provided:

• XSL files to transform MARCXML to different formats
• Redirection to e.g. RefWorks for direct import
• Proper mime types, disposition and file name so that the browser knows to do The Right Thing (display on screen, offer to save the result, default to a sensible file name, automatically save to Zotero etc.)
• Optionally open the results in a new window
• Provide links back to Voyager in the records
• Formats provided in the distribution:
  • HTML Reference List
  • HTML Reference List with Subjects
  • Refworks Tagged Format and a direct export to RefWorks
  • RIS (EndNote, ProCite, Reference Manager, etc.)
  • IMS Packages (Export records with added notes to virtual learning environments)
  • Raw MARCXML for advanced users

plugin_references calls xsltproc to do the actual transformation.

plugin_openurl

Displays improved OpenURL links to e.g. SFX. This plugin requires that WebVoyáge is configured to display OpenURL's. It will replace the original OpenURL with the improved one and allows to hide the button if necessary metadata is not present.
All the functionality describing how the OpenURL is created is in the code. Due to its nature it would be difficult to provide settings that would achieve the same flexibility. The code is quite simple to modify. By default the plugin will try to build a nice OpenURL and display it only if ISSN is found. The OpenURL will contain the following fields: issn, date, volume, issue, spage, epage, isbn, doi (from 036), author and title.

plugin_component_parts

Fixes the navigation in a list of component parts. There's a bug in WebVoyáge screwing up the result set when a record is displayed and breaking Next/Previous record navigation. The bug will hopefully be resolved in a future version, but meanwhile this plugin refreshes the record list in background and adds bibId parameters to Next/Previous record links to make them work properly.

Also adds a link to the host item to each component part in the results list. Most of the settings govern how the link is created.

plugin_search_order

Allows configuration of the order in which the options are displayed in the "search within" drop-down in Advanced Search.

plugin_requests

Depends on: plugin_patron_data

Provides enhancements to the patron request screens. Some of the features could be achieved with modifications to the xsl files, but they are provided here as a convenience.

plugin_elinks

Provides links from 856 fields of bib and/or mfhd records in the search results lists. No changes to template files are needed, but the added links can be styled as necessary using css. The containing div has class lineElink and the span that contains the link text has class elinkText.

Writing New Plugins

New plugins can be easily added. plugin_test can be used as a template. Start by copying it to a new name and changing the package line to reflect the new name. WebVoyáge Enhancer calls preprocess function before the request is sent to Tomcat, and/or postprocess function before the response is sent to Apache. If preprocess return value is non-zero, the request will not be passed to Tomcat or further plugins, and the connection will be closed without further processing. Same applies for postprocess: if postprocess return value is non-zero, the original response will not be passed to Apache or further plugins.

Parameters passed to preprocess:

Currently any modifications to request_attrs_ref are not reflected in the request when it's passed to Tomcat. Modifications to headers_ref and body_ref are passed to Tomcat, but this has not yet been tested, and crafting them properly is somewhat complicated (e.g. Content-Length in headers must match the length of body data in bytes).

Parameters passed to postprocess:

Currently any modifications to response_attrs_ref are not reflected in the response when it's passed to Apache. Modifications to response_headers_ref and content_ref are passed to Apache, but as response_headers_ref is the raw message, crafting it properly is somewhat tedious.

WvEnhancerUtils class provides functions that help building plugins. The functions are documented in wv_enhancer_utils.pm. See documentation of parse_request and parse_response for information regarding typical attributes contained in request_attrs_ref and response_attrs_ref. You can also see the attributes in the log file when debug setting is enabled.

TO DO list

• More plugins

---

Comments

Please comment if there's anything you'd like to see added, fixed or so