Tag Archives: JISC

Conducting a User Trial

One of the aims of the Bayesian Feed Filter Project was to test the ability of the recommender service to identify new journal papers of interest to researchers based on a knowledge of papers which they have recently read. The recommender service used was sux0r a blogging package, an RSS aggregator, a bookmark repository, and a photo publishing platform with a focus on Naive Bayesian categorization and probabilistic content .

As well as creating an API for sux0r, the project created a Bayesian Feed Filter theme which included simplifying the sux0r interface so that user saw only the RSS Aggregator with Bayesian Filtering. The Bayesian Feed Filter uses Bayes’ theorem to attempt to predict whether or not a new item in a feed is relevant to an individual’s research interests based on previous categorization of items by the user. This explicit categorization by the user is known as training; the system also allows for other text documents to be used as training material.

Twenty researchers from Engineering and Science based schools within Heriot-Watt University volunteered to participate in the trial to test the ability of the Bayesian Feed Filter to identify new journal papers of interest to them based on knowledge of papers which they have recently read. The volunteers were asked to provide a list of journals that they follow or would like to follow if they had the time. Each volunteer was set up with an account on Bayesian Feed Filter, which was preloaded with RSS Feeds of the journals they said they were interested in and contained two categories for training: Interesting and Not Interesting.

An API was developed during the project which included the feature Return RSS Items for a User, which was used to create personalised RSS feeds for each user. The feeds could be filtered by category (interesting or not interesting) and by threshold (likelihood to belong to a particular category).

Stage 1: Initial Questionnaire

The first stage of the trial involved a short questionnaire to gauge the researchers’ methods of current awareness and their expectations of a service filtering journal articles matching their interests. (Results of Initial Questionnaire).

Stage Two: Demonstration of the Bayesian Feed Filter
Volunteers were each given a demonstration of how to mark items as relevant to their interests or not relevant to their interests. These items typically include the title and abstract of the journal article. The users were also shown how to use the train document feature which would allow them to include text not in the RSS feeds such as the full text of articles they had written, cited or read. (How to use Bayesian Feed Filter)

Stage Three: Training the Bayesian Feed Filter
The volunteers had access to the Bayesian Feed Filter for 6 weeks and they were asked to train the system by categorizing items as either “interesting” or “not interesting” periodically and to supplement the interesting items with other documents relevant to their interests. (User Activity).

At the end of the six week training period access to the Bayesian Feed Filter was suspended and all articles in the system were removed. The system would continue to run for 4 weeks, automatically catagorizing new articles as being “interesting” or “not interesting” to the researchers based upon the training provided. Unfortunately, two of our volunteers were not able to continue with the trial, therefore the trial continued with 18 volunteers.

Stage Four: Returning the Filtered Feeds
The users were presented with the two feeds. One feed comprised articles rated by the feed filter with at least a 50% chance of being of interest to them and the other feed articles rated with at least a 50% chance of not being of interest to them. The feeds were presented to the user using Thunderbird (an email and RSS client). Users were then asked to mark each article from both feeds with a star if they found it to be of interest. Thus the feeds represent the Bayesian Feed Filters categorization of items into “interesting” and “not interesting” and the stars show the users opinion of whether the items are relevant to their research interests of not.

The number of false positives (items in the interesting feed not starred) and number of false negatives (items in the not interesting feed starred) could then be calculated for each user. A successful scenario would be for the interesting feed to contain a significantly higher proportion of interesting articles than an unfiltered feed with few items of interest wrongly filtered into the “not interesting” feed. The success of the filtering seems to be dependant on the training provided, with users who trained over 150 items seeming to get a reasonable measure of success. (Statistics from the User Trials).

Stage 5: Follow Up Questionnaire
The final stage of the of the trial was a follow up questionnaire, in order to gauge the user’s satisfaction with the filtering process and whether they would be interested in using a similar system in the future and what the advantages of doing so would be. (Results of the Follow Up Satisfaction Survey).


Comments Off on Conducting a User Trial

Filed under trialling

How to Install the Bayesian Feed Filter

The Bayesian Feed Filter (BayesFF) is an optional interface for the popular sux0r software package. To be able to use the BayesFF interface you only need to follow the normal process for installing sux0r and make a few edits in the sux0r configuration file.

The BayesFF interface will allow you to use the API and the web interface developed by the BayesFF project. In general, installing sux0r is a simple process that takes less than 30 minutes to complete, depending of the type of PHP configuration found in your web server. You may want to ask your IT support team to install sux0r for you, if you are not familiar with installing and configuring PHP packages that would require access to the web server configuration files. However, if you wish to install sux0r yourself, the following detailed installation guide would help you.

A. Prerequisites

* Configuring PHP to enable mb, gd, and PDO libraries:
– mb is non-default extension and you need to explicitly enable it with the configure option. See http://www.php.net/manual/en/mbstring.installation.php webpage for details
– gd represents the GD library that you will need to install (available at http://www.libgd.org/) and enable with the configure PHP command. See http://www.php.net/manual/en/image.installation.php webpage for details
– PDO driver is enabled by default as of PHP 5.1.0, but you may need to enable it to work with MySQL. Please consult the documentation at http://www.php.net/manual/en/pdo.installation.php and http://www.php.net/manual/en/ref.pdo-mysql.php web pages to find out more about PDO installation.

* MySQL 5.0.x, set to support UTF characters
(Further information on http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html)

* Apache 2.x webserver with mod_rewrite module enabled
(a simple but good tutotial on enabling mod_rewrite can be found at http://www.tutorio.com/tutorial/enable-mod-rewrite-on-apache)

B. Installation

To install sux0r code on your web server:
1. Login to your server and go to the directory where you want to install sux0r
2. Execute the following Unix command:
svn export https://sux0r.svn.sourceforge.net/svnroot/sux0r/branches/icbl/
3. Execute these two commands:
chmod 777 ./data
chmod 777 ./temporary

To create the MySQL database and tables for sux0r:
4. Create a database named “sux0r” on your MySQL server
5. Import ./supplemental/sql/db-mysql.sql into MySQL

C. Configuartion

1. From the shell, execute these commands:
mv ./sample-config.php ./config.php
mv ./sample-.htaccess ./.htaccess

2. Edit ./config.php and ./.htaccess appropriately (follow the instructions included inside these files.) The changes you need to make are pretty obvious.

Edit Database Connection: $CONFIG[‘DSN’]
Edit URL for your intallation of sux0r: $CONFIG[‘URL’]
Edit Title: $CONFIG[‘TITLE’]
If you want to use the BayesFF interface, you will need to change the default value of the $CONFIG[‘PARTITION’] configuration parameter found in config.php,
$CONFIG[‘PARTITION’] = ‘sux0r’;
$CONFIG[‘PARTITION’] = ‘bayesff’;

3. To check your installation, run the ./supplemental/dependencies.php script from your browser. Example:
http://yourwebsite/sux0r210/supplemental/dependencies.php (If there are no errors OK will be returnes with a link to your new installation.

4. If the previous step didn’t produce any error, point your web browser to http://yourwebsite/sux0r210/supplemental/root.php’ and follow the onscreen instructions to make yourself a sux0r root user.

5. Setup a CRON job to fetch RSS feeds every x minutes (we recommend you to start by running the CRON every 60 minutes). The PHP script that fetches the feeds is already provided by sux0r and it is available at http://yourwebsite/sux0r210/modules/feeds/cron.php
For example:
0 * * * * /bin/nice /usr/bin/wget -q -O /dev/null “http://yourwebsite/sux0r210/modules/feeds/cron.php” > /dev/null 2>&1

6. Delete the ./supplemental directory from the webserver.

Sux0r should now be successfully installed on your website.

1 Comment

Filed under dissemination, technical

BayesFF: Final post

Diagram of prototype: schematically we can show how the prototype supports the aggregation and of RSS feeds comprising table of contents information from selected journals and filters them (using pre-existing software called sux0r) into two feeds, one of which is has information about those papers that are predicted to be relevant to a user’s research interests. The project has added the ability to interact with sux0r through third-party software.

Our work has shown how effectively this works for a trial group of researchers; in most cases, after sufficient training of the system, the outgoing feeds were successfully filtered so that one contained a significantly higher concentration of interesting items than the raw feeds and the other did not contain a significant number of interesting items.

End User of Prototype:
We have an installation of sux0r which people are welcome to register on and which can be used to set up feeds for aggregation (you will not automatically be given sufficient privileged to approve feeds, so it is best to contact the project about this). The base URL for the API for this installation is http://icbl.macs.hw.ac.uk/sux0rAPI/icbl/ and the API calls which have been implements are documented in the following posts on this blog: Return RSS items for a user and ReturnVectors and ReturnCategories. Also available: a summary of other features for the API have been scoped. The latest update was 08 December 2009.

Here’s a screen cast of Lisa using the API

(NB the version at the end of the link is a whole lot clearer than the embedded YouTube version, especially if you click on the view in HD option).

The code for our work on the API is in a branch of the main sux0r repository on sourceForge.

Project Team
Phil Barker, philb@icbl.hw.ac.uk, Heriot-Watt University (project manager)
Santiago Chumbe, S.Chumbe@hw.ac.uk, Heriot-Watt University (developer)
Lisa J Rogers, l.j.rogers@hw.ac.uk, Heriot-Watt University (researcher)

Project Website: http://www.icbl.hw.ac.uk/bayesff/
PIMS entry: https://pims.jisc.ac.uk/projects/view/1360

Table of Content for Project Posts
Development work

User trialling

Community Engagement

Project Mangement


Filed under management

User Trials Follow Up Satisfaction Survey

The user trials consisted of 5 main stages.

  • An initial meeting to demonstrate the system.
  • An initial questionnaire to gather expectations
  • Training: Users spent between 4-6 weeks training the system
  • A follow up meeting to indicate how successfully their interests had been matched; and;
  • A follow up questionnaire to gauge the users’ satisfaction.

The results of the follow up survey are discussed in more detail below.
Question 1.

Were enough “Not Interesting” articles filtered out of the “Interesting” feed to make reading this feed manageable?

Though the percentages of interesting items delivered to each user were in general lower than the users had indicated would be acceptable in the initial questionnaire. The users seemed to be happy with this result and in most cases the percentage of “not interesting” in the “interesting” feed was greatly reduced.

13 users answered yes, 4 answered no and 1 was not sure.

Question 2.

If the “Not Interesting” feed wrongly contained “interesting” articles, was the percentage small enough to tolerate?

The majority of the users were able to tolerate some “interesting” articles being filtered out into the “not interesting” feed.

15 users answered yes; 3 answered no.

Question 3.

Would you consider using a similar tool in the future?

The majority of users indicated that they would consider using a similar tool in the future. This gives us a certain confidence that the concept of applying Bayesian filtering to journal articles is worth investigating further.

15 users answered yes; 2 answered maybe; 1 answered no.

Question 3 cont…

If yes, which of the following would you consider?

[a] A stand alone tool?
[b] A tool integrated into an existing tool you use everyday i.e. in an email client/feed reader/iGoogle?
[c] Integrated into a library or research tool such as web of science?

Users were able to enter more than one choice.

There were 6 votes for [a]; 13 votes for [b]; 12 votes for [c]

Users were then asked which of the above would be their preferred option?

3 voted for [a]; 6 voted for [b]; 7 voted for [c]; 1 user thought daily/weekly email alerts would be a better option.

The strong preference for integration into other tools (options b and c) rather than use as a stand alone tool is interesting as it validates our supposition that an API would be useful, i.e. that it would be desirable to be able to integrate interact with sux0r into other tools.

Question 4.
If you would consider using a similar tool in the future, what do you think the advantages of doing so would be?

The main advantages offered by the users included time saving by filtering out unwanted articles, the ability to scan more journals and a single place to scan the latest articles form interesting journals. Only one user considered a similar tool not to have any advantages.

A selection of responses follow below:

If trained sufficiently the tool would save time in showing the searches from interesting results, with keywords on saved interests.

To flag up interesting articles without the user having to actively search for them i.e. it would help with horizon scanning.

Make e-journals more helpful when filtering interesting articles and not interesting ones.

1. One advantage would be a single place to find interesting reserach articles. 2. If the feed is trained well, then less time is spent on uninteresting articles. 3. If it is integrated into broader serach tools like iGoogle it would have wider reach.

As it highlights interesting/prospectively interesting journals that you may not be able to find easily using databases search such as science direct.

Quicker sorting of interesting and not interesting articles

Keeping up to date with new articles. But disadvantage is the guilt of seeing all the interesting things you should read but don’t have time to.

Saving time. However I am not sure I would be completely confident in the results I would get.

Screening for new articles would become more organised rather than my random search at the moment which only happens when I need to find information.

Tend to search on the basis of keywords; this appears to work better.

It does appear to throw up interesting articels that I might otherwise miss.

Time saving and effective worktime

Obviously it will save a lot of time

Simultaneous filtering of many journals

Make looking for papers more fun because much of the clutter is removed compared to reading journal indexes. And I find more interesting articles compared to googling or searching by keyword.

a) save time, reduce number of articles. b) We can create research group feed of interest

Even with uninteresting articles in the mix it still allowed me to find dozens of articles that would have passed me by otherwise. I felt it was worth the effort & still a lot less effort than reading all the tables of contents would have been. A key advantage for me was that it effectively allowed me to, in a similar length of time, scan the contents of a far greater number of journals than I would have studied by hand. A worthwhile tool if you can be bothered to train it.

Get an overview of recently published articles with at least some relevance to me, which at the moment I’m not getting.


Filed under trialling

SWOT Analysis

Here are the Strengths, Weaknesses, Opportunities & Threats of the project, as estimated by Lisa and Phil during an informal project meeting over coffee. Following standard SWOT procedures (I used info and templates from businessballs.com and CIPD for guidance), Strengths and Weaknesses are internal and Opportunities and Threats are external. We think the “internals” of the project comprise the project team (our skills and connections to others) and the idea itself and the approach to realising it; the “externals” are the users, the sux0r project, the JISC environment and others (e.g. commercial interests, our host institution and the wider HE system).

(The points are numbered for ease of referencing, not for ordering.)


  1. We think we’re starting with a good idea, at least in principle; an innovative solution to a recognized need.
  2. Using sux0r as a starting point has given us access to existing OS code and put us in contact with a knowledgeable developer.
  3. We have a settled team who have worked well together on a number of previous projects over the last 4-10 years.
  4. We have good existing links with experts in JISC, CETIS, the IE, UKOLN, JISC services and projects (and we’re not afraid to use them).
  5. We have previous experience in related projects dealling with Journal ToC and other RSS feeds (e.g. PerX, TicTocs, GoldDust . . .).
  6. We work in close proximity to our intended test user group (which should help with encouraging engagement for the trials).


  1. We have lots of new stuff to learn: this is the most deliberately RESTful development we have undertaken; we’re using a project management technique that is new to us; this is first time we’ve worked on a branch of an existing OSS project; we need more robust user trials than we’ve previously managed.
  2. We have all that to learn in a short project time frame (six months, all the team are working part time on this project).
  3. Bayesian filtering is not a complete solution. Other techniques (e.g. popularity from usage data analysis; manual over-rides to specify that that everything from some authors is important, no matter what the topic) would help identify important items but are out of scope.
  4. Bayesian filtering might not work for our users with the type of data and sources we have (see threats), though as a good academic I think this is not so much a weakness as a potential research finding.


  1. Working with sux0r provides an opportunity to work with an existing user base and experienced developer.
  2. Other projects in the information environment provide additional/alternative usage scenarios (but see threat 2).
  3. It may be possible to embed the output of this project into other services, e.g. TicTocs, TechXtra, JISC IE or commercial services.
  4. There is good support for RESTful development approaches.
  5. There is a good developer community in the JISCRI projects.


  1. Lack of user engagement. We don’t know that users will be as enthusiastic about this approach as we are, they might just resent disruptive technologies.
  2. Expectation mismatch (see opportunity 2 & weakness 3), possibly leading to scope creep.
  3. There might be some unexpected conflict with the sux0r project (over approach or priorities).
  4. There might be a lack of table of content information from the right journals in RSS form, or what there is might be polluted (garbage in garbage out).
  5. Competing demands on time from other projects/tasks that the team are working on (see weakness 2).

I guess some mitigation of the negative factors is called for, that will come later, but a quick reflection is that engagement with the project externals is going to be important.

The programme guidance documentation suggests that the SWOT analysis is best to be undertaken in small steps, throughout the duration of the project; and the other guidance I read suggested that it should draw on as many view points as possible. So, hopefully this isn’t the last on SWOT, and please comment on anything that has been overlooked.

1 Comment

Filed under management