One of the aims of the Bayesian Feed Filter Project was to test the ability of the recommender service to identify new journal papers of interest to researchers based on a knowledge of papers which they have recently read. The recommender service used was sux0r a blogging package, an RSS aggregator, a bookmark repository, and a photo publishing platform with a focus on Naive Bayesian categorization and probabilistic content .
As well as creating an API for sux0r, the project created a Bayesian Feed Filter theme which included simplifying the sux0r interface so that user saw only the RSS Aggregator with Bayesian Filtering. The Bayesian Feed Filter uses Bayes’ theorem to attempt to predict whether or not a new item in a feed is relevant to an individual’s research interests based on previous categorization of items by the user. This explicit categorization by the user is known as training; the system also allows for other text documents to be used as training material.
Twenty researchers from Engineering and Science based schools within Heriot-Watt University volunteered to participate in the trial to test the ability of the Bayesian Feed Filter to identify new journal papers of interest to them based on knowledge of papers which they have recently read. The volunteers were asked to provide a list of journals that they follow or would like to follow if they had the time. Each volunteer was set up with an account on Bayesian Feed Filter, which was preloaded with RSS Feeds of the journals they said they were interested in and contained two categories for training: Interesting and Not Interesting.
An API was developed during the project which included the feature Return RSS Items for a User, which was used to create personalised RSS feeds for each user. The feeds could be filtered by category (interesting or not interesting) and by threshold (likelihood to belong to a particular category).
Stage 1: Initial Questionnaire
The first stage of the trial involved a short questionnaire to gauge the researchers’ methods of current awareness and their expectations of a service filtering journal articles matching their interests. (Results of Initial Questionnaire).
Stage Two: Demonstration of the Bayesian Feed Filter
Volunteers were each given a demonstration of how to mark items as relevant to their interests or not relevant to their interests. These items typically include the title and abstract of the journal article. The users were also shown how to use the train document feature which would allow them to include text not in the RSS feeds such as the full text of articles they had written, cited or read. (How to use Bayesian Feed Filter)
Stage Three: Training the Bayesian Feed Filter
The volunteers had access to the Bayesian Feed Filter for 6 weeks and they were asked to train the system by categorizing items as either “interesting” or “not interesting” periodically and to supplement the interesting items with other documents relevant to their interests. (User Activity).
At the end of the six week training period access to the Bayesian Feed Filter was suspended and all articles in the system were removed. The system would continue to run for 4 weeks, automatically catagorizing new articles as being “interesting” or “not interesting” to the researchers based upon the training provided. Unfortunately, two of our volunteers were not able to continue with the trial, therefore the trial continued with 18 volunteers.
Stage Four: Returning the Filtered Feeds
The users were presented with the two feeds. One feed comprised articles rated by the feed filter with at least a 50% chance of being of interest to them and the other feed articles rated with at least a 50% chance of not being of interest to them. The feeds were presented to the user using Thunderbird (an email and RSS client). Users were then asked to mark each article from both feeds with a star if they found it to be of interest. Thus the feeds represent the Bayesian Feed Filters categorization of items into “interesting” and “not interesting” and the stars show the users opinion of whether the items are relevant to their research interests of not.
The number of false positives (items in the interesting feed not starred) and number of false negatives (items in the not interesting feed starred) could then be calculated for each user. A successful scenario would be for the interesting feed to contain a significantly higher proportion of interesting articles than an unfiltered feed with few items of interest wrongly filtered into the “not interesting” feed. The success of the filtering seems to be dependant on the training provided, with users who trained over 150 items seeming to get a reasonable measure of success. (Statistics from the User Trials).
Stage 5: Follow Up Questionnaire
The final stage of the of the trial was a follow up questionnaire, in order to gauge the user’s satisfaction with the filtering process and whether they would be interested in using a similar system in the future and what the advantages of doing so would be. (Results of the Follow Up Satisfaction Survey).
Bayesian Feed Filter 
You must be logged in to post a comment.