Lisa and Phil had a coffee meeting this morning to plan work package 2 of the Bayesian Feed Filter project. The aim of this work package is to
“Test the ability of the recommender service to identify new journal papers of interest to researchers based on a knowledge of papers which they have recently read”.
We shall recruit around 20 volunteers, who have an interest in research, based locally here at Heriot-Watt. The trialling process will involve the researchers selecting a list of journals they are interested in, and marking some articles as either “Interesting” or “Not Interesting” using sux0r (a tool for Bayesian Filtering of RSS feeds). The users will be asked to continue marking items over a period of a month. If the users cannot find many articles that represent their interests they will be asked to top up the interesting articles by submitting the abstracts of other articles they have written, cited or are of particular interest to them. After the initial training period of one month, users will not access their accounts for a further month, allowing a collections of articles to build up and the filter to try to determine if these articles are of interest to that user or not. A follow up session will then take place allowing the users to confirm how accurate the filter has been.
The deadline for completion of this project is 30th November 2009, giving a tight schedule for the trialling process.
By Tuesday 11th August we need to formalise the evaluation process and prepare a questionnaire for users (i.e. user’s expectations, current practice in monitoring journal tables of contents etc.)
From week commencing the 10th of August volunteers will be recruited.
In the following two weeks accounts for each user will need to be created and journals added for each user.
By the 28th of August an initial meeting will be held with users, (either in small groups or individually) to explain what we want them to do and to make sure they have filled in the questionnaire. Users will be asked to train the system between the first meeting and the cut off date.
On the 25th of September will be the date users should stop training articles.
The final week in October is the proposed date for holding follow up meetings with the users. They will be asked in this meeting to indicate how accurate the system was at determining the interestingness of articles. The number of false positives and false negatives shall be recorded. A follow up questionnaire asking how the system matched their expectations shall also be given at this stage.
This schedule gives the project the whole of November to analyse and write up the findings of the trials.