User Trials Follow Up Satisfaction Survey

The user trials consisted of 5 main stages.

  • An initial meeting to demonstrate the system.
  • An initial questionnaire to gather expectations
  • Training: Users spent between 4-6 weeks training the system
  • A follow up meeting to indicate how successfully their interests had been matched; and;
  • A follow up questionnaire to gauge the users’ satisfaction.

The results of the follow up survey are discussed in more detail below.
Question 1.

Were enough “Not Interesting” articles filtered out of the “Interesting” feed to make reading this feed manageable?

Though the percentages of interesting items delivered to each user were in general lower than the users had indicated would be acceptable in the initial questionnaire. The users seemed to be happy with this result and in most cases the percentage of “not interesting” in the “interesting” feed was greatly reduced.

13 users answered yes, 4 answered no and 1 was not sure.

Question 2.

If the “Not Interesting” feed wrongly contained “interesting” articles, was the percentage small enough to tolerate?

The majority of the users were able to tolerate some “interesting” articles being filtered out into the “not interesting” feed.

15 users answered yes; 3 answered no.

Question 3.

Would you consider using a similar tool in the future?

The majority of users indicated that they would consider using a similar tool in the future. This gives us a certain confidence that the concept of applying Bayesian filtering to journal articles is worth investigating further.

15 users answered yes; 2 answered maybe; 1 answered no.

Question 3 cont…

If yes, which of the following would you consider?

[a] A stand alone tool?
[b] A tool integrated into an existing tool you use everyday i.e. in an email client/feed reader/iGoogle?
[c] Integrated into a library or research tool such as web of science?

Users were able to enter more than one choice.

There were 6 votes for [a]; 13 votes for [b]; 12 votes for [c]

Users were then asked which of the above would be their preferred option?

3 voted for [a]; 6 voted for [b]; 7 voted for [c]; 1 user thought daily/weekly email alerts would be a better option.

The strong preference for integration into other tools (options b and c) rather than use as a stand alone tool is interesting as it validates our supposition that an API would be useful, i.e. that it would be desirable to be able to integrate interact with sux0r into other tools.

Question 4.
If you would consider using a similar tool in the future, what do you think the advantages of doing so would be?

The main advantages offered by the users included time saving by filtering out unwanted articles, the ability to scan more journals and a single place to scan the latest articles form interesting journals. Only one user considered a similar tool not to have any advantages.

A selection of responses follow below:

If trained sufficiently the tool would save time in showing the searches from interesting results, with keywords on saved interests.

To flag up interesting articles without the user having to actively search for them i.e. it would help with horizon scanning.

Make e-journals more helpful when filtering interesting articles and not interesting ones.

1. One advantage would be a single place to find interesting reserach articles. 2. If the feed is trained well, then less time is spent on uninteresting articles. 3. If it is integrated into broader serach tools like iGoogle it would have wider reach.

As it highlights interesting/prospectively interesting journals that you may not be able to find easily using databases search such as science direct.

Quicker sorting of interesting and not interesting articles

Keeping up to date with new articles. But disadvantage is the guilt of seeing all the interesting things you should read but don’t have time to.

Saving time. However I am not sure I would be completely confident in the results I would get.

Screening for new articles would become more organised rather than my random search at the moment which only happens when I need to find information.

Tend to search on the basis of keywords; this appears to work better.

It does appear to throw up interesting articels that I might otherwise miss.

Time saving and effective worktime

Obviously it will save a lot of time

Simultaneous filtering of many journals

Make looking for papers more fun because much of the clutter is removed compared to reading journal indexes. And I find more interesting articles compared to googling or searching by keyword.

a) save time, reduce number of articles. b) We can create research group feed of interest

Even with uninteresting articles in the mix it still allowed me to find dozens of articles that would have passed me by otherwise. I felt it was worth the effort & still a lot less effort than reading all the tables of contents would have been. A key advantage for me was that it effectively allowed me to, in a similar length of time, scan the contents of a far greater number of journals than I would have studied by hand. A worthwhile tool if you can be bothered to train it.

Get an overview of recently published articles with at least some relevance to me, which at the moment I’m not getting.



Filed under trialling

2 responses to “User Trials Follow Up Satisfaction Survey

  1. Pingback: BayesFF: Final post « Bayesian Feed Filter

  2. Pingback: Conducting a User Trial « Bayesian Feed Filter