Library Perceptions Gossip 2010 January 26
I finally had a chance to look over Marshall Breeding’s Perceptions 2009: An International Survey of Library Automation. As Dan Scott mentions on his blog post In which I perceive that gossip is not science (and Breeding acknowledges in the comments), it is not scientific. I do think it is a little more than gossip… at least with vendors where the n → 50.
That said, one thing I do think is that comparing the numbers from one vendor or ILS to another doesn’t mean very much, For example, Apollo was rated very high, but the needs and expectations of someone running Apollo is much different from someone running Voyager or Aleph. In fact, they don’t even serve the same audiences as Apollo is marketed to small and medium public libraries while Voyager and Aleph is marketed more towards Academic, Special, and national libraries.
What I do find interesting is trends. While the people filling out the survey are not the same every year, it is interesting if one ILS or company’s score goes up or down. Being that I’m on the ELUNA Steering Committee and have worked with Aleph and Voyager more than any other ILS, I focused on the scores they received over the last three years. The three questions I looked at were:
- How satisfied is the library with your current Integrated Library System (ILS)?
- How satisfied is the library overall with the company from which you purchased your current ILS?
- How satisfied is this library with this company’s customer support services?
Breeding also asked “How likely is it that this library will purchase its next ILS from this company?” and about likelihood of migrating to an Open Source ILS, but I think there are many other factors in those questions that don’t reflect on the quality of the software or company providing support so I’m ignoring them. For example, Ex Libris is coming out with the URM. If I’m a smaller academic or special library, I may decide I do not want or need URM so I may go elsewhere if I were choosing an ILS no matter how much I like the company. Likewise, if Ex Libris is the only one in the URM market place when I’m looking at a new system, I may go with them even if I am happy with whatever company provides the ILS I’m currently using.
With three years of responses to look at, what is the trend for Ex Libris’ two ILSs? I’m happy to report, that over the last three years in every case (excepting one) the numbers improved. They might not have in regards to a statistical difference, but still that is a positive trend. The one case where it did go down was that Voyager users felt slightly less happy about Voyager in 2009 vs. 2008 (5.91 vs. 6.01). It was still higher than 2007 (5.53)). That is a relatively minor fluctuation and is within standard deviation. So overall, the trends for comparing Ex Libris with it’s self is positive.
On the Aleph side, Ex Libris had some pretty high jumps from 2008 to 2009. In fact from 2007 to 2009, the perception of customer support for Aleph went up over a point (4.87 to 5.90).
So while the scores for Aleph were generally in the middle tier, and by just looking at numbers, Breeding is correct that “Voyager did not fare quite as well on the survey” a look at the trends show Ex Libris in a more positive light.
That said, Dan Scott does have a good point that this is not science and gossip. But usually there is some truth in gossip.
I dunno. If you repeat a study with a flawed methodology, and you note differences between the first time you ran that study with the same flawed methodology, I don’t think you should try to draw any meaningful conclusions.
I see where your coming from, but I guess I don’t think the survey method used here is as flawed as you do when looking at a product compared to itself over time. While my mathematics background leads to believe there are obvious statistical issues, my LIS background knows that surveys with a lot more flaws are used in the field and presented at conferenced and in some cases published in journals.
Fair enough, Ed. I suppose a positive approach to the situation would be to celebrate the examples of good research we come across in our field. Let’s find (or create) some!
I’m all for good research, but I am a bit at a loss how to do it in the context of which ILS is “better.” There are too many variables. Sure, maybe you can do an in-depth analysis on one part of the ILS, such as the cataloging functionality or discovery layer, but not sure about the ILS as a whole. That said, please share if you ever come across anything as I would be extremely interested.
For one thing, i was talking about any good research in the field of libraries and information studies… not just limited to evaluations of library systems.
That said, it would be trivial to design a better research methodology than a self-selected survey with one responder per library, with all responses mixed in between all library types.
I responded to Marshall over at http://coffeecode.net/archives/209-In-which-I-perceive-that-gossip-is-not-science.html#c2459 with some thoughts on what such a methodology would look like.