This is a guest post contributed by Ian Mulvany.
Use of Altmetrics in US-based academic libraries
Stacy Konkiel – Altmetric, Sarah Sutton – Emporia State University & Michael Levine-Clark, University of Denver
Stacy presented the results of a wide ranging survey of librarians. They delved into questions such as how aware are librarians of ALMs in comparison to other metrics? How are they using ALMs? Are they being used to enhance library services? The core question they were interested in is are people actually using ALMs in the library at all? It’s easy to create a story around how you might use ALMs in the library, e.g. to understand usage, or for collection development, but are they being used?
They asked every librarian working in a specific class of research library in the US. They sent a survey to 1300 people, and got response from 400 (which is an astonishingly high response rate for a survey).
The scholarly communication and support librarians had a higher level of awareness of these tools, but the base line was that about 30% of libraries responded with familiarity with ALMs, where baseline familiarly on things like citations was at about 80 – 90 %. (It was surprising to me that citation awareness was not close to 100%)
Apart from awareness they also asked how often libraries were likely to discuss these kinds of things when providing reference services. ALMs were brought up rarely. (It would be great to do a longitudinal study on this to determine the rate of change and uptake in this community).
For collections development libraries the usage is very low, about 5%. the most used are usage statistics (40%) (of course I’d argue that usage is kind of an ALM. This also raises the question that what are the 60% of people using who don’t even use usage statistics when doing collections development?).
This survey forms a good baseline, but as with any complex issue it begins to raise many interesting questions.
Stacy starts asking questions of the room. We find out that about 1/2 of the audience are librarians. No one in the audience thinks that ALMs should not be used for collection development. There might be a selection bias in the audience.
Q: When working in the context of a library where there are deals, is there not a responsibility on the publisher to present some of this information? Even a the journal level A: yes
Understanding impact through alternative metrics, library-based assessment services
Kristi Holmes, Director, Galter Health Sciences Library at Northwestern University, Feinberg School of Medicine
There is a highly motivated audience within libraries to plug in to this space. The Northwestern library NUCATS has a focus on supporting translational medicine.
In 2000 is was taking 18 years to get form discovery to dissemination. By 2015 that had come down to 7 years, we are getting smarter about this, but there is still room for improvement.
The NIH provides the largest program for translational medicine. There is a lot of pressure to get these awards.
At NUCTS they are measuring a lot of things, for example influence of a research output e.g. time to publication, number of technology transfer products, ROI of pilot awards, the number of collaborations. They are trying to do this in a meaningful way.
They want to understand both productivity as well as impact, and they need great data to be able to do that. They want to know who is paying attention, what conversations are happening around a work.
Even when looking at data from a great source like Scopus, there is still a lot of missing data – the finding that led to the work, collaborations, who is citing the work. There is a lot of low hanging fruit here.
What is impact? There are a lot of definitions, what is critical is the context. It’s not just the paper or the tweets, but understanding what it leads to. If a paper leads to a new methodology, a new standard of care, if it gets used in the med school curriculum – these are very impactful. If you can bill for the work this is also meaningful. These can serve as indicators that there is a change in the way that we are delivering healthcare. It’s messy, but it can be really meaningful.
They are trying to build an ecosystem with the library as a partner. The library is a trusted neutral space, they have a tradition of service and support, the people in the library really work for the mission of the organisation.
They launched a metrics and impact core. They help people track their publications. They are helping their researchers put their NIH biosketch together. They have helped hundreds of researcher with this.
They call it a core because this is terminology that resonates with the people who they are serving.
They are using tools like altmetrics.com, plumb analytics, bibliometrics analysis, social network analysis. They are using a lot of surveys.
They found the case studies that were produces as part of the REF magnificent, and they want to make that work in their site.
They are creating dashboards to provide insight over a large number of facets of what is happening with research in the library.
They are mainly trying to shine a light on what is going on.
You need to know who cares, who the people are who you need to make sure are paying attention to their perspectives. You need to know who is going to be the champion in your organisation. Find out whats missing. Ask what you can do today!
They have a google group – firstname.lastname@example.org. There will be MOASIC meeting in Toronto with the MLA in May 2016.
Q: can you say more about your stakeholders?
A: it really is about getting out an having conversations, the dean is generally the last person to know, they work a lot with their students and junior faculty. JF are highly motivated to explain why what they are doing is worthwhile.
Q: Can you show whether AMLs add to, or not, to PubMed
A: everyone cares about improved health, and it seems like there really are good data in this space, e.g. patient care organisations. Medical libraries are like their own little thing, people are very motivated and also under a lot of publication and impact pressure.
Altmetrics opportunities for librarians
Wouter Gerritsma, Deputy Librarian, Digital Services & Innovation at Vrije Universiteit Amsterdam
Librarians have a natural role to play in the altmetrics space.
Wouter discusses where you are likely to encounter ALMs today. Basically, all over the palce; Researchers find ALMs often on journal and article pages. If you are using a Primo discovery system ALM indicators are integrated. DSpace can also integrate this. Web Of Science provides usage data. Scopus has also introduced a whole dashboard. Wouter finds that he can’t escape these in a library environment. He needs to understand these so that he can explain them to his users.
The first role of librarians is library outreach. His experience is to start small and gain experience. It’s important to know that ALMs are not only about social networks.
He also recommends that if you make presentations on this topic then do share them for the wider community.
The field is in rapid development, and we should follow it and see what is happening in this space.
He wants libraries to be the managers of research information systems within the universities. There is a big transition happening in Dutch universities. The focus on RIM systems that exists in Europe is different to that in the US. There is an opportunity to use these platforms as a location for aggregating usage and ALM data, and there should also be connections to IRs.
He likes the idea of getting rid of the pure concept of the Institutional Repository, and think of something bigger.
He likes to talk about the Institutional Bibliography.
The challenge is how do you make use of that collection of information? Collecting the information without making wider use of it is a wasted opportunity. You have to make use of the collection that you have, and you can then start to do analytics on this data. There are opportunities to do recommendations, citation analysis, visualisations, ALMs, bibliometrics. These are all built on top of a comprehensive collection of data.
What is needed is that librarians need to develop new skill sets. Get you CRIS in good shape, only when you have a good collection of data you can start to do the analysis.
What we have not mentioned in the room yet is that we collect ALMs at a single article level. We want to aggregate to a researcher. It’s a sensitive subject but people are already doing this based only on citations. You can also roll up to the department or university level.
We need to break beyond looking at only peer reviewed articles. These are of course the mainstay of the output, but don’t forget about books, conference, thesis, non peer reviewed output. Collecting ALMs on these items is an important challenge.
The way to do this is the use the DOI. ORCID is also critical. All Dutch universities have ISNIS, but they don’t know about it, they have VIAF, but they don’t know about it. They are being asked to fill in ORCIDS now.
Don’t collect ALMs for peer reviewed articles only.
Stay away from predictors of citations, it’s about allowing researchers to tell the story of their work. Those stories are important because we are increasingly looking for evidence that sits along side citations, and not only citations.
Q: in what department has story telling worked best?
A: in the Netherlands and in the REF they are looking for societal impact. ALMs can give information around things like where news articles have been written about research, and that helps crating stories that describe societal impact. (in fact Wouter kind of side stepped the question, and it would be great to see some solid cases studies presented at a future meeting).
Q: What about things that are not even published, like software?
A: Github can give you usage.
Alternative metrics in Dutch university libraries
Alenka Princic, TU Delft Library, Netherlands
Likes to thing of these a CoMetrics, or CoMet, or complimentary metrics (not conmetrics).
The role of the library is changing from being a collection provider to a partner in science.
Research assessment is becoming a new service area for librarians. The librarians are the key group who are using these bibliometrics. In Delft they do not have a dedicated analyst for this.
At Delft researchers are publishing for impact.
In Delft they are doing some bibliometric analysis, but it’s somewhat ad-hoc. They have a research support portal, there is information in this portal to help researchers understand how to improve the footprint and impact of their researcher. This is an initiative that is in development, and they aim to expand the services that they provide.
For ALMs it is currently bottom up. They are looking mostly at the major tools, and they are experimenting and creating pilot programs. They are keen to do this in collaboration with others on this.
Within the Dutch libraries, getting 10 respondents from 13 Dutch libraries, looking at this qualitative data what they can say is that
- About 1/3 have bibliometrics services rolled out for individuals
- About 1/3 have these services for group assessment
- About 1/3 don’t have these in place
- none have this in place on the institutional level.
General awareness of the tools is quite good, but actual usage is low at this time, however the low numbers in this survey are hard to interpret.
There are a lot of people who are experimenting and observing. Quite a few have deferred looking at this until they implement a new CRIS (current research information systems).
In terms of future vision, about 1/3 have not had the time to create a future vision for ALMs. 1/3 have said that it needs maturity, but it needs potential.
The speed is considered a great benefit of ALMs. It might be able to quantify the success in achieving goals set by the researcher, specifically around reaching target groups or audiences.
It could be part of an open peer review system. Can evaluate non-traditional outputs. It could also be used to demonstrate exclusiveness of an institution, and could provide additional data for strategic purposes, e.g. how international an organisation is, how is the institution doing on open science indicators.
Q: why are Dutch Libraries not doing workshops in engaging early stage researchers on these topics?
A: does not see the topic as so black and white, in that it’s not the case that they don’t do anything at all. There is a lot of engagement and advice on science engagement. Perhaps it’s not solely the role of science communication and solely the role of the librarian, but sits in the middle?
Q: Have we got past what we call this thing? Is it really just part of telling the story, and what you call it is about who you are speaking to.
A: The speaker agrees.