Public librarian seeks statistics input
December 13, 2017 10:25 AM Subscribe
What information and statistical tests do I use to find out if moving biographies to their own special display actually increased their popularity? I want to take a rigorous approach that includes confidence levels.
The idea was that it would make the books easier to find and hence increase circulation. The analysis software available to us can hint at things, but it unfortunately does not incorporate the concept of statistical significance. Nor is it very clear about what data it is basing its results on.
However, more detailed database reports can be run. These can be tailored in many ways to show almost all specific collection data imaginable. I want to run some of these detailed reports and statistically analyze them in a spreadsheet to find out whether the data truly supports the decision to separate out biographies into their own area.
The class range is autobiographies and biographies, beginning at 920 and ending at the end of 921. Before Jul 2016, this range was in the usual position within the non-fiction collection. In Jul 2016, the range was taken out and placed in its own special display, near but not spatially integrated into the rest of non-fiction. So we have over a year of data for the new location.
I am starting with the idea of looking at biographies as a percent of total checkouts for a year, 07/01/2015-06/30/2016, before the change, versus a year, 08/01/2016-07/31/2017, after the change. I would leave out July 2016 because the books were being moved then which would cause the numbers to be off. I would normalize on total checkouts because the collection was weeded and decreased in size over those dates.
Three measurable factors would be total checkouts, total items, and checkouts per item. Three possible sets are overall non-fiction including biography, non-fiction minus biography, and biography alone.
Confounding factors are circulation varies seasonally, and there have been reductions in the overall number of non-fiction books, and people are checking out fewer books. Also, focus goes to retaining popular books, so remaining items might have higher circs per item.
If there is statistically significantly more activity after the change, I will consider it to be a valid move. If there is no change, then it is a pointless move. If there is a decrease, then it is a harmful move. Making changes is a big commitment and I want to find out a way to mathematically test the results of our choices.
Help me figure out what to normalize on, whether to compare month-by-month samples or just boil it all down to the overall numbers for a year before versus a year after, whether there is a better time frame than the one I proposed, and what is the appropriate statistical test to run once I've gotten the data sets.
Also if any librarians want to share qualitative, not quantitative, observations about what has worked at your library, I would love to hear it.
Brief answers are ok. Pointed in the right direction, I can do the research to fill in the rest.
I appreciate your help and I look forward to doing math!
The idea was that it would make the books easier to find and hence increase circulation. The analysis software available to us can hint at things, but it unfortunately does not incorporate the concept of statistical significance. Nor is it very clear about what data it is basing its results on.
However, more detailed database reports can be run. These can be tailored in many ways to show almost all specific collection data imaginable. I want to run some of these detailed reports and statistically analyze them in a spreadsheet to find out whether the data truly supports the decision to separate out biographies into their own area.
The class range is autobiographies and biographies, beginning at 920 and ending at the end of 921. Before Jul 2016, this range was in the usual position within the non-fiction collection. In Jul 2016, the range was taken out and placed in its own special display, near but not spatially integrated into the rest of non-fiction. So we have over a year of data for the new location.
I am starting with the idea of looking at biographies as a percent of total checkouts for a year, 07/01/2015-06/30/2016, before the change, versus a year, 08/01/2016-07/31/2017, after the change. I would leave out July 2016 because the books were being moved then which would cause the numbers to be off. I would normalize on total checkouts because the collection was weeded and decreased in size over those dates.
Three measurable factors would be total checkouts, total items, and checkouts per item. Three possible sets are overall non-fiction including biography, non-fiction minus biography, and biography alone.
Confounding factors are circulation varies seasonally, and there have been reductions in the overall number of non-fiction books, and people are checking out fewer books. Also, focus goes to retaining popular books, so remaining items might have higher circs per item.
If there is statistically significantly more activity after the change, I will consider it to be a valid move. If there is no change, then it is a pointless move. If there is a decrease, then it is a harmful move. Making changes is a big commitment and I want to find out a way to mathematically test the results of our choices.
Help me figure out what to normalize on, whether to compare month-by-month samples or just boil it all down to the overall numbers for a year before versus a year after, whether there is a better time frame than the one I proposed, and what is the appropriate statistical test to run once I've gotten the data sets.
Also if any librarians want to share qualitative, not quantitative, observations about what has worked at your library, I would love to hear it.
Brief answers are ok. Pointed in the right direction, I can do the research to fill in the rest.
I appreciate your help and I look forward to doing math!
Best answer: Since you have actual, complete numbers, rather than a sample, you don’t need to worry about “statistical significance.” It’s not possible for you to know whether any change in these books’ circulation was due to their change in placement or other factors, and you may decide that a change is too small to be meaningful, but this is a judgment call and does not involve complex math.
I would glance at month-to-month comparisons across years to see if there are any huge anomalies that could be explained by something other than the change; if there’s nothing obvious, I would stick with full-year data.
Percent of total checkouts is useful, but if their placement meant that people are checking out more books in general instead of checking out biographies rather than other books, that can affect your results (making them seem less strong). Look at total number of checkouts for the collection across both periods as well.
If cultural factors have led to certain insanely popular biographies in this period that you might expect people to go searching for even if they weren’t moved- like Hamilton biographies- you may want to exclude them and see how that affects your data.
Depending on how deep you want to go with this, you can look at a subset of, say, biographies that had fewer than five checkouts in the five years prior to the change, and see if they had more circulation.
If you know whether a checkout is from a request rather than someone coming in and getting it themself, that can lead to other interesting analyses.
I assume you don’t save user-specific circulation data. But if you did, you could look at whether people who never before checked out biographies began doing so.
posted by metasarah at 10:53 AM on December 13, 2017 [5 favorites]
I would glance at month-to-month comparisons across years to see if there are any huge anomalies that could be explained by something other than the change; if there’s nothing obvious, I would stick with full-year data.
Percent of total checkouts is useful, but if their placement meant that people are checking out more books in general instead of checking out biographies rather than other books, that can affect your results (making them seem less strong). Look at total number of checkouts for the collection across both periods as well.
If cultural factors have led to certain insanely popular biographies in this period that you might expect people to go searching for even if they weren’t moved- like Hamilton biographies- you may want to exclude them and see how that affects your data.
Depending on how deep you want to go with this, you can look at a subset of, say, biographies that had fewer than five checkouts in the five years prior to the change, and see if they had more circulation.
If you know whether a checkout is from a request rather than someone coming in and getting it themself, that can lead to other interesting analyses.
I assume you don’t save user-specific circulation data. But if you did, you could look at whether people who never before checked out biographies began doing so.
posted by metasarah at 10:53 AM on December 13, 2017 [5 favorites]
I would tabulate and visualize your data first - you might be able to see an effect that you consider good enough to justify the change.
You have kind of a difficult experiment - a checkout of a book may not be independent of the next - one patron especially interested in Abraham Lincoln might just happen by chance in one year and not the next, and I'm guessing non-fiction readers check out more than one book on the same subject. If an instructor assigned a class project on the founding fathers this year (but not last year), your checkouts will light up but it won't be because you moved the books. Also, once a book is checked out, it's unavailable to the next patron, and what if it's renewed twice or never returned?
Does inter-library borrowing count as a checkout?
Anyways, if I were to take a crack at this, I would consider the unit-of-analysis as number of checkouts of a given book in a given year. I would use a poisson regression with an offset for days available for checkout and I would model the expected number of checkouts for all non-fiction books, with an indicator variable for biography (y/n), an indicator variable for year (2016 vs 2017), and an interaction term for biography * year. The direction and p-value for the interaction term can give you an idea if there was a real change for biographies from 2016 to 2017.
I would ignore the non-independence issue and assume seasonality will be taken care of because the unit of analysis is over the whole year. The effect of culling will hopefully be captured by the year term and that the probability of being culled is not, like, wildly higher or lower for biographies. I would also not read real far into the size or significance of the interaction term - mostly what you get is direction (increased or decreased) and whether it might be real (p-value less than .15 or .10 or something). To make a statement about statistical significance, an approach that is more rigorous to dependence would be required.
Finally, what this model helps out with is telling you *if* there was a change, but not *why* there was a change.
also, metasarah above has really good suggestions of things to explore
posted by everythings_interrelated at 12:00 PM on December 13, 2017 [1 favorite]
You have kind of a difficult experiment - a checkout of a book may not be independent of the next - one patron especially interested in Abraham Lincoln might just happen by chance in one year and not the next, and I'm guessing non-fiction readers check out more than one book on the same subject. If an instructor assigned a class project on the founding fathers this year (but not last year), your checkouts will light up but it won't be because you moved the books. Also, once a book is checked out, it's unavailable to the next patron, and what if it's renewed twice or never returned?
Does inter-library borrowing count as a checkout?
Anyways, if I were to take a crack at this, I would consider the unit-of-analysis as number of checkouts of a given book in a given year. I would use a poisson regression with an offset for days available for checkout and I would model the expected number of checkouts for all non-fiction books, with an indicator variable for biography (y/n), an indicator variable for year (2016 vs 2017), and an interaction term for biography * year. The direction and p-value for the interaction term can give you an idea if there was a real change for biographies from 2016 to 2017.
I would ignore the non-independence issue and assume seasonality will be taken care of because the unit of analysis is over the whole year. The effect of culling will hopefully be captured by the year term and that the probability of being culled is not, like, wildly higher or lower for biographies. I would also not read real far into the size or significance of the interaction term - mostly what you get is direction (increased or decreased) and whether it might be real (p-value less than .15 or .10 or something). To make a statement about statistical significance, an approach that is more rigorous to dependence would be required.
Finally, what this model helps out with is telling you *if* there was a change, but not *why* there was a change.
also, metasarah above has really good suggestions of things to explore
posted by everythings_interrelated at 12:00 PM on December 13, 2017 [1 favorite]
Seems like Australian library language is a little different but hopefully you will get what I mean.
The traditional measure of a collection's performance is turnover - so number of items divided by the number of loans. Your plan to compare turnover before and after the move is a good one. Monthly if you want to be really rigorous, but depends how much other work you want to do, because you should really compare them with how your collections as a whole are performing. At least loans, if not turnover. If your overall loans across all collections are down for a month, but your biography loans are not, that's more meaningful than the biog turnover stats on their own. Up to you if you want to exclude collections that usually perform differently (eg, foreign language books, AV, etc) but I'd go broader than just non-fiction.
I don't know if you can run these stats retrospectively, but you could also measure the percentage of dead items in the collection before and after the move. Dead items are those that haven't been loaned in a defined time period, say six months. Given you have a year's worth of data, you may want to reduce that to three months but you need to be able to make that consistent with your pre-move data.
Measuring the reserves placed on biogs is a good idea too, and if you want to get super detailed (and depending on the size of your collection) you could do stats per item.
If you take suggestions from the public, you could have a look at how many were for biographies. That doesn't necessarily tell you how the separation went, but may be an indication.
posted by Athanassiel at 12:22 PM on December 13, 2017 [1 favorite]
The traditional measure of a collection's performance is turnover - so number of items divided by the number of loans. Your plan to compare turnover before and after the move is a good one. Monthly if you want to be really rigorous, but depends how much other work you want to do, because you should really compare them with how your collections as a whole are performing. At least loans, if not turnover. If your overall loans across all collections are down for a month, but your biography loans are not, that's more meaningful than the biog turnover stats on their own. Up to you if you want to exclude collections that usually perform differently (eg, foreign language books, AV, etc) but I'd go broader than just non-fiction.
I don't know if you can run these stats retrospectively, but you could also measure the percentage of dead items in the collection before and after the move. Dead items are those that haven't been loaned in a defined time period, say six months. Given you have a year's worth of data, you may want to reduce that to three months but you need to be able to make that consistent with your pre-move data.
Measuring the reserves placed on biogs is a good idea too, and if you want to get super detailed (and depending on the size of your collection) you could do stats per item.
If you take suggestions from the public, you could have a look at how many were for biographies. That doesn't necessarily tell you how the separation went, but may be an indication.
posted by Athanassiel at 12:22 PM on December 13, 2017 [1 favorite]
Response by poster: I could start with visualizing circs (loans) per item in biography, normalized against loans per item in all of non-fiction (or even a larger inclusion of the overall collection, as was suggested), for 5 years before the change, versus for 1 year after the change.
You are all helping me develop my understanding of statistics and the vocabulary around it. I am also glad to learn from other library workers.
Inter-Library Loan won't count as a check-out in this case. I can't get past holds (requests) info. Doing an offset based on number of days available for check-out is a creative idea but I am afraid I can't get that info either.
Doing the Poisson idea (if it proceeds past visual inspection and to a more mathematical stage) sounds compelling and I shall read up on that. The T-test seems more accessible given my background but I am up for investigating new areas.
posted by Nissake at 1:22 PM on December 13, 2017
You are all helping me develop my understanding of statistics and the vocabulary around it. I am also glad to learn from other library workers.
Inter-Library Loan won't count as a check-out in this case. I can't get past holds (requests) info. Doing an offset based on number of days available for check-out is a creative idea but I am afraid I can't get that info either.
Doing the Poisson idea (if it proceeds past visual inspection and to a more mathematical stage) sounds compelling and I shall read up on that. The T-test seems more accessible given my background but I am up for investigating new areas.
posted by Nissake at 1:22 PM on December 13, 2017
If you have data going back further on those books, it might be worth looking at to establish check out rates for those books in general. In other words, what if check outs for biographies has been going up steadily 3% for the past 5 years? If that were true, it would affect what you're trying to establish as a growth rate. Ideally, you'd see that the rate at which biographies have been checked out in the past year is much higher than what you'd expect the rate to be given no change.
posted by LKWorking at 1:39 PM on December 13, 2017
posted by LKWorking at 1:39 PM on December 13, 2017
Response by poster: To take into account growth (or, sadly but quite possibly, shrinkage) rates, I believe I would make a regression line and see if the "after" data-point falls above or below the line, and if so, is it far enough away to look important.
posted by Nissake at 2:01 PM on December 13, 2017
posted by Nissake at 2:01 PM on December 13, 2017
I'd've thought 5 years of comparative data is perhaps overkill, but up to you. A semantic point: normalising data usually means that you are smoothing out discrepancies to provide a consistent dataset. So metadata that is in several different schemas might be normalised into MARC, for example. When you compare the most recent year's data (post-move) to the previous years' data (pre-move), you are not normalising anything. You are just comparing. Normalising would be removing outliers, such as audiobook biographies, since the format difference might be a complicating factor.
I forgot to mention before, but another stat we frequently report on is the percentage of the collection that was purchased in the last 5 years. Might also be worth measuring that to see if it has changed - if it has, it may be another factor that influenced loans one way or the other. Even though biographies are frequently of dead people, who are still dead 5 years later, there are always new interpretations that come out of historical figures etc, plus I think the way biographies tend to be written has also changed over time.
I don't know how collaborative your library network is, but here we also tend to benchmark against other library services and share/compare data. If you have libraries nearby that are similar, it's probably worth asking if they have any data on how their biographies are performing. If there are any local library associations that report on trends in public libraries in your state/other geographical area, they may not have exactly the kind of specific data you need but can still provide a bigger picture. Overall, I think the trend across public libraries is for declining loans of physical items.
posted by Athanassiel at 2:24 PM on December 13, 2017
I forgot to mention before, but another stat we frequently report on is the percentage of the collection that was purchased in the last 5 years. Might also be worth measuring that to see if it has changed - if it has, it may be another factor that influenced loans one way or the other. Even though biographies are frequently of dead people, who are still dead 5 years later, there are always new interpretations that come out of historical figures etc, plus I think the way biographies tend to be written has also changed over time.
I don't know how collaborative your library network is, but here we also tend to benchmark against other library services and share/compare data. If you have libraries nearby that are similar, it's probably worth asking if they have any data on how their biographies are performing. If there are any local library associations that report on trends in public libraries in your state/other geographical area, they may not have exactly the kind of specific data you need but can still provide a bigger picture. Overall, I think the trend across public libraries is for declining loans of physical items.
posted by Athanassiel at 2:24 PM on December 13, 2017
Response by poster: I was thinking normalizing in the sense of expressing biography checkouts not in absolute terms, but as a ratio to or percentage of total non-fiction checkouts, to counteract effects from overall growth or shrinkage.
I guess then I wouldn't be doing this "normalizing" on checkouts per item, only on absolute number of checkouts.
Thanks for the further info about how your library does things. It is all new info to me and these are things I can bring to others at work.
posted by Nissake at 2:35 PM on December 13, 2017
I guess then I wouldn't be doing this "normalizing" on checkouts per item, only on absolute number of checkouts.
Thanks for the further info about how your library does things. It is all new info to me and these are things I can bring to others at work.
posted by Nissake at 2:35 PM on December 13, 2017
Best answer: This is a t-test. You are comparing two means. Your p value should be .05. A Poisson regression assumes a Poisson distribution, which is really unlikely in your data.
You may be interested in this book, Applications of Social Research to Questions in Library and Information Science. It covers all kinds of methods you might use to ask a question like this one in library settings and is very accessible. Disclaimer: my doctoral advisor wrote it.
Source: I teach research methods in library and information science at the masters level.
posted by k8lin at 4:21 PM on December 13, 2017 [1 favorite]
You may be interested in this book, Applications of Social Research to Questions in Library and Information Science. It covers all kinds of methods you might use to ask a question like this one in library settings and is very accessible. Disclaimer: my doctoral advisor wrote it.
Source: I teach research methods in library and information science at the masters level.
posted by k8lin at 4:21 PM on December 13, 2017 [1 favorite]
Response by poster: I am definitely interested in your book recommendation. Looking at the "look inside" feature on Amazon, I see what you are saying about how it focuses on practical applications and that it is accessible to the beginner. Learning more for possible future questions is one of my goals so I will try to get ahold of it.
Glad you are working with such an informative doctoral advisor!
Out of the types of t-tests, it looks like the paired t-test most matches my situation since the data are linked, before-and-after observations.
I am having trouble figuring out what the "n=" variable would be. Biography circulations per month, for each of the 12 months, so I could make pairs on up through n=12? Paired observations on a daily basis, going up to n=365?
posted by Nissake at 6:28 PM on December 13, 2017
Glad you are working with such an informative doctoral advisor!
Out of the types of t-tests, it looks like the paired t-test most matches my situation since the data are linked, before-and-after observations.
I am having trouble figuring out what the "n=" variable would be. Biography circulations per month, for each of the 12 months, so I could make pairs on up through n=12? Paired observations on a daily basis, going up to n=365?
posted by Nissake at 6:28 PM on December 13, 2017
Response by poster: I mean, what are the "n=" samples.
posted by Nissake at 6:41 PM on December 13, 2017
posted by Nissake at 6:41 PM on December 13, 2017
Best answer: The real answer will depend on what you're trying to do and what you want to get out of this, and as you've seen this gets complicated quickly. Worse, it gets complicated in ways that are more thorny research design questions and not just how to do ugly math.
Are you asking "Did putting our biographies in their own section work?" Then your population of interest is those biographies. Comparing the percent of checkouts of biographies is fine, and you don't really need any measures of statistical significance since you have that full population.
Are you asking "Does putting biographies in their own section work?" Then your population of interest is all possible biographies and things get more complex. That is, your population includes a biography that you didn't move because you won't have it until next year. The obvious thing to do, which isn't really the best since your collection of biographies is not a random sample of all possible biographies, is to compute a confidence interval for the percent of checkouts before the move, and again after the move. If they don't overlap, yay, it clearly had an effect (but maybe in the wrong direction). If they overlap a lot, boo, it obviously didn't have any effect. If they only overlap a leeeedle beet then it gets more complicated and you'd want to do a difference of proportions test, but there's no reason to wrestle with that if the CIs don't overlap in the first place. Your N for each proportion is just the total number of checkouts during that time.
Are you asking "Does putting books in their own section to highlight them work generally?" Then your population of interest is all possible books and things get more complicated yet because you moved one kind of book and are trying to make inferences about other kinds of books, and I'm guessing you think what kind of book a book is makes a difference. Maybe you'd want to do an anova with type of book as a contrast, since type of book is perfectly correlated with moving the book to its own section? Psychologists have devised about ninety bazillion different kinds of anovas for different circumstances.
If you were doing this from scratch and wanted easy statistics, I would have recommended shelving a random half of the biographies in their own section and leaving the other random half shelved with the other nonfiction.
I still wouldn't at all recommend doing this unless you really really want to spend a long time learning the background material and another long time learning enough R to run the models and another long time interpreting the results, which can require doing Yet More Math, and a final long time figuring out how to explain the results to people who hadn't done that work. And, more generally, it's swatting a fly with a sledgehammer. But:
A Poisson regression assumes a Poisson distribution, which is really unlikely in your data.
Surely poisson or negative binomial is roughly what we'd expect as a null distribution for a count variable like number of checkouts, unless checkout statistics are weird in some way?
posted by GCU Sweet and Full of Grace at 7:37 AM on December 14, 2017 [2 favorites]
Are you asking "Did putting our biographies in their own section work?" Then your population of interest is those biographies. Comparing the percent of checkouts of biographies is fine, and you don't really need any measures of statistical significance since you have that full population.
Are you asking "Does putting biographies in their own section work?" Then your population of interest is all possible biographies and things get more complex. That is, your population includes a biography that you didn't move because you won't have it until next year. The obvious thing to do, which isn't really the best since your collection of biographies is not a random sample of all possible biographies, is to compute a confidence interval for the percent of checkouts before the move, and again after the move. If they don't overlap, yay, it clearly had an effect (but maybe in the wrong direction). If they overlap a lot, boo, it obviously didn't have any effect. If they only overlap a leeeedle beet then it gets more complicated and you'd want to do a difference of proportions test, but there's no reason to wrestle with that if the CIs don't overlap in the first place. Your N for each proportion is just the total number of checkouts during that time.
Are you asking "Does putting books in their own section to highlight them work generally?" Then your population of interest is all possible books and things get more complicated yet because you moved one kind of book and are trying to make inferences about other kinds of books, and I'm guessing you think what kind of book a book is makes a difference. Maybe you'd want to do an anova with type of book as a contrast, since type of book is perfectly correlated with moving the book to its own section? Psychologists have devised about ninety bazillion different kinds of anovas for different circumstances.
If you were doing this from scratch and wanted easy statistics, I would have recommended shelving a random half of the biographies in their own section and leaving the other random half shelved with the other nonfiction.
I still wouldn't at all recommend doing this unless you really really want to spend a long time learning the background material and another long time learning enough R to run the models and another long time interpreting the results, which can require doing Yet More Math, and a final long time figuring out how to explain the results to people who hadn't done that work. And, more generally, it's swatting a fly with a sledgehammer. But:
A Poisson regression assumes a Poisson distribution, which is really unlikely in your data.
Surely poisson or negative binomial is roughly what we'd expect as a null distribution for a count variable like number of checkouts, unless checkout statistics are weird in some way?
posted by GCU Sweet and Full of Grace at 7:37 AM on December 14, 2017 [2 favorites]
« Older Police probably lost accident report after an... | Where should I ride my bike next? Newer »
This thread is closed to new comments.
posted by k8t at 10:49 AM on December 13, 2017 [3 favorites]