How to I find information about this analysis problem?

October 3, 2018 3:41 PM

I want to figure out how to compare time-to-resolution (TTR) for requests before and after a procedural change. There's a subtlety I came up against that others must have addressed, but I'm not sure how to find information it.

I'm measuring time to resolution for help desk tickets. Let's say on 9/1/2018 we made a procedural change that we think will shorten TTR. I want to measure before and after TTR.

So I use two groups of tickets for the analysis. One group is all tickets created in August, the other all tickets created in September (before and after the change). I can create a boxplot, for example, to summarize the TTR data for these two groups. What I realized, though, is that the tickets created in August have a higher maximum TTR than tickets created in September. A ticket created 8/1 might not have been resolved until 9/27 (so, 58 days). But a ticket created on 9/1 can't be open more than ~33 days, because it's only 10/3 now.

I think I can address this by limiting my analysis to requests that were created and resolved in the same month, but I can't be the first person to come up against this. I'm struggling with figuring out how to make this something I can search for online.

Any thoughts ?

posted by Gorgik to Science & Nature (6 answers total)

Wait?
Don’t look at mean TTR but at groupings (1-10 days, 11-20, 21-30, 30+) or whatever is meaningful?
posted by OrangeVelour at 3:48 PM on October 3, 2018 [1 favorite]

Best answer: You could use time-to-event analysis methods (e.g. Kaplan-Meier estimates). Censor any tickets that persist beyond a certain time, e.g. 30 days in your example. Though, I work in medical research, so this might be a case of just hammering away with my preferred method.

At any rate, "survival analysis" is a useful search term which will get you lots of examples from medical research, and "failure time analysis" will get examples from industrial research (e.g. how long until the widget fails).
posted by esoterrica at 3:54 PM on October 3, 2018 [4 favorites]

You could look at the percent that were resolved in some period, like 7 days. That would give you comparable numbers.
posted by Monday at 5:58 PM on October 3, 2018

In my experience with helpdesk, the super-long TTR tickets were probably escalated to another group, awaiting delivery of hardware, or delayed by some other thing outside the helpdesk's control. If your change was within the helpdesk itself, filter out tickets that were sent to other groups (this is what we used to do) or just throw out tickets open longer than a certain period (as others have suggested). Factoring in tickets outside your control will just confound your data anyway.
posted by pocams at 6:36 PM on October 3, 2018

Best answer: It’s also an inapt comparison because Aug and Sept are probably different every year, due to seasonal cycles.

Good statistical analysis is hard; that’s why it takes 4-10 years of dedicated study to learn how to it properly, or at least well.

The place to start is with a null hypothesis (e.g. no change due to process change) and then work out what you would need to do to reject that. Since this is not a controlled experiment, you can’t use much of the standard machinery.

If it were me I would consider starting an A/B test in Mid October that lasts for a month and tickets are randomly assigned to the new or old process. At least that way you can proceed with analytic confidence.

Another option would be to hire a statistician with experience in your field.
posted by SaltySalticid at 7:09 PM on October 3, 2018 [1 favorite]

I agree with Salty that seasonality may affect the results.

I would consider picking a statistic such as the median time to resolution i.e, if you had 100 tickets and 50 were resolved in 17 days, then the median time to resolution would be 17. This has the advantage of not needing the data for tickets that required more than 17 days to resolve. I'd be concerned that the hard-to-resolve tickets, being difficult by definition, will be more random and frustrate the use of typical stats like mean and standard deviation.
posted by SemiSalt at 3:56 PM on October 4, 2018 [1 favorite]