What criteria should we use to judge our product success or failure?

July 3, 2008 10:35 AM

How do I best create and measure success criteria for a business test?

So we are going to be launching a new product test for a piece of software and want to create a list of criteria that we can measure to indicate success. This goes beyond simply we sold it to 5000 people, but items such as, perhaps...we had 10 mentions in blogs, of those 5000 people 1/2 came back for more, our server never crashed, etc.

So what would you measure if you had a new web-based software product and you wanted to find out if it was worth putting the time and expense into doing a full launch? Any resources I should look at?

Also FYI...the software works currently, but parts are a bit hacky so it would take a good long while and a decent amount of coin to make it really fly as it will need to if we expand. There are some parts that are completed 100% so those would be the ones we are checking on the technical side for success.

posted by anonymous to Work & Money (2 answers total)

It seems like you are already on the right track, and it's hard to get more specific without knowing your product. Just think of a funnel. Measure things at each stage of the funnel. How many visitors come to the site once, how many convert to registered users, how many return once, twice, etc. How many registered users use feature X. Feature Y? How many registered users send invitations/referrals. How many of the recipients of those invitations/referrals register, etc.

For X dollars in ad sense spending, how many people visit, how many convert to registered users register, etc. How do the conversion rates of users from source X (ie Ad Sense) compare to the conversion rates of users from source Y.

As for assessing those metrics, you can take a few different approaches.

You could look at the metrics for comparable companies. They could be competitors, or they could just be companies that draw users from the same market, charge similar prices, etc. No one comp is going to be perfect, even for a single metric, but if you can find a handful of comps for each metric you can start getting a feel.

How do you find our how the comps are doing? Research. They'll often brag about some metric or another. Or some analyst will do so, or it may be something you can estimate externally (for example, pick 100 flickr users randomly, how many have pro accounts, how many have public photos?) Or you might be able to find traffic estimates for a site using compete, alexa, or quantcast (all imperfect) and then look at that vs the number of registered users (or some other visible metric).

You can also look at your metrics in terms of your business model. If $X spend on AdSense, and $Y spend on PR yield N users, how much revenue do you think you'll get from N users. How much marketing spend would it take to bring that revenue to a level that it could be a sustainable business with a reasonable return on your investment?

Metrics of time spent on scaling existing features vs implementing new features could also be useful, as would measurement of uptake of new features.

Also, seriously consider whether you should be rolling out hacky features, or whether you would have an attractive offering to a reasonably sized audience if you didn't launch those features right away. You could then try rolling out those still-hacky features to a small number of your initial userbase and take advantage of their input, rather than working on them in a vacuum.
posted by Good Brain at 11:21 AM on July 3, 2008

any hints as to the audience and purpose of this software? That might help get you more metrics that are relevant.. I could suggest number of referred sign-ups, but if your software doesn't have a referral system (or a sign-up system).. then that's not gonna help ya.
posted by mhh5 at 1:12 PM on July 3, 2008