Ask MetaFilter questions tagged with Stats

Linear regression is... uh... something to do with residuals? (Shit.)

veol — Thu, 12 Nov 2009 19:19:02 -0800

I have some interviews coming up. Anybody with a statistics-related profession.... please help me out with answering technical questions in my job interviews! (A bit on the long side...) I am in my school's Co-op program and I am currently pursuing a bachelor degree in Statistics. The number of entry level jobs for statisticians are really next to none, and I really want to do well on this next interview.

I have already had three interviews for statistics related jobs with different companies and I pretty much flunked each one. I can't seem to wrap my head around the technical questions that are asked during interviews. Most of the time, my mind draws a blank and useless garble comes out. All the generic questions about me, my interests, etc. are easy, but when it comes to solving technical questions on the fly, I lose all confidence and the interview goes downhill. Sometimes I just don't know the answer, sometimes I just can't process the question, and other times I just say "I don't know" and doom myself joblessness.

So now I have a phone interview with company X tomorrow morning. X is a rather large company (one that I've interviewed for before, actually, for a different position by a different person) and some students (myself included) were recommended for the position by my school's Co-op office. The position I am being interviewed for is Statistical Methods Analyst, and doing a quick google search gives me a similar job description to the one I received.

In more detail, my job description says that I will need to be responsible for collaborating with stakeholders to gather and analyze requirements, produce process analysis reports, adhoc reports and data queries, validate and analyze data, develop tools and communicate analysis results.

Under essential skills, it says I should be adept at MS Excel, especially data analysis and process analysis, experienced in VBA Macro or similar programming languages, have knowledge of Statistical tools (such as Minitab, Stata, SAS) and the ability to correctly interpret statistical test results, valuing the nature of the data. Also, familiarity with basic statistical concepts and ability to relate them to industry settings, able to detect and diagnose process problems. Knowing SQL was noted as an asset.

I feel under-qualified for this job and doomed for failure at yet another job interview, but my co-op coordinator assures me that I will be able to pick up the skills once I am on the job. Soo... please help me land this job! Please make my years of studying statistics useful!

My questions are (yes, I know. Finally, eh?) :

What kind of technical questions should I be expecting?
What should I study before going into the interview tomorrow?
If I don't know the answer to a technical question, what should I do?
Any job interview tips?

Many thanks!

How many pugs and pug owners are there both in the U.S. and worldwide?

JPigford — Sat, 17 Oct 2009 12:24:49 -0800

How many pugs and pug owners are there both in the U.S. and worldwide? I'm trying to get a rough estimation on both the number of pugs and pug owners in the U.S. and worldwide.

Any ideas on sources for that sort of info?

Where to find good state-by-state savings statistics?

ejfox — Thu, 24 Sep 2009 07:06:46 -0800

Where can I find solid nationwide savings statistics/data? I am trying to make a map illustrating income & savings in the united states. However, the only data I am finding related to savings is based on measuring the entire United States as one unit. Does anyone know if I could find a state-by-state breakdown?

I have been looking through the "Saving and Investment" section of the BEA website (BEA NIPA Tables) to no avail.

If this information is not available, any advice on how to show a geographic breakdown of income/savings would be appreciated.

Where is a website where I can download an excel-readable file showing the monthly impressions of popular websites?

ejfox — Wed, 09 Sep 2009 11:46:07 -0800

I want to compare the monthly hits of popular websites, where could I download a spreadsheet or similar data for making a graph out of? I am looking to make a graph showing the amount of impressions various sites have been getting. If I could download this information as a .csv or something similar, so I could then import it into illustrator to graph it, that would be very helpful.

Hopefully from a respectable source, please.

Any resources or tricks are welcome!

How can I get college football box scores as a feed or in a flat file?

gpen — Fri, 04 Sep 2009 10:14:49 -0800

Is there an easy (or at least relatively straightforward) way to get all of the college football box scores for a weekend (or more) as a feed or in a flat file (txt, csv, xls, etc.)? I'm running an offline fantasy football league, and I would like to automate the scoring if possible. (Solutions that require scripting or macros would be fine.)

Parvovirus: odds of a puppy getting it?

zippy — Tue, 01 Sep 2009 23:34:45 -0800

I understand the danger parvovirus poses to puppies, but what are the odds of a puppy contracting the disease in the US (specifically Alameda County, California)? I have been reading about parvovirus in dogs (including this discussion), and understand how serious the illness is.

What I can't seem to find is any indication of risk or prevalance. What are the odds a dog will get parvo, and how many cases of it are there a year in my area?

The more mathematical and bounded the answer, the better. I know I can't be assured to the fifth decimal place about anything, but I want to know: Parvo, this terrible disease, are the odds 1%, 10%, or 100%?

More details below, in the hope that they may allow more exact bounding of the answer.

My dog is five weeks old. He was one of the larger dogs in the litter (with two or three brothers and a sister), which I understand tends to confer longer maternal immunity. I intend to start him on a full vaccine series for parvo.

He's 3/4 Australian Cattle Dog, 1/4 Fox Terrier. He was born in a remote rural area of Humboldt County, California, and as of a few days ago now lives in a semi-urban area in Alameda County.

I keep him mostly indoors, with trips to the back and front yard for exercise. I understand that completely preventing exposure to parvo is impossible (as the virus hardy and survives for long periods in the soil), but also that minimizing exposure to parvo greatly reduces the chances for infection.

I would like to know:

How common is parvo in Humbolt County and in Alameda County? Or, if these specific numbers aren't available, then whatever numbers are available for California or the US. A link to numbers of cases per year would be ideal.

What are the odds of a puppy getting parvo between the ages of 5 and 16 weeks if he's allowed to socialize with a: known dogs (with shots), or b: occasionally visit parks and meet other non-wild dogs.

Links to scholarly papers are fine, and links to the dog equivalent to the CDC would also appreciated.

If this is too specific, or if there isn't enough information, please let me know. Also, I do know how bad the illness itself is.

Good websites for tennis stats

peacheater — Wed, 03 Jun 2009 11:42:34 -0800

What are good websites for tennis statistics? What's a good website for search-able tennis stats? For instance, I was looking for a complete list of Rafael Nadal's (pro-) career losses. Where should I go for something like that? I googled but couldn't find anything with search-able stats.

(Asking for a friend)

Who's reading?

pseudostrabismus — Fri, 15 May 2009 13:06:40 -0800

How can I juice up a foo.wordpress.com site? Specifically I want a better statcounter, but any other cool stuff you recommend would be nice, too. I have a blog at nameofblog.wordpress.com. I'm planning to move it to nameofblog.com, but that's not high on my priority list at the moment. In the meantime, I'd like to know more about my stats, but I'm not sure which is the best free stat tracker for a Wordpress.com site. Right now all I see on the default tracker are pageviews, referring links, and the posts that get the most clicks. But those "most clicks" numbers only add up to about 20% of my daily pageviews, which... huh?

Specifically, I'd like to know:
How many people are following on RSS
Stats by city & country
Average length of visit
Any other stats you'd think I might find interesting.

All this stuff used to be easy to track on Blogger blogs with StatCounter, but I don't understand how to put a different stat tracker on Wordpress (I'm using the Contempt template, and I don't know how to edit CSS). Anyone have a suggestion of an easy, free way for me to satisfy my stats curiosity until I have time to customize the whole site? Ease of use is key, though- my knowledge of programming is scant, by which I mean I know maybe 10 HTML tags and that's about it.

While we're at it, I guess I'd also be interested in hearing about any Wordpress.com site goodies you know about that are cool & easy to use, but not widely known.
Thanks!

Lazy Pseudo Statistician Looking for Computer To Do Work For Him

theichibun — Tue, 17 Mar 2009 20:05:04 -0800

How can I take a Myers-Briggs test entered into SPSS in a per question format and get a Myers-Briggs Type for each of 92 participants? I've tried Google and am getting nothing. I downloaded a trial so I can work on it at home while I'm on Spring Break, and the Help files didn't install or something to where the end result is that I can't use them. I'm also kicking myself for not getting my text back after letting someone borrow it for my school's stats class.

I have 92 surveys, part of which is a Myers-Briggs Personality Test. I've entered the data on a question by question basis, where each of the 20 Myers-Briggs questions has an option value of either 1 or 2 (representing E or I, S or N, T or F, or J or P depending on the question).

I know I can get a frequency to get the personality type of the entire sample population. But how can get SPSS to score he Myers-Briggs for me on a survey by survey basis?

If it would be easier, I also have the surveys available and can go score them all by hand. But I'd rather not to that if I don't have to.

Math for the fantasy sports enthusiast

reenum — Tue, 17 Mar 2009 09:34:53 -0800

Looking for resources regarding math in sports. I'm an avid fantasy baseball player, and bought the Baseball Prospectus this year. The use of statistical analysis in that book is amazing.

To that end, I want to learn more about new statistics or metrics being used in sports and also how I can learn more about these methods (e.g. regression analysis), so I can better understand the rationale behind each formula. I am a math amateur in every regard.

Any books, websites, or other resources that can help me understand how these metrics are created, and perhaps that can give me enough of an understanding to try and do some independent research would be much appreciated.

Thanks!

Help me find a web site that I should have signed up for years ago.

niles — Mon, 02 Mar 2009 19:21:17 -0800

[Find-that-website] So. There's this website where you would enter everything you do everyday - what you had for breakfast, what color shirt you wore, what TV shows you watched, whatever. After you had compiled enough data, it would come up with dubious statistics relating things you have done against world news and other users. For example, I might find out that every time I have eggs for breakfast there's an earthquake in SF, or that some guy in California wears a purple shirt every time I forget to floss. I might have found this on the blue or in PC World, and it was probably sometime around 2006. I've been looking for this site forever. Help.

Per VHost Apache Bandwidth Stats

Brian Puccio — Tue, 16 Sep 2008 06:56:14 -0800

Linux/Unix Admin Filter: What is your preferred method for monitoring bandwidth use per virtual host in Apache? I could just log everything and run AWStats/Webalizer/whatever, but that seems overkill when all I really want is a MRTG or RRD Tool graph for each virtual host to indicate bandwidth used the last day, week, month and year. Do you use the impossible to find mod_watch or something else?

Help with GIS

wfrgms — Wed, 23 Jul 2008 09:38:11 -0800

Noob with GIS: Help me get more out of my data, craft reports, etc. I'm really new to GIS in general and ArcGIS in particular. I've got my shapefiles together and plotted my various points without too much trouble. I've even overlaid census data. (I'm using ArcGIS 9.2 - almost exclusively ArcMap.)

Now I'm wondering what else I can get out of it. Specifically:

- I've plotted several hundred points on a map of Chicago. Each point is the location of a financial institution. I'd like to generate some sort of report either in ArcGIS or outside that says something like:

X percent of institutions are within X distance (feet, miles, whatever) of census tracts (or blocks) with median household income of X.

I know methods for generating this type of statistical analysis exists, because I see it in papers all the time! ;-)

I have no experience with statistics or GIS. I'm really in over my head here, but I think I can trudge along and produce something worthwhile. I'm not good with building data queries or scripts either, but I can get help with that.

What I'm looking for is advice, suggestions, or just general direction on where to go for answers. Am I completely lost? Is this beyond the scope of ArcGIS?

Thanks in advance!

Configured, Boiled Down Numbers Please

rokusan — Wed, 14 May 2008 21:36:20 -0800

A good, modern, local website traffic stats solution? I used to know this stuff, but it's been many years and technology changes, yadda yadda, and this has some special odd requirements... so I prefer some hive mind opinions...

I have a colleague looking for a good solution to traffic stats reporting. He is responsible for dozens of websites, with new ones being added/removed every couple of weeks -- promotions and projects from various departments. He has access to combined/extended Apache logs (regular + referrer) already, but since he controls the websites, he is also open to a javascript-triggered thingie on each page, and a custom server that logs things itself. Either is fine as long as it churns out the results.

For results, he'd need an assortment of the usual hits/views/referrers stuff (visitors from Chile, most popular pages this week, hits on this certain link), but boiled down nicely like so:

1) takes raw logs and turns them into something human-readable.
2) scheduled (weekly or monthly) reports automatically saved as fixed files (auditable, this not recreated each time it's run).
3) reports showing just the info he wants* e-mailed to him as attachments (PDF, ideally, but text files would work) so he can present to management.
4) the ability to request custom reports (the period from Sept 1 to Sept 14, users from China only) on demand, presumably from some kind of web interface -- an on-the-fly config run one-off.
5) the ability to combine or separate data from diff sites/logs by configuration. (that is: these two sites are the same, but these four are all to be tracked separately)

* I don't have a list, but he says it will be one or two pages and NOT change over time. A template of certain reports?

He is considering hiring a human being part-time to do this with some offline program as a secretarial task every week... but that struck me as crazy since it's automatable, right? Right? Hm.

It can't be Google Analytics, sadly for me (I know how to use that) and for privacy/security reasons he'd prefer something running on his own Un*x server rather than a web service.... whether it's the same server as the websites or another doesn't matter, as long as he can control access and keep the data private.

In the old days, I would say this all called for urchin or awstats, I think... plus some magic I don't know about to turn the results into pretty reports and mail them from a big old cron job.

From my perspective (helping find/source/make/install this), I'd be fine with something very complicated that I can tweak the configuration of to produce just what he's asking for -- and no more.

Or if something does all the processing, will run reports on a schedule and produce some kind of manageable (parsable?) output like plain flat tables, I can probably manage turning that into pretty documents and mailing them out with additional scripty code that I can stumble through.

Open source and hackable preferred, because of the configurability and tweakiness required... or guidance to existing things that can be cobbled together. If someone has built/integrated one of these before, or something similar, please mefi-mail me and maybe it's easy work, too. I may have overcommitted myself in trying to help.

Looking for a smooth function from noisy observations

Sockpuppet The First — Wed, 14 May 2008 15:46:24 -0800

Statsfilter: I have a bunch of noisy measurements. Each has an (x,y) coordinate and a "score" for that location. Most of the scores are trustworthy, but with the occasonal outlier. I want to come up with a function f(x,y) that estimates what the score at (x, y) would be (whether or not I have an observation at exactly that location). I'd like the function to be smooth and resilient to noise. Can someone point me in the right direction? I'm thinking of something like a kernel density estimator that takes the score into account, but I don't know how to make that work since it estimates density rather than some other value.

Liberal-minded statistics?

jitterbug perfume — Fri, 02 May 2008 04:13:00 -0800

My boyfriend needs to find some statistics for a research paper. Specifically, pro-vegan, pro-environmentalist, anti-war, or other such statistics. Preferably all gathered in one place. We need hard facts - in the vein of those presented in "Diet for a New America" by John Robbins (for example, the statistics involving the water and grain used to feed livestock and how that detracts from feeding third-world starving nations) - but more current. As far as anti-war, looking for things such as the cost of war per day compared with what America as a country puts towards education and/or environmental issues yearly, etc.

Help me not track my visitors

Rabulah — Mon, 24 Mar 2008 05:18:19 -0800

I'm looking for a website visitor tracking service that will allow me to not track users referred from certain domains. Or, one that will allow me to not show those users when I'm viewing the stats. At the moment, I have a free Statcounter account, with the detailed info from the past 500 visitors. Someone has posted my site to StumbleUpon, which is great, but I've had nearly 300 visitors from StumbleUpon so far today, which is eating up my log really quickly.

I really like being able to see where people are coming from, but with results like I'm getting at the moment, it's hard to see. Is there a service, like Statcounter, that will either allow me to not track users referred from a given website, or one that will allow me to filter a given website from showing up in the log when I view it?

I don't mind paying a small amount per month/year to get this functionality, but free is even better.

Past performance, future results, &c

perissodactyl — Thu, 06 Mar 2008 08:34:51 -0800

Statistics-filter: I need to establish to what extent student performance on a particular standardized test is predicted by each of the following: GPA, standardized test scores and a couple of other miscellaneous numerical factors. How do I go about this? I have a fairly large database of student scores on a particular standardized test that is of primary interest to me. Each student record also contains grade point average information and that student's scores on a couple of other standardized tests.

What statistical techniques can I use to analyze how predictive each of a student's other numbers is of their ultimate performance on the standardized test of interest?

I'm handy with Excel and of a fairly technical bent, but I know almost nothing about statistical analysis. Both direct assistance and pointers to relevant information on the Web would be greatly appreciated!

Free NHL stats??

cgg — Thu, 17 Jan 2008 10:49:57 -0800

I want NHL (2007/08) stats. I want as much data as I can get my hands on, in a nice, easily parsed format. But I don't really want to pay for it. Can you help me find this data on the cheap? The downloadable stats here look like they'd be perfect. I'm not so keen on paying $89/year for them, however.

I don't suppose anyone knows where I could find this data for free?

My goal is to get these stats in a MySQL database, and I want to get them in there with as little effort as possible. As the season progress, I'll be updating the database, most likely on a daily basis. Obviously, NHL stats are available from every major sports news source, but they're usually buried in individual site pages.

If it comes down to it, I'll probably just either pony up the $$ to get them in .csv, or write something up that will scrape the boxscores/logs available on tsn.ca for every game. I am a software developer, so that's within my talents, but it just seems like... work. I'm also lazy, and I figure someone, somewhere has already done this work, and has made it publicly available... but where? Google is failing me. Unless someone else has a better idea...

On a related note, in case I'm going to have to do it myself - anyone have any strategies for easy page scraping? Any specific Perl or Python libraries I should look into?

Help me count the ways! How popular is mobile content?

DarlingBri — Thu, 06 Dec 2007 06:08:50 -0800

[MarketResearchFilter] Where can I find stats on the popularity of content sent to mobile/cell phones and other handheld devices? This may well be terribly vague but I'm happy to answer any questions.

I am looking for information on how popular different types of media are on mobile gadgets - specifically, multi-media, I guess. For example, how many people are viewing video, downloading podcasts, or listening to streaming radio on their various handheld devices?

For the purposes of this question, I am not interested in how many people are using iTunes. Music is way outside my market.

European stats preferred but at this point I'll take anything.

Help me get a C+, so I can never touch this godforsaken subject again!

Phalene — Fri, 02 Nov 2007 22:49:57 -0800

Stats Help Filter: Arrrgh! Help me pass my Intro Stats course with a C+ I made a 66% on the first half of the midterm and I think I can't have made more than a 50% on the second half of the midterm, and that's with luck and probably curve correction. Therefore the remaining 50% of my mark has to be something spectacular. Please, share the resources that helped you make the grade?

It's intro to Stats Psychology, if that helps, with a focus on things like Z scores, Regression, Correlation and Probability.

If I can't pass this course it'll scupper my GPA and doom me to math free majors like History, which while I've very good at, are not my cup of tea.

What's better than awstats?

SpecialK — Wed, 19 Sep 2007 08:03:40 -0800

I've got a website that does a lot of traffic. I'm using awStats. It's slow as hell because we're adding ten million hits a day to it's database. We don't want to use an external logging solution because we've found that they underreport dramatically (by 40% in the case of Google Analytics). Are there any free stats processors that will be able to process lots of records AND have better drill-down capability than awstats? I have plenty of hardware to throw a tthe problem.

Formulas for Food

JaySunSee — Sat, 04 Aug 2007 06:36:54 -0800

Looking for math formulas involving food. By math I mean anything from algebra to simple arithmatic. By food I mean caloric intake, macronutrient ratios, energy expenditure etc. Bonus points for bizarro stats regarding food. Thank you.

In search of free online tool for creating radial graphs!

cortex — Fri, 27 Jul 2007 15:47:20 -0800

Are there any free online tools for generating radar (aka radial, spider, star) graphs/charts? The ideal tool would be easy to use dynamically and/or against large sets of data to generate multiple graphs.

I know that Google Documents, for example, provides a lot of graph/chart options, but not this specific sort. On the other end, I'm solid with perl and aware of e.g. radial.pm, but I'd love to find a slick, existing start-to-finish tool rather than having to roll some or all of it myself.

Any ideas?

What are the odds that I can solve this problem?

reformedjerk — Tue, 19 Jun 2007 07:46:07 -0800

Probability/stat question: I'm looking for patterns in a protein sequence, and I've found a few that occur quite frequently. How do I know these are actual patterns and not just an artifact of random amino acid distribution? I have a roughly 1000 amino acid sequence, and I've used a sliding window to chop it up into overlapping 6-mers. Some of these 6-mers occur much more frequently than others and I suspect they have some sort of biological significance. Unfortunately, I don't know to test whether these are true pattern in the biological sense, or if they could just as easily have been the result of random distribution.

I've tried comparing the expected frequency of these 6-mers based on the amino acid distribution with the observed frequency; but the chance of getting any given 6-mer randomly is so low that almost anything I observe (even the ones that only show up once) seem really significant. I'll be happy to clarify things if this post is a bit messy.