Ask MetaFilter questions tagged with analysis

Simple data analysis in Excel

mxc — Wed, 15 May 2013 10:00:38 -0800

How can I easily analyse test score data in a spreadsheet with multiple independent and dependent variables in each row? I'm currently trying to do some simple data analysis on a spreadsheet of student test results that looks roughly like this in each row:

Gender | Age | Level | etc. | Section 1 result | Section 2 result | Section 3 result | etc. | Overall result

(The first few cells in each row describe attributes of the student; the remaining cells describe results attained by the student)

I'd like to see things like which gender struggles with which section, which age group does better overall, etc. At the moment I'm doing this by making a copy of the data, sorting it by the attribute that I'm examining, summarising that in cells at the bottom of the data, and then making a graph of the summary. I feel like there's probably an easier way to do this. Maybe PivotTables, but I haven't used those before, and can't seem to get them to do what I want.

Ideally, I'd also like to be able to find out things like whether people who did poorly in e.g. section 1 are more likely to have done poorly in section 3.

While my current task is analysing this particular data, any tips or resources for similar/related analyses would be appreciated; this isn't the first time I've run into a situation like this and hit a brick wall.

How do you do keyword research for EBAY?

flexiverse — Tue, 14 May 2013 06:41:36 -0800

There must be a keyword analysis tool by now that specifically works for ebay listings ? That is a keyword tool like google Adwords keyword research tool, but only works for ebay searches within ebay. Not suggestions, but rankings of ebay keywords.

Where did my brain go?

avocet — Sun, 21 Apr 2013 13:02:09 -0800

Help me rescue my analytical faculties so I can survive the last months of grad school with my dignity intact. I'm in my final semester of a research-based MA program in the social sciences, currently trying to get my thesis written so I can defend by a September deadline. My project has been incredibly intellectually and emotionally challenging – I'm researching the public commemoration of violent death among two populations in Canada – and a portion of my fieldwork was downright traumatic. That said, I know I have a compelling, stimulating and fairly original project on my hands, and numerous people both inside and outside of academia have been very supportive of my work. I'm just really not sure I'm able to carry it out.

While completing my coursework last year, I suddenly lost all confidence in my ability to comprehend and synthesize everything I was reading – I'd come to class having done the readings multiple times and always seemed to be on a totally different page from everyone else, many of whom didn't have the advantage of having a background in the discipline like I did, which led to great doubts about my abilities to critically engage texts. The crippling anxiety associated with getting it/not getting it led to me feeling totally unable to participate in what was actually an incredibly supportive small seminar environment. Everything that came out of my mouth was rambling and incoherent and frequently irrelevant. Through some miracle I managed to submit some decent written work and received As in all my classes, but the psychological barriers persisted and seemed to get worse.

I don't really know what happened this year, but it feels like I no longer understand the theory at the core of my project (and I think I've kept it pretty simple, theory has never been my strong suit) and my abilities to analyze and argue seem to have completely disappeared. I don't know how to explain my way from A to B anymore. I just got a big unsettling load of major edits back from my supervisors on everything I've written so far, pointing to an overreliance on empirics and lack of theory and absence of analysis throughout the chapter – yeah, I should have known this was coming. I've apparently totally misunderstood key texts, elided major theoretical arguments, and the whole thing reads as just so facile and simplistic. I've been following my supervisors' cues when they explain to me what my argument is all about, furiously scribbling down things I need to write about when we're in meetings together, then getting home and looking at my notes and realizing I don't understand what they said at all. This all just seems to big for my brain to process, and my mind is becoming increasingly sieve-like with respect to everything I read. I seem to forget and misunderstand so easily.

My supervisors have been wonderfully supportive but I fear they're getting really sick of me and my lack of intellectual progress (or even regression – I feel like I wasn't always this bad). I'm hugely embarrassed by the fact that I can't answer ostensibly simple questions about my project on the spot – how will I manage to get through a thesis defense if I can't string together cogent answers in meetings? I've done very good work for both of them before – it's not like I was admitted to the program by mistake – but I'm ashamed by the fact that I feel like I'm starting to waste their time. It just feels like everyone around me knows my project much better than I do. (I feel like the academic version of Guido, the protagonist of 8 1/2.)

How do I relearn how to think critically, analyze and argue in a really short period of time? I've got a big pile of substantial edits (more like a massive rewrite) due at the end of the month, and the whole thesis (most of which remains to be written) needs to be done by the beginning of August, at the latest. The government and the banks will not give me any more money if I'm not done by then, and I'll have to drop out if I'm not ready for a defense/get to a defense and end up failing.

I know this is all just a bad case of performance anxiety/impostor syndrome/self-defeating thoughts at its core. Through CBT I learned how to shut certain depression-related self-defeating thoughts down, but I don't know how to suddenly reconfigure my thought patterns to understand texts and theory and logic again. I sit down to write/lie awake at night and attempt to logically think through the questions that have been presented to me in the simplest terms possible and my mind gets stuck in this endless 20 GOTO 10 loop in which I can't think past step one of my argument, let alone follow it through to its conclusion – it's not like I've got a negative thought that I can step in and interrupt. My mind is a mess. (And yes, I'm on medication for depression, but it doesn't seem to be doing a damn thing these days. Given my previous experiences changing drugs, having to taper off and start on a new one does not seem like a good idea right now.)

Is there any way to salvage my confidence and mental faculties from this mess in time? MeMail me if you need to. Thanks for listening.

Decisions, decisions...

Seeking Direction — Mon, 04 Mar 2013 12:06:25 -0800

How can I avoid analysis paralysis in decision-making and pick what's "right for me" more consistently? How can I minimize regret? I seem to be incapable of making a good decision.

No, I'm not talking about "bad decisions" in the sense of drinking and driving, or playing hockey on a pond with a "thin ice" sign.

Instead, I'm talking about not being capable of deciding which restaurant to meet someone at, what time to make an appointment at, or what color tie to wear. In other words, decisions where there is no "right" choice, though I may indeed enjoy one of the choices better.

I know traditional "decision-making" advice tends to be things like "make a list of pros and cons, weigh the outcomes, etc.". However, I try to think too logically, and get stuck in analysis paralysis...often, over the stupidest things! (Unfortunately, the problem happens with decisions on every scale, unless there is indeed a clearly logical, "right" choice.)

Thinking vs. preference/feeling have a battle.

It's worse when someone else is involved.

I develop a sense of regret easily (especially if it is a large-scale decision).

I'd like to follow something along the lines of Steve Pavlina's "is it me" advice (which I consider very valuable), but the "logical" side of me inevitably tries to suppress it.

Any advice would be appreciated!

Car-Buying Negotiations

wolfdreams01 — Wed, 30 Jan 2013 07:53:52 -0800

I love my car. It is reliable, trustworthy, and it has carried me safely through at least one rather hairy situation. Best of all, it was cheap. When I touch the dashboard, I remember the thrill of the negotiation process, and even the salesman's sweaty palm as we shook hands on a price thousands of dollars lower than what my friend paid for the exact same model. Nevertheless, it's time to upgrade to a new car. Since it's been eight years since I last purchased an automobile, my bargaining skills in that area are rusty. (I negotiate pricing at work all the time, but that typically involves transactions in the millions, which is a totally different animal.) It's really important for me to get a good deal on this car, since (for better or worse) a lot of the emotional joy of owning it will come from knowing that I got a better price than most other people could have.

What's the best way to instantaneously determine whether you are getting a good price on a car? What is the best way to get a good price? In the past, it used to be that shopping online meant you were effectively paying "retail price" and so the only way to get a good deal was to go to a dealership and negotiate in person. Is that still the case?

Potentially significant data points:
1) I will be buying a brand new car, not used.
2) I believe in always getting the maximum warranty.
3) It will be a hybrid with a very good safety rating (most likely a Subaru, but not necessarily).
4) I can make full payment upfront.
5) I would like to "pimp it out" a little (in a classy way) in terms of getting accessories like heated seats, remote starter, etc.)
6) I have no strong preference for which car I get, as long as points 1, 2, and 3 are met.

What is the best way to go about this, maximizing my leverage in order to secure the best price?

News analysis

tavegyl — Mon, 28 Jan 2013 11:11:09 -0800

I need to do a rough and ready news analysis. I want to go to a specific news site and see the frequency of certain words and phrases before and after a certain date. Ideally, I'd be able to do this in a slightly more complex way, with combinations of words and phrases. I want to do this for free. Does such a tool exist? Say a particular politician was always considered a bit of a clown. After he saves a small child from drowning, the way in which he was spoken of in a particular news outlet shifts so he's now spoken of as a serious candidate. I want to fiddle around with a tool which would let me see if there has been such a shift.

Is this something that exists?

(I know I saw something which did something similar developed by some American university. I requested a trial and never heard back. All memory of URL, search terms etc has subsequently been wiped, argh.)

How can I do a meta-analysis of just two papers?

ocherdraco — Thu, 17 Jan 2013 10:37:41 -0800

I'm working on my first ever scientific paper: a meta-analysis of data from other studies. I've just finished my search for relevant papers. My paper is to cover two related (but different) topics (determined by my advisor). For Topic A, I have a fair number of papers (nearly 20) from which to obtain data. For Topic B, though, I only have two! Is it even possible to do a meta-analysis of two papers? If so how would I go about it? I'm asking y'all instead of my advisor because he is unavailable this week and next, and I want to go ahead and get going, at least by reviewing other meta-analyses so I can get a sense of what I need to do with the papers I've got.

Chart help needed: How to show proportion between two values?

Sock Ray Blue — Wed, 16 Jan 2013 06:48:41 -0800

What type of chart do I want to show that one number is several times as many as another? Putting a tall bar representing value x next to short bar representing value y shows that x is greater than y by x-y. (I realize that's a bit of an oversimplification.) What's the best way to show that x is greater than y by x/y times? I would love some examples.

Bonus question that I should really know the answer to: if x-y is the "linear" relationship, what is x/y? Neither the "geometric" relationship nor the "proportional" relationship sound right to me.

Seeking Truth Out of Fiction

RabbleRabble — Fri, 04 Jan 2013 11:14:55 -0800

Inspired by this post on the blue, I was wondering -- what are some examples of articles/blog posts where someone analyzed (presumably with tongue-in-cheek) a minor part of a fictional world in a serious way that treats the subject matter as truth, not fictional? I'm having difficulty explaining what I mean, but here's some more information. In the linked example above, someone looked at the Hill Valley Telegraph from Back to the Future II and critiqued it as if it were a real paper. What are some other examples of this? A related example I can think of is this wonderful analysis of urban planning in Richard Scarry's Busytown. Busytown is treated as an actual town and given serious consideration as an actual town. The humor is that the author is analyzing a children's book, but the analysis does not treat the content as fiction.

I realize there are many ways to look at this question, but I'm looking more for pieces that offer an offbeat, witty, or humorous view but in an ironic or faux-serious manner.

[first AskMe; hopefully it makes sense]

Help Me Pick Good IT Training for 2013

Irontom — Fri, 28 Dec 2012 04:22:23 -0800

Training for Senior Business Analyst - looking for recommendations. I am a senior business/requirements analyst with 14 years of experience, with 7 years in the federal healthcare IT space (as a contractor) for both the VA and the DOD. For the first time in years, I have the opportunity to take advantage of training paid for by my employer.

So, can you recommend some training courses for me? Given my preference, I would select courses that give practical skills/capabilities over resume-stuffers.

Stats-filter: Comparing individual means?

Scottie_Bob — Thu, 06 Dec 2012 08:26:20 -0800

StatsFilter: How do I compare the means of an individual at three different points? Ok, so I'm working on a project just now and I want to find out if the means of my dependent variable X for ONE person at times A, B and C are significantly from one another. I'm not especially interested in finding out if this speaker is significantly different from other speakers, since that's part of my regression analysis that I'm also doing.

To give you an idea of what I'm looking at, let's say that we interview a speaker and find out that they say the word 'bag' in a particular way and we take measurements of that. In year 1, I take 200 observations of this word, in year 2, I take 240 observations, and in year 3, I take 175 observations. Each observation has its own line in my excel sheet, who says it, what kind of word it is (noun, verb etc), what came before it, and so on. Now what I want to know is if the way that the person says 'bag' is significantly different in Year 1 compared to Year 2 compared to year 3.

The data is not normal, so ANOVA seems to be out, plus the data are unbalanced (so I have different numbers of observations at times A, B and C). I did think that Kruskal-Wallis would work, but that seems to only apply to group means while I'm just interested in comparing the value of X for one person at three different times.

In terms of stats packages, I have SPSS and R, although if the solution is with R, I'll need step-by-step instructions for it cause I'm a noob with it.

Any stats-aces out there able to help a brother out?

Give me the "why?" of the weather

awenner — Sun, 02 Dec 2012 10:21:51 -0800

Where can I find detailed weather analysis for Canadian cities? I live in Montreal. I have plenty of access to instant temperature readings, and I have a pretty good sense of the forecast thanks to Weather Underground. I have the "what?" of the weather. What I'm missing is the "how?" and the "why?". Is there a service that will tell me about the big picture weather patterns currently going on, and how they're affecting the smaller scale picture here at home? I'm looking for something like what the weatherman on TV does, but with a bit more science behind it. Searching Environment Canada doesn't seem to help much.

Is there a better/faster way to operationalize the coding of messy text data?

iamkimiam — Thu, 29 Nov 2012 11:24:13 -0800

I'm trying to tag and code about 4000+ unique paragraphs of data. These are opinion responses to two similar questions. I manually went through the first 4000 responses and it took weeks using Google Refine. I'm wondering if there's a way to operationalize this to be a bit easier and less time consuming? There are about 10 dimensions I'm coding for with tags, and the wording of the answers varies wildly. Here's an example of various responses to one of the dimensions:

"This is just how it is in my brain."
"I see it in my head like this."
"When I think about it, this is what I come up with."
"It's just the way it is, you know, inside my mind."
"Mind's eye sees it this way."

These responses would be preceded and followed by other text that's also important to me to code on other dimensions, so I don't want to replace the whole paragraph with a tag like 'in head', but rather add 'in head' to a series of tags for that response (separate by semicolons). The data is also very messy with punctuation, spelling, omissions and IPA/other character variation.

If you've done something like this, how did you go about it? I'm currently using Google Refine's cluster feature to collapse about 5% of the rows that are short and straightforward. Then manually adding semicolon-separated tags in a new column for the rest. It's very tedious.

Tools I am familiar with: Google Refine, SQL, Excel (yuck), Text Wrangler and some very limited command line stuff (terminal and awk, grep).

Trapped in a local min.

Nomyte — Fri, 02 Nov 2012 15:11:02 -0800

I sort of know Matlab, R, SPSS, and C. But how do I really learn Matlab, R, SPSS, and C? I've been scripting math/stat things for years in the context of both work and school. I've been using Matlab and SPSS for about three years. I can duplicate in R most of what I can do in SPSS. I know enough C to do problems from K&R and mock up numerical algorithms for analysis courses. Basically, I know enough to meet my needs and can look up answers to most questions I have.

The problem is that my needs are very modest, and when I look for more in-depth learning materials, I run into my lack of area expertise.

For example, the MathWorks website has lots of free recorded webinars, but they're on topics like large-scale wind farm management, circuit simulation, fluid dynamics modeling, etc. Those things are far outside my area of expertise. I am not an engineer. I don't know what the problems are in these areas, and learning to programs solutions to problems I don't understand seems backwards. (For the record, my niche is MRI, but we leave the writing of analysis software to engineers and computer scientists at NIH.)

Similarly, I am not a PhD statistician. I have 12 graduate credits' worth of statistics. I am familiar with a handful of basic statistical models and when they are best applied. This requires minimal mastery of SPSS or R. A statistician should be able to develop hybrid models to accommodate any sort of messy data from the real world, rather that complete, balanced research datasets.

I know enough C to do trivial problems and mock up simple numerical algorithms. That's not what C is really used for. But I certainly can't write a new LAPACK, and I know next to nothing about OS design principles or algorithm optimization.

How and in what directions can I make progress from my current state? How can I take my skills to the point where, for example, they might be used as a central part of a job? Or, do I absolutely need to be an expert in a domain of application before it pays off to improve software skills?

(If you are going to suggest "contributing to projects," please outline the next 2-3 steps. I have no idea what kinds of projects use these tools, or how to find ones I can contribute to meaningfully.)

Former Account Exec with MBA pariah for data jobs?

hrj — Wed, 17 Oct 2012 11:03:27 -0800

Looking for how to get skills that would make me qualified for a data analyst position. I've posted a few questions on my job search (here and here), and in searching through job postings here (Austin, TX) and thinking about what interests me I think data stuff would be a good fit as I like crunching with numbers, and technical stuff but my resume does not reflect that. I'm happy to share my resume if that would help. I took a Intro to SQL class and an advanced Access class per recommendations before, but what else do I need to break into that field. I've been in Austin for 5 months and I really need a job!

Some things about me:
- I'm a highly analytical and data oriented person
- I have my MBA in finance
- My BA is in History (I didn't really know anything about careers or real jobs when I started college at 17 - in retrospect Economics, CS, or MIS would have been a better fit)
- 8 year experience in Account Management - the last 5 spent in a company that specialized in selling data used for credit analysis and marketing purposes)
- I recently paid (overpaid) for a resume writing and my resume is oriented towards Sales Analyst positions, but I don't know how strong it is

I appreciate any help advice for what types of jobs to search for, how to strengthen my application, and any training that might help. I've spent about $1000 recently (for the classes, software, and the resume) so money is a bit tight, but I'm willing to spend for something that would seriously help me get a job (i.e. certification??)

Thanks in advance.

I'll be the most powerful man in Hill Valley, and I'm gonna clean up this data.

Homeboy Trouble — Wed, 05 Sep 2012 19:33:42 -0800

Help me find this data analysis tool, so I can process lots of cool data. I have this giant set of user-entered data describing places that I'm trying to classify. One specific task that I'm having problems with is that people enter very similar but not identical information, so each of them show up individually. So I have records like:

JOE'S GAS

JOE'S GAS STATION

JOE'S GAS STATION PINEHURST DRIVE

JOE'S GAS STATIONS INC

JOES GAS

JOES GAS STATION

but it would make my life a lot easier if they were all linked together, so I don't have to classify all of them individually.

I remember seeing, quite possibly here, a stand-alone Windows program that did this. It was a data analysis package, with a lot of other features and analytical capabilities, but there were robust functions for grouping these sorts of similar texts together using some sort of algorithm (I think I remember fuzzy clustering, but don't quote me). If memory serves, it was open-source or free or at least there was a free demo, and I seem to remember it being vaguely affiliated with Google.

I remember there was a modest hubbub when it was released; there was a series of demo videos showing cool features of the program. As I said, I think I may have seen it on the blue, but I follow enough data blogs that I may have seen it elsewhere.

What are the most marketable computer/programming skills in the Seattle job market?

brighteyes7 — Tue, 04 Sep 2012 17:35:50 -0800

What are the most marketable computer/programming skills in the Seattle job market? Hi folks,

For one reason or another, my current track isn't working out; I've got three actuarial exams passed (all within 9 months), but don't like the industry and can't see myself spending the next 30 years in it.

So I left my job, and have some college teaching lined up for math (I have an MA, and three years of teaching experience). This is greatly rewarding work in and of itself, but I would like to supplement this income with another job. I've been applying to other jobs, but the only skills I have are teaching math, and advanced Excel.

I see there are some continuing education classes being offered at the local community college. I know that Seattle is a very tech-oriented industry, so what classes would give me the most marketability? I have a good amount of experience in financial modeling and data analysis, but nothing too fancy or seriously statistical/mathematical. As evidenced by my academic background (near-perfect GPA in graduate math), I like technical/creative work.

There are classes in Java, C#, C++, Visual Basic, MyPHP, SQL, and Perl, but I'm a good self-learner, so if, say Python or Ruby were highly recommended, I wouldn't mind learning it from scratch. So far I'm leaning towards SQL, but upon reflection I really don't see why other than the fact that I've heard it's very useful in analyzing large data sets, and almost had the opportunity to use it in my old job.

can the second noun ever be the subject?

Nanukthedog — Sun, 20 May 2012 10:52:19 -0800

[Language Processiong / Grammar Question] With a pattern of noun infinitive adjective noun verb infinitive, can the second noun ever be the subject of the verb? Bonus question (below the fold): In the second case does the adverb of the verb always determine the sentiment of the second noun? I am making a foray into some language processing and I am looking for any case which invalidates what I am thinking might be a rule. With a pattern of noun infinitive adjective noun verb infinitive, can the second noun ever be the subject of the verb? Counter-examples welcome.

a: article
n: noun
i: infinitive
j: adjective
v: verb
r: adverb

As an example I using the word "new" as my key. For context I am examining words that are close (up to 4 before and 4 after) to that word.

"The vase over the new television fell on the [floor]."
"-4 -3 -2 -1 0 +1 +2 +3 +4"
"a n i a j n v i a"

Effectively, this sentence is neutral to the word "new"

I could also write:
"The vase over the new television is better."
"-4 -3 -2 -1 0 +1 +2 +3 +4"
" a n i a j n v r #"

In the latter case I am making a comparrison between (presumably) the vase and the television and I can say the sentence is negative to the word "new". In my mind I see the change from the infinitive to the verb as the driver for making "new" relevant to the meaning of the sentence.

Bonus question: In the second case does the adverb of the verb always determine the sentiment of the second noun?

Sometimes a drawing of a pipe is just a drawing

b33j — Mon, 23 Apr 2012 02:55:08 -0800

Do you know or can you find any scholarly articles about analysing adults' drawings that do not relate to health? I'm on a research project that is including a request for the participants to draw a picture in response to a question. I'm concerned that we don't have rigorous research to draw on in analysing these images. What I've found so far, going through Google Scholar is many works on: meaning of children's drawings (both healthy and with health and psychological issues); analysis of drawings by adults with health issues (including cancer, migraines, eating disorders, sexual abuse survivors); analysis of trained artists' works; analysis of media images (e.g. magazine covers in one publication over 25 years) but nothing to show a methodology of working out what ordinary adults may mean with drawings.

Things I have noticed about the drawings in question: some include profession-specific dress or tools, facial expression, differences in the relationship of the size of the drawing to the space they have to draw in, but I don't feel comfortable making conclusions from this.

Any suggestions of different ways to search (eg semiotics, semantics, etc.) also very welcome.

Looking for a blog, podcast, or website that dissects songs.

iftheaccidentwill — Sat, 21 Apr 2012 17:03:13 -0800

Looking for a blog, podcast, or website that dissects songs. I have a few friends who understand music much more fully than I do, and they can talk at length about the production of a song, the recording decisions, tunings, instruments, rhythms, and so on. I find this fascinating.

I'd love to find blogs (or whatever) that sort of dissect a song - tell the listener things to watch for, analyze different components, and so on. Genre doesn't really matter, though to be honest I'm not as interested in classical.

If you know anything like this, I'd love to hear about it!

What web survey tools can give analytical feedback to survey taker?

JiffyQ — Tue, 17 Apr 2012 02:53:00 -0800

Looking for a web based survey tool that can give feedback to survey takers at the end Hi Mefites: I need your help:

I'm looking for a web based survey tool that give feedback to the survey taker at the end on how they compare to previous takers.

I have a list of roughly 100 multiple choice questions, and at the end of the survey, the survey taker will receive a feedback report on how his responses stack up to others who have taken the test.

I know Zoomerang can do something like this with their 'poll' option, but can they do more advanced calculations of the data before giving out the final feedback report? I'm not just looking for simple feedback such as "20% of people who answered this question said A", but some mid level statistical analysis.

I doesn't have to be free, but that would be nice ;)

Thanks in advance!

Need an assist with some citation analysis

ryanshepard — Fri, 23 Mar 2012 09:08:46 -0800

Highly cited journals in online / distance education? To assist a faculty member, I need to know the top 10-15 most highly cited:

* Education journals generally
* If possible, distance / online education-related journals as well

We do not have access to Journal Citation Reports^* - can anyone assist with a search or suggest a reasonably comparable (2009-2010 stats would be acceptable, if that is all I can get) free resource?

* I'm aware of its limitations, but a rough estimate will be fine here.

Mad Men Analysis

crapples — Mon, 19 Mar 2012 15:31:52 -0800

I would like to find some high quality Mad Men analysis. Is it out there? I just started watching Mad Men. I'm at the end of Season 2. It's awesome, but I often feel as if I'm missing something. I remember back when I watched Lost there was an amazing site (called Lostpedia) that provided deep analysis of every episode. I know that Mad Men is not like Lost in that it's much more character driven. But still, there are smart music choices, symbols and metaphor, etc. It's a smart show.

However, when I search the internet for sites that provide deep analysis (as opposed to recaps) they are few and far between, and the few that are out there are pretty lame.

Is there anything better?

Walk me through a multivariate regression

PercussivePaul — Sun, 18 Mar 2012 07:36:15 -0800

Stats filter. I am doing multivariate regression for the first time and I want to understand what I'm doing, having gone beyond my formal training. I have many possible ways I could formulate the regression (different variables to include) and I want to find a model that is both the best possible fit while using the fewest number of variables. How? I have what seems like a straightforward regression set up. A given object i has inputs x_1 to x_n with coefficients A_1 to A_n, and output y_i. (I think this is multivariate regression, rather than multiple regression, because there are multiple outputs. Correct me if I'm wrong.)

In the first formulation of this problem the output is almost a direct linear combination some of the inputs (x), and I can use average coefficients which have some empirical basis. The problem I want to solve is that the inputs x_i have to be measured empirically, and some of these are much easier to collect than others -- also some of them are related to others, they are not entirely independent, and in fact some of them may be almost fully correlated -- and I want to show a good way to approximate y_i using only a few of the inputs. In particular, I want to determine analytically the best such model out of several possible models.

The answer has something to do with correlation coefficients and residuals and r-squared. I've been playing around in Matlab and am getting somewhere, but I don't have a high-level procedure in mind - just fiddling around and not converging on an answer. Can you walk me through the procedure you would use to test different models, identify which variables matter and which ones can be thrown out, and demonstrate that you have arrived at a good answer?

Salary Vs experience analysis

r2d2 — Sat, 03 Mar 2012 13:22:34 -0800

Need to do a simple analysis around pay and experience fairness. Example: Two project managers managing exactly same kind of projects and have similar performance levels: PM 1 -> Pay $130,000 ; Experience 15 years; PM2-> Pay $80,000 ; Experience 3 years; I want a mathematical way of analyzing if the person getting paid $80,000 is being under payed. Now I know there are industry standards and charts available. I want to do something that may not be perfect but gives a sense. How much weight should be given to experience and can this be factored in mathematically? Please dont get hung up on accuracy. I want a logical analysis method that does not need to be entirely accurate. Also, where can I find industry charts comparing pay and experience