Ask MetaFilter questions tagged with analysis

Automatic song identification/tagging?

mojoworkin — Sat, 19 Dec 2009 14:07:13 -0800

Any programs that can analyze and tag individual MP3 files that have NO metadata? I have a pile of untagged files, and so far no joy. I have a big, fat, messy pile of files, and would love a head-start on getting them tagged and filed. It seems there are some programs that are geared to ALBUMS, but I don't really care about album info as much as I am in accurate artist/title. There are iPhone apps that can listen and identify songs through the mic.. I have to believe there is a desktop program that would do this by the batch.

PLEASE HELP! Thanks!

What interesting statistical information can I dig out of this medical billing data?

neilkod — Thu, 19 Nov 2009 11:37:28 -0800

What interesting statistical information can I dig out of this medical billing data? I have 2000 rows of medical billing information (mysql) with the following attributes. I've normalized the data and here is a very simple description:

patient:
id
dob
city/state/zip
primary insurance provider
secondary insurance provider

provider:
id

procedure:
procedure id/code

patient_procedure:(fk to patient & provider & procedure)
procedure date
billed amount

Unfortunately, the data only spans one month so I can't provide anything interesting across a date dimension beyond aggregating certain measures by weekday.

I've already produced a lot of information such as
% of patients with no secondary insurance
avg age for procedure
top-5 most frequently occurring procedures with large avg cost
billing and frequency by zipcode
so on and so forth-measure X across dimension Y

I'm looking for suggestions for some deeper analysis than the many SQL aggregates that I've performed.

How do I automagically display trending of complex data?

Spumante — Mon, 26 Oct 2009 06:41:38 -0800

I need some suggestions for data analysis and displaying trending information. At the moment I'm doing this with a huge amount of Excel pivote tables. The goal is to highlight increasing trends and top hitters in some sort of (semi)automated way. The catch is that the data is pretty deep. For example, say I have a metric for "server rebooted for unknown reason"(which, by the way, is one of 50 things measured)...I'd like to also see the break down by server type and then by, say, location and then to see it by week. What I'm not looking for is a pointer to Tufte or "sparklines" or a generic point to something like "R".

How do I fit in to a new work culture without losing my mind?

soft and hardcore taters — Sun, 27 Sep 2009 16:54:05 -0800

I'm exhausted from trying to understand the working environment at a new job, and hoping the hive mind can help me figure out how to become better at this job. I started a new job two months ago doing business analysis for a growing company. It's a new position for the company (they've only created an IT department within the last year) and a new position for me (I've mostly done database and software development for teachers and librarians).

These aren't computer people. I don't expect them to know the intricacies of software and hardware or formal business terms to describe their requests, but I'm finding that I'm not very good at getting information out of the user base and I'm not fitting in with the corporate culture.

Here's examples of where we're clashing:
1. I received a one-sentence request from another new hire to "Install X software on his machine".
I can't find x software documented internally, or any x-ish install programs on any IT-owned servers. I send him an email asking what's X software and where can I find it? What's it used for? Do you mean y software? I get verbally spanked: asking for specificity was perceived as combative and rude.
2. I was tasked with documenting and expanding a series of linked spreadsheets, and warned that each one takes "like an hour" to open. I start working on the spreadsheets. Each one does take within 10 minutes of an hour to open, and I keep asking why the user base is expecting the work to be done in less time than would be possible to even open each spreadsheet if each one takes "like an hour". I get verbally spanked again: taking user timeframes at face value is perceived as pedantic and willfully stupid, because the reported user experience of taking an hour to open each was meant figuratively, not literally - I should have known and reported the speed as indicative of a larger problem.
3. I get a request to run a new data query from a department from a set of written specifications. They're not sure where the data is stored. Or what it should look like. Or if we even store each particular piece of information. I'm told to make a best guess at the content and pass it back to them for validation - and I hear back that the data looks good. Weeks later, I hear that the final result was both incorrect and incomplete. I get verbally spanked for expecting other departments to correctly validate information, and especially not to accept specs or validation from one particular individual to ever be correct.

Jobs are hard to come by right now and I genuinely like the people I work with (although I find the culture to be something that definitely takes some getting used to). Is this just a bad fit?

What method or type of software is best for collecting complex information for future analysis?

newlyminted — Thu, 20 Aug 2009 08:37:36 -0800

Lots of interrelated data, little idea of how to analyze it. What method or type of software is best for collecting complex information for future analysis? Specifically I'm looking to take quantitative information from medical histories and family histories to predict disease risk and look for other trends. The information being collected is from individual patients but will include information on their relatives who have diseases including the degree of relation and their ages.

Beyond a stats class years ago, my experience in this kind of research is limited, but I pick up things very quickly and want to learn the best way to approach this kind of problem, so I am willing to put in the time and effort. My predecessors approached similar projects using MS Excel, but they weren't trying to collect the ages of relatives as well as degree of relation, and I am quickly reaching the limits of excel's usefulness. Money should not be a big issue if I request software for this project as long as it is justified and will do the job. I'd like to approach this thoughtfully the first time around so that the information is in a useful format for my project and also available for other types of analysis in the future.

So, what is the best system to capture all of the data? If it's way too complicated for a beginner, who or what resources should I consult for advice?

Thank you for your suggestions!

A Graphic Equalizer for Images?

jfrancis — Sun, 10 May 2009 17:12:16 -0800

Is there such a thing as a graphic equalizer for images? Can I obtain it? In Photoshop you can high pass filter an image (edges and details) or low pass filter it (blur it). But there isn't continuous control over a range of frequencies.

Does some software exist that I as an average user can get a hold of that does the necessary Fourier analysis (or whatever is necessary) to give me fine control over any frequencies in an image I wish to address?

Sort of a graphic equalizer for photos?

Revenue per...eyeball?

xiaolongbao — Fri, 27 Mar 2009 10:03:24 -0800

Dotcom revenue statistics for dummies. I'm trying to get my head around how stats like revenue/page view, revenue/unique user, etc. are calculated and used. Help a dotcom novice understand some basics.

First off, I'm wondering if I use last twelve months revenue, how do I match that to unique users or page views? It seems the figures on sites like Compete and Quantcast are given in monthly terms. Would I look over the last twelve months and take an average of the monthly page views? Or would I just use the most recent statistic available (February '09 in this case)? Also, can anyone explain to me how Quantcast in particular calculates its "People" figure?

I'm looking at public and private companies, and obviously having more success finding info for the public companies. When I dig into their financials, often I find components of the revenue breakdown that I think should be excluded for an "apples-to-apples" comparison. That is, some companies have lines of business that fall well outside the area I'd like to learn more about, or are for a region I'd like to exclude from my analysis. Those I can easily ignore. But then there are other revenue segments like "advertising revenue" that could at least partially be attributed to the region of interest. Should I look at the regional revenue breakdown and allocate based on that?

Other suggestions on interesting statistics to examine along these lines also welcome, as well as the best source for this sort of information. If there's a book or site that covers this topic, please point me to it.

Thanks a million, Mefites!

How is sodium in foods calculated?

captain cosine — Sat, 21 Mar 2009 18:19:55 -0800

Question about nutrition information - For products like pickles and olives, which are usually packaged in jars with brine, does the total sodium content listed on the facts (amount per serving x number of servings) include the amount in the liquid? Also, for items such as tortilla chips, where a lot of salt and seasoning tends to fall off the chips and collect at the bottom of the bag during shipping, does the total sodium content include the stuff at the bottom? or do they instead take a serving size from the bag, as someone eating the chips would do, and analyze it that way?

Also, a distantly related question (linked only by nutrition labels as the subject): I know that beer companies are not required to put the nutrition info on the bottle itself, but are they required to do an analysis of their products and have the information available somewhere? I'd really like to know how many calories are in that bottle of Chimay Blue or Stone Smoked Porter.

How can I best visualize performance data by tenure?

samuelad — Sat, 07 Mar 2009 12:31:21 -0800

Excel charting help: Help me analyze sales stats based on salesperson tenure. I have an Excel file with the names, start dates, sales volumes, margins, and average unit selling prices for a sales team of 1,300 people dating as far back as 11 years.

I want to understand the performance differences between subgroups with different tenures (for example, how much better or worse on average do those with 1 -1.5 years experience perform than those with 2-2.5 years experience?).

I would really like to represent this data graphically in a chart. It seems like a histogram or a scatter plot with a trend line makes the most sense, but I am not a statistician and would describe my charting capability as “just above basic”. What is the best way to represent this stuff graphically and how do I build that chart?

Should I create a “sales performance index” of the sales metrics so that I can simplify the data into three variables: name, start date, and sales performance index? If so, how would you recommend creating such an index?

Radically remixed video

mildred-pitt — Tue, 03 Mar 2009 08:38:19 -0800

I'm looking for a website and video that described a man who wrote some software that radically remixed video. The software would take hours of video and analyze the audio from it. After the analysis, you were able to speak into a microphone plugged into the computer - and the machine would play back video with audio that most closely matched what you were saying. He used a Michael Jackson interview as one of the examples.

I think I may have seen this on Boing Boing a few years back - but fail when trying to search for it. Any ideas?

I'm having a financial existentialist crisis

MrMulan — Wed, 11 Feb 2009 09:36:19 -0800

Have I been wrong on my financial core beliefs? I have had a worldview when comes to finances for over 15 years that I am now questioning.

1. I believed 8-12% return on your money over the long term was reasonable, conservative, and realistic.
2. I believed in No-load mutual funds.
3. I believed in dollar cost averaging.
4. I believed Technical Analysis does not work.
5. I believed in A Random Walk Down Wall Street.

First there was this segment from NPR(of all places!) that blew my mind: Technical Analysis actually works (at least for oil)
Then this article from Slate on Suzy Orman. The article disses No-load, and Dollar-Cost Averaging without any references.

Was NPR sinking to a major network level of reporting? Or does TA really work?
What is the mounting evidence against No-loads as part of a passive, long term strategy?

There is no other choice for me but to be an optimist for the long term health and prosperity of our country and by proxy, the stock market. Else, the best thing to do would be to stock up and canned food and guns, and that really isn't a nice way for me to live.
Why then is Dollar-cost averaging bad, if you are doing it in mutual funds, and doing it for the long term?

There are many books out know calling into question A Random Walk Down Wall Street. Is there any truth to them?

Poetry Web Sites

Secret Life of Gravy — Wed, 04 Feb 2009 07:51:35 -0800

Internet resources for understanding poetry. My husband and I have Poetry Club every Saturday night. We choose a poem, give a short biographical intro, read the poem, and follow-up with as much insight and discussion as possible. It is the last part that is the most difficult, of course. We are surprised that there seem to be so few sources on the net that analyze and discuss poetry, modern and classic. Normally I just google the name of the poem which leads to a lot of wasted time because most often it is just a listing of the poem with no follow-up. Do you have any resources to share?

I did read this previous (2005) AskMe post.

How do i use Michael Porter's value chain analysis for an airline company (not aircraft)?

red_rika — Fri, 19 Dec 2008 01:14:58 -0800

How do i use Michael Porter's value chain analysis for an airline company (not aircraft)? Hi there, I am just wondering how do i associate the value chain analysis for an airline company? I am mainly focusing on the primary activities (inbound logistics, operations, outbound logistics, marketing & sales and service).

Thank you in advance.

LiteraryFilter: Meaning of "Buried Talents" by Richard Matheson

bangitliketmac — Sat, 29 Nov 2008 13:58:53 -0800

I recently read the short story "Buried Talents" by Richard Matheson (of "I Am Legend" fame (Amazon link). What does it mean? Short summary: a strange man is uncannily successful at beating a carnival game; when he leaves, the man running the game (who had been getting flustered), appears very ill. Extended summary: a strange man in a dark coat beats a carnival game by consistently tossing ping pong balls into a container. The fat man running the game gets flustered as the man keeps on winning; he won't take a prize and instead just wants more ping pong balls. As the game progresses, the fat man sweats more and more and gets into worse and worse shape; his voice gets a bit faint and it seems like he's having a heart attack or something. The strange tall man eventually gets shooed away by by the fat carnival man, and leaves. The crowd supports the strange man who keeps winning, and is angry when the carnival man makes him stop playing, and is also angry to discover that most of the "prizes" aren't really available, they are "display only".

When the strange man leaves, the prize of steak knives are gone. The fat carnival man can barely utter a whisper as he offers a chance to play the game to some kid. He says "God" and doubles over in pain.

My theory is that the strange man is a metaphor for death and as he keeps landing the ping pong balls in the container, he is taking life away from the fat carnival man. He may be punishing the man for his somewhat unethical carnival game, which is designed to attract players but not really to reward winners. The "God" comment and the steak knives are obviously important. Also, the fat carnival man at some point says he's running an "honest game" and the tall man better not be trying anything shady. I'm not sure how the title ties in other than the fat man probably gets "buried" shortly after the tall man's display of talent with the ping pong balls. I think there is more meaning to the story than just a simple "death" metaphor.

Thoughts?

How do I get started with data visualization?

icarus — Thu, 20 Nov 2008 11:41:31 -0800

I'd like to figure out how to do good looking data visualization for reports beyond Excel charts, but without any coding/programming skills except xhtml/css. Bonus points for also listing some data visualization techniques or websites that you find useful, attractive, user friendly, etc.

Help me find a shot-by-shot analysis of famous movie scenes.

Cool Papa Bell — Fri, 10 Oct 2008 12:01:55 -0800

Help me find some shot-by-shot breakdowns and analysis of famous movie scenes, with examples of the shots! This is the only good example I can find of what I'm looking for. I want examples from relatively mainstream movies, preferably with action. Bonus points if the site has examples of the same moment from different camera angles.

What is the linux command 'dt'?

logicpunk — Mon, 06 Oct 2008 10:05:17 -0800

Obscure linux/unix text processing command: what is 'dt'? I'm trying to figure out what one of my boss's data analysis scripts is doing, and I'm hung up on this weird little command called dt. There aren't any man or help pages for it, and googling around hasn't come up with anything.

The script I'm looking at reads in a formatted data file ('tmp') with X numbers of columns. Each row is a consecutive point in time. The line I'm stuck on reads as follows:

cat tmp | dt t 1 t 2 t 3 d 12

The numbers specify columns, but I'm not entirely sure what the 't' and 'd' arguments are doing there. I suspect the script is sorting the data in some fashion, and I could figure it out eventually (or, god forbid, ask the boss), but I'd like to know what the command is and where I can find more information about it.

Bail them out? Or let them reap what they sow? Who wins and loses?

jeanmari — Mon, 22 Sep 2008 08:04:45 -0800

Who will the current bailout plan benefit and how? I've been trying for the last few days to untangle the implications of this crazy expensive plan.

As the rest of you have seen, it's daunting.

Any resources out there which can break it down for me? The pro's and con's of this bailout versus the pro's and con's for just telling Wall Street, "Sorry, you screwed up. You're on your own."

Trying to find a set of pro's and con's for each category of the players, like:

- Executives of the companies involved
- Investment bankers
- Investors in the companies involved
- Regular Main Street banks that hold deposits
- Taxpayers
- etc.

Principal components factor analysis and scale construction (PCFASCfilter) again

proj — Mon, 18 Aug 2008 17:43:30 -0800

Principal components factor analysis and scale construction (filter) again. When constructing a scale based on results of principle components factor analysis and components have differing signs, yet there is a theoretical reason for the variables to be coded as they are, what to do?

Example. I have four variables. For each of the three variables, higher scores indicate more "desirable" scores for societies on measures of development. The fourth variable is a macro-level variable that indicates the percent of a society that is comprised of people with a certain characteristic. The factor loadings look like this:

Var1: .7588
Var2: -.8355
Var3: .7138
Var4: -.7984

When constructing the scale, do I need to recode the variables with negative signs so that all items have positive loadings?

Statistics Filter: Finding the top 10% from thousands of data points. Bonus question: inflection point value.

lrivers — Thu, 07 Aug 2008 09:49:00 -0800

Statistics Filter: Finding the top 10% from thousands of data points. Bonus question: inflection point value. Statistics Filter: I have thousands of data points (ranging in value from 1 to 0, 8 decimal points) representing a score. I graphed the data set and it's a classic long-tail distribution--looks like a hockey stick. How do I find the value of the "inflection point" and also the value/position of top 10%, in other words, which scores are in the top 10%. I've googled my little heart out and think I don't know the right questions to ask. Thanks, wise ones!

A nowhere continuous derivative?

thrako — Fri, 01 Aug 2008 19:23:47 -0800

Is there a differentiable function f(x) such that the derivative is nowhere continuous? That is, f'(x) has no point of continuity? Googling has led me to believe this does not exist, though I can't find anything like a proof, only vague references to Baire classes.

Can you tell me where I would find a proof?

good movie analysis websites

clueless22 — Fri, 25 Jul 2008 08:26:16 -0800

Are there any good website where they analyze a movie thoroughly just as though you are in a film class (from cinematic, philosophical, psychological, etc. perspectives)?

Is TV real?!?

parkerjackson — Wed, 16 Jul 2008 05:54:18 -0800

What are other sites that tell me about the "realness" of television shows? I love sites that analyze the real-world consequences or accuracy of television shows. I've found sites about the litigation problems Michael Scott would face on The Office, and I've read about how shoddy the medical facts are on House.

What are some other sites that analyze television shows in this way? Specifically, I'd like to read an expert's analysis of The Closer, but really anything similar would be interesting. The key is they have to follow through specific episodes and tell me what's legit and what's total fiction.

As a small aside- if you saw the new episode of The Closer on Monday, was their disgusting method of getting fingerprints based on any part of reality? Thanks!

Questions about Crystals

Large Marge — Sat, 05 Jul 2008 11:03:41 -0800

How do we know what makes up a gemstone? And did people know in the previous centuries when (I assume) these techniques weren't around I just finished reading The Blue Carbuncle, and in it Holmes says something to the effect of "all this fuss over a small piece of crystallized charcoal." I know that rubies are just doped sapphires, which is just crystal aluminum oxide, so my impression is that the boundary between the two might not be hard and set, yet there are all these gem stones and tons of different names and chemical compositions, so how do/did we determine what something was? How did Holmes know that diamond was carbon (although apparently he was wrong about the carbuncle, which is what got me wondering)?

Also, is there a good natural history book about all this?

Textual Analysis Primer/Examples?

signal — Fri, 20 Jun 2008 21:12:50 -0800

Looking for a general background/framework within which to define the more technical aspects of a textual analysis project, as well as interesting examples of computer-driven textual analysis. I am participating in a research project on textual analysis of historical architectural magazine articles.
We'll be inputting the titles and maybe text of articles from architectural magazines from 1930 to 1960 into a database and looking for patterns.
I am wholly ignorant of textual- or content-analysis, and am sure there's a great deal of literature and interesting analyses that other smarter people have already thought of.
I'm naively thinking word frequency, adjacency and correlation (i.e.: what percentage of articles that mention 'le corbusier' also mention 'modernism' in as given year/magazine), to figure out links between articles. etc. This could spin of into graph-theory measurements.
Our interest is in the actual content, keywords, etc., not so much in sentence structure, grammar, parsers, etc.
I program in Python, mostly, and am in charge of the 'hard' numerical part of the project (there's 3 architectural history geeks in charge of the soft conceptual part).
Books are nice, but something online might be quicker.
As we are architects, interesting visualizations are always welcome.