Ask MetaFilter posts tagged with searchengines

Extra Characters in Title Meta Tags

napdynmite — Wed, 16 Jul 2008 08:07:04 -0800

Does adding symbol characters to your title meta tags hurt your SEO? Our company has a title format of "-- Our Web Site : A Page --" and since we're going through a redesign I was wondering if it might be a good time to drop that convention.

Identifying forums on specific subjects

zaebiz — Mon, 09 Jun 2008 13:48:31 -0800

What is your formula for finding the most active online forums for discussions about very specific subjects? Benchmark subjects to test your formula out are - solar electricity, icecream lovers, driving a taxi, the Weimar Republic, analysing dreams. The forums needn't be dedicated to the particular subject but they should be the sort of forum for which information about the subject is abundant.

Seeking the Internet Clueless

gusandrews — Sun, 27 Apr 2008 10:49:10 -0800

Have you seen blog threads where commenters seem not to have read the initial post, and are way off topic? I'm seeking more examples of this for my dissertation research. Looking for blogs which have garnered comments similar to those mentioned on the earlier MeFi threads Tuesdays with Maury, Jeremy Jordan Loves Demon Dogs, and How Hawkish. Basically, commenters show up and start commenting as if they haven't read the original post at all, and they don't seem to have any idea what the discussion is about. In many cases they seem to think they're talking to a celebrity, or they are just seeking information in the wrong place. Has this happened on your blog or on blogs you've seen? If you need a better sense of what I'm looking for, check my profile for a link to a page where I'm collecting these.

Statistically comparing different search engine results

carter — Wed, 06 Dec 2006 08:12:57 -0800

Stats101Filter. I have (I think) a stats question, but little stats knowledge. Problem: The same library, and two different information retrieval systems - A and B - incorporating different metadata and search engines. I search the library with each IR system, using the same query. Is there a way to compare the similarity or difference between the different results sets I get from each system, and also assign this difference some statistical significance? For example, if I search for 'frogs,' I could get 'All about Frogs,' 'Lifecycle of the Frog,' 'Florida Frog Cam,' 'Cool Frog Pics,' etc., as results. I can see a range of scenarios for comparing A and B.

H0 - There is no difference
Scenario 1 - A and B return the same results, in the same order

H1 - There is a difference
Scenario 2 - A and B return the same results, but in different order
Scenario 3 - A and B return at least some different results
Etc.

For various reasons - basically we are replacing one engine with another that we think is more efficient and scalable - we are hoping for Scenario 1. However, we are worried that we may encounter Scenario 3. So the question is, how can we calculate any 'difference' we might encounter in Scenario 3, and how can we decide whether or not this difference is significant (and in real-world terms, likely to confuse users if we do switch our IR systems). Phew! (And thanks!)

You are here. We are not.

snailer — Tue, 17 Oct 2006 15:51:53 -0800

Why do some search engines allow you to search blank queries? Most search engines such as google, yahoo etc. allow you to search for nothing. You can enter no text into the box, then it searches for it. Sometimes the browser is brought to another page (e.g. google goes to a search results page, with just the footer), and sometimes it just refreshes the page.

Why do they do this? If you capture the data it shows a blank query going out to the server. Surely it would be less bandwidth- or cycle- intensive to highlight the error in script or similar. Does Google's API allow blank queries also?

Hardly a life-or-death question, but it's the internets, y'know.

Image search engine scraper/downloader

Moistener — Tue, 16 May 2006 22:08:34 -0800

I need a web scraper script/proggie to download and thumbnailize images for a long list of search terms. .

Input: Long list of search terms.

Output: corresponding images and thumbnails on my hard drive, arranged/accessed/organized by search term.

My already feeble google-fough (<-- note ignorance of even the spelling) is failing me.

Finding the cheapest airfare to Europe.

kimdog — Thu, 04 May 2006 07:15:41 -0800

Is there a travel search engine that will search airfares to multiple cities at once? I am planning a trip to Europe in August and so far, I am chagrined by the price of airfare. While I will eventually need to end up in Amsterdam, I am open to flying into a cheaper city and then making my way there. Is there any travel search engine that will let you search on multiple cities at once, or could help me zone in on the cheapest airfare to central Europe during a specific time frame? Randomly putting in major cities is not very efficient. I'm familiar with all the biggies- Orbitz, Travelocity, Expedia, Yahoo... but unless there is a hidden feature, I haven't found a way to do this kind of non-place specific search.

Why the discrepancy in search engine market share results?

helios — Fri, 24 Feb 2006 19:04:46 -0800

Every time i see statistics like this one which list market share for various search engines, I'm somewhat surprised. According to the article/graph, in mid-2005, Google had a 36.5% market share in online searches, with Yahoo and MSN at 30.5% and 15%, respectively. But on every website I've managed for myself or others (and on the sites of everyone I've ever spoken to about this), Google consistently accounts for 80 - 90% of all their search referrals. So, what accounts for the discrepancy? Is it just because my evidence is anecdotal, or is there something special going on in which Yahoo/MSN searchers get sidetracked before ever reaching their search results? I understand that, for example, Yahoo! Sports searches count toward Yahoo's overall total, as explained in the article. But I would assume that would be evened out by the alternative searches Google provides as well (groups, froogle, print, news, etc).

Nielsen claims Google has a 48% share, which is closer, but nowhere near 80% - 90%.

Help Me Unlove Google

alms — Mon, 13 Feb 2006 19:59:08 -0800

What search engine should I use instead of Google to protest their agreement to provide a censored product for China? Unfortunately Yahoo and MSN are not options because they are apparently similarly evil.

Which are the best search engines that aren't Google or Yahoo?

reynir — Sun, 29 Jan 2006 03:55:48 -0800

Which are the best search engines that aren't Google or Yahoo? I want to find one that is good for searching but which also isn't in the business of turning this into this. Dear Ask Mefi. I'd like to avoid using Google where possible. Yes, I know this won't make any difference to anything but it will make me feel better. I'd appreciate recommendations for other search engines that deliver good results and aren't run by companies that are snuggled up all cosy in bed with murderers, conspiring to help them better hide their crimes. So that's Yahoo! out, for starters. Looking around over the last day or so, Gigablast and Clusty seem to be getting a few recommendations. Anyone here used them much? Any other recommendations?

Regular expressions to describe incoming search engine URLs?

WCityMike — Thu, 05 Jan 2006 09:56:13 -0800

HtAccessFilter: I’m seeking regular expressions to describe URLs that encompass search engines’ image and video search engines, but not the engines themselves, so that I can block said image and video search engines using “SetEnvIfNoCase Referer” in .htaccess. Also, I’m also seeking to block all incoming requests for one particular URL which is popular with these video and image search engines but which no longer exists – but I don’t want to serve them the normal 404. To be specific, I’m seeking to block most image and video search engines from my website, while not excluding the main search engines themselves (e.g., I want Google Images and all its regional variations blocked, but not Google itself; Yahoo Video Search, but not Yahoo; etc.). I managed to get a good variation for Google Images:

SetEnvIfNoCase Referer "^https?://(www\.)?images.google.(ae|at|be|ca|ch|cl|co\.hu|co\.il|co\.in|co\.jp|co\.kr|co\.nz|co\.th|co\.uk|co\.za|com|com\.ar|com\.au|com\.br|com\.fn|com\.gr|com\.hk|com\.mx|com\.my|com\.ph|com\.pr|com\.ru|com\.sg|com\.tr|com\.tw|com\.ua|de|dk|fi|fr|gr|ie|it|lv|nl|pl|pt|ro|se|sk)" DumbSearchEngine=1

However, I did that not through my own very bad knowledge of regular expressions, but by emulating what I saw elsewhere. I just don't have the know-how or the adequate reference to stave off the other search engines, really.

I’m now hoping others have worked out similar ways of describing and/or blocking things like AltaVista Video, Yahoo Video Search, and so on, without blocking Google, AltaVista, and Yahoo themselves. I also do not want to go too wide by blocking anything with ‘images’ or ‘video’ in the name itself, for example.

I then can direct them to a Forbidden error, so that they never even hit my StatCounter, which is the ultimate goal. I'm not a heavy multimedia website -- for some reason, if I am linking to an MP3 or AVI on another website, Yahoo, AltaVista, Google are all deciding that I'M hosting the image and linking their image- or video-search to me. Bastards.

Thanks.

How to monitor search results ranking?

mattwatson — Sun, 20 Nov 2005 16:03:00 -0800

Does anyone know of a free piece of software (or internet site) that can monitor a web site's ranking in various search engines using various keywords? Ideally, I like to find a free option. I've come across several in my initial searches, but they all cost money, and I have no idea if they're worth it. Has anyone used something like this?

How best to migrate from one domain to another?

AmbroseChapel — Fri, 11 Nov 2005 14:59:06 -0800

How best to migrate from one domain to another? Let's say I have a website at project.foo.com and I want to move it to its own "project.com". At some time in the future, project.foo.com will cease to operate.

How do I best migrate? And by best I mean "with the least, shortest disruption to visits from both search engines and human beings"

I can redirect by

changing the content to inform visitors of the new URL

using HTTP-EQUIV tags in pages

sending 301 HTTP headers from a script

using .htaccess files

changing the DNS entries

but which should I do, and when?

Let's say I absolutely don't want to lose a single hit from regular human browsing. I can imagine a very careful long-term transition like:

keeping the content, but adding a message to each page on the old domain: "project.foo.com/bar.htm will shortly move to project.com/bar.htm, please note"

moving the content and leaving only the message on each page: "project.foo.com/bar.htm has moved to project.com/bar.htm, please note"

after a certain period, removing all the pages, and redirecting any hit anywhere on project.foo.com to a "this whole domain has moved" page.

That seems like a good strategy for keeping my human hits, but it looks like a bad strategy for Google, etc.

So, say I just redirect the DNS? Or make every page issue a 301?

Presumably Google and other such agents (after a few hours downtime while the change propagates) will "see" the change, and redirect their links/update their database as required. But the humans might not have noticed. They ask for the content, they get the content. Why should they notice a new URL in the URL bar? So, they keep using their old bookmark and they haven't noticed the migration.

Should I attempt to to detect the fact that they came to project.com via a request to project.foo.com, and tell only those people that they should update their bookmarks?

Bonus question: am I over-thinking this?

The economics of bogus search engines

elgilito — Wed, 22 Jun 2005 04:24:27 -0800

What are the economics of bogus search engines and other oddities found by regular search engines? Whenever I look up my website name (it's a unique made-up name) in Google or another legit search engine, some of the results are just odd: bogus search engines, pseudo sex/dating/tax/furniture/etc websites that look like datingsite.com/myfirstname.html or bodypart-enlargment.bogussearchsite.com. I understand that these "sites" are only there to get money, and I've read about spamdexing and other internet scams and but I still don't understand the economics of it. Who is paying whom for what? Is someone getting real cash from having my website (which doesn't have a large traffic) listed in a totally irrelevant way? And who is getting scammed, since no real service is provided? Also, are we talking about large sums of money or small change?

Evaluating Lucene

MrMoonPie — Wed, 27 Apr 2005 05:36:13 -0800

I’ve been assigned to evaluate the feasibility of using Lucene for our website, a large, high-use government site with rapidly-changing data. It’s not the only search engine we’re evaluating, but it’s the one I’m looking at. Anything I should know?

I know this is a very general question, but I’m not particularly technical, and I’ve just started looking into it, so I don’t even know what questions to ask.

Question number 10821

SpaceCadet — Mon, 11 Oct 2004 09:28:57 -0800

GoogleBombing: Suppose I register 10 domains, and upload an HTML page to each domain that has a named link - for example, "Click here to buy Cheap Shoes" - and these named links all point to the same webpage....when someone searches on Google for "Cheap Shoes", will the page that has 10 domains linking to it stand a better chance of being higher up in the rankings?

Question number 8095

jester69 — Sun, 20 Jun 2004 09:04:27 -0800

Did anyone else notice Google search behaving oddly yesterday? (more inside)

Question number 7529

blueshammer — Wed, 26 May 2004 11:30:29 -0800

If I want to know where a certain website ranks for a particular Google search, is there anything I can do other than just scroll through every page of results until I find that page? For instance: This is on the 29th page of Google results for "Ask Metafilter." If I was Mr. or Ms. Ofrenda, would there be a faster way for me to learn than that simply looking on every page?

Question number 7481

o2b — Mon, 24 May 2004 18:14:27 -0800

Search engine optimization... I'm happy to buy a book, but are they worth it? Do I really need to know more than the basic best-practices for Google?

Question number 6946

billsaysthis — Sun, 02 May 2004 16:07:55 -0800

Is there a way on Google to have it exclude shopping sites from a search? If I search on a book or movie title, or hotel name, most of the results are just generic sales pages and not very informative. Willing to use a different search engine if I can get the desired results.

Question number 5944

ass — Thu, 18 Mar 2004 21:16:29 -0800

Questions about search engines: 1) What if you regret something you say on your website, say bad-mouthing an ex-girlfriend, is there ways of sending this down the memory hole? You could delete the statement, but how do you get rid of search engine caches? And how many different engines are collecting these caches? 2) what if you say something like this on another website - is there any way to at least get Google to stop showing it, even though you don't run the site?

Question number 5707

MsVader — Mon, 08 Mar 2004 08:21:29 -0800

Are there any location based retail search engines out there? [more inside]

Question number 4783

vacapinta — Tue, 20 Jan 2004 17:28:00 -0800

Is there an image search engine that allows me to take a given image and see if there are other copies online (the image name may be different and perhaps even resized?) (more inside)