Fine-tuned searching - how is it formed?
September 14, 2012 6:39 AM Subscribe
What kind of search engine is this, and is it something which one has to code themselves or which websites get someone to set up for them? How does it work?
Let's say the site is a website that has a database of baked goods you can buy, and you can go on it to find the cake or pie you'd most like. Someone has the following options to chose from:
- price of the item (this will be the 'main' factor in the search) which would be a min/max price range (say someone wants a cake for between £1 - £1.50)
- which supermarket chain you want to go to (this will be the second most-important factor - it will pull up all the cakes from that chain in that price range)
- whether it contains one or more of: chocolate, cherries, nuts etc. (within the first two categories)
- whether it is one or more of: gluten free/low fat/vegan etc. (within the first two categories)
Someone can use this search engine to, say, find a cake recipe, which contains chocolate and nuts, and is vegan, by choosing from the options. So is this a common enough type of search engine that a cheapskate person can adopt an existing one to their needs for no cost? Or is the search critera involved too complex?
Let's say the site is a website that has a database of baked goods you can buy, and you can go on it to find the cake or pie you'd most like. Someone has the following options to chose from:
- price of the item (this will be the 'main' factor in the search) which would be a min/max price range (say someone wants a cake for between £1 - £1.50)
- which supermarket chain you want to go to (this will be the second most-important factor - it will pull up all the cakes from that chain in that price range)
- whether it contains one or more of: chocolate, cherries, nuts etc. (within the first two categories)
- whether it is one or more of: gluten free/low fat/vegan etc. (within the first two categories)
Someone can use this search engine to, say, find a cake recipe, which contains chocolate and nuts, and is vegan, by choosing from the options. So is this a common enough type of search engine that a
Response by poster: Yes! So first it takes price, and retailer, and only selects things that are within these categories alongside the extra attributes.
Is this the kind fo search engine websites get someone to build for them? Or are there customizable versions out there that have this capability? I know forums have search functions which allow you to search within certain parameters but I was thinking about something for more of a website.
posted by mippy at 7:02 AM on September 14, 2012
Is this the kind fo search engine websites get someone to build for them? Or are there customizable versions out there that have this capability? I know forums have search functions which allow you to search within certain parameters but I was thinking about something for more of a website.
posted by mippy at 7:02 AM on September 14, 2012
Its relatively simple if your database is set up for it. If you're using a custom built database you'll likely need a custom built search. If you're using an off-the-shelf database then you'll need a solution written for that database.
Presumably your database already has price and supermarket stored as separate information on the cake/pie record. They may already be categorised by dietary requirements. Its less likely that specific ingredients are stored in a neatly searchable way unless the database was set up for this to begin with.
posted by missmagenta at 7:04 AM on September 14, 2012
Presumably your database already has price and supermarket stored as separate information on the cake/pie record. They may already be categorised by dietary requirements. Its less likely that specific ingredients are stored in a neatly searchable way unless the database was set up for this to begin with.
posted by missmagenta at 7:04 AM on September 14, 2012
That sounds like you'd be looking for faceted search. Faceted search uses a taxonomy to allow for search results to be broadened or narrowed by the user. There is quite a good description of it in this article.
I don't know of any out of the box (or indeed free) methods because the taxonomy would need to be developed for each site (i.e. your cake recipe site would need a taxonomy that covered all the things like dietary requirements, flavourings etc. and then the recipes in the database would need those terms applied to them, probably in the metadata.)
tl;dr: You probably want Faceted search; I don't think it would be something you could easily find for free. (but I'd be interested if you did find anything that would let you do custom facet search for free)
posted by halcyonday at 7:06 AM on September 14, 2012 [1 favorite]
I don't know of any out of the box (or indeed free) methods because the taxonomy would need to be developed for each site (i.e. your cake recipe site would need a taxonomy that covered all the things like dietary requirements, flavourings etc. and then the recipes in the database would need those terms applied to them, probably in the metadata.)
tl;dr: You probably want Faceted search; I don't think it would be something you could easily find for free. (but I'd be interested if you did find anything that would let you do custom facet search for free)
posted by halcyonday at 7:06 AM on September 14, 2012 [1 favorite]
Solr and Elasticsearch are both open source projects that can do faceted search as well as full-text search. But they're backends only; you would still need to build a user interface on top of them.
I'm not aware of any ready-to-use search engine frontends that can do what you want, but it wouldn't surprise me if such a thing exists.
posted by teraflop at 8:20 AM on September 14, 2012
I'm not aware of any ready-to-use search engine frontends that can do what you want, but it wouldn't surprise me if such a thing exists.
posted by teraflop at 8:20 AM on September 14, 2012
Response by poster: I should have said that I am not a programmer or coder so I'm not sure what you mean by backend or frontend. I thought my terrible explanation would make my lack of knowledge obvious!
halcyonday - I am picturing it in my head as a taxonomy - I could draw the tree diagram showing exactly what I'd like - I just don't know what that means in coding terms.
posted by mippy at 8:32 AM on September 14, 2012
halcyonday - I am picturing it in my head as a taxonomy - I could draw the tree diagram showing exactly what I'd like - I just don't know what that means in coding terms.
posted by mippy at 8:32 AM on September 14, 2012
If the database of baked goods is properly designed, and all the data that should be there is there (eg all goods containing chocolate have been tagged 'chocolate'), this type of search should be pretty easy to provide. I mean, this is what a database does. But I don't understand how you are going to get a database-driven e-commerce website that works without getting a specialist involved at some point. And if you are talking instead about a bunch of text pages and wanting to search within those, the task becomes far more complex to achieve.
posted by Acheman at 9:19 AM on September 14, 2012
posted by Acheman at 9:19 AM on September 14, 2012
a front-end is what you (as the user) would see, so the search field/the tickyboxes. The back-end is what the user doesn't see - the database that stores all the info. The front end would send the search string to the back-end, which would process the search and then return the results, which the front end would then display back to the user in a user-friendly manner.
In code terms, your taxonomy would be the keywords (and metadata) of the item. If you, say, take your local library catalogue. What you see is the front end and when you put a keyword (say, science fiction) in to the search box, the database behind the front-end will look for all book records inside that have the term 'science fiction' in the keyword field and then return a list of what matches to the user. (nb. this is a very simplified view of how a library catalogue works! ;))
Without being a programmer, I think it would be quite hard to roll your own faceted search as acheman says, the database needs to be properly designed - they're garbage in-garbage out systems - before you can put a search on top of it.
posted by halcyonday at 10:09 AM on September 14, 2012
In code terms, your taxonomy would be the keywords (and metadata) of the item. If you, say, take your local library catalogue. What you see is the front end and when you put a keyword (say, science fiction) in to the search box, the database behind the front-end will look for all book records inside that have the term 'science fiction' in the keyword field and then return a list of what matches to the user. (nb. this is a very simplified view of how a library catalogue works! ;))
Without being a programmer, I think it would be quite hard to roll your own faceted search as acheman says, the database needs to be properly designed - they're garbage in-garbage out systems - before you can put a search on top of it.
posted by halcyonday at 10:09 AM on September 14, 2012
You are describing some things that are not be off-the-shelf options, and others that are.
Let's go with off-the-shelf first. Faceted search (as opposed to free-text search where you type in a bunch of words and come back with matches) is something you can have with any database-backed web site.
Sphinx is a well-regarded search engine where you have structured data (prices, stores, etc) and want to provide the ability to search over these elements.
You can also implement the search in the database itself, if you want. Oracle and MySQL and other databases give you the ability to query the database, and that's all a search is. Put a user-friendly interface on the most common queries, et voila, you can sort by price, select for a particular store, etc.
Now the hard part which may not be off-the-shelf is getting the data into a searchable, up-to-date form.
In your example, if it were a search over in-stock store inventory, and you didn't control the databases for the individual stores, implementing your search engine such that it's aware that a store no longer has an item, or is out of stock, requires an integration with that store's (and every other store's ) own data.
That is generally not free or off the shelf. You may have to establish a business relationship with the chain of stores, then set up a process for copying their data to your database, or alternately a way of integrating your query system with theirs so that to the user a query appears to be coming from one integrated site rather than many (this is called "federated search".)
And if you can't establish a business relationship, you have to resort to crawling their website and scraping the factual data from web pages (see for instance PadMapper wanting to do map search for Craigslist apartments. Craigslist would not give them access, so Padmapper had to figure out an alternative technical solution (which was then shut down by Craigslist's lawyers).
If your search doesn't involve proprietary data, but is more like a search over things one might see in Wikipedia, the Freebase would be a solution. Any user can edit the data on Freebase, and any user can create or modify schemas;whether those changes are promoted to canonical is another matter, but it gives you a huge sandbox for trying out search and data representation, so long as the data would be broadly interesting to the public (not interesting: your store inventory; interesting: recipes for brownies).
posted by zippy at 11:58 AM on September 14, 2012
Let's go with off-the-shelf first. Faceted search (as opposed to free-text search where you type in a bunch of words and come back with matches) is something you can have with any database-backed web site.
Sphinx is a well-regarded search engine where you have structured data (prices, stores, etc) and want to provide the ability to search over these elements.
You can also implement the search in the database itself, if you want. Oracle and MySQL and other databases give you the ability to query the database, and that's all a search is. Put a user-friendly interface on the most common queries, et voila, you can sort by price, select for a particular store, etc.
Now the hard part which may not be off-the-shelf is getting the data into a searchable, up-to-date form.
In your example, if it were a search over in-stock store inventory, and you didn't control the databases for the individual stores, implementing your search engine such that it's aware that a store no longer has an item, or is out of stock, requires an integration with that store's (and every other store's ) own data.
That is generally not free or off the shelf. You may have to establish a business relationship with the chain of stores, then set up a process for copying their data to your database, or alternately a way of integrating your query system with theirs so that to the user a query appears to be coming from one integrated site rather than many (this is called "federated search".)
And if you can't establish a business relationship, you have to resort to crawling their website and scraping the factual data from web pages (see for instance PadMapper wanting to do map search for Craigslist apartments. Craigslist would not give them access, so Padmapper had to figure out an alternative technical solution (which was then shut down by Craigslist's lawyers).
If your search doesn't involve proprietary data, but is more like a search over things one might see in Wikipedia, the Freebase would be a solution. Any user can edit the data on Freebase, and any user can create or modify schemas;whether those changes are promoted to canonical is another matter, but it gives you a huge sandbox for trying out search and data representation, so long as the data would be broadly interesting to the public (not interesting: your store inventory; interesting: recipes for brownies).
posted by zippy at 11:58 AM on September 14, 2012
This is also called a "parametric search". Judging by how different-looking every website's version of this is, I don't think there are any common out-of-the-box implementations of it. Conceptually it's straightforward, but my guess is that every website wants a little bit of domain-specific smarts sprinkled in.
I think it'd be a straightforward thing to hire a person to build for you, though.
posted by hattifattener at 12:04 PM on September 14, 2012
I think it'd be a straightforward thing to hire a person to build for you, though.
posted by hattifattener at 12:04 PM on September 14, 2012
« Older Cooking 101: Cooking for the on-the-go student. | I've been reading about locksmith scams Newer »
This thread is closed to new comments.
posted by Grither at 6:49 AM on September 14, 2012