Ask MetaFilter questions tagged with datawarehouse

Oracle Views: Are calculated columns always evaluated even when they are not considered in a query?

NailsTheCat — Tue, 21 Apr 2009 14:14:16 -0800

Oracle Views: Are calculated columns always evaluated even when they are not considered in a query? I'd like some advice regarding Oracle views. Are fields always evaluated regardless of whether they are required in the results set?

Let's say I have tables PRODUCTS and ORDERS. And I create a view called PRODUCTS_VIEW which consists of something like:

SELECT
PRODUCT.ID,
PRODUCT.NAME,
fcnGetMostRecentOrderDate(PRODUCT.ID) AS MostRecentOrderDate,
fcnGetTotalOrdersCount(PRODUCT.ID) AS TotalOrdersCount,
fcnGetMaxOrderVolumn(PRODUCT.ID) AS MaxOrderVolumn
and so on
FROM
PRODUCTS,
ORDERS

I.e. a flat view that encapsulates all possible attributes--I estimate around 20 static and 60 calculated fields.

This is for reporting purposes, either queried directly or perhaps wrapped up in an API. Maintainability is important, if I need a new attribute then I can easily add a new field as necessary. Performance is certainly a factor but, since it's only for reporting purposes, there is a degree of flexibility.

If I run a query that only considers 2 or 3 fields e.g.

SELECT PRODUCT.NAME, MostRecentOrderDate
FROM PRODUCTS_VIEW
WHERE... say....CLIENT = Heinz

Would it still evaluate *all* the other calculated fields? (I think it does unfortunately: and I'm concerned that performance would degrade as new fields are added.)

If it *does* behave thus is there a way around it? I suppose a materialized view is an option. Or is this an inappropriate use? I am avoiding dynamic SQL since I would rather consolidate the logic into something that can be made (easily) accessible through ODBC, ADO, a bundle of Web Service methods etc. Note that a data warehouse approach (e.g. a daily dump of denormalised data) is not appropriate since data updates need to reflected immediately.

Thanks.

help a layman understand multi-dimensional data modelling

goethean — Wed, 26 Dec 2007 12:02:49 -0800

I'd like to know how multi-dimensional data modelling works. My company has Cognos BI and I am working closely with the developers. I would like to understand how a star schema works, fact and dimension tables, etc. I'd like to be able to more clearly envision whats going on with the data. How is it different from relational databases? Can you recommend a book or some online articles written for the layman?

I don't necessarily need to understand how *to* model the data, just how to understand data retrieval *from* a model.

Datawarehouse 101

jontyjago — Tue, 28 Aug 2007 02:22:50 -0800

I am changing jobs in a couple of months, and need to know more about database design, and more specifically datawarehouse and datamart for reporting systems. I will be working on the ETL and reporting side, but would like to sound more knowledgeable about general database design techniques, such as normalisation, star schemas and so on, with particular emphasis on datawarehousing. The last time i looked at this Ralph Kimball was the datawarehouse reference point. Would it still be worth reading his Lifecycle Toolkit? Or is there somebody new out there that is a must-read? Online or hard-copy recommendations would be most welcome.

Datamining the public web

ilike — Mon, 30 Jul 2007 01:52:51 -0800

How do i build a data warehouse that scrapes data from public websites for my own use? Tools? Tips? Hi. I would like to track apartments on a classifieds site and use the data for analyzing the inpact of diffrent things on price. What i need is a tool or scripting language that would make it easy for me to spider the website and put the data in a database. Preferable this would be an open source solution.

I am also looking for good tools for extracting information out of longer pieces of text. For example on the site i want to mine users can put in comments on every object. I would like to be able to decide if a comment is positive, negative och neither. I have seen this be done on one online art site that i cant remember the name of right now. The artist used blog post and decided the mood of the writer by what words were used.