Timely Data

June 9, 2008

Paul Kedrosky had an interesting (at least to me) tweet today pointing to an article on the Wall Street Journal entitled “Playing a Role In Oil’s Ascent: Crude Data”. Essentially the article talks about  the market being starved for TIMELY data.

I am curious what data the market is looking for in this area and how much of it is available from open sources? 

If this data is more or less available from open sources I see an opportunity here to create an automated system to collect, collate and cluster this open data.  Of course that is not enough, we need to also use automated tools to rapidly refine the data pulling out specific data points of interest so the end consumer of the data does not have to.

Basically I am thinking I’d take some openly available crawling technology add a few of my tool (statistical and others) to gather and present realtime (or near realtime) reports.

So how about it Paul is this type of data publicly available (not an oil analyst so I am not sure precisely what data they would want) and would this be useful?

First what do I mean by actionable information?

It is information that comes from a trusted source, is about something that’s important to you, and that, once known compells you to an take action.

So how do we aqcuire actionable information from the fire hose of data we see every day?  Well while it often appears that the volume of information we have flowing through our various data pipes is inherently at odds with the ability for us to make it actionable I think this misses one key point, the fact that the inherent volume of similar data can actually be seen as a positive if the right “mining” tools are in place.

This flow of information more often than not has patterns that can be extracted statistically, syntactically or semantically or using some combination, ie. using syntactic and semantic features to improve statistical recognition.

A good overview of leveraging data using statistics can be found in a talk given by Peter Norvig at Paul Graham’s latest startup school.