Data Mining, Down Under » Research

AusDM 09 & Analytic Challenge

Shane Butler — Tue, 07 Jul 2009 13:47:52 +0000

Australian Data Mining conference (AusDM09) will be held in Melbourne next December and Dr Phil Brierley of Tiberius Data Mining has put out the call for proposals for an analytic challenge to accompany the conference. Competitions are quite popular in data mining circles and provide a good training ground for new practitioners to get access to real data and solve real problems. They also often have surprising results, such as the team who used laptop with 2GB RAM to beat IBM’s mighty clusters.

For businesses, this is a great opportunity to find out what is available by having others suggest new ideas and methods, or even to test your internally deployed models against the best of the best. So if you’re a business who has data, please consider being invloved! For further details, see the competition webpage.

Winning the DARPA Grand Challenge

Shane Butler — Sun, 17 Sep 2006 04:21:05 +0000

Sebastian Thrun of Stanford Racing gives a great a talk on what it took build an autonomous vehicle to win the DARPA Grand Challenge. There are lots of cool technical details on the use of machine learning to achieve this. You can watch it on Google Video here.

Smart SPAM & Fighting it

Shane Butler — Sat, 13 May 2006 02:26:28 +0000

For any machine learning based SPAM filters, such as the popular Bayesian methods, the key to success is the body of previously identified SPAM and HAM (valid emails) or training data. In order for the spammer to trick the filter, they must try to be more HAM-like. The way to beat this is by giving your email classifier as much training data as possible, and continually updating it. Just learning from your company’s emails is probably not fool-proof when you consider the volume and variety of SPAM on the net. Web-based email on the other hand, like Gmail and the hosted version, should never have this problem because the filter learns from thousands of user’s SPAM folders.

Researchers from University of Calgary claim that the next evolution of will be smart SPAM, which will infiltrate your computer via spyware/viruses and ‘mine’ your emails. By creating emails based on the your actual messages you’ve previously sent, the spammers hope they will be more believable to readers.

I would argue, however, that such a situation would merely make services Gmail, more attractive. Firstly because they have a truly massive body of knowledge to use to fine tune their spam filters, and secondly because it is unlikely such spyware could infiltrate a web-based system. Even if a program was distributed that waited for someone to log on and then took over, Google could have it effectively neutralised in a matter of hours.

Data Mining Cup 2006

Shane Butler — Fri, 05 May 2006 03:02:11 +0000

The Data Mining Cup (DMC2006), has launched for 2006. This year the competition focuses on eBay auctions. The target is to predict for each new auction whether the actual sales revenue is higher than the average sales revenue of the product category.

DARPA Grand Challenge

Shane Butler — Thu, 04 May 2006 02:18:23 +0000

Start your engines, the DARPA Grand Challenge is on again only this time its an urban challenge! The last two competitions were to race an autonomous vehicle through a desert, with the 2005 winner, Standford, taking home a US$2 million prize.

Stanford’s software in action: Input from GPS and many sensors feed the algorithms to determine the safe path (see tech report).

What’s in a name?

Shane Butler — Wed, 05 Apr 2006 09:19:27 +0000

Dennis Forbes gives a fantastic analysis of one of the biggest databases on the Internet – the DNS records. His analysis includes insights into domain name length, personal and family name usage and other characteristics. For example, did you know that all 2- and 3-letter domains are taken? Dennis is planning a second part so keep a look out for that too.

Got Zeitgeist? Mining Online Trends

Shane Butler — Mon, 06 Mar 2006 04:12:31 +0000

Each week, Google provides a taste of the top search queries, a site called Google Zeitgeist. At the end of each year, they compile a more comprehensive report of what people have been searching for. The 2005 Zeitgeist has been out since December and provides some interesting insights into online trends over past year. My favourites were world affairs and nature.

Beyond just being interesting, companies such as BuzzMetrics and BlogPulse have realised that analysis of Internet activity will be a useful tool for many companies. They produce tools that mine blogs in a bid to capture consumer sentiment on particular product(s), for example to improve product marketing. Matthew Hurst of BlogPulse has an interesting blog with the odd post covering Internet blogging activity, such as this pre-Oscars analysis.

Another interesting data mining application is the one pioneered by Majestic Research. They provide stock research and earnings forecasts to analysts before actual company information is released. Using web-based data mining, they track the sales of the top consumer-sensitive web companies, and then use this information to infer the company’s performance. Nice!

Profiling Amazon Users

Shane Butler — Thu, 19 Jan 2006 05:57:30 +0000

Here’s an interesting read. Data Mining 101: Finding Subversives with Amazon Wishlists takes a look at just how much information we can extract from publicly available data such as Amazon.com’s Wish List service. The wish list allows a user to bookmark items they would like either by coming back and purchase at a later date or telling people about as gift ideas. Thus it was the ideal database to extract amazon user’s political interests. More and More, the term data mining seems to be used in the USA to refer to privacy concerns of a government tool for spying on it’s citizens. That’s certainly the tone this article takes but it is still interesting to see what can be achieved using freely available data.

Searching on political and religious keywords the author generated a list of users and the books interested in. Although the user’s address information is hidden, he was able to use Yahoo! People Search to find their location and display the highlighted users on a customized Google Map. Anyway, its worth the read, so check it out.