Data Mining, Down Under

Lately I’ve been trying to come up with a generic way to deploy models on any platform.  So I’d like to share some early code that takes a PMML TreeModel and converts it to R code.  The intention is to get the R code generation working right, then extend to support generation for other languages.  Anyway, here it is (remember — early alpha, very rough still!!).  Updates to follow soon!

Australian Data Mining conference (AusDM09) will be held in Melbourne next December and Dr Phil Brierley of Tiberius Data Mining has put out the call for proposals for an analytic challenge to accompany the conference.  Competitions are quite popular in data mining circles and provide a good training ground for new practitioners to get access to real data and solve real problems.  They also often have surprising results, such as the team who used laptop with 2GB RAM to beat IBM’s mighty clusters.

For businesses, this is a great opportunity to find out what is available by having others suggest new ideas and methods, or even to test your internally deployed models against the best of the best. So if you’re a business who has data, please consider being invloved! For further details, see the competition webpage.

The DMG has released a new version of the PMML open format for representing predictive models. The new version includes support for ensembles, new model types and more built in functions to name just a few of the enhancements. For a detailed summary, see the Zementis blog.

Some really good advice here from John Elder in a series of video tutorials on data mining mistakes to avoid. Tip #5, regarding contaminating the project with future data is a good one, although sometimes it can be quite tricky (if not impossible) to ‘rewind’ the data! I believe the video series is a part of the launch of The Handbook of Statistical Analysis and Data Mining Applications. You can watch part one below or head over to YouTube for the entire series.

A forum post by Ingo Mierswa of Rapid-I indicates the upcoming RapidMiner v5 will feature two GUIs: the existing tree-based designer and a new graph-based designer! I’m quite excited about this because I’ve personally found the existing UI a bit clunky. Details and screenshots over at the
user forum.

In more R news, it appears SAS isn’t as worried about airplane safety as originally thought, and has indicated they will include R support in an upcoming update to the SAS/IML product.  For details see NYTimes & Adventures in Consulting.

The New York Times has an interesting story on the increasing use of R for data analysis within academia and industry.  Several large corporates are cited as having selected R over commercial conterparts such as S and SAS.

[via Slashdot]

Update: For more R news, see also Ajay Ohri’s interview with Dr Graham Williams, the author of Rattle.

Rapid-I has released an new and improved version of the open source data mining suite RapidMiner (formely called YALE).  I’ve been evaluating RapidMiner lately as a possible addition to my data mining toolbox.  I’ve found the biggest hurdle in learning how to use it is probably the GUI.  It is a tree-based GUI which I find much harder to understand than the graph-style approach used by many others.  However RapidMiner is quite a powerful tool, and the Community Edition is free, so there is probably a lot of benefit in getting used to the strange GUI.

The built in tutorial is a really good way to get a grasp of the system and I highly recommend spending some time on this if you are interested in learning RapidMiner.  I would also recommend a series of RapidMiner video turtorials over at Neural Market Trends that are worth checking out too.

RapidMiner 4.3 includes a 3D lift chart

The SAS Forum (Australia) was held in Sydney back in August.  I was unable to attend but luckily the presentations have been put online.  Here are some that I found interesting:

• Make Sure Your Insight is Insightful: Analytical Marketing at NAB by Antony Ugoni (National Australia Bank)
• Model Deployment and Management – The ATO Story by Warwick Graco (Australian Taxation Office)
• Putting Cheques in Place to Identify Fraud by Dr Paul Bracewell (Offlode NZ) and Flavio Palaci (Marsh Australia)
• Customer Value Creation Using Analytics by Arun VS (Satyam)
• Analysing Performance and Tuning your SAS Application by Bill Gibson (SAS)

I’ve decided to start afresh with a new blog, Data Mining Down Under.  While I have blogged previously on my personal site, I thought it would be good to start fresh with a new focus and more regular posts.  Therefore this site will not be about personal posts but rather a journal of my thinking around various data mining topics. Generally speaking, this blog will cover a wide range of data mining topics from the latest research and development efforts to all the trends and best practices for industry.  Hopefully it provides for some interesting reading!