Data Mining, Down Under

Earlier in the month local data miner Tim Manns presented at the Sydney Data Miners group.  Tim spoke on some work been doing at Optus around using mobile call patterns to establish social networks and using these networks to reduce customer churn.  Interestingly, there are also applications in many other areas, including data cleansing, for example, where one person has purchased two mobile phones and given one to their spouse or child.  Using this analysis we can try to determine which account is likely to be the actual account holder and infer the details (such as age) of the other customer.

For a full write up of Tim’s work, check out James Taylor’s PAW 2009 summary or head over to Tim’s data mining blog.

Lately I’ve been trying to come up with a generic way to deploy models on any platform.  So I’d like to share some early code that takes a PMML TreeModel and converts it to R code.  The intention is to get the R code generation working right, then extend to support generation for other languages.  Anyway, here it is (remember — early alpha, very rough still!!).  Updates to follow soon!

Australian Data Mining conference (AusDM09) will be held in Melbourne next December and Dr Phil Brierley of Tiberius Data Mining has put out the call for proposals for an analytic challenge to accompany the conference.  Competitions are quite popular in data mining circles and provide a good training ground for new practitioners to get access to real data and solve real problems.  They also often have surprising results, such as the team who used laptop with 2GB RAM to beat IBM’s mighty clusters.

For businesses, this is a great opportunity to find out what is available by having others suggest new ideas and methods, or even to test your internally deployed models against the best of the best. So if you’re a business who has data, please consider being invloved! For further details, see the competition webpage.

The DMG has released a new version of the PMML open format for representing predictive models. The new version includes support for ensembles, new model types and more built in functions to name just a few of the enhancements. For a detailed summary, see the Zementis blog.

Some really good advice here from John Elder in a series of video tutorials on data mining mistakes to avoid. Tip #5, regarding contaminating the project with future data is a good one, although sometimes it can be quite tricky (if not impossible) to ‘rewind’ the data! I believe the video series is a part of the launch of The Handbook of Statistical Analysis and Data Mining Applications. You can watch part one below or head over to YouTube for the entire series.

A forum post by Ingo Mierswa of Rapid-I indicates the upcoming RapidMiner v5 will feature two GUIs: the existing tree-based designer and a new graph-based designer! I’m quite excited about this because I’ve personally found the existing UI a bit clunky. Details and screenshots over at the
user forum.

In more R news, it appears SAS isn’t as worried about airplane safety as originally thought, and has indicated they will include R support in an upcoming update to the SAS/IML product.  For details see NYTimes & Adventures in Consulting.

The New York Times has an interesting story on the increasing use of R for data analysis within academia and industry.  Several large corporates are cited as having selected R over commercial conterparts such as S and SAS.

[via Slashdot]

Update: For more R news, see also Ajay Ohri’s interview with Dr Graham Williams, the author of Rattle.

Rapid-I has released an new and improved version of the open source data mining suite RapidMiner (formely called YALE).  I’ve been evaluating RapidMiner lately as a possible addition to my data mining toolbox.  I’ve found the biggest hurdle in learning how to use it is probably the GUI.  It is a tree-based GUI which I find much harder to understand than the graph-style approach used by many others.  However RapidMiner is quite a powerful tool, and the Community Edition is free, so there is probably a lot of benefit in getting used to the strange GUI.

The built in tutorial is a really good way to get a grasp of the system and I highly recommend spending some time on this if you are interested in learning RapidMiner.  I would also recommend a series of RapidMiner video turtorials over at Neural Market Trends that are worth checking out too.

RapidMiner 4.3 includes a 3D lift chart

The SAS Forum (Australia) was held in Sydney back in August.  I was unable to attend but luckily the presentations have been put online.  Here are some that I found interesting:

• Make Sure Your Insight is Insightful: Analytical Marketing at NAB by Antony Ugoni (National Australia Bank)
• Model Deployment and Management – The ATO Story by Warwick Graco (Australian Taxation Office)
• Putting Cheques in Place to Identify Fraud by Dr Paul Bracewell (Offlode NZ) and Flavio Palaci (Marsh Australia)
• Customer Value Creation Using Analytics by Arun VS (Satyam)
• Analysing Performance and Tuning your SAS Application by Bill Gibson (SAS)