Introduction

Data mining is the process of discovering previously unknown, actionable and profitable information from large consolidated databases and using it to support tactical and strategic business decisions.
 

The statistical techniques of data mining are familiar. They include linear and logistic regression, multivariate analysis, principal components analysis, decision trees and neural networks. Traditional approaches to statistical inference fail with large databases, however, because with thousands or millions of cases and hundreds or thousands of variables there will be a high level of redundancy among the variables, there will be spurious relationships, and even the weakest relationships will be highly significant by any statistical test. The objective is to build a model with significant predictive power. It is not enough just to find which relationships are statistically significant.
 

Consider a campaign offering a product or service for sale, directed at a given customer base. Typically, about 1% of the customer base will be "responders," customers who will purchase the product or service if it is offered to them. A mailing to 100,000 randomly-chosen customers will therefore generate about 1000 sales. Data mining techniques enable customer relationship marketing, by identifying which customers are most likely to respond to the campaign. If the response can be raised from 1% to, say, 1.5% of the customers contacted (the "lift value"), then 1000 sales could by achieved with only 66,666 mailings, reducing the cost of mailing by one-third.
 

Suggestions for Analysis

We expect that you will be using Splus or SAS for the analysis, however, not all of the methods suggested here are readily available in Splus. If you have a SAS licence, the Enterprise Miner module will conveniently automate many of the analyses and you may be able to get an evaluation copy inexpensively from SAS. IBM's Intelligent Miner is also recommended but it is less likely to be available to you.
 

For all the analyses below, you should create a training set and a validation set. As the data were stratified to 50/50 you should create an unstratified validation set with the original proportion of 1% "True" for the objective variable. You would, of course, get better validation sets if you had the complete sample of around 100,000 accounts, 99% of them non-responders, but the file is too large for us to distribute conveniently. Validation sets constructed from the 50/50 stratified sample should be adequate for the purposes of this exercise.
 

Your results should be plotted on a gains chart, either tabular or graphical. A gains chart is a plot of the % of the responders reached (ordinate) against the % of the customer base contacted (abscissa). If the campaign is directed at randomly-chosen individuals the plot will be a straight line with unit slope through the origin. If the campaign preferentially targets responders, the gains curve will lie above the diagonal except, of course, at 0% and 100% where it necessarily touches.
 

The performance of a predictive model is measured by looking at the % responders at 10%, 20% or 30% of customers mailed. A good model will get 1.5 to 3.5 times as many as random over this range so, for example, mailing to 10% of the customer base will reach 15% to 35% of the responders. Less than this means the data are not very predictive, more than this likely means that you have overfitted or there is a strong bias in the data.
 

Some things you could try with these data include:

1. Try some simple linear correlations, Spearman and Pearson, against the objective variable and reduce the number of variables. With the reduced set of variables, build logistic regression models. Don't forget to remove colinear variables.
2. Break the variables into blocks of 10-20 and build logistic models on each of the blocks. After all the models are built, pool the variables that were left in the models and create new blocks of 10-20 and redo until there is only one block of variables left. Don't forget to remove colinear variables.
3. Create PCA factors from the set of variables (don't include the objective variable!). Select a reduced set of variables from the PCA factors (using cumulative % of variation explained) and build a model from the factors. Compare this result with using all the factors, noting the effect of overfitting. Don't forget to remove colinear variables.
4. Perform a varclus with all variables. This procedure clusters the variables into hierarchical groups using the PCA factors. Select variables from the bottom-level of the hierarchical groups and build a logistic model. Don't forget to remove colinear variables.
5. Create multiple training and test samples. Use bootstrapping to estimate the error bounds on the model coefficients and gains chart performance. Try sampling with and without replacement to see how sensitive logistic regression is to the data set configuration.
6. Use SAS to construct a Radial Basis Function regression. Use all the above methods to reduce the variable set and compare RBF results to logistic.
7. It is possible to implement a decision tree with SAS using the CART algorithm. Run this algorithm against all the variables. Build multiple training sets using sampling with replacement. This should improve the tree performance by a few percent.
8. Other modeling techniques to try include neural networks and genetic algorithms.

All model results should be analyzed for gains chart performance with the following measures:

1. What is the response rate (as % of responders in the customer base reached), compared to random, for campaigns mailed to 10%, 20% or 30% of all customers? At these points, the random response would be 10%, 20% or 30% respectively. Most campaigns are mailed to 10% to 30% of the customer base. Good models can achieve 1.5 to 3.5 times the random response rate in this range.
2. Monotonicity and smoothness: do the response rates by quantile group form a smoothly decreasing profile? Any waviness is indicative of bias, overfitting or unmodelled effects.
3. Ease of model explanation. It is very important for prospective clients to understand why the model is working!

Resources

You can get copies of some of Gary Saarenvirta's work online at www.db2mag.com. He has also written The Intelligent Miner for Data Applications Guide, found at www.redbooks.ibm.com.
 

An Internet search on "data mining" will find a number of commercial products similar to Intelligent Miner and Enterprise Miner.