# Data Set

This diabetes data set comes from UCI Machine Learning Repository and can be obtained here.

Currently, 1 out of 7 American adults has diabetes, per data from the Centers for Disease Control and Prevention. By 2050, the rate is expected to skyrocket to 1 out of 3. With this info in mind, here’s what we intend to do today: study how machine learning can improve patient outcomes by helping us predict Diabetes. Let’s do this!

`import pandas as pdimport numpy as npimport matplotlib.pyplot as plt%matplotlib inlinediabetes = pd.read_csv('diabetes.csv')print(diabetes.columns)`

Index([‘Pregnancies’, ‘Glucose’, ‘BloodPressure’, ‘SkinThickness’, ‘Insulin’, ‘BMI’, ‘DiabetesPedigreeFunction’, ‘Age’, ‘Outcome’], dtype=’object’)

The diabetes data set consists of 768 data points, with 9 features each:

`print("dimension of diabetes data: {}".format(diabetes.shape))`

dimension of diabetes data: (768, 9)

“Outcome” is what we intend to predict, 0 refers to “No diabetes”, 1 refers to “Has diabetes”. Of 768 data points, 500 are as 0 and 268 are as 1:

`print(diabetes.groupby('Outcome').size())`
`import seaborn as snssns.countplot(diabetes['Outcome'],label="Count")`
`diabetes.info()`

# k-Nearest Neighbors

First, let us examine if we can confirm a relationship between complexity of the model and accuracy of predictions:

K-NN is a simple algorithm from machine learning. Constructing a model requires only storing the training data set. To generate a prediction for new points in the data, the algorithm locates the closest points from the training data — its “nearest neighbors.”

`from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(diabetes.loc[:, diabetes.columns != 'Outcome'], diabetes['Outcome'], stratify=diabetes['Outcome'], random_state=66)from sklearn.neighbors import KNeighborsClassifiertraining_accuracy = []test_accuracy = []# try n_neighbors from 1 to 10neighbors_settings = range(1, 11)for n_neighbors in neighbors_settings:    # build the model    knn = KNeighborsClassifier(n_neighbors=n_neighbors)    knn.fit(X_train, y_train)    # record training set accuracy    training_accuracy.append(knn.score(X_train, y_train))    # record test set accuracy    test_accuracy.append(knn.score(X_test, y_test))plt.plot(neighbors_settings, training_accuracy, label="training accuracy")plt.plot(neighbors_settings, test_accuracy, label="test accuracy")plt.ylabel("Accuracy")plt.xlabel("n_neighbors")plt.legend()plt.savefig('knn_compare_model')`

The plot displayed above is the accuracy of the training and test set on the y-axis against the n_neighbors on the x-axis. If we choose an individual nearest neighbor, the prediction is perfect for the the training set. However if we choose more neighbors, then accuracy drops for the model, showing that using an individual nearest neighbor produces a model that is too complicated. 9 neighbors is approximately the best performance.

The plot recommends selecting n_neighbors=9. Here we are:

`knn = KNeighborsClassifier(n_neighbors=9)knn.fit(X_train, y_train)print('Accuracy of K-NN classifier on training set: {:.2f}'.format(knn.score(X_train, y_train)))print('Accuracy of K-NN classifier on test set: {:.2f}'.format(knn.score(X_test, y_test)))`

Accuracy of K-NN classifier on training set: 0.79

Accuracy of K-NN classifier on test set: 0.78

# Logistic regression

Logistic Regression is among the most common algorithms for classifying things.

`from sklearn.linear_model import LogisticRegressionlogreg = LogisticRegression().fit(X_train, y_train)print("Training set score: {:.3f}".format(logreg.score(X_train, y_train)))print("Test set score: {:.3f}".format(logreg.score(X_test, y_test)))`

Training set accuracy: 0.781

Test set accuracy: 0.771

The default value of C=1 provides with 78% accuracy on the training and 77% accuracy on the test set.

`logreg001 = LogisticRegression(C=0.01).fit(X_train, y_train)print("Training set accuracy: {:.3f}".format(logreg001.score(X_train, y_train)))print("Test set accuracy: {:.3f}".format(logreg001.score(X_test, y_test)))`

Training set accuracy: 0.700

Test set accuracy: 0.703

Using C=0.01 results in lower accuracy on both the training and the test sets.

`logreg100 = LogisticRegression(C=100).fit(X_train, y_train)print("Training set accuracy: {:.3f}".format(logreg100.score(X_train, y_train)))print("Test set accuracy: {:.3f}".format(logreg100.score(X_test, y_test)))`

Training set accuracy: 0.785

Test set accuracy: 0.766

Using C=100 yields slightly higher accuracy on the training set and slightly lower accuracy on the test set, which confirms that less regularization and a more complicated model may not improve over the default.

Thus, it’s best for us to select a default of C=1.

Next, let us visualize the coefficients that the models learned under the 3 different configurations of the regularization parameter C.

More intense regularization (C=0.001) pushes coefficients further towards zero. Examining the plot more in detail, we also can see that the feature “DiabetesPedigreeFunction”, with C=100, C=1 and C=0.001, has a coefficient that is positive. This shows that high “DiabetesPedigreeFunction” is connected to a sample being “diabetes”, regardless of the model we examine.

`diabetes_features = [x for i,x in enumerate(diabetes.columns) if i!=8]plt.figure(figsize=(8,6))plt.plot(logreg.coef_.T, 'o', label="C=1")plt.plot(logreg100.coef_.T, '^', label="C=100")plt.plot(logreg001.coef_.T, 'v', label="C=0.001")plt.xticks(range(diabetes.shape), diabetes_features, rotation=90)plt.hlines(0, 0, diabetes.shape)plt.ylim(-5, 5)plt.xlabel("Feature")plt.ylabel("Coefficient magnitude")plt.legend()plt.savefig('log_coef')`

# Decision Tree

`from sklearn.tree import DecisionTreeClassifiertree = DecisionTreeClassifier(random_state=0)tree.fit(X_train, y_train)print("Accuracy on training set: {:.3f}".format(tree.score(X_train, y_train)))print("Accuracy on test set: {:.3f}".format(tree.score(X_test, y_test)))`

Accuracy on training set: 1.000

Accuracy on test set: 0.714

The accuracy on the training set is 100%, while the test set accuracy is much worse. This is an indicative that the tree is overfitting and not generalizing well to new data. Therefore, we need to apply pre-pruning to the tree.

We set max_depth=3, limiting the depth of the tree decreases overfitting. This leads to a lower accuracy on the training set, but an improvement on the test set.

`tree = DecisionTreeClassifier(max_depth=3, random_state=0)tree.fit(X_train, y_train)print("Accuracy on training set: {:.3f}".format(tree.score(X_train, y_train)))print("Accuracy on test set: {:.3f}".format(tree.score(X_test, y_test)))`

Accuracy on training set: 0.773

Accuracy on test set: 0.740

# Feature Importance in Decision Trees

Feature importance rates how important each feature is for the decision a tree makes. It is a number between 0 and 1 for each feature, where 0 means “not used at all” and 1 means “perfectly predicts the target”. The feature importances always sum to 1:

`print("Feature importances:\n{}".format(tree.feature_importances_))`

Feature importances: [ 0.04554275 0.6830362 0. 0. 0. 0.27142106 0. 0. ]

Then we can visualize the feature importances:

`def plot_feature_importances_diabetes(model):    plt.figure(figsize=(8,6))    n_features = 8    plt.barh(range(n_features), model.feature_importances_, align='center')    plt.yticks(np.arange(n_features), diabetes_features)    plt.xlabel("Feature importance")    plt.ylabel("Feature")    plt.ylim(-1, n_features)plot_feature_importances_diabetes(tree)plt.savefig('feature_importance')`

Feature “Glucose” is by far the most important feature.

# Random Forest

Let’s apply a random forest consisting of 100 trees on the diabetes data set:

`from sklearn.ensemble import RandomForestClassifierrf = RandomForestClassifier(n_estimators=100, random_state=0)rf.fit(X_train, y_train)print("Accuracy on training set: {:.3f}".format(rf.score(X_train, y_train)))print("Accuracy on test set: {:.3f}".format(rf.score(X_test, y_test)))`

Accuracy on training set: 1.000

Accuracy on test set: 0.786

The random forest gives us an accuracy of 78.6%, better than the logistic regression model or a single decision tree, without tuning any parameters. However, we can adjust the max_features setting, to see whether the result can be improved.

`rf1 = RandomForestClassifier(max_depth=3, n_estimators=100, random_state=0)rf1.fit(X_train, y_train)print("Accuracy on training set: {:.3f}".format(rf1.score(X_train, y_train)))print("Accuracy on test set: {:.3f}".format(rf1.score(X_test, y_test)))`

Accuracy on training set: 0.800

Accuracy on test set: 0.755

It did not, this indicates that the default parameters of the random forest work well.

# Feature importance in Random Forest

`plot_feature_importances_diabetes(rf)`

Similarly to the single decision tree, the random forest also gives a lot of importance to the “Glucose” feature, but it also chooses “BMI” to be the 2nd most informative feature overall. The randomness in building the random forest forces the algorithm to consider many possible explanations, the result being that the random forest captures a much broader picture of the data than a single tree.

`from sklearn.ensemble import GradientBoostingClassifiergb = GradientBoostingClassifier(random_state=0)gb.fit(X_train, y_train)print("Accuracy on training set: {:.3f}".format(gb.score(X_train, y_train)))print("Accuracy on test set: {:.3f}".format(gb.score(X_test, y_test)))`

Accuracy on training set: 0.917

Accuracy on test set: 0.792

We are likely to be overfitting. To reduce overfitting, we could either apply stronger pre-pruning by limiting the maximum depth or lower the learning rate:

`gb1 = GradientBoostingClassifier(random_state=0, max_depth=1)gb1.fit(X_train, y_train)print("Accuracy on training set: {:.3f}".format(gb1.score(X_train, y_train)))print("Accuracy on test set: {:.3f}".format(gb1.score(X_test, y_test)))`

Accuracy on training set: 0.804

Accuracy on test set: 0.781

`gb2 = GradientBoostingClassifier(random_state=0, learning_rate=0.01)gb2.fit(X_train, y_train)print("Accuracy on training set: {:.3f}".format(gb2.score(X_train, y_train)))print("Accuracy on test set: {:.3f}".format(gb2.score(X_test, y_test)))`

Accuracy on training set: 0.802

Accuracy on test set: 0.776

Both methods of decreasing the model complexity reduced the training set accuracy, as expected. However, in this case, none of these methods increased the generalization performance of the test set.

We can visualize the feature importances to get more insight into our model even though we are not really happy with the model:

`plot_feature_importances_diabetes(gb1)`

We can see that the feature importances of the gradient boosted trees are somewhat similar to the feature importances of the random forests, it gives weight to all of the features in this case.

# Support Vector Machine

`from sklearn.svm import SVCsvc = SVC()svc.fit(X_train, y_train)print("Accuracy on training set: {:.2f}".format(svc.score(X_train, y_train)))print("Accuracy on test set: {:.2f}".format(svc.score(X_test, y_test)))`

Accuracy on training set: 1.00

Accuracy on test set: 0.65

The model overfits quite substantially, with a perfect score on the training set and only 65% accuracy on the test set.

SVM requires all the features to vary on a similar scale. We will need to re-scale our data that all the features are approximately on the same scale:

`from sklearn.preprocessing import MinMaxScalerscaler = MinMaxScaler()X_train_scaled = scaler.fit_transform(X_train)X_test_scaled = scaler.fit_transform(X_test)svc = SVC()svc.fit(X_train_scaled, y_train)print("Accuracy on training set: {:.2f}".format(svc.score(X_train_scaled, y_train)))print("Accuracy on test set: {:.2f}".format(svc.score(X_test_scaled, y_test)))`

Accuracy on training set: 0.77

Accuracy on test set: 0.77

Scaling the data made a huge difference! Now we are actually underfitting, where training and test set performance are quite similar but less close to 100% accuracy. From here, we can try increasing either C or gamma to fit a more complex model.

`svc = SVC(C=1000)svc.fit(X_train_scaled, y_train)print("Accuracy on training set: {:.3f}".format(    svc.score(X_train_scaled, y_train)))print("Accuracy on test set: {:.3f}".format(svc.score(X_test_scaled, y_test)))`

Accuracy on training set: 0.790

Accuracy on test set: 0.797

Here, increasing C allows us to improve the model, resulting in 79.7% test set accuracy.

# Deep Learning

`from sklearn.neural_network import MLPClassifiermlp = MLPClassifier(random_state=42)mlp.fit(X_train, y_train)print("Accuracy on training set: {:.2f}".format(mlp.score(X_train, y_train)))print("Accuracy on test set: {:.2f}".format(mlp.score(X_test, y_test)))`

Accuracy on training set: 0.71

Accuracy on test set: 0.67

The accuracy of the Multilayer perceptrons (MLP) is not as good as the other models at all, this is likely due to scaling of the data. deep learning algorithms also expect all input features to vary in a similar way, and ideally to have a mean of 0, and a variance of 1. We must re-scale our data so that it fulfills these requirements.

`from sklearn.preprocessing import StandardScalerscaler = StandardScaler()X_train_scaled = scaler.fit_transform(X_train)X_test_scaled = scaler.fit_transform(X_test)mlp = MLPClassifier(random_state=0)mlp.fit(X_train_scaled, y_train)print("Accuracy on training set: {:.3f}".format(    mlp.score(X_train_scaled, y_train)))print("Accuracy on test set: {:.3f}".format(mlp.score(X_test_scaled, y_test)))`

Accuracy on training set: 0.823

Accuracy on test set: 0.802

Let’s increase the number of iterations:

`mlp = MLPClassifier(max_iter=1000, random_state=0)mlp.fit(X_train_scaled, y_train)print("Accuracy on training set: {:.3f}".format(    mlp.score(X_train_scaled, y_train)))print("Accuracy on test set: {:.3f}".format(mlp.score(X_test_scaled, y_test)))`

Accuracy on training set: 0.877

Accuracy on test set: 0.755

Increasing the number of iterations only increased the training set performance, not the test set performance.

Let’s increase the alpha parameter and add stronger regularization of the weights:

`mlp = MLPClassifier(max_iter=1000, alpha=1, random_state=0)mlp.fit(X_train_scaled, y_train)print("Accuracy on training set: {:.3f}".format(    mlp.score(X_train_scaled, y_train)))print("Accuracy on test set: {:.3f}".format(mlp.score(X_test_scaled, y_test)))`

Accuracy on training set: 0.795

Accuracy on test set: 0.792

The result is good, but we are not able to increase the test accuracy further.

Therefore, our best model so far is default deep learning model after scaling.

Finally, we plot a heat map of the first layer weights in a neural network learned on the diabetes data set.

`plt.figure(figsize=(20, 5))plt.imshow(mlp.coefs_, interpolation='none', cmap='viridis')plt.yticks(range(8), diabetes_features)plt.xlabel("Columns in weight matrix")plt.ylabel("Input feature")plt.colorbar()`