{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# SK Part 0: Introduction to Predictive Modeling with Python and Scikit-Learn" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is the first in a series of tutorials on supervised machine learning with Python and `Scikit-Learn`. It is a short introductory tutorial that provides a bird's eye view using a binary classification problem as an example and it is actually is a simplified version of the tutorial **SK Part 1**. The reference textbook in these tutorials is below and it can be accessed [here](https://machinelearningbook.com/).\n", "\n", "The classifiers illustrated are as follows:\n", "* **Nearest neighbors** (Chapter on Similarity-based learning)\n", "* **Decision trees** (Chapter on Information-based learning)\n", "* **Random forests ensemble method** (Chapter on Information-based learning)\n", "* **Naive Bayes** (Chapter on Probability-based learning)\n", "* **Support vector machines** (Chapter on Error-based learning)\n", "\n", "As an overview, we shall cover various aspects of `scikit-learn` in the following tutorials:\n", "\n", "- **SK Part 0 (\"SK-Intro\"):** Introduction to machine learning with Python and scikit-learn (this tutorial)\n", "- **SK Part 1 (\"SK-Basics\"):** Basic model fitting\n", "- **SK Part 2 (\"SK-FS\"):** Feature selection and ranking\n", "- **SK Part 3 (\"SK-Eval\"):** Model evaluation (using performance metrics other than simple accuracy)\n", "- **SK Part 4 (\"SK-CV\"):** Cross-validation and hyper-parameter tuning\n", "- **SK Part 5 (\"SK-Pipes\"):** Machine learning pipeline, statistical model comparison, and model deployment\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Binary Classification Example: Breast Cancer Wisconsin Data\n", "\n", "This dataset is concerned with predicting whether a cell tissue is cancerous or not using the cell's measurement values. It contains 569 observations and 30 input features. The target feature, \"diagnosis\", has two classes: 212 \"malignant\" and 357 \"benign\", denoted by \"M\" and \"B\" respectively.\n", "\n", "The dataset has no missing values and all features are numeric other than the target feature (which is binary)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reading Breast Cancer Dataset from the Cloud\n", "\n", "We load the data directly from the following github account." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import warnings\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import io\n", "import requests\n", "\n", "# so that we can see all the columns\n", "pd.set_option('display.max_columns', None) \n", "\n", "# how to read a csv file from a github account\n", "url_name = 'https://raw.githubusercontent.com/akmand/datasets/master/breast_cancer_wisconsin.csv'\n", "url_content = requests.get(url_name, verify=False).content\n", "df = pd.read_csv(io.StringIO(url_content.decode('utf-8')))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's check the shape of this dataset to make sure it has been downloaded correctly." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(569, 31)" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's have a look at the first 5 rows in this raw dataset." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
mean_radiusmean_texturemean_perimetermean_areamean_smoothnessmean_compactnessmean_concavitymean_concave_pointsmean_symmetrymean_fractal_dimensionradius_errortexture_errorperimeter_errorarea_errorsmoothness_errorcompactness_errorconcavity_errorconcave_points_errorsymmetry_errorfractal_dimension_errorworst_radiusworst_textureworst_perimeterworst_areaworst_smoothnessworst_compactnessworst_concavityworst_concave_pointsworst_symmetryworst_fractal_dimensiondiagnosis
017.9910.38122.801001.00.118400.277600.30010.147100.24190.078711.09500.90538.589153.400.0063990.049040.053730.015870.030030.00619325.3817.33184.602019.00.16220.66560.71190.26540.46010.11890M
120.5717.77132.901326.00.084740.078640.08690.070170.18120.056670.54350.73393.39874.080.0052250.013080.018600.013400.013890.00353224.9923.41158.801956.00.12380.18660.24160.18600.27500.08902M
219.6921.25130.001203.00.109600.159900.19740.127900.20690.059990.74560.78694.58594.030.0061500.040060.038320.020580.022500.00457123.5725.53152.501709.00.14440.42450.45040.24300.36130.08758M
311.4220.3877.58386.10.142500.283900.24140.105200.25970.097440.49561.15603.44527.230.0091100.074580.056610.018670.059630.00920814.9126.5098.87567.70.20980.86630.68690.25750.66380.17300M
420.2914.34135.101297.00.100300.132800.19800.104300.18090.058830.75720.78135.43894.440.0114900.024610.056880.018850.017560.00511522.5416.67152.201575.00.13740.20500.40000.16250.23640.07678M
\n", "
" ], "text/plain": [ " mean_radius mean_texture mean_perimeter mean_area mean_smoothness \\\n", "0 17.99 10.38 122.80 1001.0 0.11840 \n", "1 20.57 17.77 132.90 1326.0 0.08474 \n", "2 19.69 21.25 130.00 1203.0 0.10960 \n", "3 11.42 20.38 77.58 386.1 0.14250 \n", "4 20.29 14.34 135.10 1297.0 0.10030 \n", "\n", " mean_compactness mean_concavity mean_concave_points mean_symmetry \\\n", "0 0.27760 0.3001 0.14710 0.2419 \n", "1 0.07864 0.0869 0.07017 0.1812 \n", "2 0.15990 0.1974 0.12790 0.2069 \n", "3 0.28390 0.2414 0.10520 0.2597 \n", "4 0.13280 0.1980 0.10430 0.1809 \n", "\n", " mean_fractal_dimension radius_error texture_error perimeter_error \\\n", "0 0.07871 1.0950 0.9053 8.589 \n", "1 0.05667 0.5435 0.7339 3.398 \n", "2 0.05999 0.7456 0.7869 4.585 \n", "3 0.09744 0.4956 1.1560 3.445 \n", "4 0.05883 0.7572 0.7813 5.438 \n", "\n", " area_error smoothness_error compactness_error concavity_error \\\n", "0 153.40 0.006399 0.04904 0.05373 \n", "1 74.08 0.005225 0.01308 0.01860 \n", "2 94.03 0.006150 0.04006 0.03832 \n", "3 27.23 0.009110 0.07458 0.05661 \n", "4 94.44 0.011490 0.02461 0.05688 \n", "\n", " concave_points_error symmetry_error fractal_dimension_error \\\n", "0 0.01587 0.03003 0.006193 \n", "1 0.01340 0.01389 0.003532 \n", "2 0.02058 0.02250 0.004571 \n", "3 0.01867 0.05963 0.009208 \n", "4 0.01885 0.01756 0.005115 \n", "\n", " worst_radius worst_texture worst_perimeter worst_area worst_smoothness \\\n", "0 25.38 17.33 184.60 2019.0 0.1622 \n", "1 24.99 23.41 158.80 1956.0 0.1238 \n", "2 23.57 25.53 152.50 1709.0 0.1444 \n", "3 14.91 26.50 98.87 567.7 0.2098 \n", "4 22.54 16.67 152.20 1575.0 0.1374 \n", "\n", " worst_compactness worst_concavity worst_concave_points worst_symmetry \\\n", "0 0.6656 0.7119 0.2654 0.4601 \n", "1 0.1866 0.2416 0.1860 0.2750 \n", "2 0.4245 0.4504 0.2430 0.3613 \n", "3 0.8663 0.6869 0.2575 0.6638 \n", "4 0.2050 0.4000 0.1625 0.2364 \n", "\n", " worst_fractal_dimension diagnosis \n", "0 0.11890 M \n", "1 0.08902 M \n", "2 0.08758 M \n", "3 0.17300 M \n", "4 0.07678 M " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Partitioning Dataset into the Set of Descriptive Features and the Target Feature" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we partition `cancer_df` columns into the set of descriptive features and the target." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# The \".values\" part below converts the data frame to a 2-dimensional numpy array\n", "Data = df.drop(columns = 'diagnosis').values\n", "target = df['diagnosis']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Encoding Target" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Keep in mind that `scikit-learn` always requires all data to be numeric, so the target needs to be encoded as 0 and 1." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from sklearn import preprocessing\n", "\n", "target = preprocessing.LabelEncoder().fit_transform(target)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that the `LabelEncoder` labels in an alphabetical order. That is, \"B\" is labeled as 0 whereas \"M\" as labeled as 1 (see the code below)." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(array([0, 1]), array([357, 212]))" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.unique(target, return_counts = True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Scaling Descriptive Features" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It's always a good idea to scale your descriptive features before fitting any models. Here, we use the \"min-max scaling\" so that each descriptive feature is scaled to be between 0 and 1. In the rest of this tutorial, we work with scaled data." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "Data = preprocessing.MinMaxScaler().fit_transform(Data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Spliting Data into Training and Test Sets \n", "\n", "We split the descriptive features and the target feature into a `training set` and a `test set` by a ratio of 70:30. That is, we use 70% of the data to build our classifiers and evaluate their performance on the remaining 30% of the data. This is to ensure that we measure model performance on unseen data in order to avoid overfitting. We also set a random state value so that we can replicate our results later on." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "D_train, D_test, t_train, t_test = train_test_split(Data, \n", " target, \n", " test_size = 0.3,\n", " random_state=999)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fitting a Nearest Neighbor Classifier" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's fit a nearest neighbor classifier with 5 neighbors using the Euclidean distance. We fit the model on the train data and evaluate its performance on the test data.\n", "\n", "Below, the `score` method returns the accuracy of the classifier on the test data. Accuracy is defined as the ratio of correctly predicted observations to the total number of observations." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9707602339181286" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.neighbors import KNeighborsClassifier\n", "\n", "knn_classifier = KNeighborsClassifier(n_neighbors=5, p=2)\n", "knn_classifier.fit(D_train, t_train)\n", "knn_classifier.score(D_test, t_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you would like to see which parameters are available for a classifier, just type the name of the classifier followed by a question mark, e.g., \"KNeighborsClassifier?\"." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# KNeighborsClassifier?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fitting a Decision Tree Classifier" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's fit a decision tree classifier with the entropy split criterion and a maximum depth of 4 on the train data, and then evaluate its performance on the test data. " ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9298245614035088" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.tree import DecisionTreeClassifier\n", "\n", "dt_classifier = DecisionTreeClassifier(criterion='entropy', max_depth=4)\n", "dt_classifier.fit(D_train, t_train)\n", "dt_classifier.score(D_test, t_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fitting a Random Forest Classifier" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An ensemble method is a collection of many sub-classifiers. The final outcome is determined by a majority voting of the sub-classifiers. Random forest classifier is a popular ensemble method based on the idea of \"bagging\" where the sub-classifiers are decision trees. Let's fit a random forest classifier with 100 decision trees." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9590643274853801" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.ensemble import RandomForestClassifier\n", "\n", "rf_classifier = RandomForestClassifier(n_estimators=100)\n", "rf_classifier.fit(D_train, t_train)\n", "rf_classifier.score(D_test, t_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fitting a Gaussian Naive Bayes Classifier" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another model we would like to fit to the breast cancer dataset is the Gaussian Naive Bayes classifier with a variance smoothing value of $10^{-3}$." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.935672514619883" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.naive_bayes import GaussianNB\n", "\n", "nb_classifier = GaussianNB(var_smoothing=10**(-3))\n", "nb_classifier.fit(D_train, t_train)\n", "nb_classifier.score(D_test, t_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fitting a Support Vector Machine" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One last model we fit is the SVM with all the default values." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9883040935672515" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.svm import SVC\n", "\n", "svm_classifier = SVC()\n", "svm_classifier.fit(D_train, t_train)\n", "svm_classifier.score(D_test, t_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Making Predictions with a Fitted Model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once a model is built, a prediction can be made using the `predict` method of the fitted classifier.\n", "\n", "For example, suppose we would like to use the fitted nearest neighbor classifier as our model, and we would like to find out the model's prediction for the first three rows in the input data. Of course, we already know the labels of these rows (which are all malignant), so this is just to illustrate how you would make a prediction for a new observation." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([1, 1, 1])" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_obs = Data[0:3]\n", "knn_classifier.predict(new_obs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The model's prediction for these three rows is that they are all \"1\", that is, they are all \"malignant\". Thus, in this particular case, we observe that the model correctly predicts the first three rows in the input data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "This tutorial illustrates that Python and `Scikit-Learn` together provide a unified interface to model fitting and evaluation and they greatly simplify the machine learning workflow.\n", "\n", "Of course, there is a whole lot more to supervised machine learning than what is shown in here, such as \n", "1. Other classification algorithms\n", "2. Solving prediction problems where the target feature is numeric (a.k.a. regression problems)\n", "3. Using other model performance metrics (e.g., precision, recall, mean squared error for regression, etc.)\n", "4. More sophisticated model performance assessment methods (such as cross-validation)\n", "5. How model parameters can be optimized (also known as hyperparameter tuning)\n", "\n", "We cover these topics in the subsequent tutorials." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 4 }