评估Scikit-Learn模型/估算器的三种方法:
- 估算器的内置score()方法
- scoring 参数
- 特定问题的度量函数
import pandas as pd import numpy as np # 1. 数据读取 heart_disease = pd.read_csv(r'UsersAdministratorDesktopdataheart-disease.csv') heart_disease.head()
| age | sex | cp | trestbps | chol | fbs | restecg | thalach | exang | oldpeak | slope | ca | thal | target | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 63 | 1 | 3 | 145 | 233 | 1 | 0 | 150 | 0 | 2.3 | 0 | 0 | 1 | 1 |
| 1 | 37 | 1 | 2 | 130 | 250 | 0 | 1 | 187 | 0 | 3.5 | 0 | 0 | 2 | 1 |
| 2 | 41 | 0 | 1 | 130 | 204 | 0 | 0 | 172 | 0 | 1.4 | 2 | 0 | 2 | 1 |
| 3 | 56 | 1 | 1 | 120 | 236 | 0 | 1 | 178 | 0 | 0.8 | 2 | 0 | 2 | 1 |
| 4 | 57 | 0 | 0 | 120 | 354 | 0 | 1 | 163 | 1 | 0.6 | 2 | 0 | 2 | 1 |
heart_disease.isnull().sum()
age 0 sex 0 cp 0 trestbps 0 chol 0 fbs 0 restecg 0 thalach 0 exang 0 oldpeak 0 slope 0 ca 0 thal 0 target 0 dtype: int64
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
np.random.seed(42)
X = heart_disease.drop('target', axis = 1)
y = heart_disease['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
clf = RandomForestClassifier()
clf.fit(X_train, y_train)
clf.score(X_test, y_test)
0.8524590163934426
y_preds = clf.predict(X_test) y_preds
array([0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0,
1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1,
1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0], dtype=int64)
np.sum(y_test == y_preds) / len(y_test)
0.8524590163934426
由此可知，分类器RandomForestClassifier内置的score是求accuracy，也就是准确率，即预测正确的个数占总体的比例，还可以用sklearn内置的方法求同样的值
from sklearn.metrics import accuracy_score accuracy_score(y_test, y_preds)
0.8524590163934426让我们在回归问题上使用 score()
让我们使用加州住房（California Housing）数据集，内置于sklearn中
# 获取加州住房数据 from sklearn.datasets import fetch_california_housing housing = fetch_california_housing() housing
{'data': array([[ 8.3252 , 41. , 6.98412698, ..., 2.55555556,
37.88 , -122.23 ],
[ 8.3014 , 21. , 6.23813708, ..., 2.10984183,
37.86 , -122.22 ],
[ 7.2574 , 52. , 8.28813559, ..., 2.80225989,
37.85 , -122.24 ],
...,
[ 1.7 , 17. , 5.20554273, ..., 2.3256351 ,
39.43 , -121.22 ],
[ 1.8672 , 18. , 5.32951289, ..., 2.12320917,
39.43 , -121.32 ],
[ 2.3886 , 16. , 5.25471698, ..., 2.61698113,
39.37 , -121.24 ]]),
'target': array([4.526, 3.585, 3.521, ..., 0.923, 0.847, 0.894]),
'frame': None,
'target_names': ['MedHouseVal'],
'feature_names': ['MedInc',
'HouseAge',
'AveRooms',
'AveBedrms',
'Population',
'AveOccup',
'Latitude',
'Longitude'],
'DESCR': '.. _california_housing_dataset:nnCalifornia Housing datasetn--------------------------nn**Data Set Characteristics:**nn :Number of Instances: 20640nn :Number of Attributes: 8 numeric, predictive attributes and the targetnn :Attribute Information:n - MedInc median income in block groupn - HouseAge median house age in block groupn - AveRooms average number of rooms per householdn - AveBedrms average number of bedrooms per householdn - Population block group populationn - AveOccup average number of household membersn - Latitude block group latituden - Longitude block group longitudenn :Missing Attribute Values: NonennThis dataset was obtained from the StatLib repository.nhttps://www.dcc.fc.up.pt/~ltorgo/Regression/cal_housing.htmlnnThe target variable is the median house value for California districts,nexpressed in hundreds of thousands of dollars ($100,000).nnThis dataset was derived from the 1990 U.S. census, using one row per censusnblock group. A block group is the smallest geographical unit for which the U.S.nCensus Bureau publishes sample data (a block group typically has a populationnof 600 to 3,000 people).nnAn household is a group of people residing within a home. Since the averagennumber of rooms and bedrooms in this dataset are provided per household, thesencolumns may take surpinsingly large values for block groups with few householdsnand many empty houses, such as vacation resorts.nnIt can be downloaded/loaded using then:func:`sklearn.datasets.fetch_california_housing` function.nn.. topic:: Referencesnn - Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions,n Statistics and Probability Letters, 33 (1997) 291-297n'}
housing_df = pd.DataFrame(housing['data'], columns = housing['feature_names']) housing_df
| MedInc | HouseAge | AveRooms | AveBedrms | Population | AveOccup | Latitude | Longitude | |
|---|---|---|---|---|---|---|---|---|
| 0 | 8.3252 | 41.0 | 6.984127 | 1.023810 | 322.0 | 2.555556 | 37.88 | -122.23 |
| 1 | 8.3014 | 21.0 | 6.238137 | 0.971880 | 2401.0 | 2.109842 | 37.86 | -122.22 |
| 2 | 7.2574 | 52.0 | 8.288136 | 1.073446 | 496.0 | 2.802260 | 37.85 | -122.24 |
| 3 | 5.6431 | 52.0 | 5.817352 | 1.073059 | 558.0 | 2.547945 | 37.85 | -122.25 |
| 4 | 3.8462 | 52.0 | 6.281853 | 1.081081 | 565.0 | 2.181467 | 37.85 | -122.25 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 20635 | 1.5603 | 25.0 | 5.045455 | 1.133333 | 845.0 | 2.560606 | 39.48 | -121.09 |
| 20636 | 2.5568 | 18.0 | 6.114035 | 1.315789 | 356.0 | 3.122807 | 39.49 | -121.21 |
| 20637 | 1.7000 | 17.0 | 5.205543 | 1.120092 | 1007.0 | 2.325635 | 39.43 | -121.22 |
| 20638 | 1.8672 | 18.0 | 5.329513 | 1.171920 | 741.0 | 2.123209 | 39.43 | -121.32 |
| 20639 | 2.3886 | 16.0 | 5.254717 | 1.162264 | 1387.0 | 2.616981 | 39.37 | -121.24 |
20640 rows × 8 columns
housing_df['target'] = housing['target'] housing_df.head()
| MedInc | HouseAge | AveRooms | AveBedrms | Population | AveOccup | Latitude | Longitude | target | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 8.3252 | 41.0 | 6.984127 | 1.023810 | 322.0 | 2.555556 | 37.88 | -122.23 | 4.526 |
| 1 | 8.3014 | 21.0 | 6.238137 | 0.971880 | 2401.0 | 2.109842 | 37.86 | -122.22 | 3.585 |
| 2 | 7.2574 | 52.0 | 8.288136 | 1.073446 | 496.0 | 2.802260 | 37.85 | -122.24 | 3.521 |
| 3 | 5.6431 | 52.0 | 5.817352 | 1.073059 | 558.0 | 2.547945 | 37.85 | -122.25 | 3.413 |
| 4 | 3.8462 | 52.0 | 6.281853 | 1.081081 | 565.0 | 2.181467 | 37.85 | -122.25 | 3.422 |
from sklearn.ensemble import RandomForestRegressor
np.random.seed(42)
X = housing_df.drop('target', axis = 1)
y = housing_df['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = RandomForestRegressor()
model.fit(X_train, y_train)
model.score(X_test, y_test)
0.8065734772187598
回归问题的score默认方式就是决定系数R^2:
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-zNE2QhP0-1659622556163)(attachment:eb910708-1bb2-4b17-b092-ae864a6bfb39.png)]
决定系数示意图: 线性回归（右侧）的效果比起平均值（左侧）越好，决定系数的值就越接近于1。 蓝色正方形表示线性回归的残差的平方，红色正方形数据表示对于平均值的残差的平方。
- 决定系数R^2是回归函数与Y=平均值这条水平线误差的比较；
- 只要样本点固定，SS_Total(红色分母，左侧图)是固定不变的，回归函数或模型误差越小则SS_Regression(蓝色分子,右侧图)这一项越小，R2就越趋近于1
上述clf所求的就是决定系数 r-squared , 接下来直接调用sklearn的包，也可以直接求出对应值
from sklearn.metrics import r2_score
# 先用model预测想赢值
y_preds = model.predict(X_test)
r2_score(y_true = y_test,
y_pred = y_preds)
0.8065734772187598使用 scoring 参数评估一个模型 让我们看看 scoring 参数在分类问题中的应用
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
np.random.seed(42)
X = heart_disease.drop('target', axis = 1)
y = heart_disease['target']
clf = RandomForestClassifier()
# 如果 scoring=None, 则使用 esitmator 的默认评分计算指标（分类模型的准确性）---即 accuracy
cv_acc = cross_val_score(clf, X, y, cv=5, scoring = None)
# 交叉验证的准确性
print(f"The cross-validated accuracy is: {np.mean(cv_acc)*100:.2f}%")
The cross-validated accuracy is: 82.48%
scoring = 'accuracy' 将得到一样的结果，可以看下
np.random.seed(42)
cv_acc = cross_val_score(clf, X, y, cv=5, scoring="accuracy")
print(f"The cross-validated accuracy is: {np.mean(cv_acc)*100:.2f}%")
The cross-validated accuracy is: 82.48%
# Precision
np.random.seed(42)
cv_precision = cross_val_score(clf, X, y, cv = 5, scoring = 'precision')
print(f"The cross-validated precision is: {np.mean(cv_precision)*100:.2f}%")
The cross-validated precision is: 83.30%
# Recall
np.random.seed(42)
cv_recall = cross_val_score(clf, X, y, cv=5, scoring="recall")
print(f"The cross-validated recall is: {np.mean(cv_recall)}")
The cross-validated recall is: 0.8545454545454545让我们看看 scoring 参数在回归问题中的应用 回归模型评价指标
我们要涉及的是:
- R^2 (pronounced r-squared) 决定系数
- Mean absolute error (MAE) 平均绝对误差
- Mean squared error (MSE) 平均平方误差
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestRegressor
np.random.seed(42)
X = housing_df.drop('target', axis = 1)
y = housing_df['target']
model = RandomForestRegressor()
cv_r2 = cross_val_score(model, X, y, cv=3, scoring=None)
np.mean(cv_r2)
0.6545458023124507
mse 具是预测值和真实值差的平方取平均
交叉验证返回负的 mse 值
# Mean squared error cv_mse = cross_val_score(model, X, y, cv=5, scoring="neg_mean_squared_error") np.mean(cv_mse)
-0.4303784827892982
MSE是实际值和预测值之间误差的平方的平均值
# Mean absolute error cv_mae = cross_val_score(model, X, y, cv=5, scoring="neg_mean_absolute_error") np.mean(cv_mae)
-0.46680736586240323分类模型评价指标
- 准确度
- ROC曲线下的面积
- 混淆矩阵
- 分类报告
之前搞过了，跟内置的score一样的结论
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
np.random.seed(42)
X = heart_disease.drop('target', axis = 1)
y = heart_disease['target']
clf = RandomForestClassifier()
cross_val_score = cross_val_score(clf, X, y, cv = 5)
np.mean(cross_val_score)
print(f"心脏病分类器交叉验证的准确性: {np.mean(cross_val_score) *100:.2f}%")
心脏病分类器交叉验证的准确性: 82.48%(AUC/ROC)
- 曲线下面积 (AUC)
- ROC 曲线
ROC曲线是一个模型的真阳性率（tpr）与一个模型的假阳性率（fpr）的比较
- 真阳性 True positive = 模型在真相为1时预测为1
- 假阳性 False positive = 模型在真相为0时预测为1
- 真阴性 True negative = 模型在真相为0时预测为0
- 假阴性 False negative = 模型在真相为1时预测为0
from sklearn.metrics import roc_curve clf.fit(X_train, y_train) y_probs = clf.predict_proba(X_test) y_probs[:10]
array([[0.87, 0.13],
[0.41, 0.59],
[0.45, 0.55],
[0.89, 0.11],
[0.19, 0.81],
[0.14, 0.86],
[0.4 , 0.6 ],
[0.96, 0.04],
[1. , 0. ],
[0.43, 0.57]])
ROC曲线
该曲线通过绘制两个变量向我们展示了每个阈值的分类器行为：真阳性率 (TPR) 和假阳性率 (FPR)。
真阳性率通常称为召回率/灵敏度，定义为：
而假阳性率定义为：
TPR = TP/P = 94/100 = 94%
TNR = TN/N = 850/900 = 94.4%
FPR = FP/N = 50/900 = 5.5%
FNR = FN/p =6/100 = 6%
# 计算fpr、tpr和阈值 y_probs_positive = y_probs[:, 1] fpr, tpr, thresholds = roc_curve(y_test, y_probs_positive) fpr
array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0.03448276, 0.03448276, 0.03448276,
0.03448276, 0.13793103, 0.13793103, 0.17241379, 0.17241379,
0.17241379, 0.17241379, 0.20689655, 0.20689655, 0.24137931,
0.34482759, 0.4137931 , 0.51724138, 0.5862069 , 0.65517241,
0.65517241, 0.72413793, 0.82758621, 0.89655172, 0.93103448,
1. ])
# 创建一个用于绘制ROC曲线的函数
import matplotlib.pyplot as plt
def plot_roc_curve(fpr, tpr):
plt.plot(fpr, tpr, color = 'orange', label = 'ROC')
plt.plot([0, 1], [0, 1], color = 'darkblue', linestyle = '--', label = 'Guessing')
plt.xlabel('False positive rate (fpr)')
plt.ylabel('True positive rate (tpr)')
plt.legend()
plt.show()
plot_roc_curve(fpr, tpr)
# 根据预测分数计算接收者操作特征曲线下的面积（ROC AUC） from sklearn.metrics import roc_auc_score roc_auc_score(y_test, y_probs_positive)
0.9251077586206896
# 绘制完美的 ROC 曲线和 AUC 分数 fpr, tpr, thresholds = roc_curve(y_test, y_test) plot_roc_curve(fpr, tpr)
# 完美的 AUC 分数 roc_auc_score(y_test, y_test)
1.0混淆矩阵
评估一个分类模型的下一个方法是使用[混淆矩阵]
混淆矩阵是一种比较模型预测的标签和它应该预测的实际标签的快速方法。从本质上讲，让你了解模型在哪些方面出现了混乱。
from sklearn.metrics import confusion_matrix y_preds = clf.predict(X_test) confusion_matrix(y_test, y_preds)
array([[24, 5],
[ 4, 28]], dtype=int64)
这可能更容易被可视化。
一种方法是用pd.crosstab()来做。
pd.crosstab(y_test,
y_preds,
rownames = ['Actual Label'],
colnames = ['Predicted Label'])
| Predicted Label | 0 | 1 |
|---|---|---|
| Actual Label | ||
| 0 | 24 | 5 |
| 1 | 4 | 28 |
Scikit-Learn有多种不同的绘制混淆矩阵的实现方法:
- sklearn.metrics.ConfusionMatrixDisplay.from_estimator(estimator, X, y) - 这需要一个拟合的估计器（如我们的clf模型）、特征（X）和标签（y），然后它使用训练好的估计器对X进行预测，并通过显示混淆矩阵将预测与y进行比较。
- sklearn.metrics.ConfusionMatrixDisplay.from_predictions(y_true, y_pred) - 这需要真实标签和预测标签，并通过显示混淆矩阵对它们进行比较。
from sklearn.metrics import ConfusionMatrixDisplay ConfusionMatrixDisplay.from_estimator(estimator = clf, X = X_test, y = y_test);
clf 是我们之前的RandomForestClassifier
# 根据预测结果绘制混淆矩阵
ConfusionMatrixDisplay.from_predictions(y_true = y_test,
y_pred = y_preds);
Classification Report
from sklearn.metrics import classification_report print(classification_report(y_test, y_preds))
precision recall f1-score support
0 0.86 0.83 0.84 29
1 0.85 0.88 0.86 32
accuracy 0.85 61
macro avg 0.85 0.85 0.85 61
weighted avg 0.85 0.85 0.85 61
总结一下分类指标:
- Accuracy 是一个很好的衡量标准，如果所有类别都是平衡的（例如，相同数量的样本被标记为0或1），则可以从这个标准开始。
- Precision and recall 类别不平衡的情况下变得更加重要。
- 如果假阳性预测比假阴性预测更糟糕，则应追求更高的精度。
- 如果假阴性预测比假阳性预测更差，则应追求更高的召回率。
- F1-score 精度和召回率的组合。
