业务目标：上游模型——预测该区域房价中位数，下游——决定是否投资
当前方案——团队调查信息，误差0.2，是否有监督——数据含f，有
多重回归——多个特征进行预测，一元回归——只预测某区域的单标签值
学习方法——无在线数据，数据量小，批量学习
性能指标：回归误差衡量——errsqr
验证假设：要具体价格，还是含糊类别
获取数据：自动获取函数：fetch_housing_data 自动加载函数：load_housing_data
数据结构：.head()特征名称 .info()是否缺失值，是否含分类属性，
可视化：
测试集（计算Eout）：法一：自定义函数split_train_test，在运行后即保存
法二：设置随机种子np.random.seed(42)再调用np.random.permutation()
法三：自定义函数test_set_check设置标识符，确保测试集哈希值<0.2，再split_train_test_by_id
分层抽样：pd.cut()分五层，StratifiedShuffleSplit过程（按比例切train/test、对切分数据按各箱比例入箱，分n组）
保存训练集strat_train_set测试集strat_test_set
对训练集探索：某特征的散点图、各特征集合散点图、corr()两两特征相关系数矩阵，scatter_metrix()绘制相关性直方图，观察最重要特征
创建组合属性，得相关性
自动清洗：处理缺失值drop、dropna、fillna、SimpleImputer
文本分类属性——对文本：PrdinalEncoder转数字，对类别：OneHOtEncoder将类别fit_transform独热向量，太多得表征学习
自定义组合属性类CombinedAttributesAdder
特征缩放：归一化MinMaxScaler、StandardScaler标准化
流水线训练数据：Pipiline(转换器估算器).fit_transform(数值数据)
ColumnTransformer(转换器).fit_transform(全数据)
模型+训练：np.sqrt( mean_squared_error( f, LinearRegression().fit(D,f).predict(D) ) )
交叉验证：np.sqrt(- cross_val_score(model, D, f, mode, cv) ).mean()/.std()
交叉验证超参所有组合：GridSearchCV(回归实例、组合、n、mode).fit(D,f) .best_params_ / .best_estimator_ / .cv_results_
网格搜索: RandomizedSearchCV
上测试集：走流程、泛化误差0.1的95%置信：stats.t.interval()
from jupyterthemes import jtplot jtplot.style()
import matplotlib.pyplot as plt import numpy as np import pandas as pd import sklearn.linear_model
def prepare_country_stats(oecd_bli, gdp_per_capita):
oecd_bli = oecd_bli[oecd_bli["INEQUALITY"]=="TOT"]
oecd_bli = oecd_bli.pivot(index="Country", columns="Indicator", values="Value")
gdp_per_capita.rename(columns={"2015": "GDP per capita"}, inplace=True)
gdp_per_capita.set_index("Country", inplace=True)
full_country_stats = pd.merge(left=oecd_bli, right=gdp_per_capita,
left_index=True, right_index=True)
full_country_stats.sort_values(by="GDP per capita", inplace=True)
remove_indices = [0, 1, 6, 8, 33, 34, 35]
keep_indices = list(set(range(36)) - set(remove_indices))
return full_country_stats[["GDP per capita", 'Life satisfaction']].iloc[keep_indices]
import os import tarfile import urllib
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedShuffleSplit
from pandas.plotting import scatter_matrix
DOWNLOAD_ROOT = 'https://raw.githubusercontent.com/ageron/handson-ml2/master/'
HOUSING_PATH = os.path.join('datasets', 'housing')
HOUSING_URL = DOWNLOAD_ROOT = 'datasets/housing/housing.tgz'
def load_housing_data(housing_path=HOUSING_PATH):
csv_path = os.path.join(housing_path, 'housing.csv')
return pd.read_csv(csv_path)
def split_train_test(data, test_ratio):
shuffled_indices = np.random.permutation(len(data))
test_set_size = int(len(data) * test_ratio)
test_indices = shuffled_indices[:test_set_size]
train_indices = shuffled_indices[test_set_size:]
return data.iloc[train_indices], data.iloc[test_indices]
from zlib import crc32
def test_set_check(identifier, test_ratio):
return crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32
def split_train_test_by_id(data, test_ratio, id_column):
ids = data[id_column]
in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio))
return data.loc[~in_test_set], data.loc[in_test_set]
oecd_bli = pd.read_csv('datasets/lifesat/oecd_bli_2015.csv', thousands=',')
gdp_per_capita = pd.read_csv('datasets/lifesat/gdp_per_capita.csv', thousands=',', delimiter='t',
encoding='latin1', na_values='n/a')
country_stats = prepare_country_stats(oecd_bli, gdp_per_capita) x = np.c_[country_stats['GDP per capita']] y = np.c_[country_stats['Life satisfaction']]
country_stats.plot(kind='scatter',x='GDP per capita',y='Life satisfaction') plt.show()
model = sklearn.linear_model.LinearRegression() model.fit(x,y)
LinearRegression()
# 机器学习流程技术 x_new = [[22587]] print(model.predict(x_new))
[[5.96242338]]
housing = load_housing_data() housing.head()
| longitude | latitude | housing_median_age | total_rooms | total_bedrooms | population | households | median_income | median_house_value | ocean_proximity | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | -122.23 | 37.88 | 41.0 | 880.0 | 129.0 | 322.0 | 126.0 | 8.3252 | 452600.0 | NEAR BAY |
| 1 | -122.22 | 37.86 | 21.0 | 7099.0 | 1106.0 | 2401.0 | 1138.0 | 8.3014 | 358500.0 | NEAR BAY |
| 2 | -122.24 | 37.85 | 52.0 | 1467.0 | 190.0 | 496.0 | 177.0 | 7.2574 | 352100.0 | NEAR BAY |
| 3 | -122.25 | 37.85 | 52.0 | 1274.0 | 235.0 | 558.0 | 219.0 | 5.6431 | 341300.0 | NEAR BAY |
| 4 | -122.25 | 37.85 | 52.0 | 1627.0 | 280.0 | 565.0 | 259.0 | 3.8462 | 342200.0 | NEAR BAY |
housing.info()
RangeIndex: 20640 entries, 0 to 20639 Data columns (total 10 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 longitude 20640 non-null float64 1 latitude 20640 non-null float64 2 housing_median_age 20640 non-null float64 3 total_rooms 20640 non-null float64 4 total_bedrooms 20433 non-null float64 5 population 20640 non-null float64 6 households 20640 non-null float64 7 median_income 20640 non-null float64 8 median_house_value 20640 non-null float64 9 ocean_proximity 20640 non-null object dtypes: float64(9), object(1) memory usage: 1.6+ MB
housing.describe()
| longitude | latitude | housing_median_age | total_rooms | total_bedrooms | population | households | median_income | median_house_value | |
|---|---|---|---|---|---|---|---|---|---|
| count | 20640.000000 | 20640.000000 | 20640.000000 | 20640.000000 | 20433.000000 | 20640.000000 | 20640.000000 | 20640.000000 | 20640.000000 |
| mean | -119.569704 | 35.631861 | 28.639486 | 2635.763081 | 537.870553 | 1425.476744 | 499.539680 | 3.870671 | 206855.816909 |
| std | 2.003532 | 2.135952 | 12.585558 | 2181.615252 | 421.385070 | 1132.462122 | 382.329753 | 1.899822 | 115395.615874 |
| min | -124.350000 | 32.540000 | 1.000000 | 2.000000 | 1.000000 | 3.000000 | 1.000000 | 0.499900 | 14999.000000 |
| 25% | -121.800000 | 33.930000 | 18.000000 | 1447.750000 | 296.000000 | 787.000000 | 280.000000 | 2.563400 | 119600.000000 |
| 50% | -118.490000 | 34.260000 | 29.000000 | 2127.000000 | 435.000000 | 1166.000000 | 409.000000 | 3.534800 | 179700.000000 |
| 75% | -118.010000 | 37.710000 | 37.000000 | 3148.000000 | 647.000000 | 1725.000000 | 605.000000 | 4.743250 | 264725.000000 |
| max | -114.310000 | 41.950000 | 52.000000 | 39320.000000 | 6445.000000 | 35682.000000 | 6082.000000 | 15.000100 | 500001.000000 |
%matplotlib inline
plt.figure(figsize=(15,9)) housing.hist(bins=50) plt.tight_layout() plt.show()
train_set, test_set = split_train_test(housing, 0.2) len(train_set) len(test_set)
4128
housing_with_id = housing.reset_index()
train_set, test_set = split_train_test_by_id(housing_with_id, 0.2, 'index')
housing_with_id['id'] = housing['longitude'] * 1000 + housing['latitude'] train_set, test_set = split_train_test_by_id(housing_with_id, 0.2, 'id')
train_set, test_set = train_test_split(housing, test_size=0.2, random_state=42)
housing['income_cat'] = pd.cut(housing['median_income'],
bins=[0, 1.5, 3., 4.5, 6., np.inf],
labels=[1,2,3,4,5])
housing['income_cat'].hist()
split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing['income_cat']):
strat_train_set = housing.loc[train_index]
strat_test_set = housing.loc[test_index]
strat_test_set['income_cat'].value_counts() / len(strat_test_set)
3 0.350533 2 0.318798 4 0.176357 5 0.114341 1 0.039971 Name: income_cat, dtype: float64
housing['income_cat']
0 5
1 5
2 5
3 4
4 3
..
20635 2
20636 2
20637 2
20638 2
20639 2
Name: income_cat, Length: 20640, dtype: category
Categories (5, int64): [1 < 2 < 3 < 4 < 5]
for set_ in (strat_train_set, strat_test_set):
set_.drop('income_cat', axis=1, inplace=True)
housing = strat_train_set.copy()
housing.plot(kind='scatter', x='longitude',y='latitude', alpha=0.4,
s=housing['population']/100, label='population', figsize=(10,7),
c='median_house_value', cmap=plt.get_cmap('jet'), colorbar=True)
plt.legend()
corr_matrix = housing.corr()
corr_matrix['median_house_value'].sort_values(ascending=False)
median_house_value 1.000000 median_income 0.687151 total_rooms 0.135140 housing_median_age 0.114146 households 0.064590 total_bedrooms 0.047781 population -0.026882 longitude -0.047466 latitude -0.142673 Name: median_house_value, dtype: float64
attributes = ['median_house_value', 'median_income', 'total_rooms',
'housing_median_age']
scatter_matrix(housing[attributes], figsize=(12,8))
array([[,
,
,
],
[,
,
,
],
[,
,
,
],
[,
,
,
]],
dtype=object)
housing.plot(kind='scatter', x='median_income', y='median_house_value', alpha=0.1)
housing['rooms_per_household'] = housing['total_rooms']/housing['households'] housing['bedrooms_per_room'] = housing['total_bedrooms']/housing['total_rooms'] housing['population_per_household'] = housing['population']/housing['households']
corr_matrix = housing.corr()
corr_matrix['median_house_value'].sort_values(ascending=False)
median_house_value 1.000000 median_income 0.687151 rooms_per_household 0.146255 total_rooms 0.135140 housing_median_age 0.114146 households 0.064590 total_bedrooms 0.047781 population_per_household -0.021991 population -0.026882 longitude -0.047466 latitude -0.142673 bedrooms_per_room -0.259952 Name: median_house_value, dtype: float64
housing = strat_train_set.drop('median_house_value', axis=1)
housing_labels = strat_train_set['median_house_value'].copy()
housing.dropna(subset=['total_bedrooms'])
housing.drop('total_bedrooms', axis=1)
median = housing['total_bedrooms'].median()
housing['total_bedrooms'].fillna(median, inplace=True)
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy='median')
housing_num = housing.drop('ocean_proximity', axis=1)
imputer.fit(housing_num)
SimpleImputer(strategy='median')
imputer.statistics_
array([-118.51 , 34.26 , 29. , 2119. , 433. ,
1164. , 408. , 3.54155])
housing_num.median().values
array([-118.51 , 34.26 , 29. , 2119. , 433. ,
1164. , 408. , 3.54155])
X = imputer.transform(housing_num)
housing_tr = pd.DataFrame(X, columns=housing_num.columns,
index=housing_num.index)
housing_cat = housing[['ocean_proximity']]
housing_cat.head(10)
| ocean_proximity | |
|---|---|
| 12655 | INLAND |
| 15502 | NEAR OCEAN |
| 2908 | INLAND |
| 14053 | NEAR OCEAN |
| 20496 | <1H OCEAN |
| 1481 | NEAR BAY |
| 18125 | <1H OCEAN |
| 5830 | <1H OCEAN |
| 17989 | <1H OCEAN |
| 4861 | <1H OCEAN |
from sklearn.preprocessing import OrdinalEncoder
ordinal_encoder = OrdinalEncoder() housing_cat_encoded = ordinal_encoder.fit_transform(housing_cat) housing_cat_encoded[:10]
array([[1.],
[4.],
[1.],
[4.],
[0.],
[3.],
[0.],
[0.],
[0.],
[0.]])
ordinal_encoder.categories_
[array(['<1H OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEAN'],
dtype=object)]
from sklearn.preprocessing import OneHotEncoder
cat_encoder = OneHotEncoder() housing_cat_1hot = cat_encoder.fit_transform(housing_cat) housing_cat_1hot
<16512x5 sparse matrix of type '' with 16512 stored elements in Compressed Sparse Row format>
housing_cat_1hot.toarray()
array([[0., 1., 0., 0., 0.],
[0., 0., 0., 0., 1.],
[0., 1., 0., 0., 0.],
...,
[1., 0., 0., 0., 0.],
[1., 0., 0., 0., 0.],
[0., 1., 0., 0., 0.]])
cat_encoder.categories_
[array(['<1H OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEAN'],
dtype=object)]
from sklearn.base import BaseEstimator, TransformerMixin
rooms_ix, bedrooms_ix, population_ix, households_ix = 3,4,5,6
class CombinedAtributesAdder(BaseEstimator, TransformerMixin):
def __init__(self, add_bedrooms_per_room=True):
self.add_bedrooms_per_room = add_bedrooms_per_room
def fit(self, X, y=None):
return self
def transform(self,X):
rooms_per_household = X[:, rooms_ix] / X[:, households_ix]
population_per_household = X[:, population_ix] / X[:, households_ix]
if self.add_bedrooms_per_room:
bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
return np.c_[X, rooms_per_household, population_per_household, bedrooms_per_room]
else:
return np.c_[X, rooms_per_household, population_per_household]
attr_adder = CombinedAtributesAdder(add_bedrooms_per_room=False)
housing_extra_attribs = attr_adder.transform(housing.values)
from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler
num_pipeline = Pipeline([
('imputer', SimpleImputer(strategy='median')),
('attribs_adder', CombinedAtributesAdder()),
('std_scaler', StandardScaler())
])
housing_num_tr = num_pipeline.fit_transform(housing_num)
housing_num.dtypes
longitude float64 latitude float64 housing_median_age float64 total_rooms float64 total_bedrooms float64 population float64 households float64 median_income float64 dtype: object
from sklearn.compose import ColumnTransformer
num_attribs = list(housing_num)
cat_attribs = ['ocean_proximity']
full_pipeline = ColumnTransformer([
('num', num_pipeline, num_attribs),
('cat', OneHotEncoder(), cat_attribs)
])
housing_prepared = full_pipeline.fit_transform(housing)
from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(housing_prepared, housing_labels)
LinearRegression()
some_data = housing.iloc[:5]
some_labels = housing_labels.iloc[:5]
some_data_prepared = full_pipeline.transform(some_data)
print('predictions:', lin_reg.predict(some_data_prepared))
predictions: [ 85657.90192014 305492.60737488 152056.46122456 186095.70946094 244550.67966089]
print('Labels:', list(some_labels))
Labels: [72100.0, 279600.0, 82700.0, 112500.0, 238300.0]
from sklearn.metrics import mean_squared_error
housing_predictions = lin_reg.predict(housing_prepared)
lin_mse = mean_squared_error(housing_labels, housing_predictions)
lin_rmse = np.sqrt(lin_mse)
lin_rmse
68627.87390018745
from sklearn.tree import DecisionTreeRegressor
tree_reg = DecisionTreeRegressor()
tree_reg.fit(housing_prepared, housing_labels)
housing_predictions = tree_reg.predict(housing_prepared)
tree_mse = mean_squared_error(housing_labels, housing_predictions)
tree_rmse = np.sqrt(tree_mse)
tree_rmse
from sklearn.model_selection import cross_val_score
scores = cross_val_score(tree_reg, housing_prepared, housing_labels,
scoring='neg_mean_squared_error', cv=10)
tree_rmse_scores = np.sqrt(-scores)
def display_scores(scores):
print('Scores:', scores)
print('Mean:', scores.mean())
print('Standard deviation:', scores.std())
display_scores(tree_rmse_scores)
Scores: [73208.8495113 70410.53805223 68567.71015747 72147.10419032 70315.69842661 77336.57489448 73226.32275652 72536.22956953 68469.27027015 70308.72475141] Mean: 71652.70225800324 Standard deviation: 2510.3302542700344
lin_scores = cross_val_score(lin_reg, housing_prepared, housing_labels,
scoring='neg_mean_squared_error', cv=10)
lin_rmse_scores = np.sqrt(-lin_scores)
display_scores(lin_rmse_scores)
Scores: [71774.15825125 64114.99166359 67771.17124356 68627.7242824 66848.58938823 72527.24321966 74023.75820358 68800.50640085 66443.28836884 70147.43163952] Mean: 69107.88626614638 Standard deviation: 2886.049299226037
from sklearn.ensemble import RandomForestRegressor
forest_reg = RandomForestRegressor()
forest_reg.fit(housing_prepared, housing_labels)
forest_predictions = forest_reg.predict(housing_prepared)
forest_mse = mean_squared_error(housing_labels, forest_predictions)
forest_rmse = np.sqrt(forest_mse)
forest_rmse
forest_scores = cross_val_score(forest_reg, housing_prepared, housing_labels,
scoring='neg_mean_squared_error', cv=10)
forest_rmse_scores = np.sqrt(-forest_scores)
display_scores(forest_rmse_scores)
Scores: [51611.68353822 48919.83045026 46652.19250233 51872.20826211 47564.13086453 52204.60300844 52511.88574329 49866.90415642 48168.12462235 53794.71131954] Mean: 50316.6274467482 Standard deviation: 2288.041919636827
from sklearn.model_selection import GridSearchCV
param_grid = [
{'n_estimators':[3,10,30], 'max_features': [2,4,6,8]},
{'bootstrap':[False], 'n_estimators':[3,10], 'max_features':[2,3,4]}
]
forest_reg = RandomForestRegressor()
grid_search = GridSearchCV(forest_reg, param_grid, cv=5,
scoring='neg_mean_squared_error',
return_train_score=True)
grid_search.fit(housing_prepared, housing_labels)
GridSearchCV(cv=5, estimator=RandomForestRegressor(),
param_grid=[{'max_features': [2, 4, 6, 8],
'n_estimators': [3, 10, 30]},
{'bootstrap': [False], 'max_features': [2, 3, 4],
'n_estimators': [3, 10]}],
return_train_score=True, scoring='neg_mean_squared_error')
grid_search.best_params_
{'max_features': 8, 'n_estimators': 30}
grid_search.best_estimator_
RandomForestRegressor(max_features=8, n_estimators=30)
cvres = grid_search.cv_results_
for mean_score, params in zip(cvres['mean_test_score'], cvres['params']):
print(np.sqrt(-mean_score), params)
63104.34052345203 {'max_features': 2, 'n_estimators': 3}
55044.54815870831 {'max_features': 2, 'n_estimators': 10}
52479.70137042046 {'max_features': 2, 'n_estimators': 30}
60731.76038620718 {'max_features': 4, 'n_estimators': 3}
52868.393285151826 {'max_features': 4, 'n_estimators': 10}
50711.051576613936 {'max_features': 4, 'n_estimators': 30}
58683.36172378717 {'max_features': 6, 'n_estimators': 3}
52342.272196172285 {'max_features': 6, 'n_estimators': 10}
50218.35499652776 {'max_features': 6, 'n_estimators': 30}
58751.627370128 {'max_features': 8, 'n_estimators': 3}
52112.39293130457 {'max_features': 8, 'n_estimators': 10}
49934.63197516629 {'max_features': 8, 'n_estimators': 30}
63074.771657032594 {'bootstrap': False, 'max_features': 2, 'n_estimators': 3}
54106.1640459132 {'bootstrap': False, 'max_features': 2, 'n_estimators': 10}
59668.76583074217 {'bootstrap': False, 'max_features': 3, 'n_estimators': 3}
52201.004448067884 {'bootstrap': False, 'max_features': 3, 'n_estimators': 10}
58989.599193443675 {'bootstrap': False, 'max_features': 4, 'n_estimators': 3}
51432.98586544147 {'bootstrap': False, 'max_features': 4, 'n_estimators': 10}
feature_importances = grid_search.best_estimator_.feature_importances_
feature_importances
array([6.79881161e-02, 6.15216910e-02, 4.44877452e-02, 1.51136911e-02,
1.51213976e-02, 1.48061388e-02, 1.38968897e-02, 3.56355010e-01,
4.74741464e-02, 1.11377081e-01, 7.19685834e-02, 4.00569389e-03,
1.70150096e-01, 1.58285049e-04, 2.40818102e-03, 3.16725327e-03])
extra_attribs = ['room_per_hhold', 'pop_per_hhold', 'bedrooms_per_room'] cat_encoder = full_pipeline.named_transformers_['cat'] cat_one_hot_attribs = list(cat_encoder.categories_[0]) attributes = num_attribs + extra_attribs + cat_one_hot_attribs sorted(zip(feature_importances, attributes), reverse=True)
[(0.3563550104523296, 'median_income'), (0.17015009643629045, 'INLAND'), (0.1113770805110796, 'pop_per_hhold'), (0.0719685834481056, 'bedrooms_per_room'), (0.06798811610731767, 'longitude'), (0.061521690993818885, 'latitude'), (0.04747414641876038, 'room_per_hhold'), (0.04448774518521582, 'housing_median_age'), (0.015121397601006564, 'total_bedrooms'), (0.015113691086082844, 'total_rooms'), (0.01480613882529324, 'population'), (0.01389688970335908, 'households'), (0.004005693892097333, '<1H OCEAN'), (0.0031672532720703026, 'NEAR OCEAN'), (0.002408181018221732, 'NEAR BAY'), (0.0001582850489508359, 'ISLAND')]
final_model = grid_search.best_estimator_
x_test = strat_test_set.drop('median_house_value', axis=1)
y_test = strat_test_set['median_house_value'].copy()
x_test_prepared = full_pipeline.transform(x_test)
final_predictions = final_model.predict(x_test_prepared)
final_mse = mean_squared_error(y_test, final_predictions)
final_rmse = np.sqrt(final_mse)
from scipy import stats
confidence = 0.95
squared_errors = (final_predictions - y_test) ** 2
np.sqrt(stats.t.interval(confidence, len(squared_errors) -1,
loc=squared_errors.mean(), scale=stats.sem(squared_errors)))
array([45942.19072814, 49791.91438289])
