Weekly Summary1（1） (Starting on the 6th,the first week)
Abstract
This holiday season began to learn computer vision, which can be said to start from scratch.At first, it was just to understand what areas computer vision applications are in, what applications are there, such as image recognition, image segmentation, edge detection, medical imaging, unmanned driving, security systems, gesture recognition, gesture recognition, speech recognition, drones in agriculture, 3D impact field and so on.It is not enough to stay in the application field, we need to understand how to apply in these areas,what role computer vision plays,what is the basis of what algorithms are used to architect these applications.
We also need to know the principles of algorithms, the application of algorithms, how to rely on Python or other programming languages to implement these algorithms.
Behind the understanding of the principle of the algorithm, whether you can find out the advantages of each algorithm to combine, to generalize more practical problems that need to be solved; whether you can find out the shortcomings of the algorithm to improve, and then iterate to make the accuracy rate close to probability 1, optimize the previous model;whether you can use different programming methods to visualize the application of each algorithm by using different libraries.
The advent of deep learning has brought great changes to the field of computer vision. In the 1990s, convolutional neural networks, which were popular in the Imagenet large-scale visual recognition competition in 2012, pushed a big step forward in the field of computer vision, and the recognition error rate dropped significantly. The historical background of the field of computer vision is not much to repeat, you want to know can go to the specific check, there are each classic algorithm, such as Alexnet, VGG, Resnet and so on.
The following is my learning and understanding of the kNN algorithm, as well as the initial understanding of convolutional neural networks(CNN), as well as the learning of linear regression algorithms and gradient descent algorithms.
一.kNN algorithms
(1)Introduction:
As a simple regression model and the simplest classifier, kNN input instance feature vector, corresponding to the point output instance class of feature space, in the space plane can be divided into feature space, resulting in decision boundaries, as well as different categories, the algorithm is simple, there is no specific learning process, is the simplest algorithm in supervised learning. But there are also some problems, such as the need to take into account distance measures, the impact of different k-value choices on the results, dimensional disasters, etc. How to optimize this simple classifier, how to handle the overfitting problem well, and how to iteratively increase the learning rate through hyperparameters, will be discussed below.
(2)kNN algorithm principle
Compare the input image pattern with many exemplars
find the closest exemplar class
classify the input image
train the algorithm by changing the hyperparameters
(3)influencing factors of kNN algorithm
1.k value selection
From this, we can see that the value of k has different effects on the classification results
Selecting a smaller value of k is equivalent to predicting with training instances in a smaller neighborhood, and the learning approximation error decreases. Only the training instances that are similar to the one have an effect on the prediction result, but the learning estimation error will increase, that is, The test results are very sensitive to the instance points of the nearest neighbors. Another way of thinking is that some of the characteristics of the training samples will also be used as the characteristics of the test data, the generalization ability will decrease, and overfitting will occur.
Conversely, if the value of k is too large, the model will become simpler, and the features of the test data that are different from the training data will also be predicted, resulting in classification errors.
2.distance measure
The distance metric is the distance between two instance points in the feature space, reflecting the similarity of the two instance points.
Lp distance derivation formula：
p=1(Manhattan distance)
2D plane： d(x,y)=||+||
p=2(Euclidean distance)
p=(chebyshev distance)
3.classification decision rules(divide the feature space into subspaces)
the division of kNN to neighboring models
Separate the two categories with each sample as a region to form a Voronoi cell.
The discontinuous line in the middle constitutes a non-linear curve in two dimensions, and a decision surface in three dimensions, increasing in higher dimensions. These planar decision surfaces are vertical bisectors (or analogs in multidimensional space) of straight lines connecting different class pairs of training patterns that lie on the boundaries of class regions.
（4）algorithm flaws and algorithm improvements
There are 5 questions as follows：
（1）Optimizing k value selection
The value of k has a great influence on the calculation result. If the value of k is small, the classifier will search in a small field. If there is noise in the area, it will cause the decision boundary to be wrong and the classification to be wrong. This leads to a decrease in generalization ability and overfitting.
（2）prone to overfitting
reference (1)
（3）The distance metric is not suitable for comparing image pixels, and the vectorized distance function is not suitable for expressing the visual similarity between images.
（4）computer storage problem
reference (5)
（5）dimensional disaster
The kNN classifier uses the training data to divide the space into several blocks. If we want the classifier to perform well, we need the training data to be densely distributed in the space. Otherwise, the distance between the nearest neighbors may be very far, which means that the similarity of the samples to be tested is not high.The cost of denseness is that as the dimension increases, the size of the data set will explode exponentially, and the amount of information stored by the computer will also increase, resulting in a dimensional disaster.
（5）algorithm improvement measures
（1). In a way, instead of fitting the dataset as much as possible, let our classifier or our method perform better on unknown data other than the training set. But we can also normalize and centralize the data, similar to the edge filling of the convolutional neural network, to reduce the feature edge features, improve the learning efficiency of the classifier, and simplify the calculation.
(2).process the dataset as follows
| train | validation | test |
Validate the algorithm with different hyperparameters on the training set, evaluate it on the validation set, then use a set of hyperparameters to select the one that performs best on the validation set, and debugging is complete. Then the best performing classifier is tested on the validation set. After running, this is the data written in the paper.
（3).take cross-validation
Divide the dataset D into k subsets of similar size, use the union of (k-1) subsets as the training set each time, and the remaining subsets as the test set, perform k training tests, and return k test results do the mean.
(4).For suitably low error rates, a large number of training set patterns are often required, but can lead to severe storage and computational problems. We can replace the original dataset by using a piecewise linear or other functional classifier, and if this flat surface system is simplified in any convenient way, the computational load will be further reduced. One can try to do this by finding training procedures (which are used to update the decision surface positions immediately as each new training set pattern is received), although it must be confirmed that the chosen training procedure converges fast enough, but greatly reduces storage need.
(5).dimensionality reduction
Schematic diagram of low-dimensional embedding
MDS algorithm can solve this problem very well.You can see ML book in detail.
（6）code(pycharm)
mnist database
#打包好的数据集和代码
import torch
from torch.utils.data import DataLoader
import torchvision.datasets as dsets
import torchvision.transforms as transforms
batch_size = 100
# MNIST dataset
#root指定了数据集存放的路径，transform指定导入数据集时需要进行何种变换操作
# train设置为True说明导入的是训练集合，否则为测试集合。
train_dataset = dsets.MNIST(root = '/ml/pymnist', #选择数据的根目录
train = True, #选择训练集
transform = None, #不考虑使用任何数据预处理
download = True) #从网络上下载图片
test_dataset = dsets.MNIST(root = '/ml/pymnist', #选择数据的根目录
train = False, #选择测试集
transform = None, #不考虑使用任何数据预处理
download = True) #从网络上下载图片
#加载数据
train_loader = torch.utils.data.DataLoader(dataset = train_dataset,
batch_size = batch_size,
shuffle = True) #将数据打乱
test_loader = torch.utils.data.DataLoader(dataset = test_dataset,
batch_size = batch_size,
shuffle = True)
print("train_data:", train_dataset.train_data.size())
print("train_labels:", train_dataset.train_labels.size())
print("test_data:", test_dataset.test_data.size())
print("test_labels:", test_dataset.test_labels.size())
import matplotlib.pyplot as plt
digit = train_loader.dataset.train_data[0] #取第一个图片的数据
plt.imshow(digit,cmap=plt.cm.binary)
#plt.cm //matplotlib 包中的 cm 模块就是找到颜色图的地方
#binary 二进制
handling simple classification tasks
# 导入画图工具
import matplotlib.pyplot as plt
# 导入数组工具
import numpy as np
# 导入数据集生成器
from sklearn.datasets import make_blobs
# 导入KNN 分类器
from sklearn.neighbors import KNeighborsClassifier
# 导入数据集拆分工具
from sklearn.model_selection import train_test_split
# 生成样本数为200，分类数为2的数据集
data=make_blobs(n_samples=200, n_features=2,centers=2, cluster_std=1.0, random_state=8)
X,Y=data
# 将生成的数据集进行可视化
# plt.scatter(X[:,0], X[:,1],s=80, c=Y, cmap=plt.cm.spring, edgecolors='k')
# plt.show()
clf = KNeighborsClassifier()
clf.fit(X,Y)
# 绘制图形
x_min,x_max=X[:,0].min()-1,X[:,0].max()+1
y_min,y_max=X[:,1].min()-1,X[:,1].max()+1
xx,yy=np.meshgrid(np.arange(x_min,x_max,.02),np.arange(y_min,y_max,.02))
z=clf.predict(np.c_[xx.ravel(),yy.ravel()])
z=z.reshape(xx.shape)
plt.pcolormesh(xx,yy,z,cmap=plt.cm.Pastel1)
plt.scatter(X[:,0], X[:,1],s=80, c=Y, cmap=plt.cm.spring, edgecolors='k')
plt.xlim(xx.min(),xx.max())
plt.ylim(yy.min(),yy.max())
plt.title("Classifier:KNN")
# 把待分类的数据点用五星表示出来
plt.scatter(6.75,4.82,marker='*',c='red',s=200)
# 对待分类的数据点的分类进行判断
res = clf.predict([[6.75,4.82]])
plt.text(6.9,4.5,'Classification flag: '+str(res))
plt.show()
handling multivariate classification tasks
# 导入画图工具
import matplotlib.pyplot as plt
# 导入数组工具
import numpy as np
# 导入数据集生成器
from sklearn.datasets import make_blobs
# 导入KNN 分类器
from sklearn.neighbors import KNeighborsClassifier
# 导入数据集拆分工具
from sklearn.model_selection import train_test_split
# 生成样本数为500，分类数为5的数据集
data=make_blobs(n_samples=500, n_features=2,centers=5, cluster_std=1.0, random_state=8)
X,Y=data
# 将生成的数据集进行可视化
# plt.scatter(X[:,0], X[:,1],s=80, c=Y, cmap=plt.cm.spring, edgecolors='k')
# plt.show()
clf = KNeighborsClassifier()
clf.fit(X,Y)
# 绘制图形
x_min,x_max=X[:,0].min()-1,X[:,0].max()+1
y_min,y_max=X[:,1].min()-1,X[:,1].max()+1
xx,yy=np.meshgrid(np.arange(x_min,x_max,.02),np.arange(y_min,y_max,.02))
z=clf.predict(np.c_[xx.ravel(),yy.ravel()])
z=z.reshape(xx.shape)
plt.pcolormesh(xx,yy,z,cmap=plt.cm.Pastel1)
plt.scatter(X[:,0], X[:,1],s=80, c=Y, cmap=plt.cm.spring, edgecolors='k')
plt.xlim(xx.min(),xx.max())
plt.ylim(yy.min(),yy.max())
plt.title("Classifier:KNN")
# 把待分类的数据点用五星表示出来
plt.scatter(0,5,marker='*',c='red',s=200)
# 对待分类的数据点的分类进行判断
res = clf.predict([[0,5]])
plt.text(0.2,4.6,'Classification flag: '+str(res))
plt.text(3.75,-13,'Model accuracy: {:.2f}'.format(clf.score(X, Y)))
plt.show()
reference books：
1.Computer Vision priniciple,algorithms,applications,learning,fifth edition(E.R.Davies)
2.Mashine learning(Zhou zhihua)
3.Statistical Learning Methods（Li hang）
