OpenCV python

简介
图像篇
- 图片的读取、显示和保存
- 画图
- 填充凸多边形
- 通道分离和合并
- 图像的运算
- - 加法
  - numpy的加法
  - 与运算
  - 或运算
  - 异或运算
  - 非运算
  - 权重叠加
- 图像边界填充
- 图像阈值处理
- 滤波
- - 均值滤波
  - 方框滤波
  - 高斯滤波
  - 中值滤波
  - 腐蚀和膨胀
  - 梯度运算
  - 开运算、闭运算、礼帽、黑帽
  - 梯度计算
  - - sobel算子
    - scharr算子
    - laplacian算子
  - candy边缘检测
  - 拉普拉斯计算
- 轮廓检测
- - 轮廓的外接矩形
  - 轮廓的外接圆
- 均衡化
- - 直方图统计灰度图像素值
  - 直方图统计彩色图像素值
  - 部分图像的的像素值直方图
  - 均衡化
  - 自适应直方图均衡化
- 傅里叶变换
视频篇
- 视频读取、打开摄像头、保存图像
应用篇
- 1. OCR文字识别
- 2. 角点检测
- 3. 图像拼接
- 4. 车位检测
- 5. 背景建模
- 6. 光流估计
- 7. 用dnn模块加载模型
- 8. 多目标追踪

简介

本系列旨在记录人工智能边缘计算的基础知识，共分为三部分：

OpenCV-python:：图像、视频数据的处理、一些应用
Qt：软件界面设计
SDK调用：调用已有的人工智能模型

所需python工具包：

pip install numpy
pip install opencv-python
pip install matplotlib

图像篇

图片的读取、显示和保存

import numpy as np
import cv2
import matplotlib.pyplot as plt

# 读取图片  彩色（默认） 1、灰色 0、带alpha通道 -1
img = cv2.imread("img.png", 1)
cv2.imshow("img", img)
cv2.waitKey(0)

#打印图片属性
print(img.shape, img.size, img.type)

# 用plt显示
plt.imshow(img[:100, :100, ::-1])
plt.title("plt显示结果"), plt.xticks([]), plt.yticks([])
# 汉字防止出现乱码
plt.rcParams['font.sans-serif']=['SimHei']
plt.rcParams['axes.unicode_minus'] = False
plt.show()
# 保存图像
cv.imwrite('img_leftTop.png',img[:100, :100, :])

cv2显示结果：

图片属性：

(281, 500, 3)：对应（高、宽、通道）
421500 ：281 * 500 * 3，数据量
uint8：无符号8bit整数

plt这里截取了左上角一块100*100的部分图片，由于cv2读取的图片数据格式为BGR，所以对img数组的最后一维进行了倒置，显示结果：

画图

# 在img上画一条从[60, 60]到[440,60]宽度为5的蓝色（这里是BGR）直线
cv2.line(img, [60, 60], [440, 60], [255, 0, 0], 5)
# 在img上画一个从[240,100]到[260,150]的红色实心矩形,当画笔粗细为-1时画实心图形
cv2.rectangle(img, (240, 100), (260, 150), (0, 0, 255), -1)
# 在img上画一个圆心为（250，200），半径为30线条宽度为5的黄色空心圆
cv2.circle(img, [250, 200], 30, (0, 255, 255), 5)
# 在img上（250，450）位置写上Jack,字体为cv2.FONT_HERSHEY_SIMPLEX,大小为 1，颜色为绿色，线条粗细为22，这里不支持中文
cv2.putText(img, 'Jack', (400, 250), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)
# 深拷贝
import copy
a = copy.deepcopy(img[125:175, 100:150, :])
# 交换两个区域的部分图片
img[125:175, 100:150, :] = img[125:175, 350:400, :]
img[125:175, 350:400, :] = a
# 将通道由BGR转变为RGB
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
plt.imshow(img)
plt.show()

填充凸多边形

# 在img上填充凸多边形
area1 = [[150, 80], [115, 160], [195, 160]]
# 填充单个
cv2.fillConvexPoly(img, np.array(area1), (0, 255, 0))
plt.imshow(img[:, :, ::-1])
plt.show()
area2 = [[350, 80], [305, 160], [395, 160]]
# 填充多个
cv2.fillPoly(img, [np.array(area1), np.array(area2)], (0, 0, 0))
plt.imshow(img[:, :, ::-1])
plt.show()

单个：

多个：

通道分离和合并

# 通道拆分
b,g,r = cv2.split(img)
print(b.shape)
b_img = np.zeros((281, 500, 3), np.uint8)
b_img[:, :, :1] = np.expand_dims(b, 2)
plt.imshow(b_img[:, :, ::-1])
plt.show()
g_img = np.zeros((281, 500, 3), np.uint8)
g_img[:, :, 1:2] = np.expand_dims(g, 2)
plt.imshow(g_img[:, :, ::-1])
plt.show()
r_img = np.zeros((281, 500, 3), np.uint8)
r_img[:, :, 2:3] = np.expand_dims(r, 2)
plt.imshow(r_img[:, :, ::-1])
plt.show()
# 通道合并
img = cv2.merge((b, g, r))
plt.imshow(img)
plt.show()

B：

R:

合并：

图像的运算

这里需保证参与运算的图片shape相等

加法

img1 = cv2.imread("img_1.png")
img2 = cv2.imread("img_2.png")
plt.imshow(img1[:, :, ::-1])
plt.show()
plt.imshow(img2[:, :, ::-1])
plt.show()
img = cv2.add(img1, img2)
plt.imshow(img[:, :, ::-1])
plt.show()

img1原图:

img2原图:

相加后：（当相加和大于255时取255）

numpy的加法

img1 = cv2.imread("img_1.png")
img2 = cv2.imread("img_2.png")
img = img1 + img2
plt.imshow(img[:, :, ::-1])
plt.show()

和超过255时取255余数：

与运算

img = cv2.bitwise_and(img1, img2)
plt.imshow(img[:, :, ::-1])
plt.show()

1 & 1 = 1, 其他为0:

或运算

img = cv2.bitwise_or(img1, img2)
plt.imshow(img[:, :, ::-1])
plt.show()

** 0 | 0 = 0, 其他为1:**

异或运算

img = cv2.bitwise_xor(img1, img2)
plt.imshow(img[:, :, ::-1])
plt.show()

1 ^ 1, 0 ^ 0 为1，其他为0:

非运算

img = cv2.bitwise_not(img1)
plt.imshow(img[:, :, ::-1])
plt.show()

!x = 255 - x:

权重叠加

img = cv2.addWeighted(img1, 0.7, img2, 0.3, 0)
plt.imshow(img[:, :, ::-1])
plt.show()

0.7img+0.3img2+0：

图像边界填充

img = cv2.imread("img.png")
replicate = cv2.copyMakeBorder(img, 50, 50, 50, 50, cv2.BORDER_REPLICATE)
reflect = cv2.copyMakeBorder(img, 50, 50, 50, 50, cv2.BORDER_REFLECT_101)
wrap = cv2.copyMakeBorder(img, 50, 50, 50, 50, cv2.BORDER_WRAP)
constant = cv2.copyMakeBorder(img, 50, 50, 50, 50, cv2.BORDER_CONSTANT, value=(255, 0, 0))
plt.subplot(1, 5, 1), plt.imshow(img[:, :, ::-1]), plt.title("original")
plt.subplot(1, 5, 2), plt.imshow(replicate[:, :, ::-1]), plt.title("replicate")
plt.subplot(1, 5, 3), plt.imshow(reflect[:, :, ::-1]), plt.title("reflect")
plt.subplot(1, 5, 4), plt.imshow(wrap[:, :, ::-1]), plt.title("wrap")
plt.subplot(1, 5, 5), plt.imshow(constant[:, :, ::-1]), plt.title("constant")
plt.show()

图像阈值处理

# 阈值处理
_, thresh1 = cv2.threshold(img, 128, 200, cv2.THRESH_BINARY)
_, thresh2 = cv2.threshold(img, 128, 200, cv2.THRESH_BINARY_INV)
_, thresh3 = cv2.threshold(img, 128, 200, cv2.THRESH_TRUNC)
_, thresh4 = cv2.threshold(img, 128, 200, cv2.THRESH_TOZERO)
_, thresh5 = cv2.threshold(img, 128, 200, cv2.THRESH_TOZERO_INV)
plt.subplot(2, 3, 1), plt.imshow(img[:, :, ::-1]), plt.title("原图")
plt.subplot(2, 3, 2), plt.imshow(thresh1[:, :, ::-1]), plt.title("大于128的取200")
plt.subplot(2, 3, 3), plt.imshow(thresh2[:, :, ::-1]), plt.title("二值的反转")
plt.subplot(2, 3, 4), plt.imshow(thresh3[:, :, ::-1]), plt.title("大于128的设为128")
plt.subplot(2, 3, 5), plt.imshow(thresh4[:, :, ::-1]), plt.title("小于128的设为0")
plt.subplot(2, 3, 6), plt.imshow(thresh5[:, :, ::-1]), plt.title("to0的反转")
plt.rcParams['font.sans-serif']=['SimHei']#汉字防止出现乱码
plt.rcParams['axes.unicode_minus'] = False
plt.show()

滤波均值滤波

img = cv2.imread("img.png")
# 均值滤波
img_average = cv2.blur(img, (3, 3))
plt.subplot(1, 2, 1), plt.imshow(img[:, :, ::-1]), plt.title("原图"), plt.xticks([]), plt.yticks([])
plt.subplot(1, 2, 2), plt.imshow(img_average[:, :, ::-1]), plt.title("均值滤波"), plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams['axes.unicode_minus'] = False
plt.show()

方框滤波

img_box = cv2.boxFilter(img, -1, (5, 5), normalize=False)
plt.subplot(1, 2, 1), plt.imshow(img[:, :, ::-1]), plt.title("原图"), plt.xticks([]), plt.yticks([])
plt.subplot(1, 2, 2), plt.imshow(img_box[:, :, ::-1]), plt.title("方框滤波-未求均值"), plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams['axes.unicode_minus'] = False
plt.show()

当normal为true时与均值滤波相同，当为false时不取均值，和大于255的取255:

高斯滤波

img_gauss = cv2.GaussianBlur(img, (5, 5), 1)
plt.subplot(1, 2, 1), plt.imshow(img[:, :, ::-1]), plt.title("原图"), plt.xticks([]), plt.yticks([])
plt.subplot(1, 2, 2), plt.imshow(img_gauss[:, :, ::-1]), plt.title("高斯滤波"), plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams['axes.unicode_minus'] = False
plt.show()

中值滤波

img_median = cv2.medianBlur(img, 5)
plt.subplot(1, 2, 1), plt.imshow(img[:, :, ::-1]), plt.title("原图"), plt.xticks([]), plt.yticks([])
plt.subplot(1, 2, 2), plt.imshow(img_median[:, :, ::-1]), plt.title("中值滤波"), plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams['axes.unicode_minus'] = False
plt.show()

腐蚀和膨胀

kernel = np.ones((3, 3), np.uint8)
img_erosion = cv2.erode(img, kernel, iterations=8)
img_dilate = cv2.dilate(img, kernel, iterations=8)
# 水平拼接
pk = np.hstack((img, img_erosion, img_dilate))
plt.imshow(pk[:, :, ::-1]), plt.title("原图 vs 腐蚀 vs 膨胀"), plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams['axes.unicode_minus'] = False
plt.show()

梯度运算

img_erosion = cv2.erode(img, kernel, iterations=1)
img_dilate = cv2.dilate(img, kernel, iterations=1)
td = img_dilate - img_erosion
gradient = cv2.morphologyEx(img, cv2.MORPH_GRADIENT, kernel)
pk = np.hstack((td, gradient))
plt.imshow(pk[:, :, ::-1]), plt.title("梯度运算：膨胀-腐蚀 vs  gradient"), plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams['axes.unicode_minus'] = False
plt.show()

开运算、闭运算、礼帽、黑帽

img_open = cv2.morphologyEx(img, cv2.MORPH_OPEN, kernel)
img_close = cv2.morphologyEx(img, cv2.MORPH_CLOSE, kernel)
img_limao = cv2.morphologyEx(img, cv2.MORPH_TOPHAT, kernel)
img_heimao = cv2.morphologyEx(img, cv2.MORPH_BLACKHAT, kernel)
pk = np.hstack((img_open, img_close, img_limao, img_heimao))
plt.imshow(pk[:, :, ::-1])
plt.title("开运算：先腐蚀后膨胀 vs  闭运算：先膨胀再腐蚀 vsn"
    "礼帽：img-开运算结果 vs 黑帽：img-闭运算结果"), plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams['axes.unicode_minus'] = False
plt.show()

梯度计算 sobel算子

sobel_x = cv2.Sobel(img, cv2.CV_64F, 1, 0, ksize=3)
# 取结果的绝对值
sobel_x = cv2.convertScaleAbs(sobel_x)
sobel_y = cv2.Sobel(img, cv2.CV_64F, 0, 1, ksize=3)
sobel_y = cv2.convertScaleAbs(sobel_y)
sobel_xy = cv2.addWeighted(sobel_x, 0.5, sobel_y, 0.5, 0)
pk = np.hstack((sobel_x, sobel_y, sobel_xy))
plt.imshow(pk)
plt.title("梯度计算:x vs y vs xy"), plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams['axes.unicode_minus'] = False
plt.show()

scharr算子

scharr_x = cv2.Scharr(img, cv2.CV_64F, 1, 0)
scharr_x = cv2.convertScaleAbs(scharr_x)
scharr_y = cv2.Scharr(img, cv2.CV_64F, 0, 1, ksize=3)
scharr_y = cv2.convertScaleAbs(scharr_y)
scharr_xy = cv2.addWeighted(scharr_x, 0.5, scharr_y, 0.5, 0)
plt.imshow(scharr_xy)
plt.title("scharr算子"), plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams['axes.unicode_minus'] = False
plt.show()

比sobel算子效果更明显

laplacian算子

Laplacian = cv2.Laplacian(img, cv2.CV_64F, ksize=3)
Laplacian = cv2.convertScaleAbs(Laplacian)
plt.imshow(Laplacian)
plt.title("Laplacian算子"), plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams['axes.unicode_minus'] = False
plt.show()

candy边缘检测

img_candy = cv2.Canny(img, 50, 100)
plt.imshow(img_candy), plt.title("candy"),plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams['axes.unicode_minus'] = False
plt.show()

大致流程：

这里使用的双阈值为（50，100），阈值越小细节越多

拉普拉斯计算

上采样和下采样：

img = cv2.imread("img_3.png")
print("img：",  img.shape)
img1 = cv2.pyrDown(img)
print("img1：",  img1.shape)
img2 = cv2.pyrUp(img)
print("img2：",  img2.shape)

这里是用的高斯内核卷积，放大或缩小2倍，运行结果：

img： (313, 500, 3)
img1： (157, 250, 3)
img2： (626, 1000, 3)

拉普拉斯计算：

laplas = img- cv2.pyrUp(cv2.pyrDown(img))[:313]
plt.subplot(1, 2, 1), plt.imshow(img[:, :, ::-1]), plt.title("原图")
plt.xticks([]), plt.yticks([])
plt.subplot(1, 2, 2), plt.imshow(laplas[:, :, ::-1]), plt.title("laplas=img-up(down(img))")
plt.xticks([]), plt.yticks([])
plt.rcParams["font.sans-serif"] = ["SimHei"]
plt.rcParams["axes.unicode_minus"] = False
plt.show()

轮廓检测

img = cv2.imread("img_2.png")
# 预处理
img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
_, thresh = cv2.threshold(img_gray, 127, 255, cv2.THRESH_BINARY)
contours, hierarchy = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
# drawContours会改变原图，所以这里复制一份
img_draw = img.copy()
img_draw = cv2.drawContours(img_draw, contours, -1, (0, 0, 255), 2)
pk = np.hstack((img, img_draw))
plt.imshow(pk[:, :, ::-1]), plt.xticks([]), plt.yticks([]),
plt.show()

轮廓的外接矩形

con = contours[300]
# 边界矩形
x, y, w, h = cv2.boundingRect(con)
img_rect = cv2.rectangle(img, (x, y), (x+w, y+h), (0, 0, 255), 2)
plt.imshow(img_rect[:, :, ::-1]), plt.xticks([]), plt.yticks([]),
plt.show()

轮廓的外接圆

(x, y), radius = cv2.minEnclosingCircle(con)
img_circle = cv2.circle(img, (int(x), int(y)), int(radius), (0, 0, 255), 2)
plt.imshow(img_circle[:, :, ::-1]), plt.xticks([]), plt.yticks([]),
plt.show()

均衡化直方图统计灰度图像素值

img = cv2.imread("img.png", 0)
# 直方图统计
hist = cv2.calcHist([img], [0], None, [256], [0, 256])
print(hist.shape)
plt.hist(img.ravel(), 256)
plt.show()

直方图统计彩色图像素值

# 彩色直方图
hist1 = cv2.calcHist([img], [0], None, [256], [0, 256])
plt.plot(hist1, color='b'), plt.xlim([0, 256])
hist2 = cv2.calcHist([img], [1], None, [256], [0, 256])
plt.plot(hist2, color='g'), plt.xlim([0, 256])
hist3 = cv2.calcHist([img], [2], None, [256], [0, 256])
plt.plot(hist3, color='r'), plt.xlim([0, 256])
plt.show()

部分图像的的像素值直方图

img = cv2.imread("img.png", 0)
mask = np.zeros(img.shape[:2], np.uint8)
mask[50:250, 100:400] = 255
img_masked = cv2.bitwise_and(img, mask)
plt.imshow(img_masked, "gray"), plt.xticks([]), plt.yticks([])
plt.show()
hist = cv2.calcHist([img], [0], mask, [256], [0, 256])
plt.plot(hist), plt.xlim([0, 256])
plt.show()

添加mask后的图片：

掩码后的图像像素直方图：

均衡化

各个像数值从小到大的累积概率 * 255取整

img_equal = cv2.equalizeHist(img)
plt.hist(img_equal.ravel(), 256)
plt.show()
pk = np.hstack((img, img_equal))
plt.imshow(pk, "gray"), plt.xticks([]), plt.yticks([])
plt.show()

均衡化后的直方图;

均衡化后的图像对比，均衡化后的图片（右）明显更亮：

自适应直方图均衡化

clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
img_clahe = clahe.apply(img)
pk = np.hstack((img, img_clahe))
plt.imshow(pk, "gray"), plt.xticks([]), plt.yticks([])
plt.show()

将图像分成8 * 8份，分别进行均衡化后拼接到一起，连接处插值处理，相比于整体的均衡化，保留了更多细节：

傅里叶变换

img = cv2.imread("img.png", 0)
img_f32 = np.float32(img)
dft = cv2.dft(img_f32, flags=cv2.DFT_COMPLEX_OUTPUT)
dft_shift = np.fft.fftshift(dft)
# 将数值放大到0~255
dft_magnitude = 20 * np.log(cv2.magnitude(dft_shift[:, :, 0], dft_shift[:, :, 1]))
plt.subplot(121), plt.imshow(img, "gray"), plt.title("img"), plt.xticks([]), plt.yticks([])
plt.subplot(122), plt.imshow(dft_magnitude, "gray"), plt.title("dft_magnitude"), plt.xticks([]), plt.yticks([])
plt.show()
rows, cols = img.shape
center_x, center_y = int(cols/2), int(rows/2)
# 低通滤波,使图像模糊
mask = np.zeros((rows, cols, 2), np.uint8)
mask[center_y-30:center_y+30, center_x-30:center_x+30] = 1
# 傅里叶反变换
f_shift = dft_shift * mask
f_ishift = np.fft.ifftshift(f_shift)
img_trans = cv2.idft(f_ishift)
img_trans = cv2.magnitude(img_trans[:, :, 0], img_trans[:, :, 1])
plt.subplot(121), plt.imshow(img, "gray"), plt.title("img"), plt.xticks([]), plt.yticks([])
plt.subplot(122), plt.imshow(img_trans, "gray"), plt.title("img_trans"), plt.xticks([]), plt.yticks([])
plt.show()
# 高通滤波,保留细节
mask = np.ones((rows, cols, 2), np.uint8)
mask[center_y-30:center_y+30, center_x-30:center_x+30] = 0
f_shift = dft_shift * mask
f_ishift = np.fft.ifftshift(f_shift)
img_trans = cv2.idft(f_ishift)
img_trans = cv2.magnitude(img_trans[:, :, 0], img_trans[:, :, 1])
plt.subplot(121), plt.imshow(img, "gray"), plt.title("img"), plt.xticks([]), plt.yticks([])
plt.subplot(122), plt.imshow(img_trans, "gray"), plt.title("img_trans"), plt.xticks([]), plt.yticks([])
plt.show()

原图与空间频域图对比：

原图与低通滤波对比：

原图与高通滤波对比：

视频篇

视频读取、打开摄像头、保存图像

import cv2

# 读取图像
video = cv2.VideoCapture()
if not video.open("my_map.mp4"):
    print("can not open the video")
    exit(1)
else:
    print("帧率：", video.get(cv2.CAP_PROP_FPS)) # cv2.CAP_PROP_FPS==5
    print("帧数：", video.get(cv2.CAP_PROP_FRAME_COUNT)) # cv2.CAP_PROP_FRAME_COUNT==7
    print("宽：", video.get(cv2.CAP_PROP_FRAME_WIDTH), "高：", video.get(cv2.CAP_PROP_FRAME_HEIGHT))
# 按帧读取视频
count = 0
while video.isOpened():               # 当视频被打开时：
    success, frame = video.read()         # 读取视频，读取到的某一帧存储到frame，若是读取成功，ret为True，反之为False
    if success:                         # 若是读取成功
        count += 1
        cv2.putText(frame, str(count)+"th", (20, 150), cv2.FONT_HERSHEY_DUPLEX, 0.9,
                (147, 58, 31), 1)
        cv2.imshow('frame', frame)  # 显示读取到的这一帧画面
        key = cv2.waitKey(25)       # 等待一段时间，并且检测键盘输入
        if key == ord('s'):         # 若是键盘输入's',则保存当前帧
            cv2.imwrite(str(count)+".jpg", frame)
        if key == ord('q'):         # 若是键盘输入'q',则退出，释放视频
            video.release()           # 释放视频
            break
    else:
        video.release()
cv2.destroyAllWindows()             # 关闭所有窗口
print(count)



# 打开摄像头
cap = cv2.VideoCapture(0)
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
size = (int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)), int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)))
# 创建VideoWriter
out = cv2.VideoWriter('camera.mp4', fourcc, cap.get(5), size)
# 按帧读取视频
count = 0
while cap.isOpened():               # 当视频被打开时：
    success, frame = cap.read()         # 读取视频，读取到的某一帧存储到frame，若是读取成功，ret为True，反之为False
    # 翻转
    frame = cv2.flip(frame, 1)
    if success:                         # 若是读取成功
        count += 1
        # 写入视频
        out.write(frame)
        cv2.putText(frame, str(count)+"th", (20, 150), cv2.FONT_HERSHEY_DUPLEX, 0.9,
                (147, 58, 31), 1)
        cv2.imshow('frame', frame)  # 显示读取到的这一帧画面
        key = cv2.waitKey(25)       # 等待一段时间，并且检测键盘输入

        if key == ord('q'):         # 若是键盘输入'q',则退出，释放视频
            cap.release()           # 释放视频
            break
    else:
        cap.release()
out.release()
cv2.destroyAllWindows()             # 关闭所有窗口
print(count)

第495帧的图像：

摄像头录制保存成功：

应用篇

1. OCR文字识别

这里识别主要是利用最外边的大框将图片转正，然后用tesseract进行ocr识别，特殊情况需特殊处理，这里仅针对这种含有外边框的特殊情况，进行其他图片的识别可能结果并不是很好。
tesseract下载地址：https://digi.bib.uni-mannheim.de/tesseract/
tesseract的git地址：https://github.com/tesseract-ocr/tesseract/
安装tesseract后需要在所在环境pip install pytesseract
然后在python环境的site-packages包中找到pytesseract.py,更改其中的tesseract为绝对路径：

然后需以管理员身份运行pycharm。

import numpy as np
import cv2
import argparse

ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True, help="扫描图像的路径")
args = vars(ap.parse_args())
print(args)


def order_points(pts):
    rect = np.zeros((4, 2), dtype="float32")
    s = pts.sum(axis=1)
    rect[0] = pts[np.argmin(s)]
    rect[2] = pts[np.argmax(s)]

    diff = np.diff(pts, axis=1)
    rect[1] = pts[np.argmin(diff)]
    rect[3] = pts[np.argmax(diff)]

    return rect


def four_points_transform(image, pts):
    rect = order_points(pts)
    (tl, tr, br, bl) = rect

    widthA = np.sqrt(((br[0] - bl[0]) ** 2) + ((br[1] - bl[1]) ** 2))
    widthB = np.sqrt(((tr[0] - tl[0]) ** 2) + ((tr[1] - tl[1]) ** 2))
    max_width = max(int(widthA), int(widthB))
    heightA = np.sqrt(((tr[0] - br[0]) ** 2) + ((tr[1] - br[1]) ** 2))
    heightB = np.sqrt(((tl[0] - bl[0]) ** 2) + ((tl[1] - bl[1]) ** 2))
    max_height = max(int(heightA), int(heightB))

    dst = np.array([
        [0, 0],
        [max_width - 1, 0],
        [max_width - 1, max_height - 1],
        [0, max_height - 1]
        ], dtype="float32")

    M = cv2.getPerspectiveTransform(rect, dst)
    wraped = cv2.warpPerspective(image, M, (max_width, max_height))

    return wraped


def resize(image, width=None, height=None, inter=cv2.INTER_AREA):
    dim = None
    (h, w) = image.shape[:2]
    if width is None and height is None:
        return image
    if width is None:
        r = height / float(h)
        dim = (int(w * r), height)
    else:
        r = width / float(w)
        dim = (width, int(h * r))
    resized = cv2.resize(image, dim, interpolation=inter)
    return resized


img = cv2.imread(args["image"])
ratio = img.shape[0] / 1000.0
orig = img.copy()

img = resize(orig, height=1000)

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# 高斯滤波去除噪音点
gray = cv2.GaussianBlur(gray, (5, 5), 0)
# 边缘检测
edged = cv2.Canny(gray, 75, 200)
cv2.imshow("image", img)
cv2.imshow("edge", edged)
cv2.waitKey(0)
cv2.destroyAllWindows()
# 轮廓检测
cnts = cv2.findContours(edged.copy(), cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)[0]
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[:5]
# 遍历轮廓
for c in cnts:
    # 计算轮廓周长
    peri = cv2.arcLength(c, True)
    # 为轮廓近似外接化
    approx = cv2.approxPolyDP(c, 0.02 * peri, True)
    if len(approx) == 4:
        screenCnt = approx
        break
cv2.drawContours(img, [screenCnt], -1, (255, 0, 0), 2)
cv2.imshow("lined", img)
cv2.waitKey(0)
cv2.destroyAllWindows()

warped = four_points_transform(orig, screenCnt.reshape(4, 2) * ratio)
warped = cv2.cvtColor(warped, cv2.COLOR_BGR2GRAY)

cv2.imshow("warped", warped)
cv2.waitKey(0)
cv2.destroyAllWindows()
cv2.imwrite("scan.jpg", warped)

import pytesseract
from PIL import Image


text = pytesseract.image_to_string(Image.open("scan.jpg"))
print(text)

原图：

边缘检测：

轮廓检测，最大的轮廓：

图像转换：

ocr文字识别结果，识别结果有些许瑕疵，但整体效果还不错：

2. 角点检测

import cv2

img = cv2.imread("img_1.png")
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

dst = cv2.cornerHarris(gray, 2, 3, 0.04)

img[dst > 0.05*dst.max()] = [0, 0, 255]
cv2.imshow("corner", img)
cv2.waitKey(0)
cv2.destroyAllWindows()

3. 图像拼接

用sift函数获取关键点：

img = cv2.imread("img_2.png")
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

sift = cv2.SIFT_create()
kp = sift.detect(gray, None)
img = cv2.drawKeypoints(gray, kp, img)

cv2.imshow("keyPoint", img)
cv2.waitKey(0)
cv2.destroyAllWindows()

匹配特征相似的关键点：

img1 = cv2.imread("img_2.png")
img2 = cv2.imread("img_3.png")
gray1 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)
gray2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)

sift = cv2.SIFT_create()
keypoint1, descriptors1 = sift.detectAndCompute(gray1, None)
keypoint2, descriptors2 = sift.detectAndCompute(gray2, None)

# 蛮力匹配，计算特征向量欧式距离
bf = cv2.BFMatcher(crossCheck=True)

match = bf.match(descriptors1, descriptors2)
match = sorted(match, key=lambda x: x.distance)
img = cv2.drawMatches(gray1, keypoint1, gray2, keypoint2, match[-20:], None, flags=2)

cv2.imshow("match", img)
cv2.waitKey(0)
cv2.destroyAllWindows()

#k对最佳匹配, 1个点匹配多个，由k决定
bf = cv2.BFMatcher()
match = bf.knnMatch(descriptors1, descriptors2, k=2)
good = []
for m, n in match:
    if m.distance < 0.75 * n.distance:
        good.append(m)
good = np.expand_dims(good, 1)
img = cv2.drawMatchesKnn(gray1, keypoint1, gray2, keypoint2, good[:20], None, flags=2)

cv2.imshow("knn_match", img)
cv2.waitKey(0)
cv2.destroyAllWindows()

蛮力匹配：

k对最佳匹配：

图像拼接：

# 图像拼接
MIN_MATCH_COUNT = 10
imgA, imgB = cv2.imread("img_4.jpg"), cv2.imread("img_5.jpg")
sift = cv2.SIFT_create()
(kpsA, featuresA), (kpsB, featuresB) = sift.detectAndCompute(imgA, None), sift.detectAndCompute(imgB, None)

matcher = cv2.BFMatcher()
good = []
matches = matcher.knnMatch(featuresB, featuresA, k=2)
for m, n in matches:
    if m.distance < n.distance * 0.75:
        good.append(m)
good1 = np.expand_dims(good, 1)
matching = cv2.drawMatchesKnn(imgA, kpsA, imgB, kpsB, good1[:20], None, matchColor=(0, 255, 0), flags=2)
if len(good) > MIN_MATCH_COUNT:
    src_pts = np.float32([kpsB[m.queryIdx].pt for m in good]).reshape(-1, 1, 2)
    dst_pts = np.float32([kpsA[m.trainIdx].pt for m in good]).reshape(-1, 1, 2)
    # 求转置矩阵
    H, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 5.0)
    # 求B转变后的图像
    wrap = cv2.warpPerspective(imgB, H, (imgB.shape[1] + imgB.shape[1], imgB.shape[0] + imgB.shape[0]))
    plt.imshow(wrap[:, :, ::-1]), plt.xticks([]), plt.yticks([])
    plt.show()
    wrap[0:imgB.shape[0], 0:imgB.shape[1]] = imgA
    # 去除黑色无用部分
    rows, cols = np.where(wrap[:, :, 0] != 0)
    min_row, max_row = min(rows), max(rows) + 1
    min_col, max_col = min(cols), max(cols) + 1
    result = wrap[min_row:max_row, min_col:max_col, :]

    plt.imshow(matching[:, :, ::-1]), plt.title("matching"), plt.xticks([]), plt.yticks([])
    plt.show()
    plt.imshow(result[:, :, ::-1]), plt.title("result"), plt.xticks([]), plt.yticks([])
    plt.show()

imgB经过转换后的图片：

特征匹配图:

图像拼接结果：

4. 车位检测

为图片制作掩膜：

img = cv2.imread("img_1.png")
lower = np.uint8([120, 120, 120])
upper = np.uint8([255, 255, 255])
#   [0,0,0] < lower < [255,255,255] < upper < [0,0,0]
mask = cv2.inRange(img, lower, upper)
showImage("mask", mask)

用mask提取图片关键部分：

masked_img = cv2.bitwise_and(img, img, mask=mask)
showImage("masked", masked_img)

边缘检测：

gray_img = cv2.cvtColor(masked_img, cv2.COLOR_BGR2GRAY)
# 边缘检测
edge_img = cv2.Canny(gray_img, 50, 200)
showImage("edge", edge_img)

搜索边缘中的直线并进行筛选：

# 直线检测 rho:距离精度， theta：角度精度，threshold:阈值,minLineLength:最小线长，maxLineGap：最大间隔
lines = list(cv2.HoughLinesP(edge_img, rho=0.1, theta=np.pi/10, threshold=15, minLineLength=80, maxLineGap=50))
line_img = img.copy()
# 过滤
good = []
for line in lines:
    for x1, y1, x2, y2 in line:
        if abs(x2-x1) <= 1 and abs(y2-y1) >= 180:
            good.append((x1, y1, x2, y2))
            cv2.line(line_img, (x1, y1), (x2, y2), [255, 0, 0], thickness=2)
showImage("lined", line_img)

按行聚簇停车位：

import operator
# 按照y1排序
list1 = sorted(good, key=operator.itemgetter(1))

# 按行聚簇
clusters = {}
dIndex = 0
clus_dist = 10
for i in range(len(list1) - 1):
    distance = abs(list1[i+1][1] - list1[i][1])
    if distance < clus_dist:
        if not dIndex in clusters.keys():
            clusters[dIndex] = []
        clusters[dIndex].append(list1[i])
        clusters[dIndex].append(list1[i+1])
    else:
        dIndex += 1
rects = {}
i = 0
for key in clusters:
    all_list = clusters[key]
    cleaned = list(set(all_list))
    if len(cleaned) > 3:
        cleaned = sorted(cleaned, key=lambda tup: tup[0])
        avg_x1 = cleaned[0][0]
        avg_x2 = cleaned[-1][0]
        avg_y1 = 0
        avg_y2 = 0
        for tup in cleaned:
            avg_y1 += tup[1]
            avg_y2 += tup[3]
        avg_y1 = avg_y1/len(cleaned)
        avg_y2 = avg_y2/len(cleaned)
        rects[i] = (avg_x1, avg_y1, avg_x2, avg_y2)
        i += 1

rects_img = img.copy()
for key in rects:
    tup_topLeft = (int(rects[key][0]), int(rects[key][1]))
    tup_botRight = (int(rects[key][2]), int(rects[key][3]))
    cv2.rectangle(rects_img, tup_topLeft, tup_botRight, (0, 255, 0), 3)
showImage("rect", rects_img)

制作停车位标签并保存每个停车位的图片以用于后续模型训练：

# 分割,制作标号字典
spot_dict = {}
for key in rects:
    x1, y1, x2, y2 = rects[key]
    dis_x = abs(x2 - x1)/6
    dis_y = abs(y1 - y2)/2
    for i in range(6):
        spot_dict[i+1+int(key)*6*2] = (int(x1 + i*dis_x), int(y2), int(x1 + (i+1)*dis_x), int(y2 + dis_y))
        spot_dict[i+7+int(key)*6*2] = (int(x1 + i*dis_x), int(y2+dis_y), int(x1 + (i+1)*dis_x), int(y1))


# 写入训练图片
import os


def writeImage():
    cwd = os.getcwd()
    for key in spot_dict:
        (x1, y1, x2, y2) = spot_dict[key]
        filename = "img/train/spot_" + str(key) + ".jpg"
        spot_img = img[y1:y2, x1:x2]
        cv2.imwrite(filename, spot_img)
writeImage()

这里用的是pytorch框架训练的模型，由于判断0，1问题比较简单，这里仅展示模型结构
模型结构：

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.initial = nn.Sequential(
            nn.Conv2d(3, 64, 3, 2, 1, padding_mode="reflect"),
            nn.BatchNorm2d(64),
            nn.ReLU()
        )
        self.body = nn.Sequential(
            nn.Conv2d(64, 32, 3, 2, 1, padding_mode="reflect"),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.Conv2d(32, 16, 3, 2, 1, padding_mode="reflect"),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.Conv2d(16, 4, 3, 2, 1, padding_mode="reflect"),
            nn.BatchNorm2d(4),
            nn.ReLU(),
            nn.AvgPool2d(4, 7)
        )
        self.fn = nn.Linear(4, 2)

    def forward(self, x):
        x = self.initial(x)
        x = self.body(x)
        x = torch.flatten(x, 1)
        return self.fn(x)

更换图片进行测试，图片由同样的方法进行车位分割，并由模型对每个位置进行判断，处理代码与上面相同，除了不再调用writeImage() ：

# 更换图片后，判断
# 加载模型
from train import MyModel, DEVICE
from utils import load_checkpoint
from dataset import trans
import torch


def empty_detect():
    empty = 0
    result_img = img.copy()
    model = MyModel().to(DEVICE)
    load_checkpoint("spoting_judge.pth.tar", model)
    model.eval()
    with torch.no_grad():
        for key in spot_dict:
            (x1, y1, x2, y2) = spot_dict[key]
            spot_img = img[y1:y2, x1:x2]
            spot_img = cv2.cvtColor(spot_img, cv2.COLOR_BGR2RGB)
            spot_img = trans(spot_img).unsqueeze(0).to(DEVICE)
            output = model(spot_img)
            _, not_empty = torch.max(output.data, 1)  # 获得最大值索引
            if not not_empty:
                empty += 1
                cv2.rectangle(result_img, (x1, y1), (x2, y2), (0, 0, 255), 2)
    cv2.putText(result_img, "empty:{}/{}".format(empty, len(spot_dict)), (10, 20), cv2.FONT_HERSHEY_SIMPLEX,  0.75, (0, 0, 255), 1, cv2.LINE_AA)
    showImage("result", result_img)


empty_detect()

左上角显示空车位个数，并将空车位用红框标记显示：

5. 背景建模

所谓背景建模就是通过计算前后帧的像素变化来分离出运动的物体，这里用的方法是高斯混合模型：createBackgroundSubtractorMOG2()

import numpy as np
import cv2

cap = cv2.VideoCapture("video.mp4")
# 开运算需要的kernel：3*3的椭圆结构
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (3, 3))

# 创建混合高斯模型用于背景建模,参数介绍：
# history：用于训练背景的帧数，默认帧数为500帧;
# varThreshold:方差阈值，用于判断当前像素是前景还是背景,一般默认为16.如果光照变化明显，如阳光下的水
# 面，建议设为25，值越大灵敏度越低;
# detectShadows：是否检测影子，设为true为检测，false为不检测，一般设置为false;
fgbg = cv2.createBackgroundSubtractorMOG2()

while 1:
    ret, frame = cap.read()
    fgmask = fgbg.apply(frame)
    # 开运算去噪点
    fgmask = cv2.morphologyEx(fgmask, cv2.MORPH_OPEN, kernel)
    # 寻找轮廓
    contours, hierarchy = cv2.findContours(fgmask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

    for c in contours:
        # 计算周长
        perimeter = cv2.arcLength(c, True)
        if perimeter > 500:
            x, y, w, h = cv2.boundingRect(c)
            cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)
    cv2.imshow("dance", frame)
    cv2.imshow("dance people", fgmask)
    k = cv2.waitKey(100) & 0xff
    if k == 27:
        break
cap.release()
cv2.destroyAllWindows()

其中一帧：

6. 光流估计

这里主要用calcOpticalFlowPyrLK()进行光流估计，根据前后帧的特征点跟踪特征点的位置。
参数介绍：

prevImg
前一帧的灰度图；
nextImg
当前图像的灰度图；
prevPts
前一帧图像的特征点；
nextPts
当前图像的特征点（输出）；
status：输出状态向量（无符号char），如果在当前图像中能够光流得到标定的特征点位置改变，则设置status的对应位置为1，否则设置为0
err：输出错误向量；
winSize：在每个金字塔水平搜寻窗口的尺寸；
maxLevel：金字塔的高度，初始为3层

import numpy as np
import cv2

cap = cv2.VideoCapture("video.mp4")

feature_params = dict(
    maxCorners=100,
    qualityLevel=0.3,
    minDistance=7
)

lk_params = dict(winSize=(15, 15), maxLevel=2)

color = np.random.randint(0, 255, (100, 3))
# 从570帧开始检测
for _ in range(570):
    cap.read()

ret, old_frame = cap.read()
old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
p0 = cv2.goodFeaturesToTrack(old_gray, mask=None, **feature_params)
mask = np.zeros_like(old_frame)

while True:
    ret, frame = cap.read()
    frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)

    good_new = p1[st == 1]
    good_old = p0[st == 1]

    for i, (new, old) in enumerate(zip(good_new, good_old)):
        a, b = new.ravel()
        c, d = old.ravel()
        mask = cv2.line(mask, (int(a), int(b)), (int(c), int(d)), color[i].tolist(), 2)
        frame = cv2.circle(frame, (int(a), int(b)), 5, color[i].tolist(), -1)
    img = cv2.add(frame, mask)
    # cv2.putText(img, "frame:{}th".format(j), (10, 20), cv2.FONT_HERSHEY_SIMPLEX,  0.75, (0, 0, 255), 1, cv2.LINE_AA)
    cv2.imshow("frame", img)
    if cv2.waitKey(100) & 0xff == 27:
        break
    old_gray = frame_gray.copy()
    p0 = good_new.reshape(-1, 1, 2)

cap.release()
cv2.destroyAllWindows()

一些关键点的轨迹：

7. 用dnn模块加载模型

这里以pytorch模型为例，加载的是风格迁移的模型，git地址：https://github.com/jcjohnson/fast-neural-style。
blobFromImage()用于对图像进行预处理，参数：

image:输入图像（1、3或者4通道）

可选参数

scalefactor:图像各通道数值的缩放比例
size:输出图像的空间尺寸,如size=(200,300)表示高h=300,宽w=200
mean:用于各通道减去的值，以降低光照的影响(e.g. image为bgr3通道的图像，mean=[104.0, 177.0, 123.0],表示b通道的值-104，g-177,r-123)
swapRB:交换RB通道，默认为False.(cv2.imread读取的是彩图是bgr通道)
crop:图像裁剪,默认为False.当值为True时，先按比例缩放，然后从中心裁剪成size尺寸
ddepth:输出的图像深度，可选CV_32F 或者 CV_8U.

import cv2

model_base_dir = "./models/total_model/"
# 为不同风格模型制作字典
modelName_map = {
    1: "udnie",
    2: "la_muse",
    3: "the_scream",
    4: "candy",
    5: "mosaic",
    6: "feathers",
    7: "starry_night",
    8: "composition_vii",
    9: "the_wave"
}


def get_model_from_style(style: int):
    """
    加载指定风格的模型
    :param style: 模型编码
    :return: model
    """

    model_name = modelName_map.get(style)
    model_path = model_base_dir + model_name + ".t7"
    model = cv2.dnn.readNetFromTorch(model_path)

    return model


for i in range(len(modelName_map)):
    net = get_model_from_style(i + 1)

    img = cv2.imread("cat.jpg")
    (h, w) = img.shape[:2]
    blob = cv2.dnn.blobFromImage(img, 1.0, (w, h), (103.939, 116.779, 123.680), swapRB=False, crop=False)

    net.setInput(blob)
    output = net.forward()

    output = output.reshape((3, output.shape[2], output.shape[3]))
    output[0] += 103.939
    output[1] += 116.779
    output[2] += 123.680
    output = output.transpose(1, 2, 0)
    cv2.imwrite(modelName_map[i+1]+'.jpg', output)

各种风格的合影，最左边为原图，右边风格依此按modelName_map排列：

8. 多目标追踪

暂时懒得写了，有空再更，找了个相似的：
https://blog.csdn.net/zxjoke/article/details/125657110

参考资料：
https://www.bilibili.com/video/BV1PV411774y
http://www.eepw.com.cn/article/202007/415158.htm

OpenCV python

Python相关栏目本月热门文章