- 前言
- Camera preview 总体概括
- Establish snapdragon surfaceview surface
- Camera Device Create Preview Session
- Camera preview 详细流程
- Camera Buffer Flow
- Camera Service dequeue buffer
- Graphic buffer is transmit by binder to surfaceflinger process
- Display module 显示data image
- 总结
- Camera sensor获取自然图像通过ISP处理之后到显示器显示，图像代表着image data buffer。为弄清楚snapdragon camera APP preview功能下image buffer是怎么从camera module传输到display module进行显示，本文详细介绍总体流程其中包括buffer flow和各部分组件功能。
- 主要涉及四个部分：1.snapdragon camera APP：发送image buffer request，设置surfaceview；2.camera framework and camera service：处理request回传result；3.surfaceflinger：处理buffe queue；4.surface view：建立显示surface，触发preview buffer。
Fig1是Google提供的camera 2API调用流程，主要关注画红线camera preview buffer flow流程，下面分为几个部分介绍：
首先，Snapdragon camera preview功能通过surfaceview创建surface，通过其显示image buffer。camera device createCaptureSession把创建成功surface传输下去。下面分析具体代码流程。
首先初始化创建surfaceview，并为其设置状态回调callback。
// display the view
mSurfaceView = (AutoFitSurfaceView) mRootView.findViewById(R.id.mdp_preview_content);
mSurfaceHolder = mSurfaceView.getHolder();
mSurfaceHolder.addCallback(callback);
mSurfaceView.addOnLayoutChangeListener(new View.OnLayoutChangeListener() {
@Override
public void onLayoutChange(View v, int left, int top, int right,
int bottom, int oldLeft, int oldTop, int oldRight,
int oldBottom) {
int width = right - left;
int height = bottom - top;
if (mFaceView != null) {
mFaceView.onSurfaceTextureSizeChanged(width, height);
}
if (mStatsNNFocusRenderer != null) {
mStatsNNFocusRenderer.onSurfaceTextureSizeChanged(width, height);
}
if (mT2TFocusRenderer != null) {
mT2TFocusRenderer.onSurfaceTextureSizeChanged(width, height);
}
}
});
当surfaceview创建成功时回调surfaceCreated，通过调用previewUIReady()去通知CaptureModule开启预览的操作，
private SurfaceHolder.Callback callback = new SurfaceHolder.Callback() {
// SurfaceHolder callbacks
@Override
public void surfaceCreated(SurfaceHolder holder) {
Log.v(TAG, "surfaceCreated");
mSurfaceHolder = holder;
previewUIReady();
if(mTrackingFocusRenderer != null && mTrackingFocusRenderer.isVisible()) {
mTrackingFocusRenderer.setSurfaceDim(mSurfaceView.getLeft(),
mSurfaceView.getTop(), mSurfaceView.getRight(), mSurfaceView.getBottom());
}
if(mT2TFocusRenderer != null && mT2TFocusRenderer.isShown()) {
mT2TFocusRenderer.setSurfaceDim(mSurfaceView.getLeft(), mSurfaceView.getTop(),
mSurfaceView.getRight(), mSurfaceView.getBottom());
}
if(mStatsNNFocusRenderer != null && mStatsNNFocusRenderer.isShown()) {
mStatsNNFocusRenderer.setSurfaceDim(mSurfaceView.getLeft(), mSurfaceView.getTop(),
mSurfaceView.getRight(), mSurfaceView.getBottom());
}
}
};
判断当前时刻CaptureUI的预览surface view是否创建成功并可用，获取此预览surface。
Surface surface = null;
try {
waitForPreviewSurfaceReady();
} catch (RuntimeException e) {
Log.v(TAG,
"createSession: normal status occur Time out waiting for surface ");
}
surface = getPreviewSurfaceForSession(id);
开始准备预览流显示surface通过create session配置。
ListCamera Device Create Preview Sessionsurfaces = mFrameProcessor.getInputSurfaces(); for(Surface surs : surfaces) { mPreviewRequestBuilder[id].addTarget(surs); list.add(surs); }
- 回到Fig1，随后Camera device创建capture session之后，通过setRepeatingRequest不断的发送request请求Show preview stream(想象stream是image buffer在不断的循环)，request在发送的过程中包含在queue的结构（repeating request list -> pending request queue -> in-progress queue），保证每时每刻camera HAL都能够有足够request buffer处理。
- Camera Device hardware对image buffer指向地址填充完毕之后，通过output stream（configured outputs）传输到surfaceview 创建的surface显示，自此整个体系形成一个生产-消费-归还的循环。
Fig2展现camera preview显示图像总体流程，并且表达出各部分之间的关系，一供分为三个层次结构。
- 顶部JAVA layer包含Java APP、Java framework，包括Snapdragon camera APP（未画出），surface view and display区域。
- 中部Native layer，包括camera service process and surfaceflinger process。
- 底部HAL layer，HIDL只是作为中间连接层，包含camera provider process，其中包含Camx module。
由于Fig2复杂性，首先介绍camera buffer flow勾勒出camera sensor获取image，处理之后传输到display显示的主线。之后在对其进行补充。
camera的buffer在使用功能上主要分为两类：output stream buffer and input stream buffer。Output and input主语是camera hardware，如果从camera HAL输出就是output stream。
- output buffer是仅有一块buffer内存，里面没有具体数据，HAL会将生成的数据填入到output buffer中。
- input buffer是有实际内容的buffer，通过reprocess流程将其送到HAL Layer处理。reprocess的使用情况举例，如使用HDR algorithm：native layer算法需要使用raw image进行处理合成，会通过output stream获取。处理完之后再通过input stream把raw image送给HAL layer，对其raw image转为YUV，再对其进行后续ISP algorithm process。Native layer algorithm use raw image处理会有更高的精度和无细节损失，但是Native layer并不能直接把raw image传输给Java layer使用，需要使用ISP硬件转换处理，所以再次传输到HAL。
Fig3显示一个生产消费者模型，buffer在其中是一个循环的流程（具体参考：BufferQueue 学习总结）。
Fig3应用到camera service 、 surfaceflinger and surfaceview之间关系的情境下，figure转化为Fig4。
真正的producer是surfaceview创建的native 对象ANativeWindow， nativewindow创建出preview stream buffer，surface and camera service之间有一个dequeue、queue循环buffer过程。camera service output stream作为中间bridge，通过binder调用到surfaceflinger，surfaceflinger使用buffer queue结构管理graphic buffer。
本文只介绍camera捕获image到display显示，所以只分析Fig4从surface、camera service到surfaceflinger单向流程。
函数调用逻辑如图Fig5显示，Camera3Device中threadloop函数一旦run之后就一直循环，不断下发camera preview request。Camera3OutputStream直接使用Anativewindow dequeue buffer，之后传递buffer handle让camera provider填充image data到申请好的buffer地址中。
status_t Camera3OutputStream::getBufferLockedCommon(ANativeWindowBuffer** anb, int* fenceFd) {
ATRACE_HFR_CALL();
status_t res;
if ((res = getBufferPreconditionCheckLocked()) != OK) {
return res;
}
bool gotBufferFromManager = false;
if (mUseBufferManager) {
sp gb;
res = mBufferManager->getBufferForStream(getId(), getStreamSetId(), &gb, fenceFd);
if (res == OK) {
// Attach this buffer to the bufferQueue: the buffer will be in dequeue state after a
// successful return.
*anb = gb.get();
res = mConsumer->attachBuffer(*anb);
if (shouldLogError(res, mState)) {
ALOGE("%s: Stream %d: Can't attach the output buffer to this surface: %s (%d)",
__FUNCTION__, mId, strerror(-res), res);
}
if (res != OK) {
checkRetAndSetAbandonedLocked(res);
return res;
}
gotBufferFromManager = true;
ALOGV("Stream %d: Attached new buffer", getId());
} else if (res == ALREADY_EXISTS) {
// Have sufficient free buffers already attached, can just
// dequeue from buffer queue
ALOGV("Stream %d: Reusing attached buffer", getId());
gotBufferFromManager = false;
} else if (res != OK) {
ALOGE("%s: Stream %d: Can't get next output buffer from buffer manager: %s (%d)",
__FUNCTION__, mId, strerror(-res), res);
return res;
}
}
if (!gotBufferFromManager) {
sp currentConsumer = mConsumer;
mLock.unlock();
nsecs_t dequeueStart = systemTime(SYSTEM_TIME_MONOTONIC);
res = currentConsumer->dequeueBuffer(currentConsumer.get(), anb, fenceFd);
nsecs_t dequeueEnd = systemTime(SYSTEM_TIME_MONOTONIC);
mDequeueBufferLatency.add(dequeueStart, dequeueEnd);
mLock.lock();
if (mUseBufferManager && res == TIMED_OUT) {
checkRemovedBuffersLocked();
sp gb;
res = mBufferManager->getBufferForStream(
getId(), getStreamSetId(), &gb, fenceFd, true);
if (res == OK) {
// Attach this buffer to the bufferQueue: the buffer will be in dequeue state after
// a successful return.
*anb = gb.get();
res = mConsumer->attachBuffer(*anb);
gotBufferFromManager = true;
ALOGV("Stream %d: Attached new buffer", getId());
if (res != OK) {
if (shouldLogError(res, mState)) {
ALOGE("%s: Stream %d: Can't attach the output buffer to this surface:"
" %s (%d)", __FUNCTION__, mId, strerror(-res), res);
}
checkRetAndSetAbandonedLocked(res);
return res;
}
} else {
ALOGE("%s: Stream %d: Can't get next output buffer from buffer manager:"
" %s (%d)", __FUNCTION__, mId, strerror(-res), res);
return res;
}
} else if (res != OK) {
if (shouldLogError(res, mState)) {
ALOGE("%s: Stream %d: Can't dequeue next output buffer: %s (%d)",
__FUNCTION__, mId, strerror(-res), res);
}
checkRetAndSetAbandonedLocked(res);
return res;
}
}
if (res == OK) {
checkRemovedBuffersLocked();
}
return res;
}
getBufferLockedCommon function本文情况下运行，mUseBufferManager为FALSE，因为OutputBuffer的申请有Camera3BufferManager 申请和Anativewindow 申请两种情况，选择在snapdragon APP中设置：
- snapdragon APP中调用OutputConfiguration(@NonNull Surface surface)函数，surfaceGroupId默认赋值SURFACE_GROUP_ID_NONE(-1)，mUseBufferManager等于FALSE不使用buffer manager分配buffer。
- 直接使用ANativeWindow分配preview buffer，运行 if (!gotBufferFromManager) 中code
Camera service获取到camera provider填充完成的image buffer之后要传递给display module显示经过Fig6、Fig7函数调用过程。
在cameraserver process中由surface module发起（后面会解释为什么是surface作为发起人），由hook_queueBuffer调用到surface。binder transact and ontransact结构，跨进程把从ANativewindow中申请的graphic buffer传输给surfaceflinger process，surfaceflinger再把buffer 传输到BufferQueue结构中进行管理，等待surfaceflinger process acquire use。
SurfaceView建立native surface为真正的producer，surfaceflinger通过layer建立的BufferQueueLayer为真正的consumer，BufferQueueLayer调用updatetexImage从BufferQueue中acquire buffer，bindTextureImageLocked function Bind the new buffer to the GL texture（添加纹理之后才送到GPU进行渲染）。
以上内容介绍完cameraserver process到surfaceflinger process preview buffer 传输过程，最后介绍surfaceflinger 怎么把申请graphic buffer显示到display。
总结
