本論文提出了一個基於幀的運動偵測(Motion Detection)視覺感測器，此感測器採用了新提出的時間對比像素(Temporal Contrast Pixel)架構和曝光補償機制(Exposure Compensation Scheme)，利用全域快門和幀差異脈衝寬度調變(Pulse Width Modulation)操作實現了像素內時間對比運算和動態事件報告，並且此架構僅使用了6個電晶體及1個電容器，是目前為止像素內幀差異運算中最簡單的架構。另一方面，縱向平行區域二值化提供了空間特徵擷取並且無需先執行類比數位轉換，進而節省了大量功耗，而透過幀差異和區域二值化的結合，此晶片實現了時空間特徵擷取訊息，在針對障礙物判斷和避障的應用中，此時空間訊息可以被用來計算動態立體視覺，而這也是首次提出計算動態物體深度並過濾掉靜態物體深度的方法。最後，感興趣區域(Region of Interest)擷取也在此晶片上實現並用於資料切割和定位，並且此感興趣區域擷取不僅支援原始影像模式(Image Mode)，同時也支援了幀差異模式(Frame Difference Mode)和動態事件報告模式(Event Report Mode)以定位動態區域。
一個0.56V/0.8V多功能視覺感測器搭載126x126時間對比脈衝寬度調變影像陣列採用了0.18µm 1P6M標準互補式金氧半導體工藝製造，晶片面積為2.35x3.19mm2；此晶片支援5種主要操作模式，包括10位元原始影像模式、10位元幀差異模式、1.5位元動態事件報告模式、8位元區域二值化模式和感興趣區域模式，所有的操作模式都可以相互組合以支援複雜的應用情景。量測結果顯示此晶片在原始影像模式和動態事件報告模式以及區域二值化模式下，達到的最大幀率分別為每秒540/819/540幀，功率消耗分別為390/162.6/151.9µW，正規化優質指標分別為每幀每像素45.5/12.5/17.7pJ。

This thesis presents a frame-based motion detection (MD) vision sensor with a new proposed temporal contrast pixel (TCP) structure and exposure compensation scheme (ECS), which realizes the in-pixel temporal contrast calculation and event reporting with global shutter and frame difference pulse-width-modulation (PWM) operations using only 6 transistors and 1 capacitor (6T1C), and the structure is the simplest architecture in in-pixel frame difference to date. On the other hand, the column-parallel local binary pattern (LBP) extraction provides spatial feature extraction without performing ADC first saving lots of power. With the combination of frame difference and LBP, the temporal-spatial feature information is achieved. For the application of obstacle judgment and avoidance, this temporal-spatial information can be used to calculate dynamic stereo vision, which is first proposed to calculate dynamic objects’ depth and filter out static objects’ depth. Last, region of interest (ROI) extraction is also implemented on-chip for data windowing and location. Moreover, The ROI not only supports raw image (IM) mode but also frame difference (FD) mode and event report (ER) mode to locate motion region.
A 0.56V/0.8V multi-mode vision sensor with 126x126 6T1C TCP has been fabricated in 0.18um 1P6M standard CMOS process with chip area 2.35 x 3.19 mm2. The chip supports five main operation modes including 10-bit IM mode, 10-bit FD mode, 1.5-bit ER mode, 8-bit LBP mode and ROI mode. All the operation modes can be combined with each other to support complicated application scenarios. The measurement results show the achieved max frame rate in IM/ER/LBP mode is 540/819/540fps and the power consumption is 390/162.6/151.9µW with iFoM 45.5/12.5/17.7pJ/pixel∙frame respectively.

ABSTRACT ----------------------------------------------II
CONTENT ----------------------------------------------IV
LIST OF FIGURES --------------------------------------VII
LIST OF TABLES ----------------------------------------------X
Chapter 1 Introduction ------------------------------1
1.1 Motivation --------------------------------------1
1.2 Thesis Contribution ------------------------------3
1.3 Thesis Organization ------------------------------4
Chapter 2 Background Information ----------------------6
2.1 Fundamentals of CMOS Image Sensor --------------6
2.1.1 Basic Pixel Structures ------------------------------7
2.1.2 Pixel Readout --------------------------------------13
2.1.3 Basic Terms in CMOS Image Sensor --------------16
2.2 Motion Detection ------------------------------18
2.2.1 Event-based DVS --------------------------------------19
2.2.2 Frame-based Imager ------------------------------22
2.3 Stereo Vision --------------------------------------25
2.4 Summary ----------------------------------------------27
Chapter 3 Proposed 6T1C PWM Pixel with Exposure Compensation Scheme ------------------------------------------------------29
3.1 TVC PWM Pixel --------------------------------------29
3.2 In-pixel Motion Detection ----------------------31
3.3 Temporal Contrast Pixel with Exposure Compensation Scheme --------------------------------------------------------------35
3.4 Summary ----------------------------------------------38
Chapter 4 Circuit Implementation ----------------------40
4.1 System Architecture ------------------------------40
4.1.1 Pixel Structure --------------------------------------42
4.1.2 Local Binary Pattern Extraction ----------------------43
4.1.3 Column Counter --------------------------------------44
4.1.4 Region of Interest (ROI) ----------------------45
4.1.5 Row and Column Scanner ------------------------------46
4.1.6 Ramp and Vth Generator ------------------------------47
4.2 Operation Mode --------------------------------------47
4.2.1 Image Mode --------------------------------------48
4.2.2 Frame Difference Mode and Event Report Mode ------50
4.2.3 Local Binary Pattern Mode ----------------------51
4.2.4 Region of Interest Mode ------------------------------52
4.3 Summary ----------------------------------------------54
Chapter 5 Measurement Results ----------------------55
5.1 Chip Implementation ------------------------------55
5.2 Environment setup for Measurement --------------57
5.3 Measurement Results for Each Mode --------------59
5.3.1 One-chip Measurement ------------------------------59
5.3.2 Two-chip Measurement ------------------------------62
5.4 Comparison Table and Specification --------------64
5.5 Summary ----------------------------------------------65
Chapter 6 Conclusion and Future Work --------------66
6.1 Conclusion --------------------------------------66
6.2 Future Work --------------------------------------68
Bibliography ----------------------------------------------70

[1] J. Ohta, "Smart CMOS Image Sensor and Applications," CRC Press.
[2] J. Nakamura, "Image Sensor and Signal Processing for Digital Still Camera, Taylor & Francis Group, 2006.
[3] P. Lichtsteiner and T. Delbruck et al., "A 128 X 128 120 dB 15 μs Latency Asynchronous Temporal Contrast Vision Sensor," IEEE Journal of Solid-State Circuits, vol. 43, no. 2, pp. 566-576, Feb. 2008.
[4] C. Li, et al., "A 132 by 104 10μm-Pixel 250μW 1kefps Dynamic Vision Sensor with Pixel-Parallel Noise and Spatial Redundancy Suppression," IEEE Symp. VLSI Circuits, pp. 216-217, June 2019.
[5] B. Son, et al., "A 640×480 Dynamic Vision Sensor with a 9μm Pixel and 300Meps Address-Event Representation," IEEE ISSCC Dig. Tech. Papers, pp. 66-67, Feb. 2017.
[6] M. Gottardi, et al., " A 100 μ W 128 × 64 Pixels Contrast-Based Asynchronous Binary Vision Sensor for Sensor Networks Applications," IEEE Journal of Solid-State Circuits, vol. 44, no. 5, pp. 1582-1592, May 2009.
[7] T. Finateu et al., "A 1280×720 Back-Illuminated Stacked Temporal Contrast Event-Based Vision Sensor with 4.86µm Pixels, 1.066GEPS Readout, Programmable Event-Rate Controller and Compressive Data-Formatting Pipeline," IEEE ISSCC Dig. Tech. Papers, pp. 112-114, Feb. 2020.
[8] T. Ohmaru et al., "25.3μW at 60fps 240×160-pixel vision sensor for motion capturing with in-pixel non-volatile analog memory using crystalline oxide semiconductor FET," IEEE ISSCC Dig. Tech. Papers, pp. 1-3, Feb. 2015.
[9] T. Ohmaru et al., "A 25.3μW at 60 fps 240×160 Pixel Vision Sensor for Motion Capturing With In-Pixel Nonvolatile Analog Memory Using CAAC-IGZO FET , " IEEE Journal of Solid-State Circuits, vol.51, no. 9, September 2016.
[10] J. Choi, et al., "A Spatial-Temporal Multiresolution CMOS Image Sensor With Adaptive Frame Rates for Tracking the Moving Objects in Region-of-Interest and Suppressing Motion Blur," IEEE Journal of Solid-State Circuits, vol. 42, no. 12, pp. 2978-2989, Dec. 2007.
[11] J. Choi, et al., "A 1.36μW adaptive CMOS image sensor with reconfigurable modes of operation from available energy/illumination for distributed wireless sensor network," IEEE ISSCC Dig. Tech. Papers, pp. 112-114, Feb. 2012.
[12] J. Choi, et al., "A 3.4µW CMOS image sensor with embedded feature-extraction algorithm for motion-triggered object-of-interest imaging," IEEE ISSCC Dig. Tech. Papers, pp. 478-479, Feb. 2013.
[13] J. Choi, et al., "A 3.4-µW Object-Adaptive CMOS Image Sensor With Embedded Feature Extraction Algorithm for Motion-Triggered Object-of-Interest Imaging," IEEE Journal of Solid-State Circuits, vol. 49, no. 1, pp. 289-300, Jan. 2014.
[14] X. Zhong, et al., "A Fully Dynamic Multi-Mode CMOS Vision Sensor With Mixed-Signal Cooperative Motion Sensing and Object Segmentation for Adaptive Edge Computing," IEEE Journal of Solid-State Circuits, vol. 55, no. 6, pp. 1684-1697, June 2020.
[15] K. Lee, et al., "A 272.49 pJ/pixel CMOS image sensor with embedded object detection and bio-inspired 2D optic flow generation for nano-air-vehicle navigation," IEEE Symp. VLSI Circuits, pp. C294-C295, June 2017.
[16] S. Park, et al., "Low-Power, Bio-Inspired Time-Stamp-Based 2-D Optic Flow Sensor for Artificial Compound Eyes of Micro Air Vehicles," IEEE Sensors Journal, vol. 19, no. 24, pp. 12059-12068, Dec. 2019.
[17] G. Kim et al., "A 467nW CMOS visual motion sensor with temporal averaging and pixel aggregation," IEEE ISSCC Dig. Tech. Papers, pp. 480-481, Feb. 2013.
[18] T. -H. Hsu et al., "A 0.8 V Multimode Vision Sensor for Motion and Saliency Detection With Ping-Pong PWM Pixel," IEEE Journal of Solid-State Circuits, vol. 56, no. 8, pp. 2516-2524, Aug. 2021.
[19] T.-H. Hsu et al., "A 0.8 V multimode vision sensor for motion and saliency detection with ping-pong PWM pixel," IEEE ISSCC Dig. Tech. Papers, pp. 110-112, Feb. 2020.
[20] K. D. Choo et al., "Energy-Efficient Low-Noise CMOS Image Sensor with Capacitor Array-Assisted Charge-Injection SAR ADC for Motion-Triggered Low-Power IoT Applications," IEEE ISSCC Dig. Tech. Papers, pp. 96-98, Feb. 2019.
[21] X. Zhong, et al., "A 2PJ/Pixel/Direction MIMO Processing Based CMOS Image Sensor for Omnidirectional Local Binary Pattern Extraction and Edge Detection," IEEE Symp. VLSI Circuits, pp. 247-248, June 2018.
[22] Chih-Hao Lin, et al., "A Dual-Mode CMOS Imager for Free-Space Optical Communication with Signal Light Source Tracking and Background Cancellation, " International Image Sensor Workshop (IISW), Jun. 2015.
[23] C. Kim, et al., "A 43.7 mW 94 fps CMOS image sensor-based stereo matching accelerator with focal-plane rectification and analog census transformation," IEEE ISCAS, pp. 1418-1421, May 2016.
[24] C. Kim, et al., "A CMOS Image Sensor-Based Stereo Matching Accelerator With Focal-Plane Sparse Rectification and Analog Census Transform," in IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 63, no. 12, pp. 2180-2188, Dec. 2016.
[25] A. Y. Chiou, et al., "An ULV PWM CMOS Imager With Adaptive-Multiple-Sampling Linear Response, HDR Imaging, and Energy Harvesting, " IEEE J. Solid-State Circuits, vol. 54, no. 1, pp. 298-306, Jan. 2019.
[26] Meng-Ting Chung, et al., "A 0.5V 4.95µW 11.8fps PWM CMOS Imager With 82dB Dynamic Range and 0.055% Fixed-Pattern-Noise, " IEEE ISSCC Dig. Tech. Papers, pp. 114-116, Feb. 2012.
[27] J. T. Philip, et al., "A comparative study of block matching and optical flow motion estimation algorithms," Annual International Conference on Emerging Research Areas: Magnetics, Machines and Drives (AICERA/iCMMD), pp. 1-6, 2014.
[28] J. Huang, et al., "A Motion Sensor with On-Chip Pixel Rendering Module for Optical Flow Gradient Extraction," IEEE ISCAS, pp. 1-5, May 2018.
[29] R. Chang, et al., "An experimental evaluation of balance strategy based obstacle avoidance," 14th International Conference on Control, Automation, Robotics and Vision (ICARCV), pp. 1-6, 2016.
[30] P. Gao, et al., "Obstacle avoidance for micro quadrotor based on optical flow," 29th Chinese Control And Decision Conference (CCDC), pp. 4033-4037, 2017.
[31] K. Seyid, et al., "FPGA-Based Hardware Implementation of Real-Time Optical Flow Calculation," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 1, pp. 206-216, Jan. 2018.
[32] M. Liu, et al., "Block-matching optical flow for dynamic vision sensors: Algorithm and FPGA implementation," IEEE ISCAS, pp. 1-4, May 2017.
[33] A. J. Sanchez-Garcia, et al., "Decision making for obstacle avoidance in autonomous mobile robots by time to contact and optical flow," International Conference on Electronics, Communications and Computers (CONIELECOMP) pp. 130-134, 2015.
[34] Y. Kaneta, et al., "Determination of time to contact and application to timing control of mobile robot," IEEE International Conference on Robotics and Biomimetics, pp. 161-166, 2010.
[35] A. J. S. Garcia, et al., "Estimation of Time-to-Contact for Navigation of Autonomous Robots Using Parallel Processing," International Conference on Mechatronics, Electronics and Automotive Engineering (ICMEAE), pp. 26-31, 2016.
[36] J. Engel, et al., "Semi-dense Visual Odometry for a Monocular Camera," IEEE International Conference on Computer Vision, pp. 1449-1456, 2013.
[37] skylee, "世平安森美半導體 CMOS感光元件曝光技術/果凍效應的解決方法—淺談 Global shutter & Rolling shutter," 2021, [Online], Available: https://www.wpgdadatong.com/tw/blog/detail?BID=B2747&fbclid=IwAR2zKt_ciB5cXZkr5nFLer56DMSGD5uC-EQepg0Wfn31Br8GANkjicYl9oc.
[38] Monitor Hub, "What Is HDR? HDR Vs. SDR : Everything You Should Know About, " [Online], Available: https://bestmonitorhub.com/what-is-hdr-hdr-vs-sdr/?fbclid=IwAR3Ww8tko4jLzEbeqi4OwWWz1_wcxYUKp_4jV47uaapx6y_uAc5KyE9Kheg.
[39] AI研習社(Zach、Suen), "無人駕駛：如何使用立體視覺實現距離估計？" [Online], Available: https://codingnote.cc/zh-tw/p/261296/?fbclid=IwAR36itCOMpPpP_mnCXj93H3kLbzTeLlwTehE3GF2RgYa0wBeY6_KRBDy_H4.