Abstract
A 3D camera uses a modulated visible light source for depth imaging and includes a processor operable to perform time multiplexing between image detection and depth or time-of-flight (ToF) detection using the same photodetectors. The camera can alternate between the image detection mode and the ToF detection mode to produce a continuous stream of color and depth images that can be overlaid without the need for any post-processing software. The camera can also be configured to determine time-of-flight using analog integration modules, thereby minimizing the circuitry necessary for analog-to-digital conversions and ToF calculations in the digital domain.
Claims
1. A camera, comprising: a light source configured to transmit a pulsed light signal TX toward an object or scene; a photodetector array including at least one photodetector configured and arranged to receive a pulsed light signal RX reflected from the object or scene, each photodetector configured to generate an output signal indicative of the detection of said pulsed light signal RX; and a processor connected to each photodetector of said photodetector array and configured to alternate between (i) an image detection mode in which the processor processes the output signal of each photodetector for image information, and (ii) a time-of-flight (ToF) mode in which the processor processes the output signal of each photodetector to determine the ToF for the pulsed light signal TX to the object or scene and the reflected pulsed light signal RX to the corresponding photodetector, wherein the processor is further configured to alternate between the image detection mode and the ToF mode in successive pulses of the reflected pulsed light signal RX.
2. The camera of claim 1, wherein the processor is further configured to activate said light source.
3. The camera of claim 2, wherein the processor is further configured in the ToF mode to activate said light source to pulse the pulsed light signal TX at a frequency that is not sensible by the human eye.
4. The camera of claim 3, wherein the processor is further configured to activate said light source only in the ToF mode.
5. The camera of claim 1, wherein the pulsed light signal TX is a visible light signal.
6. A camera, comprising: a light source configured to transmit a pulsed light signal TX toward an object or scene; a photodetector array including at least one photodetector configured and arranged to receive a pulsed light signal RX reflected from the object or scene, each photodetector configured to generate an output signal indicative of the detection of said pulsed light signal RX; and a processor connected to each photodetector of said photodetector array and configured to alternate between (i) an image detection mode in which the processor processes the output signal of each photodetector for image information, and (ii) a time-of-flight (ToF) mode in which the processor processes the output signal of each photodetector to determine the ToF for the pulsed light signal TX to the object or scene and the reflected pulsed light signal RX to the corresponding photodetector, wherein the processor is further configured to operate in the image detection mode for one or more successive pulses of the reflected pulsed light signal RX and then to operate in the ToF mode for one or more successive pulses of the reflected pulsed light signal RX.
7. The camera of claim 1, wherein: the output signal from each photodetector is an output charge; and the processor includes an integrator circuit having a capacitor circuit configured and operable to receive and accumulate the output charge from each photodetector.
8. The camera of claim 7, wherein: the capacitor circuit includes two capacitors for each photodetector; and the integrator circuit includes a transfer gate for each capacitor, each transfer gate activated by said processor to accumulate the output charge in the corresponding capacitor according to a timing sequence.
9. The camera of claim 8, wherein said processor is further configured to activate each transfer gate according to the same timing sequence in the image detection mode and to activate each transfer gate according to different timing sequences in the ToF mode.
10. The camera of claim 6, wherein the processor is further configured to activate said light source.
11. The camera of claim 10, wherein the processor is further configured in the ToF mode to activate said light source to pulse the pulsed light signal TX at a frequency that is not sensible by the human eye.
12. The camera of claim 11, wherein the processor is further configured to activate said light source only in the ToF mode.
13. The camera of claim 6, wherein the pulsed light signal TX is a visible light signal.
14. The camera of claim 6, wherein: the output signal from each photodetector is an output charge; and the processor includes an integrator circuit having a capacitor circuit configured and operable to receive and accumulate the output charge from each photodetector.
15. The camera of claim 14, wherein: the capacitor circuit includes two capacitors for each photodetector; and the integrator circuit includes a transfer gate for each capacitor, each transfer gate activated by said processor to accumulate the output charge in the corresponding capacitor according to a timing sequence.
16. The camera of claim 15, wherein said processor is further configured to activate each transfer gate according to the same timing sequence in the image detection mode and to activate each transfer gate according to different timing sequences in the ToF mode.
Description
DESCRIPTION OF THE DRAWINGS
(1) FIG. 1 is a chart of known 3D imaging methods.
(2) FIG. 2 is a diagram of a flash time-of-flight (ToF) 3D camera.
(3) FIG. 3 is a diagram of a 3D camera and Lidar system.
(4) FIG. 4 is a graph of transmitted and received signals using the system shown in FIG. 3.
(5) FIG. 5 is a diagram of time multiplexing for time-of-flight and RGB detection implemented by the system and method described herein.
(6) FIG. 6 is a circuit diagram configured for a time-of-flight determination.
(7) FIG. 7 is a timing diagram for operating the circuit of FIG. 6.
(8) FIG. 8a is a diagram of a pixel array for use in the 3D camera of FIG. 3.
(9) FIG. 8b is a circuit diagram configured for RGB light detection and ToF determination for the 3D camera of FIG. 3 according to one aspect of the present disclosure.
(10) FIG. 8c is a circuit diagram of a capacitor circuit used in the circuit shown in FIG. 8b.
(11) FIG. 9 is a timing diagram for operating the circuit of FIG. 8b to generate an RGB image.
(12) FIG. 10 is a timing diagram for operating the circuit of FIG. 8b to generating time-of-flight or depth information for the 3D camera of FIG. 3 according to one aspect of the present disclosure.
(13) FIG. 11 is a graph of ToF measurements using successive approximation according to one aspect of the present disclosure.
(14) FIG. 12 is circuit diagram of a circuit for performing the successive approximation shown in the graph of FIG. 11, according to one aspect of the present disclosure.
(15) FIG. 13 is a graph of ToF measurements using sigma-delta modulation according to another aspect of the present disclosure.
(16) FIG. 14 is circuit diagram of a circuit for performing the sigma-delta modulation shown in the graph of FIG. 13, according to another aspect of the present disclosure.
(17) FIG. 15 is a graph of the sigma-delta modulation method using narrow modulation pulses.
(18) FIG. 16 is a chart explaining the operation of an integration-based constant fraction discriminator according to one feature of the present disclosure.
(19) FIG. 17 is a diagram of a non-square wave transmitted signal illustrating a feedback loop for the integration based constant fraction discriminator to determine the integral symmetry point of the signal for use in the ToF measurements disclosed herein.
(20) FIG. 18 is a diagram of a device for performing the feedback loop process shown in FIG. 17.
(21) FIG. 19 is a diagram of a device for performing the feedback loop process shown in FIG. 17 incorporating an analog feedback loop.
(22) FIG. 20 is a diagram of a device for performing the feedback loop process shown in FIG. 17 for use with the ΣΔ time-of-flight measurement disclosed herein.
DETAILED DESCRIPTION
(23) For the purposes of promoting an understanding of the principles of the disclosure, reference will now be made to the embodiments illustrated in the drawings and described in the following written specification. It is understood that no limitation to the scope of the disclosure is thereby intended. It is further understood that the present disclosure includes any alterations and modifications to the illustrated embodiments and includes further applications of the principles disclosed herein as would normally occur to one skilled in the art to which this disclosure pertains.
(24) Time Multiplexing Color Imaging Device
(25) The present disclosure contemplates sensing time-of-flight (3D) with the visible light spectrum and multiplexing the image sensor, such as the RX sensor in FIG. 3, between 2D color imaging and 3D time-of-flight imaging modes as a function of time. For simplicity, the present system is referred to as an RGBvD system, corresponding to RGB color sensor with Visible light spectrum 3D imaging system or sensor. In one aspect of the present disclosure, the RGBvD system uses a modulated visible light source, such as the flash of a typical 2D camera on a cell phone, for depth or ToF imaging and for time multiplexing between color and depth imaging. During the color imaging the flash or TX light can either be continuously off (e.g. when imaging in daylight) or continuously on (e.g. imaging at night), whereas during the depth imaging the intensity of the light source can be modulated for time-of-flight (ToF) measurement. The modulating frequency of the light source is outside the bandwidth of the human eye so that the visible light source during 3D imaging will appear to be a constant faint light coming from the camera.
(26) In addition, the camera is configured to alternate between two modes to produce a continuous stream of color and depth images that can be overlaid without the need for any post-processing software, as depicted in FIG. 5. Thus, the 3D camera, such as the camera in FIG. 3, and particularly the RX and processing circuitry, or RX processor, of the camera, can be controlled, such as by circuitry or software, to multiplex between obtaining a color image in an imaging mode, such as in Frame N and Frame N+2, and obtaining time-of-flight (ToF) information in a ToF mode, such as in the intermediate Frame N+1, from the same transmitted TX light and the same RX light reflected from the object or scene being detected. The RGBvD system and methods disclosed herein can implement methods to measure 3D depth or ToF as described in more detail below, or can use other known measurement methods for ToF such as the methods described in the documents incorporated by reference above. The camera can have the architecture depicted in FIG. 3 in which the return signal is directed to a photodiode array A in which the output of each photodiode is processed by the RX processor C. In accordance with the RGBvD camera disclosed herein, the RX processor C is adapted to alternate between frames to store color images in the imaging mode and to generate and store ToF information in the ToF mode, all from the same TX signal.
(27) A more detailed explanation starts with the basic circuit shown in FIG. 6 for measuring time-of-flight between the TX light and the RX light shown in the graph of FIG. 4. The circuit shown in FIG. 6 relates to a single light detector 10 at the RX sensor side of the camera shown in FIG. 3. The detector 10 can be a conventional RGB light detector or photodiode. The output of the detector 10 is supplied to a pair of transfer gates TG.sub.1, TG.sub.2 which, together with corresponding capacitors C.sub.FD1, C.sub.FD2 form an integrator for integrating or collecting charge from the detector 10. Charge Q.sub.1, Q.sub.2 is collected in each capacitor when the corresponding transfer gate is opened. When the gates are closed the charge in each capacitor can be outputted through a buffer BUF as corresponding voltages V.sub.1, V.sub.2. These voltages can be fed to corresponding A/D converters for use in calculating the time-of-flight within the RX processor C or within an external device to which the voltages V.sub.1, V.sub.2 (or the A/D converted digital values) are provided.
(28) The transfer gates TG.sub.1, TG.sub.2 are activated or opened according to the timing chart shown in FIG. 7 in order to perform the integration necessary to determine the time of flight. As shown in FIG. 7, at the transmission of the TX light pulse the first transfer gate TG.sub.1 is opened for a period equal to the length or time duration of the TX pulse T. As long as the transfer gate TG.sub.1 is open, charge from the detector 10 collects as charge Q.sub.1 in the capacitor C.sub.FD1. At the end of the period T.sub.P the second gate TG.sub.2 is opened for an equal time period T.sub.P, during which time the charge Q.sub.2 is collected in the capacitor C.sub.FD2. It can be appreciated that when the transfer gates TG.sub.1, TG.sub.2 are opened, charge only accumulates in the corresponding capacitors when the RX sensor detects the reflected light. Thus, as shown in FIG. 7, the RX light is received at a time offset from the commencement of the TX pulse—i.e., the time-of-flight (ToF). As represented by the two shaded portions of the RX light in FIG. 7, the leading portion of the received RX pulse overlaps the TG.sub.1 pulse by the charge Q.sub.1, and the trailing portion of the RX pulse overlaps the TG.sub.2 pulse by the second charge Q.sub.2.
(29) The circuitry for a 3-D camera according to the present disclosure is shown in FIGS. 8a-8c. The 3-D camera includes a plurality of pixel arrays, such as the pixel array shown in FIG. 8a. In the illustrated embodiment, a 2×2 array includes a detector tuned to detect red, green and blue light (RGB). The additional detector in the example can be calibrated to green light or to other frequencies or frequency bands. Each detector outputs a charge to a corresponding pair of transfer gates TG.sub.1, TG.sub.2 as discussed above, according to the circuit shown in FIG. 8b. The corresponding transfer gates pass the charge to a capacitor circuit DIN, which is shown in more detail in FIG. 8c. As shown in FIG. 8c, the capacitor C.sub.FD is connected to a base voltage VDD through a reset transistor and the charge of the capacitor is output to a data bus COL through a buffer including a select transistor SEL. As shown in FIG. 8b, each detector includes a pair of transfer gates TG.sub.1, TG.sub.2 and a pair of capacitor circuits as shown in FIG. 8c, and the output of each capacitor circuit is fed to a line COL(0), COL(1) . . . COL(7), of the data bus COL. Alternatively, the data bus can be a single data line with the output of each capacitor circuit providing a serial signal to the bus line COL.
(30) According to the present disclosure, the circuitry of FIGS. 8b-8c is alternately used to detect the RGB image and to determine the time-of-flight to enable the 3-D aspect of the camera. In particular, the present disclosure contemplates multiplexing the operation of the circuitry over a series of TX pulses between image detection and ToF measurement, as discussed above with respect to FIG. 5. In particular, the input lines SEL, RESET, VDD and currents to the transfer gates TG.sub.1, TG.sub.2 for each capacitor circuit DIN are controlled by the RX processor C. The RX processor C can also control when the TX light is activated.
(31) When the circuitry of FIGS. 8b-8c is operated in the RGB imaging mode, RX processor C activates the RESET, TG.sub.1, TG.sub.2 and SEL inputs according to the timing chart shown in FIG. 9. It can first be appreciated that in the RGB imaging mode the TX light need not be activated, as indicated by the null TX Light signal in FIG. 9. In this instance, the image detection occurs under ambient lighting conditions in order to provide an accurate depiction of the captured scene. However, it is understood that in some cases additional light may be added to the scene, in which case the TX light may be activated continuously over the image sampling period.
(32) In the standard case, the RESET signal resets all of the capacitor circuits DIN so that no charge Q.sub.1, Q.sub.2 is present in any of the capacitors corresponding to the RGB detectors. Concurrent with the reset signal, the two transfer gates TG.sub.1, TG.sub.2 are activated so that charge from the corresponding detector is collected in each of the capacitors, C.sub.FD1, C.sub.FD2 (FIG. 6). The charges are collected for a predetermined time that is particularly suited to accept enough light for accurate detection of the intensity of each of the red, green and blue components of light reflected from the scene. The transfer gates are closed after that predetermined time, and the SEL input is activated to pass the capacitor charge to the respective data bus line COL. In this color imaging mode it is anticipated that the charges in the two capacitors for each detector will be the same or substantially the same since the transfer gates for each capacitor circuit are opened for the same time period. Once the capacitor charges have been output to the data bus, the RESET signal restarts the cycle. The charge outputs in the data bus COL are passed through an A/D converter and supplied to the RX and Processing circuitry C (FIG. 2) to generate the color image of the detected scene and/or to an external device for further processing.
(33) In order to provide depth to the color image, the time of flight information is generated in the second mode of operating the circuit of FIGS. 8b-8c. When the circuitry is operated in the ToF mode, the RX processor C activates the RESET, TX light, TG.sub.1, TG.sub.2 and SEL inputs according to the timing chart shown in FIG. 10. The RESET signal clears the capacitor circuits DIN of each detector and then upon transmission of the TX light signal, the transfer gate TG.sub.1 of the first capacitor C.sub.FD1 of each detector is open for the time period T.sub.P of the TX light pulse. The first transfer gate is closed and the second gate TG.sub.2 opened at the end of the period T.sub.P and charge is accumulated in the second capacitor C.sub.FD2 in the manner illustrated in the graph of FIG. 7. At some time after the TX pulse is transmitted, the detectors receive the reflected light signal RX, and it is this signal that is integrated through the two capacitors C.sub.FD1, C.sub.FD2, as discussed above. The SEL signal is provided at a predetermined time after transmission of the TX signal to discharge the capacitors to the corresponding bus line COL. The voltages on the bus lines for all of the detectors (namely the RGBG detectors in the present example) can be fed through an A/D converter and the digital information used to calculate the ToF for each detector.
(34) It can be appreciated that the same circuit shown in FIGS. 8b-8c generates its corresponding color signal and a ToF signal for each detector, depending upon the timing of the signals in FIGS. 9 and 10. Thus, the present disclosure contemplates that the RX and Processing Circuitry C of the 3D camera (FIG. 3) can be configured to generate the RESET, TX light, TG.sub.1, TG.sub.2 and SEL signals according to the standard imaging mode of FIG. 9 interleaved with generation of the RESET, TX light, TG.sub.1, TG.sub.2 and SEL signals according to the ToF mode of FIG. 10. The multiplexing cycle can occur at uniform intervals, such as alternating between modes at each RESET signal, or the ToF timing of FIG. 10 can be activated after 2, 3 or more cycles of the imaging mode timing of FIG. 9.
(35) As seen in FIG. 10, the TX light signal is pulsed in order to set a baseline for measurement of the ToF between the TX light signal and the reflected RX light signal. The duration T.sub.P and frequency of the transmitted TX light pulse can be calibrated so that the light is not sensed by the observers. On the other hand, the duration T.sub.P is long enough so that sufficient light energy or photons are received by the detectors to charge the capacitors and provide an accurate measure of the timing difference ToF.
(36) The multiplexing feature of the present disclosure can be implemented with various light sources for the TX light. The light source can be, for example, a single white light source or separate light sources with light centered on a particular wavelength or wavelength band (such as red, green, blue). Alternatively, the light source can be outside the visible spectrum with one of the light detectors in the pixel array of FIG. 8a corresponding to the non-visible light source, with the understanding that the alternative light source is only activated during the ToF part of the cycle. The light sources in the TX portion of the camera of FIG. 3 can be LEDs, VCESLs, lasers or other suitable light transmitters. The light detectors can be standard photodiodes or other devices capable of detecting the particular light signals. All of the components, including the circuits of FIGS. 8b-8c, can be provided on a common circuit board using conventional semiconductor fabrication techniques.
(37) The multiplexing approach disclosed herein allows for the use of visible light, the same light source and the same light detectors for both image generation and ToF calculation. Since the same detectors that acquire the color image also provide ToF data, the ToF information is correct for each pixel array of the color camera—i.e., there is no parallax effect between the color image and depth ToF measurement. The multiplexing approach of the present disclosure thus allows for the use of a standard CMOS image sensor, CIS, or ASIC used for color image detection, with the control of the components dedicated to the RX processor or software. The processing circuitry must be coupled to the 3D camera light source (such as the flash of a smart phone or the like) to control the TX light pulse. The RX and processing circuitry C can be configured to receive the signal on the bus line COL and can correlate the ToF information for each sensor with the RGB information for that particular sensor to generate a 3D image.
(38) Successive Approximation Time-of-Flight Measurement
(39) In one embodiment of the present disclosure, a 3D camera, such as the camera depicted in FIG. 3, can incorporate RX processor C configured and adapted to perform a first method of the present disclosure for time-of-flight measurement identified as “Integral Symmetry Point (ISP) Successive-Approximation ToF (SA-ToF) Measurement.” The processing circuitry C can include electronic circuitry, a microprocessor programmable to execute software instructions or a combination thereof, as well as circuitry dedicated to each photodiode or group of photodiodes. Prior 2D or 3D cameras rely on collecting charges from photodiodes, converting the collected charges to digital data and performing the time-of-flight calculations in the digital domain. On the other hand, the present disclosure determines ToF in the analog domain using circuitry dedicated to one or more photodiodes to integrate the collected charges from the photodiode(s). The ISP SA-ToF method relies on finding a point (i.e., a time instant) in the received signal RX during a single modulation period in which the integral under the signal curve to the right and to the left of the point within that period is equal—hence the integral symmetry point. For instance, with square modulation as depicted in FIG. 4, this point is in the center of the “high” state of the modulation. The ToF can be measured by comparing the timing of this ISP point in the received signal RX to the timing of the similar ISP point in the transmit signal TX. The ISP for the transmitted signal TX is known a priori because the transmit signal is generated and controlled in the camera.
(40) In accordance with the present disclosure, the procedure for finding the exact position of the integral symmetry center for the received signal RX is performed in successive steps over multiple modulation cycles, hence the successive-approximation aspect of the SA-ToF measurement. The present method enables reuse of the same circuitry that in turn reduces the circuit complexity and footprint, in particular for use in flash 3D cameras. The modulation frequency is often orders of magnitude larger than the frame rate, hence performing the measurement in successive periods does not add any practical limitations.
(41) The successive approximation method of the present disclosure is illustrated in the graph of FIG. 11. The present system and method is operable to find the ToF by measuring the timing of the integral symmetry point (ISP) of the received signal with reference to the same ISP point on the transmit signal TX. For simplicity it can be assumed that the modulation of the TX light and RX light both have a square-wave shape, but the present method can be applied to any other modulation waveform. The transmit signal that is sent to the target is shown on the top of FIG. 11 as TX light and the time-delayed received signal or RX light is shown below the TX signal. The integral symmetry point of the RX light is marked by a dot labeled ISP and is measured with reference to the integral symmetry point of the TX light, which is known a priori as the mid-point of the square wave. In accordance with the present disclosure, the result of the measurement is a binary number that can be compared to a full-scale binary number equal to a predefined maximum time-of-flight ToF.sub.max that the system is capable of measuring. The ToF.sub.max is necessarily less than the modulation period T.sub.M of the TX signal.
(42) In the first step of the measurement, two integration time windows W.sub.1 and W.sub.2 are defined, each having a predetermined time duration equal to half of ToF.sub.max, as shown in “Step 1” of FIG. 11. The time windows determine the time period over which the charge generated by the photodiode 10 in response to receiving the RX signal is accumulated or integrated, using analog circuitry as described herein. The initial time window pair W.sub.1, W.sub.2 commence at the ISP of the TX signal. The integral or accumulation of the photodiode charge within each window is determined, with the integral of the first time window W.sub.1 named Q.sub.1 and integral of the RX signal within the second time window W.sub.2 named Q.sub.2. In the embodiment illustrated in FIG. 11, Q.sub.2>Q.sub.1 so it can be concluded that the RX signal has more overlap with the second time window than the first window. Hence it is then known that the ISP should occur in the second window W.sub.2. In accordance with the present method, the presence of the ISP in the second window W.sub.2 sets a first binary digit for this time-wise integration as a binary “1”, as reflected below the timeline in FIG. 11. Note that if the ISP was present within the first window W.sub.1, the first binary digit would have been a binary “0”. The ensuing steps seek to shift the pair of time windows of ½ ToF.sub.max length to a point where the integral of the RX signal in each of the windows is equal, substantially equal or within a predetermined range.
(43) Thus, in the second step it is determined whether the ISP is in the first half of the window W.sub.2 or in the second half of that window. To make this determination, the time windows W.sub.1, W.sub.2 are redefined at the next TX signal in the same manner described above, and two new time windows W.sub.3 and W.sub.4 are also defined in a similar manner—i.e., having a time duration equal to half the ToF.sub.max. However, unlike the first two windows, the new windows W.sub.3, W.sub.4 are defined so that the trailing edge of window W.sub.3 and the leading edge of window W.sub.4 are aligned in the middle of the redefined window W.sub.2, as shown in “Step 2” of FIG. 11. In this example, the integral of the RX signal received by the photodiode 10 within new time window W.sub.3, designated Q.sub.3 (corresponding to the accumulated charge during this time window), is larger than the integral of the RX signal in time window W.sub.4, designated Q.sub.4, so the second binary digit is chosen to be “0”.
(44) In the third step, the time windows W.sub.1, W.sub.2, W.sub.3 and W.sub.4 are redefined at the next TX signal in the same manner as in the second step. New time windows W.sub.5 and W.sub.6 are defined in “Step 3” in the same manner as windows W.sub.3, W.sub.4 (i.e., each having a duration of ½ ToF.sub.max), except that the boundaries of the two new windows are set to middle of the third window W.sub.3, since the integral Q.sub.3 is larger than Q.sub.4. In the third step the integral of charge generated by photodiode 10 in response to the RX signal within time window W.sub.5, designated Q.sub.5, is larger than the integral of the RX signal within time window W.sub.6, designated Q.sub.6, hence the third digit is also set to binary “0”. At the end of three steps, the ToF is with three digit precision is represented by a binary “100”. At three digit precision, the binary value for the maximum time-of-flight ToF.sub.max is a binary “111”. In this example and at three digit precision the integral symmetry point ISP for the received signal RX, from which the ToF value can be obtained, occurs in the fifth window or octant of the maximum measurable ToF.sub.max time interval. (It is noted that the first octant corresponds to “000” and the eighth octant corresponds to “111”).
(45) The precision of the measurement can be increased indefinitely by continuing these steps in which the boundary of the pair of integration windows moves closer and closer to the actual ISP as represented by “Step n” in FIG. 11. It is appreciated that after the first integrations, the boundary of each successive pair of integration windows W.sub.2n−1 and W.sub.2n is set at the middle of the time window in the prior pair of time windows for which its integral, or accumulated charge, is greater than the integral for the other window of the pair. For instance, in the illustrated example leading windows W.sub.3, W.sub.5, etc. of the window pairs will always have the greater integral, at least until the boundary between the pair of windows reaches the ISP of the RX signal. On the other hand, if in the initial step of the process the integral of window W.sub.1 had been greater than the integral of window W.sub.2, the boundary of successive pairs of integration windows would have been aligned with the even-numbered windows W.sub.2, W.sub.4, etc.
(46) In some embodiments, the measurement precision will be limited by the electronic circuit that performs the comparison of the integral values, Q.sub.2n−1 and Q.sub.2n at “step n”. Also the timing precision of the windows will be limited by the electronic clock or delay elements that are used to shift the integration time windows. In one specific example, adequate convergence of the window boundary line to the true ISP can be achieved after ten steps (n=10), which yields a 10 bit binary number from which the ToF can be calculated.
(47) The relation between maximum time-of-flight ToF.sub.max and the modulation period, T.sub.M can be considered, so that in a hypothetical case if the ToF is zero, the first integration window in each step would move ahead on the time axis until the integration time window occurs entirely before the ISP of the transmit TX signal. In another scenario if the ToF is at its maximum then the second window would shift forward in time until it would occur entirely after ToF.sub.max ahead of the ISP of the TX signal. Since each window is equal to ½ ToF.sub.max, this means the integration interval (consisting of two windows) for each step can extend from ½ ToF.sub.max before the ISP of the transmit signal up to ½ ToF.sub.max after the ISP+ToF.sub.max. Therefore, the modulation period T.sub.M of the transmit and receive signals TX, RX should be at least equal to 2×ToF.sub.max for the present method. If necessary, this extra required time can be eliminated by adding redundancy to the integration windows and also pixel circuitry in which case the modulation period T.sub.M can shrink to be equal to ToF.sub.max.
(48) An exemplary circuit implementing the methods described above is shown in FIG. 12 The exemplary circuit can be associated with each pixel or photodiode or a photodiode array, or can be associate with groups of pixels/photodiodes. It can be appreciated that the present disclosure contemplates performing the ToF determination using analog circuitry, without the necessity of performing an analog-to-digital conversion as in prior camera ToF systems. In one embodiment, the circuitry includes two gated integration blocks 12, 14 that receive the signal or charge from the photodetector 10 generated in response to reception of the RX signal. It is understood that the integration blocks can be constructed according to the circuitry shown in FIG. 6 that uses transfer gates and capacitors to store charge from the photodetector. The transfer gates of integration blocks 12, 14 are activated in accordance with each of the pairs of integration time windows W.sub.1, W.sub.2, etc., to integrate or accumulate the photodiode charge within the window, as discussed above. The integration block 12 is activated during time window W.sub.2k. 1, while the second integration block 14 is activated during the later time window W.sub.2k, where “k” represents the “step” number (see FIG. 11). Thus, the integration blocks can be implemented using switch-capacitor circuits to integrate the photocurrent from the sensor (photodetector 10) into charges Q.sub.2k−1 and Q.sub.2k. The integrated values or charges Q.sub.2k−1 and Q.sub.2k are then compared using a comparator 16 and the analog voltage corresponding to a binary “0” or “1” is sent to a control module 18. The control module 18 increments the “step” number “k” and sets the times for the two integration windows W.sub.2k+1, W.sub.2k+2 for the next cycle that commences on receipt of the next TX signal. The new time windows are provided to the two integration modules 12, 14. One exemplary way to implement this function of the control module 18 is to connect a high-frequency clock signal to a digital counter and create the time windows W.sub.2k+1, W.sub.2k+2 at particular numbers at the output of the counter. These particular numbers can then be updated digitally in order to set the timing for W.sub.2k+1 and W.sub.2k+2.
(49) The control module 18 can be configured and operable to convert the analog output of the comparator 16 to a binary “0” or “1”. A memory block or shift register 20 can hold the comparison result from each step for final binary output for the pixel, with the result of the first comparison at “Step 1” occupying the most significant bit position and the last comparison occupying the least significant bit position. It is understood that the output will be a binary number of n bit length, where n corresponds to the number of integrations performed by the integration blocks 12, 14. This output is supplied to a processor within the Rx circuitry C to perform the ToF calculation.
(50) Alternatively, the shift register 20 can include series capacitors corresponding to the digits of the binary number, with the analog output of the comparator supplied to the “k.sup.th” series capacitor under control of the control module 18. The series capacitors can be converted to the binary number of n bit length using a sample-and-hold peak detector or similar component.
(51) For the time-of-flight (ToF) calculation, the binary output number b.sub.0b.sub.1 . . . b.sub.n can be compared to the binary value for the maximum ToF measurable by the subject device. By definition, the maximum binary value for the maximum ToF is 111 . . . 1.sub.n with this binary value representing the actual time value for the maximum measurable ToF. The measured ToF is thus the ratio of the binary output b.sub.0b.sub.1 . . . b.sub.n from the circuit shown in FIG. 12 and the maximum binary ToF number 111 . . . 1.sub.n multiplied by the actual maximum ToF.sub.max time value that is known by design. Thus, the processor within the RX processor C can be configured to compare the two binary values, perform the multiplication of the ratio to the maximum ToF.sub.max time value and provide an output indicative of the actual time-of-flight value for the particular pixel/photodiode or group of pixels/photodiodes. This output can be a sensible output, such as a visual display, or can be an output signal provided to a processing device, such as a Lidar, or an image or video processing device. In the three digit example discussed above, the binary value in the output register 20 is “100”, compared to the maximum ToF binary value of “111”. The binary ratio is thus “100/111” or decimal 4/7, which means that the actual measured ToF is 4/7 of the predetermined time for ToF.sub.max.
(52) One variant of this circuit can integrate or accumulate both charges Q.sub.2k−1 and Q.sub.2k on the same capacitor, such as the capacitor C.sub.FD1 in FIG. 6, but with different polarity for the charge accumulated in the later time window, so that the charge accumulated in the capacitor essentially subtracts the second charge from the first charge. The final charge over the two time windows in the capacitor is compared to a reference potential equal to the initial potential of that capacitor to determine if the charge with positive or negative polarity was larger. This technique can have two benefits. Firstly, it can cancel the effect of background light in the subtraction process, which reduces the required range for the comparator common mode. Secondly, it eliminates the mismatch error across the two integrating channels by using the same integration element (capacitor) for both time windows.
(53) The algorithm disclosed herein as embodied in the graph of FIG. 11 and as performed by the circuit shown in FIG. 12 relies on the overall time for the integration steps being short enough so that the objects in the scene do not have noticeable motion to disturb the measurement. A sufficiently short overall time is possible because the modulation period T.sub.M can be only a fraction of a microsecond and the frame-rate of most 3D cameras can be tens of millisecond. In one specific embodiment, the number of integration steps that sets the number of bits for the ToF measurement can be less than 10. Hence the time spent per conversion step can be much larger than a fraction of microsecond. This feature allows repeating the integration windows for each conversion step multiple times and accumulating more electric charge before performing the comparison between the charges. Larger accumulated charge can relax the requirements on the comparator design and positively influence critical parameters such as power consumption for each pixel. This can be important when the circuit is implemented in a flash camera where the amount of light per pixel is small (hence less charge can be accumulated) and millions of pixels work simultaneously, hence their power consumption should be limited. Furthermore, the repetition of the integration per conversion step can help average the white noise caused by the detection shot noise and thermal noise from the electronic front-end. This consequently improves the precision of the ToF measurement. According to the possible modulation period and typical frame rates in 3D cameras, tens of thousands of integration windows can be accumulated before performing the comparison for each conversion step.
(54) The control module 18 of the circuit in FIG. 12 can include circuitry and/or firmware that is configured and operable to determine the time windows at each step “k”. The outputs W.sub.2k+1, W.sub.2k+2 can represent activation windows for the transfer gates TG.sub.1, TG.sub.2 (FIG. 6) for each integration module 12, 14. The durations of the time windows are pre-determined based on the length of ToF.sub.max and can be “hard-wired” into the control module or accessible to the control module 18. The control module is further configured and operable to determine which of the prior time windows is bisected by the boundary between the next pair of time windows. Certain aspects of the algorithm represented by FIG. 11 executed by the control module 18 can be delegated to a central control module, such as look-up table for the length of ToF.sub.max.
(55) It can be appreciated that the circuit of FIG. 12 can be incorporated into the circuit shown in FIG. 8b for use in multiplexing between RGB color imaging and ToF determination. In that instance, the comparator 16 and control module 18 can be integrated between the capacitor circuit DIN and the output bus line COL, with the binary output 20 provided on the bus line. Rather than the two outputs (e.g., COL(0), COL(1)) for each pixel, a single output would be provided corresponding to the binary output 20. It can further be appreciated that the RX and processing circuitry C can be configured to actuate the components according to the ToF timing diagram of FIG. 10 for several cycles as needed to achieve convergence of the integration. For example, the circuitry C can operate to execute one RGB color imaging sequence according to timing diagram of FIG. 9 followed by seven cycles of the ToF timing sequence of FIG. 10, with this pattern repeating throughout the imaging process. Alternatively, the circuitry C can be configured to detect convergence before the predetermined number of ToF timing cycles (such as seven in the example) and to terminate the ToF timing sequence and execute the RGB color detection timing sequence.
(56) ΣΔ Charge Accumulation for Time-of-Flight Measurement
(57) A second method of the present disclosure can be identified as “ISP approximation with ΣΔ Charge Accumulation for ToF Measurement”, as depicted in FIG. 13. In this method the integration window for each step remains the same—i.e., does not shift in relation to the TX, RX signals over time. Instead, the accumulation of the photodiode charge in response to receipt of the RX signal toggles between the two windows based on the overall accumulated signal. In the first step the two time windows W.sub.1, W.sub.2 are defined as described above in relation to the integration symmetry point ISP as shown in FIG. 11, namely having ½ ToF.sub.max duration and commencing at the ISP of the transmitted TX signal. The charge of the photodiode 10 in response to the RX signal is integrated or accumulated in each time window, Q.sub.1, Q.sub.2. Both Q.sub.1 and Q.sub.2 are stored in separate memories, such as analog memories or capacitors within the respective integration modules. In the example shown in FIG. 13, since Q.sub.2>Q.sub.1, only the charge for the first window W.sub.1, Q.sub.1, gets a second accumulation, designated 2Q.sub.1, while Q.sub.2 remains the same. In other words, charge accumulation in the first time window is determined in the next step, or more precisely the incoming RX signal is only integrated in the first window and stored in its corresponding memory, while the charge for Q.sub.2 remains the same. In the second step, Q.sub.2 is compared to 2Q.sub.1, and since Q.sub.2<2Q.sub.1, (i.e., second charge accumulation of the first window exceeds the unchanged charge of the second window) the charge accumulated during the second time window W.sub.2, Q.sub.2, gets an extra accumulation, designated 2Q.sub.2, while the second accumulation for window W.sub.1, 2Q.sub.1, remains the same.
(58) The method then continues with the same logic, with two counters keeping track of the number of accumulations N.sub.1 and N.sub.2 for each of the windows W.sub.1 and W.sub.2. In particular, the accumulations N.sub.1 and N.sub.2 are only incremented when the toggle 50 directs the output charge to the corresponding integration module. The counters can be incorporated into the control module 48. This is similar to a ΣΔ modulator where the feedback ensures the convergence of the accumulated signal from the two windows for a long measurement period, compared to the sample period (i.e. oversampled system). The control module 48 determines the product of the number of accumulations and the accumulated charge for each integration module. The control module terminates the process when these products are equal, namely when N.sub.1.Math.Q.sub.1=N.sub.2.Math.Q.sub.2. At the end of the conversion the numbers stored in the counters can be used to determine the ratio of the charges according to Equation 2 below, where T.sub.P is the pulse width:
(59)
(60) One advantage of this ΣΔ ToF measurement method compared to the previous Successive Approximation ToF measurement method is that the accumulated charge does not reset to zero over time which enables achieving better precision. Furthermore, the ΣΔ method does not require a high-frequency reference clock to shift the windows back and forth as is needed in the successive approximation method. The time quantization step is essentially equal to the width of the integration windows, but the quantization noise is kept limited by means of oversampling and noise shaping through ΣΔ modulation.
(61) One implementation of this ΣΔ time-of-flight (ΣΔ−ToF) engine is shown in the circuit diagram of FIG. 14. Like the circuit in FIG. 12 for the SA-ToF measurement, the circuit in FIG. 14 can be associated with each pixel/photodiode 40 or a group of pixels/photodiodes. The circuit includes two gated integration blocks 42, 44 that receive the charge from the photodetector 40 in response to reception of the RX signal. However, unlike the SA-ToF measurement circuit, a toggle 50 determines which integration block receives the signal from the photodetector to accumulate charge. Of course, in the initial step, the toggle 50 provides the RX signal to both blocks.
(62) The outputs from the integration blocks N.sub.1.Math.Q.sub.1 and N.sub.2.Math.Q.sub.2 are provided to a comparator 46 with the result sent to a control module 48 that selects, via toggle 50, which of the integration blocks 42, 44 receives the charge from the photodetector 40 in the next step, depending on the comparison described above. The control module stores the output in a memory block 52. In this case, the output is the number of accumulations for the two windows, N.sub.1, N.sub.2, since according to Equation 2 above the ratio of these accumulations represents the ratio of measured ToF to the maximum measurable ToF.sub.max for the device. This ratio can thus be used to calculate the time-of-flight for the reflected Rx signal by multiplying the actual time value for ToF.sub.max by this ratio. This calculation can occur in the control module 48 or in a separate processor receiving the output 52. The value for the actual time of flight can be provided as a sensible output, such as a visual display, or can be an output signal provided to a processing device, such as a Lidar, or an image or video processing device.
(63) This circuit is similar to a conventional ΣΔ quantizer, except that the present circuit does not use a reference signal in its feedback digital-to-analog converter (DAC). Instead, the circuit uses the charges Q.sub.1 and Q.sub.2 as the two states that the DAC keeps toggling between. Therefore the precision of this technique does not depend on the precision of any analog parameter that should be generated on a CMOS chip, which is an important practical advantage.
(64) One variant of this circuit can store charges from both windows on the same memory element (e.g. capacitor) but with opposite polarity. Then the decision for toggling the accumulation can be made by comparing the total stored charge to the initial reference state of the capacitor while the feedback loop ensures the overall accumulated charge at long time converges to zero (N.sub.1.Math.Q.sub.1−N.sub.2.Math.Q.sub.2=0).
(65) Similar to a conventional ΣΔ modulator, the integrator (i.e. accumulation) in the feed-forward path can be replaced by a higher order transfer function to increase the order of the modulation and improve the precision at the cost of more sophisticated pixels that potentially occupy larger area on a chip. Considering modulation frequency at the range of tens of MHz (which is equivalent to a conventional ΣΔ sample rate) and a frame rate at the range of tens of Hz, the oversampling ratio is close to millions, which is large enough that does not require higher order loop filters for most of the 3D imaging applications.
(66) It can be noted that when using the ΣΔ-ToF measurement system and method described herein, the background light can add to the charge from both windows and cause inaccuracy in the measurement. This issue can be mitigated by background light suppression techniques such as that proposed in M. Davidovic, G. Zach, K. Schneider-Hornstein and H Zimmermann, “ToF Range Finding Sensor in 90 nm CMOS Capable of Suppressing 180 klx Ambient light,” in IEEE Sensors, 2010, the disclosure of which is incorporated herein by reference.
(67) The ΣΔ-ToF measurement method illustrated in FIG. 13, may not be accurate if the modulation waveform is a sharp pulse that falls under only one of the time windows W.sub.1 or W.sub.2, as illustrated in FIG. 15. Under this condition, the charge accumulated in each period is not a function of the pulse timing. In other words, a constant amount of charge is always accumulated in one of the windows, whether the pulse arrives in the beginning of that window or at the end of it. One way to extend the use of this method to such narrow pulse waveforms is to use a multi-bit ΣΔ modulator, where the width of the time quantization steps are short enough to ensure that the pulse-width will always have overlap at least with two of the steps, and hence, the charge accumulated within those steps will be an analog function of the pulse arrival time and the timing information won't be lost.
(68) Another way is to use a two-step converter, in which the first step uses a low-resolution SA-ToF to locate the approximate ISP as described above in connection with FIG. 11. In the second step the exact ratio of the charge in the two windows is quantized using a ΣΔ core to resolve the timing of the arrival signal with higher resolution, as illustrated in FIG. 13. In this case the ΣΔ converter will not have the aforementioned limitation because the SA-ToF already ensures that the two ΣΔ quantization windows have proper timing so that the return pulse RX is not enclosed by a single window and its timing is correctly reflected by the ratio of the charges accumulated in the two windows.
(69) The two step conversion technique described above could be used with SA-ToF followed by any other converter that could find the precise ratio of the charges Q.sub.1 and Q.sub.2 as done by the ΣΔ method. Furthermore, the secondary fine-conversion method does not have to be implemented in the pixel level and can be done in column, or frame-level as suited to the particular camera architecture and use-case. It can be appreciated that the circuitry shown in FIG. 14 in connection with the ΣΔ-ToF measurement method illustrated in FIG. 13 can be operated for the first step as the circuitry shown in FIG. 12 in connection with the SA-ToF method of FIG. 11. Thus, the control module 48 can be configured and operable to close the toggle 50 for both integration modules 42, 44 so that each module receives the charge output from the photodiode 40, thereby allowing the circuitry to operate in the SA-ToF mode. The control module operates in that mode for a limited number of steps, such as the first three steps shown in FIG. 11, to provide a low-resolution estimate of the location of the true ISP. At the completion of the low-resolution steps (i.e., three in the example) the control module 48 operates in the ΣΔ-ToF mode, as described above.
(70) Finding the Integration Symmetry Point
(71) In another feature of the present disclosure, a technique is provided for finding the integral symmetry point ISP of the RX waveform where the waveform is not a square wave. By way of background, a class of circuits known as a constant fractional discriminator (CFD) is often used to extract timing or phase information of a waveform independent of its amplitude and are useful in many different systems including Lidars. In a Lidar, the beams of light reflected from two objects with different reflectivity can have different amplitudes, which should not affect its timing. The measured distance to both objects should still be the same, even though the amplitude of the received signal might be different. An example of a prior art technique used to find the timing information of a waveform independent of its amplitude is disclosed in the publication by A. Rivetti, “Fast timing techniques,” INFN Sezione di Torino, Jun. 2, 2016. Many examples of prior art CFDs employ a time delay, t.sub.d, which must be smaller than the rise time of the incoming pulse where the output of the CFD is a function of the pulse arrival time and the delay, as described in the previously-cited reference and in the publication by Kevin Carnes, “Constant Fraction Discriminators,” January 2003, the entire disclosure of which is incorporated herein by reference. Since the rise time can be on the order of 100's of picoseconds or less this delay must be quite small. In addition, this delay is often a function of environmental influences which vary such as temperature and power applied to the CFD. Hence while traditional CFD's provide timing outputs that are independent of signal amplitude their timing outputs are very sensitive to variation in the delay.
(72) The methods disclosed herein for ToF measurement measure the timing of the waveform independent of its amplitude, which allows these methods to replace traditional CFD methods and circuits. One advantage of the method disclosed herein is that the technique is not based on a time delay. It only requires a fast integrator, which is often comprised of an amplifier and a capacitor, in which the fast integrator can be adaptively controlled with a feedback loop to find the ISP. Methods for an integration-based constant fraction discriminator (CFD) to determine the ISP are described with reference to FIGS. 16-20. Since the ISP point of a waveform is independent of its amplitude, the area underneath a pulse with arbitrary shape, such as the triangular waveform shown in FIG. 16, can be divided into two pieces—A.sub.1 and A.sub.2. The time stamps t.sub.1, t.sub.2 for the two areas A.sub.1 and A.sub.2 are equal and independent of the pulse height.
(73) FIGS. 17-18 illustrate how a feedback mechanism can be used to find the ISP point of the exemplary triangular pulse shape of FIG. 16, as repeated in FIG. 17. The pulse train signal p(t) (corresponding to the RX signal) under the first time window A.sub.1 is supplied to an integration block 60 (FIG. 18) where the signal is integrated or accumulated with a positive sign (D.sub.sign=1) until time t.sub.s, while the area underneath the second time window A.sub.2 is integrated in the integration block 60 with a negative sign (D.sub.sign=−1) commencing at time t.sub.s. The feedback control loop 62 moves the boundary, time t.sub.s, between the two windows by adjusting the duration of D.sub.sign=1, as reflected in the successive time intervals in FIG. 17, so that D.sub.sign is looped at 64 to the integration block to control the positive-negative sense of the integration. The feedback control loop 62 controls the duration of D.sub.sign and thus the increase in the duration of the window A.sub.1 so that the positive integration continues until the error err(t) is no longer negative, at which time the feedback control loop 62 reduces the duration of the window A.sub.1. The duration of window A.sub.1 is continuously adjusted until the sum of the two integrated values err(t) is equal to zero, as reflected in the bottom line of the graph in FIG. 17. It can be noted that this system could be implemented in the digital or analog domain. The feedback signal 64 that controls the integration boundary (which could be analog or digital) can be used as a measure for the position of the ISP on the time axis. In the example of FIG. 17, the duration of the feedback signal D.sub.sign corresponds to t.sub.ISP and it is this value that be used to locate the ISP for the SA-ToF measurement (FIGS. 11-12) or for the ΣΔ-ToF measurement (FIGS. 13-14) when the TX and RX signal waveforms are not square waves. In the embodiment shown in FIG. 19, the feedback control loop can be replaced with a filter 65, which can be an integrator or other type of filter, and analog delay 66 providing the D.sub.sign to the feedback loop 67 to the integration block 60. The analog delay 66 receives a start signal t.sub.start (see FIG. 17) so that the time delay to is adjustable.
(74) It can be appreciated that this ISP determination can be implemented by the control modules 18 or 48 at the commencement of their respective time-of-flight operations. Since the timing of both ToF measurement techniques commence at the ISP of the TX signal, the control modules 18, 48 can be configured ant operable to incorporate the integration and feedback of FIG. 18 or 19 to the TX signal to find the ISP.
(75) FIG. 20 illustrates a ΣΔ-ToF measurement method to find the ISP point. In this embodiment, the integration block 60 is followed by the filter 65 which filters the error signal err(t) generated by the integration block, as in the circuit of FIG. 19. However, the output of the filter is provided to an analog-to-digital converter which generates a total number of accumulations by the ΣΔ-ToF measurement method, and this value is used by a digital delay 74 to determine D.sub.sign. The amount of the digital delay in the feedback path 72 indicates the position of the ISP point on the time axis. The digital delay 74 is controlled by a clock signal 76 and the start time t.sub.start.
(76) The present disclosure should be considered as illustrative and not restrictive in character. It is understood that only certain embodiments have been presented and that all changes, modifications and further applications that come within the spirit of the disclosure are desired to be protected.
