OPTICAL 3-D SENSOR FOR FAST AND DENSE SHAPE CAPTURE

Abstract

An optical 3-D sensor for very fast, highly resolved and dense capture of the surface shape of objects in 3-D space. One image or a plurality of images recorded at the same time in a single shot method suffice for the recording. Using this, it is possible, as a matter of principle, to record 3-D data with the frame rate of the employed cameras, i.e. to build a 3-D video camera. The optical 3-D sensor has a projector projecting a line pattern onto the object, and K cameras, which each record an image of the object illuminated by the projector. The line pattern contains lines in a number of R directions and the K cameras are disposed to span up to K×R triangulation sensors. The triangulation sensors are coupled by a control and evaluation unit by way of the common line pattern.

Claims

1. An optical single shot 3-D sensor for shape capture of an object, the sensor comprising: a projector for projecting a line pattern onto the object, said projector having an optical axis, the line pattern having lines in a plurality of R line directions, with R≧2; a plurality of K cameras, each having a respective optical axis, where K≧2, said cameras being disposed to record an image of the object illuminated by said projector; and said K cameras being arranged to span with the projector up to K×R triangulation sensors with up to K×R triangulation angles in respect of the R line directions; and a control and evaluation unit configure to process the image; said evaluation unit coupling said triangulation sensors by way of the common line pattern.

2. The optical 3-D sensor according to claim 1, wherein: said plurality of cameras are two cameras; said projector is configured to project a cross grid with a multiplicity of lines onto the object; directions of the lines define directions of an x-axis and a y-axis of a coordinate system; the optical axes, of said cameras, span a first, a second, a third and a fourth triangulation angle with an xz-plane and an yz-plane, respectively, of the coordinate system; and the optical 3-D sensor contains four independently measuring triangulation systems T11, T12, T21, T22 with the triangulation angles which acquire two independent measurement data records of the same surface points for each line direction; and said control and evaluation unit is configured to couple the measurement data records of said independently measuring triangulation systems.

3. The optical 3-D sensor according to claim 2, wherein: the second and third triangulation angles are each selected to be larger than the first and fourth triangulation angles, in that said triangulation sensors T12 and T21 achieve a higher accuracy than said triangulation sensors T11 and T22 for the projected lines in the x-direction and in the y-direction; the first and fourth triangulation angles are selected in such a way that uniqueness of the measurement data records of said triangulation sensors T11 and T22 in the measurement volume is immediately achieved in said triangulation systems T11 and T22, and the measurement data records of the triangulation systems are coupled by forwarding the obtained information relating to the uniqueness.

4. The optical 3-D sensor according to claim 2, wherein: the second and third triangulation angles are selected in such a way that the associated said triangulation sensors T12 and T21 achieve a predetermined accuracy for the projected lines in the x-direction and in the y-direction, the first and fourth triangulation angles are selected in such a way that, for the triangulation systems T11 and T22 according to equation $L\le \frac{\Delta .Math..Math.x}{\Delta .Math..Math.z.Math.\tan .Math..Math.\theta }$ unique measurement depths of T11 and T22 in the case of a fixed line number L are such that the correct line index may be ascertained with the aid of a residual number method, analogously to reading a Vernier scale.

5. The optical 3-D sensor according to claim 1, wherein the line pattern is a line grid with only one line direction and effective triangulation angles and lie in a plane lying perpendicular to the one line direction.

6. The optical 3-D sensor according to claim 1, wherein one or more said cameras are color cameras or color cameras are additionally introduced in order to acquire information about a color texture of the object surface in addition to capturing the 3-D data.

7. The optical 3-D sensor according to claim 1, wherein a period of the projected line pattern varies locally or for different directions.

8. The optical 3-D sensor according to claim 1, wherein the sensor is configured to acquire 3-D data with a camera clock rate and produce a 3-D model of the object therefrom in each case, wherein a series of 3-D models produced in succession forms a 3-D film sequence to be played back on a display.

9. The optical 3-D sensor according to claim 1, wherein a viewing angle when displaying the acquired data as a 3-D model is controlled by a sensor in real time, said sensor being configured to ascertain a position of an observer.

10. The optical 3-D sensor according to claim 1, which further comprises a display device which forms the 3-D model produced by the control and evaluation unit from a plurality of viewing angles for simultaneous display purposes.

11. A single shot method for producing a 3-D model of an object, the method comprising: producing Q=K×R triangulation systems with K cameras and a line pattern projected onto the object with R different line directions, where K≧2; using a control and evaluation unit for producing a non-unique 3-D model for each of the Q triangulation systems from images that are obtained simultaneously from the K cameras, thereby selecting the triangulation systems such that the non-unique 3-D models are independent of one another; and coupling the non-unique 3-D models in the control and evaluation unit such that, if two or more non-unique 3-D models in a data record correspond, the corresponding data record is used as the 3-D model of the object.

Description

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

[0030] FIG. 1 is a diagrammatic view showing a system with a single camera and a projector, and visualizing the variables of Eq. 1;

[0031] FIG. 2 is a diagrammatic view of a system according to the invention with a plurality of cameras;

[0032] FIG. 3 is a diagrammatic view of an exemplary system with two cameras;

[0033] FIG. 4 illustrates how correspondences of M.sub.1 and M.sub.2 are sought in the 3-D space and the combined unique model M.sub.K is produced therefrom. Part a) shows a situation where M.sub.1 is unique but noisy, and part b) symbolically shows the ambiguous models M.sub.1 and M.sub.2 of the data records of T1, T2 based on a plane object;

[0034] FIG. 5 is a schematic view showing an embodiment in which a pattern is projected with a cross grid;

[0035] FIG. 6 shows the visualization of a plurality of perspectives on a plurality of monitors; and

[0036] FIG. 7 illustrates three individual images of a 3-D film sequence from three different, freely selected perspectives (A, B, C).

DETAILED DESCRIPTION OF THE INVENTION

[0037] Therefore, according to equation 1, a dense and simultaneously precise single shot measurement within a relatively large measurement depth is impossible using the known state of multi-line triangulation.

[0038] The invention solves this problem by introducing one or more additional cameras (10), (20), (30), . . . , and projecting line patterns (6), preferably with lines in a plurality of directions. The optical 3-D sensor consists of a sensor head (3) and a control and evaluation unit (4), as well as a display apparatus (5) for visualization purposes, according to FIG. 2. Below, an exemplary embodiment (see FIG. 3) of a 3-D sensor with two cameras (10) and (20) is intended to be described. The embodiment in which a pattern (6a) is projected with lines only along one direction is considered first. In FIG. 3, the line direction defines e.g. the y-direction of the coordinate system. In order to further increase the data density, an embodiment is further described, in which a pattern (6b) is projected with a cross grid, as shown schematically in FIG. 5. The directions of the lines define the x-direction and y-direction of the coordinate system in an exemplary manner. Without loss of generality, the projector (7) is situated at the coordinate origin. The optical axis (8) of the projector coincides with the z-axis of the coordinate system and is directed to the object (9).

[0039] For the embodiment according to FIG. 3, the two cameras (10) and (20) are arranged along the x-axis, with the optical axes (13) and (14) thereof preferably likewise being directed to the object (9) and including different triangulation angles (1) and (2) with the optical axis (8) of the projector (the z-axis). Thus, a sensor head (3) consisting of two triangulation sensors T1 and T2, which use the same projection pattern (6a), arises. The triangulation sensor T1 is defined by the projector (7) and the camera (10). The triangulation sensor T2 is defined by the projector (7) and the camera (20). If both cameras (10) and (20) each record an image of the object (9) illuminated by the pattern (6a) at the same time, the 3-D information of the object surface is acquired by two independent triangulation sensors T1 and T2 at the locations where the pattern (6a) is projected. What is important for the method is that the triangulation sensors T1, T2 measure the same object points.

[0040] A fundamental concept of the invention is that a 3-D sensor arises as a result of coupling the triangulation sensors (T1 and T2 in the exemplary embodiment of FIG. 3), said 3-D sensor acquiring the object surface both densely (many lines) and with a low measurement uncertainty δz (large triangulation angle) and, at the same time, acquiring a large measurement depth Δz in the single shot method.

[0041] By way of example, in the exemplary embodiment according to FIG. 3, this can be achieved by virtue of camera (10) being arranged at a small triangulation angle (1) and camera (20) being arranged at a large triangulation angle (2) along the x-axis. According to equation 1, a very large unique measurement depth Δz.sub.T1 emerges for T1 for a number L of projected lines and a given measurement field Δx, i.e. the lines are correctly indexed. However, the 3-D data exhibit a large measurement uncertainty δz.sub.T1 on account of the small triangulation angle (1). A lower unique measurement depth Δz.sub.T2 with, at the same time, a lower measurement uncertainty δz.sub.T2 arises for the sensor T2. However, on account of the lower unique measurement depth Δz.sub.T2, the lines cannot be correctly indexed by the triangulation sensor T2 and the data are not uniquely evaluable without additional information.

[0042] According to the invention, the coupling of these two triangulation sensors may solve this problem. The coupling may be achieved by back projection of the data records of the triangulation sensors.

[0043] Initially, a 3-D model M.sub.1 of the object is calculated from the data of T1. As described, this model exhibits a large measurement uncertainty but correct indexing of the lines. Now, all 3-D points of M.sub.1, including the index information thereof, are projected back numerically onto the image of camera (20) of T2 from the 3-D space. In a calibrated overall system, the back-projected 3-D points will virtually correspond to the line signals in the image of the camera (20). Here, the back-projected 3-D points exhibit more noise but correct indices. These indices are now assigned to the lines in the camera image of T2. Using this, the data of T2 now may be evaluated uniquely and with a low measurement uncertainty, and a correct model M.sub.K of the object may be created. The restriction applying to each one of the triangulation sensors T1, T2 individually as a result of equation 1 is therefore lifted by the coupling for the 3-D sensor.

[0044] The basic idea of using two cameras and a small triangulation angle and a large triangulation angle with a line pattern with only one direction, as shown in (6a), is mentioned in the article [Willomitzer et al., “Single-shot 3D sensing with improved data density”, Appl. Opt. 54(3), 408-417 (2015)].

[0045] The present disclosure develops the idea of advantageously using a plurality of cameras, in particular by using line patterns with a plurality of directions, which produces a large number of coupleable triangulation sensors in conjunction with a plurality of cameras.

[0046] Nor is coupling by means of back projection restricted to the described embodiment with a small triangulation angle (1) and a large triangulation angle (2). It is also possible to select both triangulation angles to have approximately the same size such that, in the case of a fixed line number L, the unique measurement depths Δz.sub.T1 and Δz.sub.T2 (according to equation 1) are such that the correct line index may be ascertained with the aid of a residual number method, analogous to reading a Vernier scale (see FIG. 4b). By way of example, the selection of Δz.sub.T1=300 mm and Δz.sub.T2=400 mm would provide a resulting unique measurement range of 1200 mm. On the basis of the plane object, FIG. 4b symbolically shows the ambiguous models M.sub.1 and M.sub.2 of the data records of T1, T2. Coupling the data by way of the back projection permits ascertaining the “correct” model M.sub.K.

[0047] In the aforementioned case of a small angle and a large angle, the ambiguous 3-D model M.sub.2 of T2 may be calculated directly in addition to the calculation of the 3-D model M.sub.1. Correspondences of M.sub.1 and M.sub.2 are sought after in the 3-D space and the combined unique model M.sub.K is produced therefrom. This is depicted symbolically in FIG. 4a, where M.sub.1 is unique but noisy. The acquisition of a plane object is assumed, and the models M.sub.1, M.sub.2 and the combined unique model M.sub.K are shown.

[0048] Furthermore, coupling of T1 and T2 is also possible by way of features of the object, and also conceivable by combining the teaching of the patent with encoding of the patterns (e.g. spatially or by color).

[0049] In order to further increase the density of the measurement data with unchanging quality, the described embodiment of FIG. 3 may be modified. In so doing, a line pattern (6b), formed as a cross grid, is now projected in place of a line pattern (6a). In this case, the two line directions lie along the x-direction and y-direction of the coordinate axes in an exemplary manner. In principle, it would now be possible to add a further 3-D sensor, rotated by 90° in relation to the z-axis, in a manner analogous to the embodiment according to FIG. 3. There is a very effective possibility to achieve this without additional cameras according to the teaching of the patent document: if the cameras (10), (20), as shown in FIG. 5, are positioned next to the x-axis and y-axis, the optical axes (13), (14) thereof, which, once again are preferably directed to the object (9), span four angles (11), (12), (21), (22) with the xz-plane and yz-plane. These always act in a pairwise manner as triangulation angles for a projected line direction. Four independent triangulation systems T11, T12, T21, T22 are formed, which acquire at least 2 independent 3-D data records of the same surface points for each line direction. As a result, the teaching of the patent is already satisfied using two cameras instead of using four cameras.

[0050] FIG. 5a shows a spatial illustration of the 3-D sensor with the object (9) and a scheme of the projected pattern (6b). FIG. 5b shows a projection of the 3-D sensor onto the xz-plane; FIG. 5c shows a projection onto the yz-plane. Here, the angles (11) and (21) are triangulation angles for the lines in the y-direction (cf. FIG. 5b); the angles (12) and (22) are the triangulation angles for the lines in the x-direction (cf. FIG. 5c). As shown in FIG. 5, this allows an embodiment with four different triangulation angles for both projected line directions to be realized using only two cameras.

[0051] All methods for coupling triangulation systems by back projection, described for the embodiment according to FIG. 3, may analogously be applied to the embodiment according to FIG. 5.

[0052] The idea of the embodiment according to FIG. 5 may be extended without constraints to systems with more cameras or more line directions. As a matter of principle, Q=K×R triangulation systems may be produced using K cameras and R line directions. Depending on the position of the cameras and the line directions or the period of the line patterns, these Q triangulation systems exhibit different properties in respect of the unique depth range, the measurement uncertainty and the data density. These properties may also vary locally. The teaching of the patent allows many illumination options. It is by no means necessary to use a periodic pattern or only have two line directions. Other patterns, as shown in an exemplary manner at (6c), (6d), (6e), (6f) in FIG. 2, allow a multiplicity of variations. This is due to the projection axis (8), the camera axes (13), (14), (15), . . . and also the line directions or line segment directions determining the properties of the triangulation sensors. Coupling the triangulation sensors by way of the common observed line pattern and combining or coupling the respective data by the evaluation unit in turn permits the production of dense and, at the same time, accurate 3-D data of the object surface to be acquired.

[0053] This is easily comprehensible: it is possible to produce 6 different triangulation sensors T1 to T6 which acquire 6 independent data records of the same object using e.g. three line directions and two cameras. Therefore, 6 independent data records are available for removing possible errors in the data records, for example on account of lack of uniqueness or noise. Here, coupling by back projection is also applicable, similar to how it is depicted for two sensors in FIG. 4a and FIG. 4b. The 6 triangulation sensors T1 to T6 generate 6 non-unique 3-D models M1 to M6. In this case, the line patterns and triangulation angles may even be selected arbitrarily. This is because the models M1 to M6 should all show the same object model if they are error-free. If at least two or more data records correspond, there is a high probability for these data records to be correct. The described statistical method is particularly suitable for coupling if the projected line patterns are very complicated and if many triangulation sensors contribute.

[0054] The projection of a line pattern as in (6c) should be highlighted. A sensor with such a line pattern allows a further increase in the data density, both with two cameras as depicted in FIG. 4 and with an additional camera.

[0055] For projecting a plurality of grid line directions, it is also possible to project a different line period for each direction, for example by using line patterns like (6e) or (6f). In this case, according to equation 1, different unique measurement depths Δz.sub.Ti would result for the same triangulation angle.

[0056] Furthermore, the crossing points may be used for indexing in the case of a plurality of grid line directions. If the correct index of the line is known, the index of all lines intersecting this line at the crossing points is known. The relevant indices may be transferred to the remaining line segments by tracking the lines in the camera image. The selection of fitting line periods is effective, particularly for this embodiment.

[0057] Crossed lines further have the advantage that it is possible to calculate the gradient of the object surface in both directions at the crossing points. Using this, a very exact interpolation of the 3-D data in the interstices between the lines is possible. Moreover, the decomposition of an object surface into triangular meshes for cross grid projection is simpler and always unique.

[0058] In order to evaluate each line correctly and with a high accuracy, it is initially necessary to separate out line directions in the evaluation unit (4). By way of example, this may be carried out by means of directional filtering of the camera images in the Fourier plane or in real space. So that data may still be evaluated at the crossing points of the lines as well, it may be advantageous to increase or reduce, e.g. also set to zero, the intensity of the projected pattern at the crossing points in relation to the remaining line segments, or to vary the width of the lines at such locations. If the possibility of color selection exists for the employed cameras, the different line directions may also be separated, as a matter of principle, by suitable color encoding of the lines.

[0059] A peculiarity of the present teaching is that complete 3-D information about the shape of the surface is available immediately after each recording clock. This opens up possibilities for processing and visualizing in real time, as are not as advantageously available in the prior art:

[0060] First of all, the single shot method allows the three-dimensional capture of extremely fast processes, including the capture of moving, not even rigid objects or the measurement of the deformation in the case of crash tests. The image frequency of the camera restricts the fast acquisition only conditionally because the image scene may be captured in three dimensions within a very short period of time, which is significantly faster than permitted by the camera clock rate, by a short-term exposure (flash) or a short electronic shutter speed, as facilitated by modern cameras. This is possible because the illumination is carried out with a “fixed pattern”. A fixed pattern which does not vary in time may be realized more effectively and significantly more easily from a technical point of view—and it allows arbitrarily short exposure. However, this does not exclude controllable pattern generators, e.g. laser projectors, which are very fast and permit a very bright projection, from being used for the pattern projection. Furthermore, in certain cases, the data density may be more important than the speed. In this case, a time sequential displacement of grids may be linked with the teaching of the patent.

[0061] The 3-D data being available immediately in time permits a real time representation and visualization such that the user of the sensor may guide the sensor interactively, for example continuously (without “stop and go”), around the object and, in the process, has control about the already supplied data at all times. Hence, the object may be measured comfortably and about the complete circumference, which would not readily be possible without the real time representation.

[0062] A photorealistic representation of the object, as may be achieved by known methods of surface decomposition into triangular meshes and rendering, even in real time, is advantageous for the interactive use of the sensor for virtual reality.

[0063] The additional acquisition of the color texture is also advantageous for the photorealistic representation. The presented method may be extended to this end, either by virtue of the cameras (10), (20), (30) . . . being color cameras or by virtue of an additional color camera being installed into the sensor. In the first case, the image data contain both the color texture and the 3-D data. The second case offers the advantage that black and white cameras generally facilitate a more accurate 3-D capture and the capture of the color texture is independent of the 3-D acquisition. Thus, it is also conceivable for the color texture to be completely independent of the 3-D data acquisition by appropriate large area, short-term white illumination at appropriate times. This is possible with both of the mentioned options.

[0064] A particularly important advantage of the single shot real-time data is the option of a visualization from different perspectives: a data record containing the complete 3-D data of the object surface is acquired in each camera clock. If a plurality of data records of a moving object are recorded in several sequential camera clocks, these may finally be visualized as a 3-D film sequence. Hence, the visualization of a 3-D film sequence (in real time or else subsequently) is possible under free selection of the observation perspective or even under changing the observation perspective during the 3-D film sequence (“free viewpoint movie”). In addition, the 3-D film sequence may be visualized simultaneously for arbitrary perspectives.

[0065] This is demonstrated in FIG. 7. Shown are, in an exemplary manner, 3 individual images of a 3-D film sequence from 3 different, freely selected perspectives (A, B, C). The 3-D data for the imaged visualizations were acquired using the 3-D sensor of the embodiment according to FIG. 5. A talking person was measured.

[0066] The model and the visualization already contain the texture of the object because the line intensity is also captured on account of the local object reflectivity in the camera images. An interpolation of the data points between the lines or surface decomposition into triangular meshes was not applied in the example of FIG. 7, although this may take place, as a matter of principle, for a photorealistic representation.

[0067] The simplest option for varying the angle of view consists of the observer setting or varying the desired perspective in real time using a control appliance, e.g. a computer mouse.

[0068] Another option consists of a sensor (23) for capturing the observer position being attached to the monitor (5). If the observer moves in front of the monitor, the captured observer position may serve to depict the shown perspective in such a way as if the observer moves around the visualized object.

[0069] However, for example as depicted in FIG. 6, it is also possible to display the visualization of a plurality of perspectives on a plurality of monitors or on a curved monitor. The monitor (16) shows the virtual model (19) of the object from the direction (16a); the monitor (17) shows the model (19) from the direction (17a); the monitor (18) shows the model (19) from the direction (18a). Expediently, the monitors may be set up in such a way that one or more observers have the option of observing the monitors simultaneously or in succession as a result of movement and thus, as it were, of walking around the visualized object.

[0070] The aforementioned options for visualization may also be obtained with the aid of so-called VR data glasses.

[0071] Using the presented optical 3-D sensor, it is possible to measure objects once extremely quickly, for example for visual examination of fast processes in industry. However, it is also possible to capture moving scenes as a 3-D film sequence, for example for virtual reality.