SURFACE RECONSTRUCTION OF AN ILLUMINATED OBJECT BY MEANS OF PHOTOMETRIC STEREO ANALYSIS

Abstract

A method for surface reconstruction may include illuminating at least one object simultaneously by light emitted by a plurality of luminaires spaced apart from another. The method may further include recording a photographic sequence comprising a plurality of individual images of the object(s). The method may further include reconstructing at least one visible object surface of the object by photometric stereo analysis.

Claims

1. A method for surface reconstruction, wherein the method comprises: illuminating at least one object simultaneously by light emitted by a plurality of luminaires spaced apart from one another; recording a photographic sequence comprising a plurality of individual images of the at least one object; and reconstructing at least one visible object surface of the object by photometric stereo analysis; wherein: the light emitted by the plurality of luminaires is modulated with different modulation frequencies; the light components of the respective luminaires that are reflected by the object are identified on the basis of their modulation frequencies (fa-fc; fa-fd) and are assigned to respective partial images; and the partial images are used configured as input images for the photometric stereo analysis.

2. The method as claimed in claim 1, wherein: each pixel of the individual images of the sequence is subjected to a respective Fourier analysis, and Fourier components obtained from the Fourier analysis, as values of the corresponding pixels, are assigned to the respective partial images.

3. The method as claimed in claim 1, wherein more than three luminaires of the plurality of luminaires with respective modulation frequencies are used.

4. The method as claimed in claim 3, wherein the object is illuminated simultaneously by the more than three differently modulated luminaires, respective partial images associated with the more than three luminaires are generated, photometric stereo analyses are carried out for different luminaire combinations with three partial images in each case, and the at least one visible object surface comprises at least two of the respectively resulting object surfaces combined to form a single final object surface.

5. The method as claimed in claim 2, wherein: the modulation frequencies of the plurality of luminaires remain constant over time, the plurality of individual images are recorded with a sampling rate of the camera that is higher than the modulation frequencies, and the Fourier analysis is carried out for a frequency range extending maximally to half the sampling rate.

6. The method as claimed in claim 2, wherein: the modulation frequencies of the plurality of luminaires remain constant over time, the plurality of individual images are recorded with a sampling rate that is lower than the modulation frequencies, and the Fourier analysis is carried out for a frequency range extending maximally to half the sampling rate.

7. The method as claimed in claim 2, wherein: the modulation frequencies of the plurality of luminaires remain constant over time, the plurality of individual images are recorded with a sampling rate that is lower than the modulation frequencies, and before the Fourier analysis, a digital low-pass filter having a cut-off frequency corresponding to a sampling rate of the camera is applied to a temporal series of respective pixels.

8. The method as claimed in claim 2, wherein: the modulation frequencies of the plurality of luminaires remains constant over time, the individual images are recorded with a sampling rate that is lower than the modulation frequencies, the Fourier analysis is carried out for a frequency range extending to the respective modulation frequency, Fourier components of this frequency range of different sequences are averaged, and the sampling rate and the modulation frequencies are not synchronized.

9. The method as claimed in claim 1, wherein at least one modulation amplitude remains constant.

10. An apparatus configured to carry out the method as claimed in claim 1.

11. The apparatus as claimed in claim 10, wherein each luminaire of the plurality of luminaires comprises at least one light emitting diode.

12. The apparatus as claimed in claim 10, wherein the apparatus is a monitoring system.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In the embodiments and figures, components which are the same or of the same type, or which have the same effect, are respectively provided with the same references. The elements represented and their size ratios with respect to one another are not to be regarded as to scale. Rather, individual elements, in particular layer thicknesses, may be represented exaggeratedly large for better understanding.

[0091] FIG. 1 shows a monitoring system in accordance with a first exemplary embodiment;

[0092] FIG. 2 shows a sequence of individual images that were recorded by a camera of the monitoring system in accordance with the first exemplary embodiment;

[0093] FIG. 3 shows a time series of measurement values of a pixel of the sequence;

[0094] FIG. 4 shows a result of a standard Fourier analysis of the time series of measurement values;

[0095] FIG. 5 shows partial images obtained from the Fourier analysis;

[0096] FIG. 6 shows a monitoring system in accordance with a second exemplary embodiment;

[0097] FIG. 7 shows a time series of measurement values of a pixel of a sequence of individual images;

[0098] FIG. 8 shows a result of a standard Fourier analysis of the time series of measurement values that were recorded by the camera of the monitoring system in accordance with the second exemplary embodiment; and

[0099] FIG. 9 shows a determination of a surface point of a reconstructed surface from a set of a plurality of surface points.

DETAILED DESCRIPTION

[0100] FIG. 1 shows a monitoring system S1 in accordance with a first exemplary embodiment. The monitoring system S1 includes three luminaires a, b and c, respectively, spaced apart from one another. The luminaires a to c are spaced apart from one another such that light emitted by them is incident on an object O at different angles of incidence. The monitoring system S1 furthermore includes a digital camera K, the field of view of which is directed into a spatial region capable of being illuminated by the luminaires a to c.

[0101] The camera K is able to record temporal sequences of individual images EB (see FIG. 2) with a specific sampling rate f_s. The object O is present or imaged in the individual images EB. A corresponding surface region or surface point OP(x,y) is assigned to each pixel BP(x,y) of an individual image EB.

[0102] The monitoring system S1 additionally includes an evaluation device A, by means of which the photographic sequence is able to be evaluated in order to reconstruct an object surface OS1, OS2 of the object O that is visible to the camera K by means of photometric stereo analysis. The evaluation device A can also be configured to carry out an object recognition and/or activity recognition on the basis of the reconstructed object surfaces OS1, OS2.

[0103] The monitoring system S1 is configured, in particular, to modulate the light La, Lb and Lc, respectively, emitted by the luminaires a to c with different modulation frequencies fa, fb and fc, respectively (for example by way of suitable drivers). The modulation frequencies fa, fb and fc are relatively prime, in particular, with the result that an occurrence of an intermodulation can be prevented particularly effectively. The luminaires a, b, and c may each include at least one light emitting diode (not illustrated), in particular at least one light emitting diode which emits white light La to Lc. The luminaires a, b and c can have an identical construction.

[0104] FIG. 2 shows a succession or sequence of n individual images EB_r, EB_r+1, EB_-r+2, EB_r+n that were recorded by the camera K of the monitoring system S. Each of the individual images EB has a plurality of pixels BP(x,y), e.g. with x and/or y from a set {1024640}.

[0105] On account of the modulation of the luminaires a to c, the illumination situations of the pixels BP(x,y) are different at the time of the recordings of the individual images EB_r to EB_r+n.

[0106] FIG. 3 shows a time series of measurement values of a specific pixel BP(x,y) from the sequence of the individual images EB_r to EB_r+n. The magnitude or the value of the pixel BP(x,y) at a respective point in time is a (pixel) measurement value PMw(x,y) and can correspond e.g. to a brightness value.

[0107] FIG. 4 shows Fourier components FT_a, FT_b, FT_c as a result of a standard Fourier analysis of the time series of measurement values PMw(tr), . . . , PMw(tr+n) at points in time tr, . . . , tr+n from FIG. 3. The standard Fourier analysis can be a fast Fourier transformation (FFT), for example.

[0108] The magnitudes or values of the Fourier components FT_a, FT_b and FT_c correspond to representative intensity measurement values Iw which include only the portions of the light components La, Lb and Lc of the luminaires a, b and c, respectively.

[0109] FIG. 5 shows three partial images TBa, TBb and TBc obtained from the Fourier analysis. The partial images TBa, TBb and Tc respectively include or include the intensity measurement values Iw of the associated Fourier components FTa, FTb and FTc at the respective pixels BP(x,y).

[0110] The three partial images TBa, TBb and TBc correspond to individual images that would have been recorded only upon illumination by a respective one of the luminaires a, b and c. As indicated by the arrow, these three partial images TBa, TBb and TBc can be used as input images for a photometric stereo analysis in order to reconstruct the object surfaces OS1, OS2.

[0111] FIG. 6 shows a monitoring system S2 in accordance with a second exemplary embodiment. The monitoring system S2 is constructed similarly to the monitoring system S1, but now includes a further luminaire d, which emits modulated light Ld having a modulation frequency fd.

[0112] FIG. 7 shows, analogously to FIG. 4, a result of a standard Fourier analysis of the time series of measurement values of a pixel BP(x,y) of a sequence of individual images EB_r, Eb_r+n that were recorded by the camera K of the monitoring system S2. Four Fourier components FT_a to FT_d corresponding to the modulation frequencies fa to fd now result.

[0113] From the four Fourier components FT_a to FT_d of all the pixels, it is possible to generate four corresponding partial images TBa to TBd (see FIG. 8).

[0114] FIG. 8 shows the corresponding partial images TBa to TBd generated from four usable different triplet groups of luminaire combinations by means of Fourier transformations. They are the partial images TBa, TBb and Tc of the luminaire combination {a, b, c},the partial images TBa, TBb and TBd of the luminaire combination {a, b, d}, the partial images TBa, TBc and TBd of the luminaire combination {a, c, d} and the partial images TBb, TBc and TBd of the luminaire combination {b, c, d}. From the respective triplets of the partial images.

[0115] However, since only three partial images are required for a photometric stereo analysis, in each case an independent photometric surface reconstruction of the object surfaces OS1, OS2 of the object O can be carried out from the triplets of the partial images TBa to TBd.

[0116] The four independent surface reconstructions can be used by means of superimposition to form a consistent end result for the surface reconstruction. This takes account of the fact that deviations of the surfaces OS1, OS2 reconstructed by means of the different triplets can occur in practice, e.g. on account of shadings or luster effects (specular albedo).

[0117] FIG. 9 shows a determination of a final pixel value PMw_final for a specific surface point OP(x,y) from a set of a plurality (here: four) of pixel values PMw_1 to PMw_4 for this same surface point OP(x,y). The four pixel values PMw_1 to PMw_4 thus correspond in each case to nominally the same surface point OP(x,y) resulting from the surfaces reconstructed by means of the four different luminaire combinations.

[0118] If one pixel value PMw_1 lies outside a predefined bandwidth B, which is indicated here by the circle, with respect to the other pixel values PMw_2 to PMw_4, it can be regarded as an outlier and be excluded from a consideration for determining the final pixel value PMw_final, as illustrated in the left-hand part of FIG. 9.

[0119] In principle, the exclusion method can also be applied to more than one of the surface points OP(x,y).

[0120] If all pixel values apart from one pixel value have been excluded, the pixel value that has remained is set as the final pixel value PMw_final.

[0121] If a plurality of pixel values PMw_2 to PMw_4 have remained, they can be averaged, for example, in order to calculate an averaged pixel value PMw_avg, which is used as the final surface point PMw_final. This is shown in the right-hand part of FIG. 9.

[0122] The consistency concepts above can be applied to all surface points OP(x,y) or reconstructed surfaces OS1, OS2.

[0123] Although the invention has been more specifically illustrated and described in detail by means of the exemplary embodiments shown, nevertheless the invention is not restricted thereto and other variations can be derived therefrom by the person skilled in the art, without departing from the scope of protection of the invention.

[0124] In this regard, the frame rate of the camera K can also be so low that undersampling is present.

[0125] Generally, a(n), one, etc., can be understood to mean a singular or a plural, in particular in the sense of at least one or one or a plurality, etc., as long as this is not explicitly excluded, e.g. by the expression exactly one, etc.

[0126] Moreover, a numerical indication can encompass exactly the indicated number and also a customary tolerance range, as long as this is not explicitly excluded.

LIST OF REFERENCE SIGNS

[0127] Evaluation device A [0128] Luminaire a [0129] Bandwidth [0130] Pixel at the position (x,y) BP(x,y) [0131] Luminaire [0132] Luminaire [0133] Luminaire [0134] Individual images r, r+n EB_r to EB_r+n [0135] Fourier component FT_a to FT_d [0136] Modulation frequencies of the luminaires a to d [0137] fa to Fd [0138] Intensity measurement value Iw [0139] Digital camera [0140] Modulated light of the luminaires a to d La to Ld [0141] Object O [0142] Surface point of the pixel BP(x,y) OP(x,y) [0143] Measurement value of the pixel BP(x,y) PMw(x,y) [0144] Averaged pixel value PMw_avg [0145] Final pixel value PMw_final [0146] Pixel values of different triplets PWm_1 to PWm_4 [0147] Object surface OS1 [0148] Object surface OS2 [0149] Sampling rate f_s [0150] Monitoring system S1 to S2 [0151] Partial image TBa to TBd