Mapping Techniques Using Probe Vehicles

Abstract

Vehicle-mounted device includes an inertial measurement unit (IMU) (8) having at least one accelerometer or gyroscope, a GPS receiver (6), a camera (10) positioned to obtain unobstructed images of an area exterior of the vehicle (16) and a control system (20) coupled to these components. The control system (20) re-calibrates each accelerometer or gyroscope using signals obtained by the GPS receiver (6), and derives information about objects in the images obtained by the camera (10) and location of the objects based on data from the IMU (8) and GPS receiver (6). A communication system (18) communicates the information derived by the control system (20) to a location separate and apart from the vehicle (16). The control system (20) includes a processor that provides a location of the camera (10) and a direction in which the camera (10) is imaging based on data from the IMU corrected based on data from the GPS receiver (6), for use in creating the map database (12). (FIG. 2)

Claims

1. A vehicle-mounted device, comprising: an inertial measurement unit (IMU) comprising at least one accelerometer and at least one gyroscope; a GPS receiver that provides a GPS-derived location; a camera positioned to obtain unobstructed images of an area exterior of the vehicle; a control system coupled to said IMU, said GPS receiver and said camera, said control system including a processor and being configured to re-calibrate said at least one accelerometer and said at least one gyroscope using signals obtained by said GPS receiver, said control system being further configured to compare output from said at least one accelerometer with the GPS-derived location and angular orientation of the vehicle, and based on the comparison, modify acceleration output from said at least one accelerometer and modify angular velocity output from said at least one gyroscope, said control system being further configured to determine position and angular orientation of the vehicle from the modified acceleration output from said at least one accelerometer and modified angular velocity output from said at least one gyroscope, said control system being further configured to determine a direction in which said camera is pointing when images are obtained by said camera based on the determined angular orientation of the vehicle, said control system deriving information about objects in the images obtained by said camera and location of the objects based on the determined position of the vehicle and the determined direction in which said camera is pointing when images including the objects are obtained by said camera; a communication system coupled to said control system and that communicates the images obtained by said camera or information derived by said control system to a location separate and apart from the vehicle; and a map database resident on the vehicle, said map database including information about and location of objects in images obtained by said camera.

2. The device of claim 1, wherein said IMU is manufactured using mass-production MEMS technology.

3. The device of claim 1, wherein said control system is further configured to re-calibrate said at least one accelerometer and said at least one gyroscope using a zero lateral and vertical speed of the vehicle and speedometer and odometer readings of the vehicle.

4. The device of claim 1, wherein said control system is further configured to correct projections of gravitational acceleration in accelerometer readings from said at least one accelerometer, when the vehicle tilts, and projections of centrifugal accelerations in readings from said at least one gyroscope during turning maneuvers.

5. The device of claim 1, wherein said IMU, said GPS receiver, said camera, said control system and said communication system are mounted in a single housing, said housing being positioned on the vehicle such that said camera images a portion of a road in front of the vehicle and terrain on both sides of the vehicle, said camera having a horizontal field of view of from about 45 to about 180 degrees.

6. The device of claim 1, further comprising a speed limiting apparatus that notifies a driver of the vehicle of a maximum speed of travel of the vehicle or automatically limits speed of the vehicle to the maximum speed of travel of the vehicle at the location at which the vehicle is travelling, said control system being coupled to said speed limiting apparatus and generating the maximum speed of travel of the vehicle based on speed and accuracy of processing of images obtained by said camera by said control system.

7. The device of claim 1, wherein said control system is further configured to determine the location of objects in the images obtained by said camera from multiple images using displacement of the vehicle between the times when the multiples images are obtained and a known orientation of said camera relative to the vehicle when each of the multiple images is obtained, the determined location of the objects in the images obtained by said camera being included in said map database.

8. The device of claim 1, wherein said control system is further configured to identify objects in images obtained by said camera and their location and determine whether the objects are present in said map database, said processor being configured to communicate the images obtained by said camera that include objects not present in said map database or information about an object derived by said control system that is not included in said map database to the location separate and apart from the vehicle.

9. The device of claim 1, wherein said control system is configured to correct data from said IMU using said map database and said camera and without use of data from said GPS receiver by comparing expected position of an object in an image obtained by said camera when the vehicle is at a specific position using said map database to actual position of the same object in an image obtained by said camera as determined by said processor when the vehicle when at the specific position, and correcting the data from said IMU when the actual vehicle position differs from the expected vehicle position.

10. The device of claim 1, wherein said communication system communicates location of the vehicle to the remote location, a determination being made at the remote station whether images of the area exterior of the vehicle at the vehicle's location communicated to the remote station using said communication system are needed to obtain information about objects in the area to include in said map database, and when it is determined that images of the area exterior of the vehicle at the vehicle's location are needed to obtain information about objects in the area to include in said map database, said camera being directed by the remote station to obtain images of the area exterior of the vehicle at the vehicle's location.

11. A method for mapping terrain using a vehicle, comprising: obtaining information about objects using one or more devices each comprising an inertial measurement unit (IMU) including at least one accelerometer and at least one gyroscope, a GPS receiver that provides a GPS-derived location, a camera positioned to obtain unobstructed images of an area exterior of the device, and a control system coupled to the IMU, the GPS receiver and the camera; re-calibrating the at least one accelerometer and at least one gyroscope using signals obtained by the GPS receiver; comparing output from the at least one accelerometer with the GPS-derived location and angular orientation of the vehicle and based on the comparison, modifying acceleration output from the at least one accelerometer and modifying angular velocity output from the at least one gyroscope; determining position and angular orientation of the vehicle from the modified acceleration output from the at least one accelerometer and modified angular velocity output from the at least one gyroscope; determining a direction in which the camera is pointing when images are obtained by the camera based on the determined angular orientation of the vehicle; deriving information about objects in the images obtained by the camera and location of the objects based on the determined position of the vehicle and the determined direction in which the camera is pointing when images including the objects are obtained by the camera; communicating the images obtained by the camera or the information about objects in the images obtained by the camera and location of the objects derived by the control system of each device to a location separate and apart from the vehicle using a communications system co-located with the device; and maintaining a map database by adding to the map database information about and location of objects in the images obtained by the camera.

12. The method of claim 11, wherein the map database maintaining step comprises identifying the same object in two or more images obtained from different locations using the processor; and positioning, using the processor, the object in the map database based on the data about the location from which the two or more images were obtained and the pointing direction of the camera on the device when the images were obtained.

13. The method of claim 12, further comprising: identifying objects in images obtained by the camera and their location using the processor; determining whether the identified objects are present in the map database using the processor; and controlling, using the processor, communication of the images obtained by the camera or of information from the device to the location separate and apart from the vehicle using the communications system such that derived information about an object is transmitted to the location separate and apart from the vehicle only when the object is not present in the map database.

14. The method of claim 12, further comprising: correcting, using the processor, data from the IMU using the map database and the camera and without use of data from the GPS receiver by comparing expected position of an object in an image obtained by the camera when the vehicle is at a specific position known from the map database to actual position of the same object in an image obtained by the camera as determined by the processor when the vehicle when at the specific position, and correcting the data from the IMU when the actual vehicle position differs from the expected vehicle position.

15. A method for obtaining position information about a vehicle, comprising: obtaining a plurality of images including a common object using a camera on the vehicle; accessing a map database on the vehicle that contains identification data of objects and their location to obtain an expected position of an object in one of the obtained images based on a known location of the vehicle and a relationship between location of the vehicle when the images were obtained and the object; comparing the expected position of the object in the obtain images to the actual position of the object in the obtained image and correcting the vehicle location when the expected position of the object in the obtain images is different than the actual position of the object in the obtained image; and correcting, using a processor, data from the IMU using the map database and the camera and without use of data from a GPS receiver by comparing the expected position of the object in the images obtained by the camera when the vehicle is at a specific position known from the map database to the actual position of the same object in an image obtained by the camera as determined by the processor when the vehicle when at the specific position, and correcting the data from the IMU when the actual vehicle position differs from the expected vehicle position.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0023] The following drawings are illustrative of embodiments of the system developed or adapted using the teachings of at least one of the inventions disclosed herein and are not meant to limit the scope of the invention as encompassed by the claims.

[0024] FIG. 1 is a diagram showing various vehicle and IMU velocities.

[0025] FIG. 2 is a logic diagram showing the combination of the GPS system, camera and an inertial measurement unit.

BEST MODE FOR CARRYING OUT INVENTION

[0026] An object of the present invention is to provide a mapping system for probe vehicles. To do this, basic engineering solutions for a GPS-corrected IMU angular and displacement location system primarily for mass produced cars are presented. This objective is accomplished by:

[0027] 1) Using a device containing a camera, a GPS or equivalent receiver, communication apparatus, IMU, and an ECU, comprising an electronic control system with the inertial measurement unit (IMU) mounted together at an appropriate location for imaging an area external to the vehicle.

[0028] 2) Manufacturing IMUs in accordance with the mass-production MEMS technology at a cost of a few dollars per unit. An IMU can comprise 3 accelerometers and 3 gyroscopes or more generally, a plurality of accelerometers and/or a plurality of gyroscopes. Sometimes, it also comprises a 3 axis magnetometer.

[0029] 3) Replacing the expensive, currently in-use self-calibration system for the correction of the changing errors in the sensor parameters and applying a procedure for re-calibration of IMU gyroscopes and accelerometers with the help of GPS signals. The cost of a GPS sensor is also a few dollars per unit.

[0030] 4) Additionally, optionally using the zero lateral and vertical speed of the vehicle, as well as speedometer and odometer readings for IMU calibration.

[0031] 5) Additionally, optionally correcting the projections of gravitational acceleration in accelerometer readings, when the vehicle tilts, and centrifugal accelerations during turning maneuvers.

[0032] Preferred specifications for an IMU exemplifying, non-limiting system for use in generating maps using probe vehicles in accordance with one embodiment of the invention are as follows:

[0033] 1) An allowable accelerometer measurement error is assumed to be about 5% or about 0.075 g.

[0034] 2) A maximum longitudinal slope of a highway and city (V<100 km/hour) road is about 5% (2.86 degrees), which corresponds to a horizontal projection of acceleration of about 0.05 g. The cross slope of a road is about 2.5% (1.43 degrees) corresponds to a horizontal projection of acceleration of about 0.025 g.

[0035] 3) Expected values for additional sources of the vehicle tilt with respect to the horizontal plane are as follows: [0036] oscillations are <about 5 degrees; [0037] uneven tire pressure is <about 1 degrees; [0038] non-uniform load is <about 2 degrees; [0039] uneven rigidity of shock absorbers is <about 1 degree; [0040] a manufacturing error in the accelerometer sensitivity axis orientation with respect to the car horizontal axis is <about 1 degrees.

[0041] Considering that road slopes and vehicle tilts have random independent values, the average maximum slope of the accelerometer axis of sensitivity with respect to a horizon plane can be obtained as:

[0042] The projection of the vertical acceleration onto the accelerometer axis of sensitivity due to the longitudinal slope is Ah=g sin 6.4=0.11 g, which is equivalent to the half of the allowable error in the measurement, even without considering the accelerometer errors.

[0043] 4) A speed vector deviation from a longitudinal axis of the vehicle during a turning maneuver (FIG. 1) is of the order of .sub.v =about 1 degree (V=30 MPH, the lateral acceleration is about 0.1 g, the turning radius is about 180 M, the vehicle axle spacing is about 4 M, the sensor is mounted at a distance of about 1 m from the front axle).

[0044] The IMU, GPS receiver, camera, ECU and communications apparatus can be mounted in a single small device similar to an android smartphone and mounted at a convenient location on the probe vehicle where the camera has a clear view of a portion of the environment surrounding the probe vehicle. Preferably this view captures a portion of the road in front of the vehicle and the terrain at least partially to both sides of the vehicle. Preferred mounting locations include on the front grill or fender above a headlight, on the roof either centrally located or on the side or any other location providing a clear view of the road ahead of the vehicle and the terrain on both sides of the vehicle. To accomplish this, the camera can have a horizontal field of view of from about 45 to about 180 degrees although smaller fields of view are also workable.

[0045] FIG. 2 illustrates a self-contained unit, 100, having a housing which can be retrofitted onto a large number of probe vehicles. The GPS satellite constellation is illustrated at 2. Optionally a source of DGPS corrections is illustrated at 4. The self-contained unit, which can appear similar to a smart phone, is illustrated generally at 100 on a probe vehicle 16 and comprises a GPS and optionally DGPS processing system 6, an inertial measurement unit 8, a camera 10, a map database 12, and an ECU 20. Initially, these units 100 can be retrofitted onto a fleet of possibly government-owned vehicles to initiate the map creation process. Afterwards, they can be retrofitted onto any number of public and privately owned vehicles. It is expected that the self-contained unit will be about the size of and cost significantly less than a smartphone.

[0046] When an image is acquired by camera 10, it can be subjected to a coding process and coded data entered into a pattern recognition algorithm such as a neural network in the ECU 20. In one preferred implementation, the pixels of each image from camera 10 are arranged into a vector and the pixels are scanned to locate edges of objects. When an edge is found by processing hardware and/or software in the ECU 20, the value of the data element in the vector which corresponds to the pixel can be set to indicate the angular orientation of the edge in the pixel. For example, a vertical edge can be assigned a 1 and a horizontal element an 8 and those at in between angles assigned numbers between 1 and 8 depending on the angle. If no edge is found, then the pixel data can be assigned a value of 0. When this vector is entered into a properly trained neural network, the network algorithm can output data indicating that a pole, tree, building, or other desired to-be-recognized object has been identified and provide the pixel locations of the object. This can be accomplished with high accuracy providing the neural network has been properly trained with sufficient examples of the objects sought to be identified. Development of the neural network is known to those skilled in the art with the understanding as found by the inventors that a large number of vectors may be needed to make up the training database for the neural network. In some cases, the number of vectors in the training database can approach or exceed one million Only those objects which are clearly recognizable are chosen as fiduciaries.

[0047] Once pixels which represent a pole, for example, have been identified, then one or more vectors can be derived extending from the camera in the direction of the pole based on the location and angle of the camera 10. When the pole is identified in two such images (from the same or different cameras 10) then the intersection of the vectors can be calculated and the pole location in space determined.

[0048] The above described neural network is based on using the edges of objects to form the vectors analyzed by the neural network in the ECU 20. This is only one of a large number of such techniques where observed object properties exhibited in the pixels are used to form the neural network vectors. Others include color, texture, material properties, reflective properties etc. and this invention should not be limited to a particular method of forming the neural network vectors or the pixel properties chosen.

[0049] The neural network can be implemented as an algorithm on a general-purpose microprocessor or on a dedicated parallel processing DSP, neural network ASIC or other dedicated parallel or serial processor, part of the ECU 20 or independent therefrom. The processing speed is generally considerably faster when parallel processors are used. Other optical methods exist for identifying objects using a garnet crystal, for example, to form the Fourier transform of an image. This is discussed in U.S. Pat. No. 7,840,355.

[0050] It is important to note that future GPS and Galileo satellite systems plan for the transmission of multiple frequencies for civilian use. As with a lens, the ionosphere diffracts different frequencies by different amounts (as in a rainbow) and thus the time of arrival of a particular frequency depends on the value of that frequency. This fact can be used to determine the amount that each frequency is diffracted and thus the delay or error introduced by the ionosphere. Thus, with more than one frequency being emitted by a particular satellite, the approximate equivalent of the DGPS corrections can be determined be each receiver and there is no longer a need for DGPS, WADGPS, WAAS, LAAS and similar systems.

[0051] All information regarding the road, both temporary and permanent, should be part of the map database 12, including speed limits, presence and character of guard rails, width of each lane, width of the highway, width of the shoulder, character of the land beyond the roadway, existence of poles or trees and other roadside objects, exactly where chosen fiduciaries are located, the location and content of traffic control signs, the location of variable traffic control devices, etc. The speed limit associated with particular locations on the maps should be coded in such a way that the speed limit can depend upon, for example, factor such as the time of day and/or the weather conditions.

[0052] Speed of travel of a probe vehicle 16 depends to some extent on the accuracy desired for the image and thus on the illumination present and the properties of the imager. In some cases where the roadway is straight, the probe vehicle 16 can travel at moderate speed while obtaining the boundary and lane location information, for example. However, where there are multiple fiduciaries in an image, the rate at which images are acquired and processed may place a limit on the speed of the probe vehicle 16. Of course where there is an intense number of fiduciaries, the images can be stored and processed later.

[0053] In this regard, a display may be provided to the driver of the probe vehicle 16 indicating the maximum speed which is determined based on the number of fiduciaries in the images being obtained by the camera 10 on the probe vehicle 16. If the probe vehicle 16 is autonomous, then its speed may be limited by known control systems the number of fiduciaries in the images being obtained by camera 10. In the same manner, the highest speed of the probe vehicle 16 may be notified to the driver or limited by control systems based on the accuracy desired for the images obtained by the camera 10, i.e., on the illumination present and the properties of the imager, as a sort of feedback technique. Data about the time and accuracy of the processing of images from the camera 10 by the ECU 20 is thus used to control a driver display (not shown) to show the highest speed or to control the autonomous vehicle speed control system.

[0054] Multiple image acquisition systems can be placed on a probe vehicle 16 when it is desired to acquire images for mapping purposes of more than one view from the vehicle 16. One might be placed looking forward and to the right and another looking forward and to the left, for example. This is especially useful for cases when the lanes are separated by a median or for one-way streets. Alternately, a single device with a wide horizontal field of view can suffice. An alternate solution is to place multiple imagers on one device to preserve a large number of pixels but also cover a large field of view. With a large number of probe vehicles, some can be set up to observe the front and right side of the vehicle and others to observe the front and left side. Special distorting lenses can also be designed which permit more efficient use of available pixels by increasing the horizontal field of view at the expense of the vertical.

[0055] Other devices can be incorporated into the system design such as a receiver 4 for obtaining DGPS corrections on the vehicle, although this is not essential for inventive mapping methods since the corrections can be made at the remote map processing station. The inventive mapping system also works without using DGPS corrections, although the convergence to any desired accuracy level requires more images. In one implementation, DGPS corrections can be obtained from an Internet connection which can be through a Wi-Fi connection or through the cell phone system or by communication from a satellite arranged for that purpose. Each camera can also have one or more associated laser pointers, or equivalent, that preferably operates in the near IR portion of the electromagnetic spectrum and in the eyesafe portion of the IR spectrum providing the cameras used are sensitive to this wavelength.

[0056] Laser pointers can also be modulated to permit the distance to the reflective point to be determined. This can be accomplished with pulse modulation, frequency modulation with one or more frequencies, noise or pseudo noise modulation or any other modulation which permits the distance to the point of reflection to be determined. Alternately, a distance can be determined without modulation provided the pointer is not co-located with the imager. In this case, the position on the image of the laser reflection permits the distance to the reflection point to be calculated by triangulation. By using two or more such laser pointers, the angle of a surface can also be estimated.

[0057] As the probe vehicle 16 traverses a roadway, it obtains images of the space around the vehicle and transmits these images, or information derived therefrom, to a remote station 14 off of the vehicle 16, using a transmitter or communications unit 18, which may be separate from or part of a vehicle-mounted communication unit or combined unit 100 (e.g., included in the housing). This communication using communications unit 18 can occur in any of a variety of ways including cellphone, Internet using a broadband connection, LEO or GEO satellites or other telematics communication system or method. The information can also be stored in memory on the vehicle for transmission at a later time when a connection is available. In such a case, the time that the image was acquired and information permitting the remote station 14 to determine the particular satellites is used to locate the vehicle 16.

[0058] Image acquisition by a probe vehicle 16 can be controlled by a remote site (e.g., personnel at the remote station 14) on an as-needed basis. If the remote station 14 determines that more images would be useful, for example if it indicates a change or error in the map, it can send a command to the probe vehicle 16 to upload one or more images. In this manner, the roads can be continuously monitored for changes and the maps kept continuously accurate. Similarly, once the system is largely operational, a probe vehicle 16 can be constantly comparing what it sees using the camera 10 with its copy of the map in map database 12 and when it finds a discrepancy in the presence or location of a fiduciary found in the image from camera 10 relative to the contents of the map database 12, for example, it can notify the control site and together they can determine whether the probe vehicle's map needs updating or whether more images are needed indicating a change in the roadway or its surrounding terrain.

[0059] Such probe equipment can be initially installed on government-owned vehicles or for those that are permitted access to restricted lanes or a special toll discount can be given as an incentive to those vehicle owners that have their vehicles so equipped. Eventually, as the system becomes more ubiquitous, the next phase of the system can be implemented whereby vehicles can accurately determine their location without the use of GPS by comparing the location of fiduciaries as found in their images with the location of the fiduciaries on the map database. Since GPS signals are very weak, they are easily jammed or spoofed, so having an alternative location system becomes important as autonomous vehicles become common.

[0060] Remote station 14 can create and maintain a map database from the information transmitted by the probe vehicles 16. When a section of roadway was first traversed by such a probe vehicle 16, the remote station 14 can request that a large number of images be sent from the probe vehicle 16 depending on the available bandwidth. Additional images can be requested from other probe vehicles until the remote station 14 determines that a sufficient set has been obtained. Once a sufficient number of images have been acquired of a particular fiduciary so that the desired level of position accuracy has been established, then, thereafter additional images can be requested only if an anomaly is detected or occasionally to check that nothing has changed.

[0061] Two images of a particular fiduciary (taken from different locations) are necessary to establish an estimate of the location of the fiduciary. Such an estimate contains errors in, for example, the GPS determination of the location of the device each second for calibration, errors in the IMU determination of its location over and above the GPS errors, errors in the determination of the angle of the fiduciary as determined by the IMU and the camera pixels and errors due to the resolutions of all of these devices. When a third image is available, two additional estimates are available when image 1 is compared with image 3 and image 2 is also compared with image 3. The number of estimates E available can be determined by the formula E=n*(n1)/2, wherein n is the number of images. Thus the number of estimates grows rapidly with the number of images. For example, if 10 images are available, 45 estimates of the position of the fiduciary can be used. Since the number of estimates increases rapidly with the number of images, convergence to any desired accuracy level is rapid. 100 images, for example, can provide almost 5000 such estimates.

[0062] The difference between two GPS location calculations made using the same set of satellites can be used to correct the IMU in the following manner. As long as the same set of satellites are used, the influence of atmospheric distortions are eliminated when calculating the changes in positions, that is, the displacements. The displacements as determined by the GPS should be very accurate and thus can be used to compare with the displacements determined by the IMU through double integration of the accelerations. Similarly, the vector between the two GPS positions can be used to correct the IMU gyroscopes when the angular velocities are integrated once and differenced to get the change in angles which are required to conform to the vector. The angular corrections can be further checked if the IMU contains a magnetometer and the earth's magnetic field is known at the location.

[0063] An important part of this process is in determining the fiduciaries. Generally, a fiduciary, as used herein, is any object which is easily observable, has a largely invariant shape when viewed from different locations and does not move. Light poles, any vertical lines on a manmade structure such as a sign or building, would all be good choices. A rock is easily observable and does not move but it may have a different profile when viewed from nearby points and so it may be difficult to find a point of the rock which is the same in multiple images and so such a rock may not be a good fiduciary. hi some locations, there may not be any natural objects which qualify as fiduciaries such as when driving close to the ocean or a field of wheat. If the road is paved, then the edges of the pavement may qualify so long as there is a visual mark in the pavement which is permanent and observable from several locations. An unpaved road may not have any permanent observables although the smoothed road edge can be used in some cases. To solve these issues, artificial fiduciaries, such as distance markers, may be necessary.

[0064] It is known that if a GPS receiver, receiver F, is placed at a fixed location that, with appropriate software, it can eventually accurately determine its location without the need for a survey.

[0065] Even within less than an hour in a good GPS reception area, the receiver can have an estimate of its location within centimeters. It accomplishes this by taking a multitude of GPS data as the GPS satellites move across the sky and applying appropriate algorithms that are known in the art. Here, this concept is extended to where the GPS readings are acquired by multiple probe vehicles at various times of the day and under varying atmospheric (ionospheric) conditions. These vehicles also can record locations of objects in the infrastructure surrounding each vehicle, the fiduciaries, increasing the completeness and detail of the map database and recording changes in the presence and positions of such objects. For example, as a probe vehicle traverses a roadway, it can determine the location of a lamp pole, for example, on the left side of the vehicle and perhaps another fixed object on the right side of the vehicle, although this is not necessary. It also records the GPS readings taken at the moment that the images of the light pole were taken. The probe vehicle 16 can transmit, using its communications unit 18, to the remote station 14, the vectors to the fiduciaries along with the vehicle position and orientation based on the GPS-corrected IMU. Thus, the remote station 14 can obtain an estimate of the direction of the lamppost from the vehicle 16 and with two such estimates can make an estimation of the location of the lamppost. In a similar manner, therefore, as with the receiver F example, the position estimate improves over time as more and more such data is received from more and more probe vehicles using data taken at different times, from different locations and with different GPS, Galileo and/or Glonass or equivalent GNSS satellites at different locations in the sky and under different ionospheric conditions.

[0066] Distances to objects need not be actually measured since as the vehicle moves and its displacement between images and orientation or pointing direction of each camera 10 when each image is obtained is known, the distances to various objects in the images can be calculated using trigonometry in a manner similar to distances determined from stereo photography. Some of these distance calculations can be made on the probe vehicle 16 to permit anomaly and map error detection locally, whereas detailed calculations are better made (additionally or alternatively) at the remote station 14 which would have greater processing power and data from more image observations of a particular fiduciary. Particular objects in an image can be considered as fiducial points and geo-tagged in the map database to aid the probe vehicles 16 in determining their location and to determine changes or errors in the map database 12. Thus, the location of many items fixed in the infrastructure can be determined and their location accuracy continuously improved as more and more probe data is accumulated.

[0067] Technicians and/or computer programs at the remote station 14 or elsewhere can then begin to construct an accurate map of the entire roadway by determining the location of the road edges and other features and objects that were not actually measured by estimating such coordinates from the images sent by the probe vehicles 16. The probe vehicles 16 can compare their ongoing measurements with the current map database 12, using the geo-tagged fiducial points for example, and when an anomaly is discovered, the remote station 14 can be informed and new images and/or measurements can be uploaded to the remote station 14. Other map features that can be desirable in such a map database 12 such as the character of the shoulder and the ground beyond the shoulder, the existence of drop-offs or cliffs, traffic signs including their text and traffic control devices, etc. can also be manually or automatically added to the database as needed to complete the effort.

[0068] By this method, an accurate map database 12 can be created and continuously verified through the use of probe vehicles 16 and a remote station 14 that creates and updates the map database 12.

[0069] Although several approaches have been discussed above this invention is not limited thereby and other methods should now be apparent to those skilled in the art in view of the disclosure herein. These include the use of a structured light pattern projected onto the infrastructure, usually from a position displaced from the imager position, in addition to or in place of the laser pointers discussed above, among others. If the size and/or position in the image of a reflected pattern vary with distance, then this can provide a method of determining the distance from the probe vehicle to one or more objects or surfaces in the infrastructure through stereographic techniques from multiple images and knowledge of the vehicle's displacements between images and orientation at each image. This is especially useful if the location of the illumination light source is displaced axially, laterally or vertically from the imager. One particularly useful method is to project the structured image so that it has a focal point in front of the imager and thus the image reflected from the infrastructure has a size on the image that varies based on distance from the imager.

[0070] When processing information from multiple images at the remote station 14, data derived from the images is converted to a map including objects from the images by identifying common objects in the images and using the satellite position information from when the images were obtained to place the objects in the map. The images may be obtained from the same probe vehicle 16, taken at different times and including the same, common object, or from two or more probe vehicles 16 and again, including the same, common object. By using a processor at the remote station 14 that is off of the probe vehicles 16, yet in communication with all of the probe vehicles 16 via communication unit 18, images from multiple vehicles or the same vehicle taken at different times may be used to form the map. In addition, by putting the processor off of the probe vehicles 16, it is possible to make DGPS corrections without having equipment to enable such corrections on the probe vehicles 18.

[0071] GPS-based position calculations on a stationary probe vehicle 16 ought to yield the same results as long as the same satellites are used. If a different satellite is used, then a jump in the position can be expected. Thus, GPS can be used to correct the IMU as long as the satellites do not change in the location calculations. When differential corrections are used, they are done on a satellite-by-satellite basis and therefore the vehicle must know which satellites are being used in the calculation. If these corrections are done at the remote station 14 separate and apart from the vehicles, then the vehicles 16 must send that information to the remote station 14 where the differential corrections are known. Alternatively, the differential corrections for all satellites can be sent to the vehicle 16 and the corrections made on the vehicle 16.

[0072] As the vehicle 16 moves, the uncorrected GPS position calculations can be compared to the position calculations made by the IMU 8 after the IMU 8 has been corrected based on consecutive GPS readings providing the same satellites area used. In the same way that a receiver F placed on the ground gradually eliminates the GPS errors, a moving receiver can also be capable of this process and therefore a properly constructed algorithm can result in the vehicle position being determinable with high accuracy even though it is moving. Such a procedure can be as follows:

[0073] 1. Make an initial calculation of the vehicle base position, P0, using uncorrected GPS signals.

[0074] 2. Make a second vehicle position calculation after the vehicle has moved using the same satellites, P1.

[0075] 3. Compare the GPS determined changes with the IMU determined changes and correct the IMU.

[0076] 4. Continue this process as long as the satellites used do not change.

[0077] 5. When the satellites being used change, use the new position calculation Pn based on the new satellites to change the value of Pn by combining by an appropriate method (which would be known or can be determined by one skilled in the art in view of the disclosure herein) the new position Pn with the old position Pn which has been determined from the IMU and Pn-1. Then use this as the new base position.

[0078] 6. By using the proper combining algorithm, the base position of the vehicle should converge to the real position to any degree of accuracy desired.

[0079] By using this process, an accurate map database 12 can automatically be constructed based on accurate vehicle positions and continuously verified without the need for special mapping vehicles containing expensive position determining apparatus.

[0080] Other map information can be incorporated in the map database 12 at the remote station 14 such as the locations, names and descriptions of natural and manmade structures, landmarks, points of interest, commercial enterprises (e.g., gas stations, libraries, restaurants, etc.) along the roadway since their locations can have been recorded by the probe vehicles 16. Once a map database 12 has been constructed using more limited data from a mapping vehicle, for example, additional data can be added using data from probe vehicles 16 that have been designed to obtain different data than the initial probe vehicles 16 have obtained thereby providing a continuous enrichment and improvement of the map database 12. Additionally, the names of streets or roadways, towns, counties, or any other such location based names and other information can be made part of the map. Changes in the roadway location due to construction, landslides, accidents etc. can now be automatically determined by the probe vehicles. These changes can be rapidly incorporated into the map database 12 and transmitted to vehicles on the roadway as map updates. These updates can be transmitted by means of cell phone towers, a ubiquitous Internet or by any other appropriate telematics method.

[0081] The probe vehicles 16 can transmit pictures or images, or data derived therefrom, from vehicle-mounted cameras along with its GPS and IMU derived coordinates. Differential corrections, for example, can be used at the remote station 14 and need not be considered in the probe vehicles 16 thus removing the calculation and telematics load from the probe vehicle 16. See, for example, U.S. Pat. No. 6,243,648 and similar techniques described in the patents assigned to the current assignee. The remote station 14, for example, can know the DGPS corrections for the approximate location of the vehicle at the time that the images or GPS readings were acquired. Over time the remote station 14 would know the exact locations of infrastructure resident features such as the lamppost discussed above in a manner similar to receiver F discussed above.

[0082] In this implementation, the remote station 14 would know the mounting locations of the vehicle-mounted camera(s) 10, the GPS receivers 6 and IMU 8 on the vehicle 16 and their positions and orientations relative to one another, the view angles of the vehicle-mounted cameras 10 and its DGPS corrected position which should be accurate within 10 cm or less, one sigma. By monitoring the movement of the vehicle 16 and the relative positions of objects in successive pictures from a given probe vehicle 16 and from different probe vehicles, an accurate three dimensional representation of the scene can be developed over time even without any laser based actual distance measurements. Of course, to the extent that other information can be made available, the map can be more rapidly improved. Such information can come from other sensors such as laser radar, range gating, radar or other ranging or distance measurement devices or systems. Images from one or more probe vehicles 16 can be combined using appropriate software to help create the three-dimensional representation of the scene.

[0083] Another aspect of this technique is based on the fact that much in the infrastructure is invariant and thus once it is accurately mapped, a vehicle with one or more mounted cameras and/or range determining devices (range meters) can accurately determine its position without the aid of GPS. In the camera case, the vehicle can contain software that can align a recently acquired image with one from the map database and from the alignment process accurately determine its location. For example, the vehicle resident map can tell the vehicle that based on its stated location, it should find a fiduciary imaged on certain pixels of its imager. If instead that fiduciary is found at a slightly different location based on image analysis, then the base position can be corrected. When there are two such discrepancies, then the IMU can be corrected. In this manner, the map can be used to accurately locate the vehicle and one or more images used to correct its base position and its IMU calibration. This can be done at whatever frequency is necessary to maintain the vehicle 16 at a high accuracy state. Such a system eliminates the necessity for GPS and thus protects against a GPS outage or spoofing.

[0084] Map improvements can include the presence and locations of points of interest and commercial establishments providing location-based services. Such commercial locations can pay to have an enhanced representation of their presence along with advertisements and additional information which may be of interest to a driver. This additional information could include the hours of operation, gas price, special promotions etc. Again, the location of the commercial establishment can be obtained from special vehicles which can specialize in identifying commercial establishments or the probe vehicles 16. The commercial establishment can pay to add additional information to the database 12.

[0085] An important part of some embodiments of the invention is the digital map that contains relevant information relating to the road on which the vehicle is traveling. The digital map, which should conform to GIS standards, usually includes the location of the edge of the road, the edge of the shoulder, the elevation and surface shape of the road, the character of the land beyond the road, trees, poles, guard rails, signs, lane markers, speed limits, etc. some of which are discussed elsewhere herein.

[0086] Examples of flow charts, logic diagrams and connections to the various components to the system are described in the above referenced patents and published patent applications and is not reproduced here.

[0087] Map database 12 can be of any desired structure or architecture. Preferred examples of the database structure are of the type discussed in U.S. Pat. No. 6,144,338 (Davies) and U.S. Pat. No. 6,247,019 (Davies).

[0088] Cameras 10 used can be ordinary color still or video cameras, high-speed video cameras, wide angle or telescopic cameras, black and white video cameras, infrared cameras, etc. or combinations thereof. In some cases, special filters are used to accentuate certain features. For example, it has been found that lane markers frequently are more readily observable at particular frequencies, such as infrared. In such cases, filters can be used in front of the camera lens or elsewhere in the optical path to block unwanted frequencies and pass desirable frequencies. Using a camera constructed to be sensitive to infrared in conjunction with general IR illumination can, by itself, improve lane absorbability either with or without special filters. Polarizing lenses have also been found to be useful in many cases. Natural illumination can be used in the mapping process, but for some particular cases, particularly in tunnels, artificial illumination can also be used in the form of a floodlight or spotlight that can be at any appropriate frequency of the ultraviolet, visual and infrared portions of the electromagnetic spectrum or across many frequencies with IR being a preferred illumination, when illumination is desired, especially when the vehicle is operating while the road is in use by others.

[0089] Laser scanners can also be used for some particular cases when it is desirable to illuminate some part of the scene with a bright spot. In some cases, a scanning laser rangemeter can be used in conjunction with the forward-looking cameras to determine the distance to particular objects in the camera view. The scanning laser rangemeter determines distance to a reflection point by time of flight or phase comparisons of a modulated beam between the transmitted and received signals. Range gating can also be used especially in poor visibility conditions to allow an image to be capture of a particular slice in space at a particular distance from the camera. If the camera is outside of the vehicle passenger compartment, the lens can be treated with a coating which repels water or resists adherence of dirt or other contaminants which may obscure the view through the lens as is known to those skilled in the art.

[0090] Finally, not all probe vehicles 16 need be identical since different camera systems highlight different aspects of the environment to be mapped.

[0091] During the map creation it may be desirable to include other information such as the location of all businesses of interest to a traveler such as gas stations, restaurants etc., which could be done on a subscription basis or based on advertising which can yield an additional revenue source for the map providing institution or company.

[0092] Another important aid as part of some of the inventions disclosed herein is to provide markers along the side(s) of roadways which can be either visual, passive or active transponders, reflectors, or a variety of other technologies including objects that are indigenous to or near the roadway, which have the property that as a vehicle passes the marker, it can determine the identity of the marker and from a database, it can determine the exact location of the marker. The term marker is meant in the most general sense. The signature determined by a continuous scan of the environment, for example, would be a marker if it is relatively invariant over time such as, for example, buildings in a city. Basically, there is a lot of invariant information in the environment surrounding a vehicle as it travels down a road toward its destination.

[0093] For the case of specific markers placed on the infrastructure, if three or more of such markers are placed along a side of the roadway, a passing vehicle can determine its exact location by triangulation.

[0094] Although the system is illustrated for use with automobiles, the same system would apply for all vehicles including trucks, trains an even airplanes taxiing on runways. It also can be useful for use with cellular phones and other devices carried by humans.

[0095] While the invention has been illustrated and described in detail in the drawings here and in the referenced related patents and patent applications and the foregoing description, the same is to be considered as illustrative and not restrictive in character, it being understood that only preferred embodiments have been shown and described and that all changes and modifications that come within the spirit of the invention are desired to be protected.

[0096] This application is one in a series of applications covering safety and other systems for vehicles and other uses. The disclosure herein goes beyond that needed to support the claims of the particular invention that is claimed herein. This is not to be construed that the inventors are thereby releasing the unclaimed disclosure and subject matter into the public domain. Rather, it is intended that patent applications have been or will be filed to cover all of the subject matter disclosed above.