A holographic calling system can capture and encode holographic data at a sender-side of a holographic calling pipeline and decode and present the holographic data as a 3D representation of a sender at a receiver-side of the holographic calling pipeline. The holographic calling pipeline can include stages to capture audio, color images, and depth images; densify the depth images to have a depth value for each pixel while generating parts masks and a body model; use the masks to segment the images into parts needed for hologram generation; convert depth images into a 3D mesh; paint the 3D mesh with color data; perform torso disocclusion; perform face reconstruction; and perform audio synchronization. In various implementations, different of these stages can be performed sender-side or receiver side. The holographic calling pipeline also includes sender-side compression, transmission over a communication channel, and receiver-side decompression and hologram output.

In one embodiment, a computing system may access a first grayscale image and a second grayscale image. The system may generate a first color image and a second color image based on the first grayscale image and the second grayscale image, respectively. The system may generate affinity information based on the first grayscale image and the second grayscale image, the affinity information identifying relationships between pixels of the first grayscale image and pixels of the second grayscale image. The system may modify the color of the first color image and the second color image based on the affinity information. The system may generate a first visual output based on the modified first color image and a second visual output based on the modified second color image.

A stereo camera array system includes: a first camera array at a first viewpoint including: a first camera configured to capture images in a first modality, the first modality being viewpoint-independent; and a second camera configured to capture images in a second modality different from the first modality; and a second camera array at a second viewpoint spaced apart along a first baseline from the first camera array at the first viewpoint, the second camera array including: a first camera configured to capture images in the first modality; and a second camera configured to capture images in the second modality.

A head-mounted device includes a plurality of front camera modules, a display device, an in-display camera module and an image-processing unit. The front camera modules include a first camera module and a second camera module that are respectively disposed at the left and right sides of the head-mounted device. The in-display camera module is disposed within the boundary of the display device. The image-processing unit is in communication connection with the front camera modules and the display device. The display device and the in-display camera module face a user side of the head-mounted device.

A head-mounted device includes a plurality of front camera modules, a display device, an in-display camera module and an image-processing unit. The front camera modules include a first camera module and a second camera module that are respectively disposed at the left and right sides of the head-mounted device. The in-display camera module is disposed within the boundary of the display device. The image-processing unit is in communication connection with the front camera modules and the display device. The display device and the in-display camera module face a user side of the head-mounted device.

Using the same image sensor to capture a two-dimensional (2D) image and three-dimensional (3D) depth measurements for a 3D object. A laser point-scans the surface of the object with light spots, which are detected by a pixel array in the image sensor to generate the 3D depth profile of the object using triangulation. Each row of pixels in the pixel array forms an epipolar line of the corresponding laser scan line. Timestamping provides a correspondence between the pixel location of a captured light spot and the respective scan angle of the laser to remove any ambiguity in triangulation. An Analog-to-Digital Converter (ADC) in the image sensor operates as a Time-to-Digital (TDC) converter to generate timestamps. A timestamp calibration circuit is provided on-board to record the propagation delay of each column of pixels in the pixel array and to provide necessary corrections to the timestamp values generated during 3D depth measurements.

Systems and method for correcting images including artifacts due to dirty camera lenses of electronic device are disclosed. Correction of images by the systems and methods includes obtaining a first raw pixel image of a scene captured with a first camera, obtaining a second raw image of the scene captured with a second camera separate from the first camera in a camera baseline direction, rectifying the first and second raw pixel images to create respective first and second rectified pixel images, determining disparity correspondence between corresponding image pixel pairs of the first and second rectified images in the camera baseline direction, mapping first and second rectified images into the same domain using the determined disparity, detect image artifact regions within each domain mapped image by comparing corresponding regions of the domain mapped images, determining correction factors for each detected image artifact region, and correcting the rectified first and second images by applying the determined correction factors.

Systems having rolling shutter sensors with a plurality of sensor rows are configured for compensating for rolling shutter artifacts that result from different sensor rows in the plurality of sensor rows outputting sensor data at different times. The systems compensate for the rolling shutter artifacts by identifying readout timepoints for the plurality of sensor rows of the rolling shutter sensor while the rolling shutter sensor captures an image of an environment and identifying readout poses each readout timepoint, as well as obtaining a depth map based on the image. The depth map includes a plurality of different rows of depth data that correspond to the different sensor rows. The system further compensates for the rolling shutter artifacts by generating a 3D representation of the environment while unprojecting the rows of depth data into 3D space using the readout poses.

A camera tracking system for computer assisted navigation during surgery operatively determines a first pose of a second extended-reality (XR) headset relative to stereo tracking cameras located on a first XR headset based on first tracking information from the stereo tracking cameras. The camera tracking system determines a second pose of eyes of a user wearing the second XR headset relative to the stereo tracking cameras located on the first XR headset based on second tracking information from the stereo tracking cameras. The camera tracking system also calibrates an eye-to-display relationship defining pose of the eyes of the user wearing the second XR headset to a display device of the second XR headset based on the determined first and second poses. The camera tracking system also controls where symbols are displayed on the display device of the second XR headset based on the eye-to-display relationship.

A method and an apparatus are provided for providing a dolly zoom effect by an electronic device. A first image with a first depth map and a second image with a second depth map are obtained. A first synthesized image and a corresponding first synthesized depth map are generated using the first image and the first depth map respectively. A second synthesized image and a corresponding second synthesized depth map are generated using the second image and the second depth map respectively. A fused image is generated from the first synthesized image and the second synthesized image. A fused depth map is generated from the first synthesized depth map and the second synthesized depth map. A final synthesized image is generated based on processing the fused image and the fused depth map.