Parallax effect when using depth camera and virtual objects

erictruitt
Contributor

Parallax effect when using depth camera and virtual objects

Hi all! 

 

I'm creating a simulation in which the user is inside a vehicle and is able to see certain critical components of the vehicle, otherwise he is looking at the depth camera dispay to be able to interact with real world objects.

 

The issue I'm having is as I'm looking around inside the vehicle, I'm experiencing a parallax effect where the depth camera seems to slide around behind the rendered objects rather than with them.

 

Does anyone have any advice for how to mitigate this effect?

 

Vive Staff

Re: Parallax effect when using depth camera and virtual objects

Hi erictruitt,

does you mean that the distance between you and objects in real world seems closer than your thought?

erictruitt
Contributor

Re: Parallax effect when using depth camera and virtual objects

No, I'm talking about the shifting position visible between objects spawned in the scene and the pass through video feed that is displayed. For example, if you open the demo app, place a ball on the ground and walk around it with pass through video enabled, the position of the ball will shift around relative to real world objects even though the ball has no velocity.

 

I've been able to sort of make it better by trying to use a real world object as a sort of "anchor" and making sure my virtual objects don't shift much relative to that object. There may be no better fix for this at the moment, though I'm hoping someone out there has a better solution.

areong
Denizen

Re: Parallax effect when using depth camera and virtual objects

Hi erictruitt,

 

We have also observed the phenomenon you mentioned. We are figuring out if the shifting problem can be fixed. If you have any suggestion please feel free to post here. Thanks!

Hanno
Settler

Re: Parallax effect when using depth camera and virtual objects

Hi erictruitt,

 

I'm no expert but I've been thinking about doing something similar, so maybe this can help.

 

I imagine this problem would happen when the latencies from the video feed and the rendered images are not the same. The video probably has more latency and, therefore, it "drags behind" the rendered image.

 

Your solution of "anchoring" the virtual object sounds like a good idea to mitigate the effect but it seems like the only real solution would be to match the latencies between the video feed and rendered image.

Assuming the video feed latency is higher, that leaves 2 options:

1) Increase rendering latency to match video feed

2) Reduce video feed latency to match rendering

 

Increasing rendering latency should be easy enough but it is obviously bad for overall immersion. 

Reducing video feed latency might be possible if you "shift" the image based on the latest position/orientation of headset just before displaying it on the headset. Something similar to the reprojection happening for the rendererd image already. Turning or moving your head quickly will then cause some visual artifacts, though. 

 

Regards,

Hanno

pablocael
Explorer

Re: Parallax effect when using depth camera and virtual objects

hi There, this shifting problem is preventing us from using Vive Pro trackers to match virtual and real objects. If any progress is made on this front please let us know! Its very frustrating to not be able to use the vive pro headset because of this problem. It is very strong shifiting and virtual and real object get totally unmatched.

 

Thanks!

pablocael
Explorer

Re: Parallax effect when using depth camera and virtual objects

After analysis very shortly, it seems it might be a update problem. It seems the camera position is updated in the C++ driver, and the unity object is updated later, so the plane that renders the image background moves with some delay in relation to the tracked camera within the unit.

 

DaKenpachi
Pioneer

Re: Parallax effect when using depth camera and virtual objects

Yeah, that´s one of the typical errors in AR, called dynamic registration error. I don´t know if there is a way to synchronize image rendering with hmd pose updates....

But @pablocael do you also have static errors? Like when the virtual object has a wrong position in general?

Becaus this is what I get, marking the middle of my room both in the virtual and real world and comparing them:

Registration_Test_1.JPGRegistration_Test_4.JPGRegistration_Test_5.JPGRegistration_Test_2.JPG

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

seems to me like a few cm of pose error...