Mono Cam Free Space

…where I estimate free space using a single camera.

Goals

Get a feeling for depth data resulting from a depth estimation DNN model. To be useful, I chose a model that is small enough to run at several frames per second and so that it can be shipped as part of a mobile app.

The Story

I am testing this on an iPhone as my test platform but assume this is a robotic use case to avoid obstacles, or maybe a navigation app for blind people that helps prevent them from running into objects.

A depth estimation DNN provides depth information from the mono camera video stream. From the depth data I am leveraging a few OpenCV functions to compute the free space area. With some projective-geometric reasoning, I can then compute a free space map to track and accumulate over time.

Takeaways

• Depending on your use case, a single camera may already give you all information required for navigation avoiding obstacles at sight’s distance!
• The resulting free space map looks quite usable on first inspection, results are relatively stable.
• I can even integrate overhanging objects by trading in some precision.
• On my iphone 15, I am getting depth estimates at more than 10fps, which I find more than usable!
• There is a lot of open source resources to get such an experiment up and running – be it for your own learning or to point your favourite agent to it.

Examples

Here are a few screenshots with the detected free space per frame overlaid on the bottom left. In the Bottom right, a map is accumulated over time from single frame results: ego position is in the center with camera pointing upwards, black is free space, white is potential obstacle, and green are potential walking directions. In the third image, you can see that the free space detection performs reasonably well with overhanging obstacles.