Mostly-working LandmarkDetector Python Bindings + Licence Question

[a-hurst]

For an academic research project I'm currently working on (a series of cognitive psych experiments involving face processing + eye tracking), I wrote some basic Python bindings for the LandmarkDetector module of OpenFace using the excellent PyBind11, as well as a Python script (built into the module) that will download all the OpenFace models to either the current working directory in an "openface_models" folder, or globally to a "~/.openface/" folder in the home directory so that the bindings can be run from anywhere on the system. The bindings work well for my purposes (could use some better documentation and type-checking), but since they only cover one of OpenFace's modules I'm not sure whether you'd be interested in a pull request.

Relatedly, a core reason for writing the bindings was that OpenFace doesn't have easily-installed binaries for macOS or Linux, and Python has a great way of distributing easily-installable cross-platform binaries in the form of binary wheels. Since I want my analysis pipelines to be as transparent and easily reproducible as possible (I want something I can share with other researchers in the field, who would largely have no idea how to compile anything), I was hoping I could eventually put up binary wheels for my LandmarkDetector bindings on PyPI or somewhere similar. However, after reading over the licence for the OpenFace project, it's not entirely clear whether that would be allowed or not. Barring a scenario in which I fully write Python bindings for the entirety of OpenFace, and the project officially puts those on PyPI itself, can you give me an idea of what my options are for putting my own LandmarkDetector-based binaries up somewhere within the confines of your licence?

Thanks in advance!

[TadasBaltrusaitis]

Hi,

pybind11 bindings for OpenFace sound like a great idea. Python bindings have long been on my wishlist of OpenFace features. I would definitely be interested in a pull request so that they could be integrated as part of an offical OpenFace repo (and even better on PyPI). I would be very happy to review the PR and possibly work with you on integrating that as part of official OpenFace repo (the risk of having separate repos or separate wheels is that they would not be updated as OpenFace gets updated), so ideally we would want them all in one place.

There will of course be some complications with making them cross-platform and making sure all the dependencies are correctly setup (and also making sure they produce the same outputs as the C++ versions).

Thanks,
Tadas

[a-hurst]

Thanks for the quick response! I have it locally as a separate repository for now, but I'll make a proper pull request over the next few days. It still needs a bit of work in terms of docstrings and type-checking, but overall it seems fairly stable and solid and I'm fairly happy with my approach.

As for multi-platform wheels, I don't know how it'll behave with cmake or whether the project's C++17 requirement is going to be a problem, but I've had good luck with the cibuildwheel project before to build binary wheels for all major python versions and platforms without much hassle. I'd be happy to help set up stuff on that end once the PR is in good shape!

[a-hurst]

@TadasBaltrusaitis just in case you don't get notifications for new pull requests, I reorganized my bindings and submitted a PR along with an overview of the structure/functionality. Let me know what you think!

Once I get confirmation that it's generally in good shape, I'll start putting together some docstrings and unit testing and CI support for it. Thanks in advance!

[TadasBaltrusaitis]

Thanks for the PR, I will try reviewing it shortly.

[a-hurst]

Sounds good! Let me know if you have any issues getting it running on your system: I have Windows 10 and Linux VMs I use for occasional software testing/development, so I can hopefully address any platform-specific issues you encounter. I can't foresee any issues with my bindings running on Linux, but I know Windows is very different in terms of how it handles dynamic linking so it might need some tweaking to work properly.

[D0miH]

@a-hurst Thanks for your great work👍🏼

@TadasBaltrusaitis Any updates on this? This would really come in handy in one of my projects.

[a-hurst]

@D0miH While we're both waiting on feedback from the maintainers, you could try downloading and installing my OpenFace fork and seeing if it works for you! I'd be helpful to me and probably to @TadasBaltrusaitis too to know if my bindings compile and work fine on someone else's system (I've been too lazy/busy to test it on Linux and Windows VMs so far).

As I mentioned in the OP, only the face detection and face landmark detection parts of the API have been properly wrapped, but that might be enough for your needs. Also, since the particularly tricky parts of the Python interface have mostly been tackled already (easy + working-dir-independent model installation, numpy <-> C++ OpenCV conversion, setting up cmake to build binary wheels), adding support for OpenFace's other functions shouldn't be too much of a headache.

[johnbone25]

Is it possible to invoke the head position or eye tracking in Python? And how to do?
Thanks

[sirish-gambhira]

Hello @a-hurst.

Thank you for your effort in building python-bindings. I tried the steps mentioned in #858 and was able to create pyopenface package. However, when I tried importing the package inside python, I face the following error.
from _openface import * ImportError: DLL load failed: The specified module could not be found..

I am using python3.7 to build the package and testing using the same. I am happy to provide any additional information

[a-hurst]

@sirish-gambhira Pretty sure OpenFace has been abandoned by its authors (no updates for 3 years), I wouldn't recommend trying to use it for new projects!

If you're wanting to use it for facial landmark detection, you can try using Google's mediapipe package which has official and well-supported Python bindings and is faster and nicer to work with.

My only problem with it last I used it (~2 years back) is that the numbering for the landmarks wasn't ordered clearly, making it hard to use as a drop-in replacement for the ordered "68 landmark" model that OpenFace and others use. To get around this, I made a map that makes it easy to get OpenFace-style landmarks out of the mediapipe FaceMesh output, see the README and code here: https://github.com/a-hurst/IDFR_Pipeline/tree/main/_Preprocessing

Hope that helps!

EDIT: Here's the file with all the mediapipe points I used to match the 68 OpenFace landmarks. Hopefully it's written clearly enough that it's useful to others! https://github.com/a-hurst/IDFR_Pipeline/blob/main/_Preprocessing/faces/_mediapipe2dlib.py

[sirish-gambhira]

@a-hurst, Thank you for getting back to me.

I agree that OpenFace repository has been abandoned. However, I couldn't find any other repository which gives head-pose, eye-gaze, facial action units as outputs. I would like to know if you are aware of any such reliable alternatives.

[a-hurst]

@sirish-gambhira Mediapipe might be able to do head-pose (it makes a 3D mesh overlay of the face so it must track head angle/position to some extent), but no clue about eye gaze or facial action units. I only wrote Python bindings for OpenFace's facial landmark code (all I needed for my project), so the other things you've mentioned would still need new bindings written for them. I do remember testing out the eye gaze stuff a bit using one of the built-in OpenFace demo apps and seemed a little glitchy, at least not on the medium quality webcam video I tried it on.

Anyway, if you need those things for your project and are interested in finishing the bindings for the action units and eye gaze stuff yourself, my guess regarding that error is that might be a platform issue: I wrote and tested the bindings on macOS and have also built/tested them successfully on Linux, but I've never tried it on Windows before so there may be some tweaks needed to my scripts or CMake in order for that to work properly. If you are on Windows, maybe try building/running it in WSL?

[brmarkus]

In Driver-Monitoring (e.g. automotive) OpenVINO is used successfully (head-pose, eye-tracking, gaze, face recognition&(re-)identification, emotions) - transparent use of available, underlying accelerators and with different bindings.
However, haven't seen AUs with OpenVINO applications.