Vowpalwabbit python wrappers do not build or crashes on OSX with Anaconda

[peterychang]

There is a known issue on OSX 10.14+ with the currently deployed version of python on Anaconda.

The problem is as follows:

• Python requires linking binaries (eg: VW) to also link against the same c++ library it did.
• Anaconda builds python against an older version of OSX and links against libstdc++
• libstdc++ was deprecated by Apple/Clang several years ago. And as of OSX 10.14, the library was removed entirely.

As a result, VW will typically fail to compile or install on the latest version of OSX. Using compiler options to force the use of libc++ will cause python to crash.

At the moment, the only resolutions available to users are to either uninstall Anaconda and use Homebrew or to copy the missing libraries from an older version of xcode.

We are currently exploring ways to resolve these compatibility issues, but there does not appear to be a quick or easy solution to the problem.

If this is a critical issue for you, please feel free to reach out to us so we can see if there is an individual solution that may work.

We also welcome anybody who would like to contribute to the resolution of this issue.

[peterychang]

For users of Homebrew having trouble installing VW from pip:

clang: error: no input files and boost-cmake errors have been resolved as of #2097
These fixes will be available via pip from our next release.
For anybody unable to wait for the next release, please install the package from source.

[scnullois]

Having same issue with pip on OSX. When is the fixed version going to be available on pip?

    [100%] Built target pylibvw
    building 'pylibvw' extension
    clang -bundle -undefined dynamic_lookup -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/openssl/lib -L/Users/scgalois/.pyenv/versions/3.7.0/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/openssl/lib -L/Users/scgalois/.pyenv/versions/3.7.0/lib -L/usr/local/opt/openssl/lib -o build/lib.macosx-10.14-x86_64-3.7/pylibvw.so
    clang: error: no input files
    error: command 'clang' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/scgalois/.pyenv/versions/3.7.0/envs/gc_all/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zx/r2fggd2x6x36_vbyqv6jxp9c0000gn/T/pip-install-5fn40s7t/vowpalwabbit/setup.py'"'"'; __file__='"'"'/private/var/folders/zx/r2fggd2x6x36_vbyqv6jxp9c0000gn/T/pip-install-5fn40s7t/vowpalwabbit/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/zx/r2fggd2x6x36_vbyqv6jxp9c0000gn/T/pip-record-z_peocph/install-record.txt --single-version-externally-managed --compile --install-headers /Users/scgalois/.pyenv/versions/3.7.0/envs/gc_all/include/site/python3.7/vowpalwabbit Check the logs for full command output.```

[scnullois]

Also if it will be more than a few days, are there instructions for installing package from source please?

I tried to clone the repo and then ran the instructions from the issue with the fix but I still get following error:

(gc_all) LOCAL ➜  vowpal_wabbit git:(master) pip install --global-option="--enable-boost-cmake" --global-option="--cmake-options=--verbose;-DCMAKE_OPTION=ON" vowpalwabbit
/Users/scgalois/.pyenv/versions/3.7.0/envs/gc_all/lib/python3.7/site-packages/pip/_internal/commands/install.py:283: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Collecting vowpalwabbit
  Using cached https://files.pythonhosted.org/packages/f4/09/50efc9630809fdd19056d2fff8e7b8e374063002b5a70b016854fe5e8492/vowpalwabbit-8.7.0.post1.tar.gz
Installing collected packages: vowpalwabbit
    Running setup.py install for vowpalwabbit ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/scgalois/.pyenv/versions/3.7.0/envs/gc_all/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zx/r2fggd2x6x36_vbyqv6jxp9c0000gn/T/pip-install-m2l1e7fa/vowpalwabbit/setup.py'"'"'; __file__='"'"'/private/var/folders/zx/r2fggd2x6x36_vbyqv6jxp9c0000gn/T/pip-install-m2l1e7fa/vowpalwabbit/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --enable-boost-cmake '--cmake-options=--verbose;-DCMAKE_OPTION=ON' install --record /private/var/folders/zx/r2fggd2x6x36_vbyqv6jxp9c0000gn/T/pip-record-v7agrd53/install-record.txt --single-version-externally-managed --compile --install-headers /Users/scgalois/.pyenv/versions/3.7.0/envs/gc_all/include/site/python3.7/vowpalwabbit
         cwd: /private/var/folders/zx/r2fggd2x6x36_vbyqv6jxp9c0000gn/T/pip-install-m2l1e7fa/vowpalwabbit/
    Complete output (6 lines):
    usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: setup.py --help [cmd1 cmd2 ...]
       or: setup.py --help-commands
       or: setup.py cmd --help

    error: option --enable-boost-cmake not recognized
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/scgalois/.pyenv/versions/3.7.0/envs/gc_all/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zx/r2fggd2x6x36_vbyqv6jxp9c0000gn/T/pip-install-m2l1e7fa/vowpalwabbit/setup.py'"'"'; __file__='"'"'/private/var/folders/zx/r2fggd2x6x36_vbyqv6jxp9c0000gn/T/pip-install-m2l1e7fa/vowpalwabbit/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --enable-boost-cmake '--cmake-options=--verbose;-DCMAKE_OPTION=ON' install --record /private/var/folders/zx/r2fggd2x6x36_vbyqv6jxp9c0000gn/T/pip-record-v7agrd53/install-record.txt --single-version-externally-managed --compile --install-headers /Users/scgalois/.pyenv/versions/3.7.0/envs/gc_all/include/site/python3.7/vowpalwabbit Check the logs for full command output.
(gc_all) LOCAL ➜  vowpal_wabbit git:(master) git pull
Already up to date.
Current branch master is up to date.
(gc_all) LOCAL ➜  vowpal_wabbit git:(master) python setup.py --enable-boost-cmake --cmake-options=--verbose;-DCMAKE_OPTION=ON
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: no commands supplied
zsh: command not found: -DCMAKE_OPTION=ON

[scnullois]

Ah managed to get it to work by just running python setup.py install from the cloned dir.

[scnullois]

However, now both examples cause segementation fault:

(gc_all) LOCAL ➜  vowpal_wabbit git:(master) ✗ python3
Python 3.7.0 (default, Apr 15 2019, 13:22:51)
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from vowpalwabbit import pyvw
>>> vw = pyvw.vw(quiet=False)
Num weight bits = 18
learning rate = 0.5
initial_t = 0
power_t = 0.5
using no cache
Reading datafile =
num sources = 0
>>> ex = vw.example('1 | a b c')
[1]    83919 segmentation fault  python3

(gc_all) LOCAL ➜  vowpal_wabbit git:(master) ✗ python3
Python 3.7.0 (default, Apr 15 2019, 13:22:51)
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> from sklearn import datasets
>>> from sklearn.model_selection import train_test_split
>>> from vowpalwabbit.sklearn_vw import VWClassifier
/Users/scgalois/.pyenv/versions/gc_all/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=DeprecationWarning)
>>>
>>> # generate some data
... X, y = datasets.make_hastie_10_2(n_samples=10000, random_state=1)
>>> X = X.astype(np.float32)
>>>
>>> # split train and test set
... X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=256)
>>>
>>> # build model
... model = VWClassifier()
>>> model.fit(X_train, y_train)
[1]    84429 segmentation fault  python3

[peterychang]

@scgalois I was unable to reproduce your segfault on my setup. Can I get some information from you?

• Which version of OSX are you running on?
• Where did you get python from?
• Version of boost/boost-python3
• Do you have Anaconda installed?

Thanks

[scnullois]

@peterychang thanks for following up. Here are the details:

OSX Mojave 10.14.6 with no anaconda installation (when I type conda in command line says command not found)

(gc_all) LOCAL ➜  vowpal_wabbit git:(master) ✗ which python3
/Users/scgalois/.pyenv/shims/python3
(gc_all) LOCAL ➜  vowpal_wabbit git:(master) ✗ python3 --version
Python 3.7.0

(gc_all) LOCAL ➜  vowpal_wabbit git:(master) ✗ brew search boost
brew==> Formulae
boost ✔                              boost-build                          boost-python ✔                       boost-python@1.59                    boost@1.57                           boost@1.60
boost-bcp                            boost-mpi                            boost-python3 ✔                      boost@1.55                           boost@1.59                           xgboost

==> Casks
boostnote                                    focus-booster                                nosqlbooster-for-mongodb                     soundboosterlite                             turbo-boost-switcher
(gc_all) LOCAL ➜  vowpal_wabbit git:(master) ✗ brew info boost boost-python boost-python3
boost: stable 1.71.0 (bottled), HEAD
Collection of portable C++ source libraries
https://www.boost.org/
/usr/local/Cellar/boost/1.71.0 (14,377 files, 608.3MB) *
  Poured from bottle on 2019-10-22 at 13:39:44
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/boost.rb
==> Dependencies
Required: icu4c ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 41,806 (30 days), 127,389 (90 days), 650,201 (365 days)
install_on_request: 14,825 (30 days), 45,118 (90 days), 210,074 (365 days)
build_error: 0 (30 days)

boost-python: stable 1.71.0 (bottled), HEAD
C++ library for C++/Python2 interoperability
https://www.boost.org/
/usr/local/Cellar/boost-python/1.71.0_1 (460 files, 5.4MB) *
  Poured from bottle on 2019-10-22 at 13:44:48
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/boost-python.rb
==> Dependencies
Required: boost ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
This formula provides Boost.Python for Python 2. Due to a
collision with boost-python3, the CMake Config files are not
available. Please use -DBoost_NO_BOOST_CMAKE=ON when building
with CMake or switch to Python 3.
==> Analytics
install: 5,433 (30 days), 12,561 (90 days), 55,587 (365 days)
install_on_request: 3,632 (30 days), 8,411 (90 days), 38,916 (365 days)
build_error: 0 (30 days)

boost-python3: stable 1.71.0 (bottled), HEAD
C++ library for C++/Python3 interoperability
https://www.boost.org/
/usr/local/Cellar/boost-python3/1.71.0_1 (472 files, 19.9MB) *
  Poured from bottle on 2019-10-22 at 13:45:24
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/boost-python3.rb
==> Dependencies
Build: numpy ✘
Required: boost ✔, python ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 2,778 (30 days), 6,485 (90 days), 21,634 (365 days)
install_on_request: 1,575 (30 days), 3,531 (90 days), 12,426 (365 days)
build_error: 0 (30 days)

[peterychang]

@scgalois From your logs, it doesn't look like you're using brew's python.
Specifically, python on brew is 3.7.4_1 compiled against Clang 6.0 (compared to your python 3.7.0 compiled against Clang 10.0.1)

Since your version of python differs from whats on homebrew, its possible the boost python library from homebrew is not binary compatible with the version of python you're running.

If this is the cause, then there are a couple of possible solutions. First, you can switch to python3 from homebrew.
The second option is compiling boost against your current environment and recompiling vowpalwabbit against those libraries

[scnullois]

@peterychang Thanks for the feedback. assuming I clone and compile boost myself, how do I make sure the vw package will link my new version rather than the boost version? Are there any detailed instructions anywhere or do I need to dig into the config scripts to figure it out?

[scnullois]

@peterychang Actually I ended up just changing my version of python and it is working fine now. Thanks for your help. But just for future reference, if there are instructions on building and using older version of boost against pyvw would appreciate it.

[peterychang]

@scgalois If you have a build of boost you want to link against, the following command should force VW to link against it. In this case, you'll want to pay attention to the build logs since I've noticed that the BOOST_ROOT variable is ignored in some cases that I'm still investigating.

python3 setup.py --cmake-options=BOOST_ROOT=/path/to/boost install -f
or via pip (after our next python release) with
pip3 install --global-option="--cmake-options=BOOST_ROOT=/path/to/boost" vowpalwabbit