Could not build wheels for pygeodiff - `pip3 install mergin-client` fails

[NewGraphEnvironment]

First off - banger tools. Game changing. Incredibly useful. You should be proud. Thank you.

Was hoping to install the command line tools but have run into a bit of a snag. Attempting to set up conda env using environment.yml file. Also tried with a straight up pip3 install mergin-client with same errors. I have attached the environment.yml file as a .txt file

System is:

System Version: macOS 12.4 (21F79)
Kernel Version: Darwin 21.5.0
Chip: Apple M1 Max

Seems to be an issue related to cmake and pygeodiff. I tried rolling back the python versions to 3.10 and 3.9). Rolling back mergin-client to 2.0.0 and <2.0.0. Installing pygeodiff and mergin-client separately with just straight pip. Although I actually installed it, I did not go so far as to add ninja to my path as I figured that would have been specified as a requirement in the readme. Anyway, here is the reoccurring error:

..........
Failed to build pygeodiff
\ 
Pip subprocess error:
  error: subprocess-exited-with-error
  
  × Building wheel for pygeodiff (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [161 lines of output]
      
      
      --------------------------------------------------------------------------------
      -- Trying 'Ninja' generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      Not searching for unused variables given on the command line.
      CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-install-6f9b5ket/pygeodiff_993321479ce24f789896c0f1effce627/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log".
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying 'Ninja' generator - failure
      --------------------------------------------------------------------------------
      
      
      
      --------------------------------------------------------------------------------
      -- Trying 'Unix Makefiles' generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      Not searching for unused variables given on the command line.
      -- The C compiler identification is AppleClang 13.1.6.13160021
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- The CXX compiler identification is AppleClang 13.1.6.13160021
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Configuring done
      -- Generating done
      -- Build files have been written to: /private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-install-6f9b5ket/pygeodiff_993321479ce24f789896c0f1effce627/_cmake_test_compile/build
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying 'Unix Makefiles' generator - success
      --------------------------------------------------------------------------------
      
      Configuring Project
        Working directory:
          /private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-install-6f9b5ket/pygeodiff_993321479ce24f789896c0f1effce627/_skbuild/macosx-12.0-arm64-3.11/cmake-build
        Command:
          /private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-build-env-oksyersm/overlay/lib/python3.11/site-packages/cmake/data/bin/cmake /private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-install-6f9b5ket/pygeodiff_993321479ce24f789896c0f1effce627/geodiff -G 'Unix Makefiles' -DCMAKE_INSTALL_PREFIX:PATH=/private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-install-6f9b5ket/pygeodiff_993321479ce24f789896c0f1effce627/_skbuild/macosx-12.0-arm64-3.11/cmake-install -DPYTHON_VERSION_STRING:STRING=3.11.0 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-build-env-oksyersm/overlay/lib/python3.11/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/opt/homebrew/Caskroom/miniconda/base/envs/dff/bin/python -DPYTHON_INCLUDE_DIR:PATH=/opt/homebrew/Caskroom/miniconda/base/envs/dff/include/python3.11 -DPYTHON_LIBRARY:PATH=/opt/homebrew/Caskroom/miniconda/base/envs/dff/lib/libpython3.11.dylib -DPython_EXECUTABLE:PATH=/opt/homebrew/Caskroom/miniconda/base/envs/dff/bin/python -DPython_ROOT_DIR:PATH=/opt/homebrew/Caskroom/miniconda/base/envs/dff -DPython_INCLUDE_DIR:PATH=/opt/homebrew/Caskroom/miniconda/base/envs/dff/include/python3.11 -DPython_FIND_REGISTRY:STRING=NEVER -DPython3_EXECUTABLE:PATH=/opt/homebrew/Caskroom/miniconda/base/envs/dff/bin/python -DPython3_ROOT_DIR:PATH=/opt/homebrew/Caskroom/miniconda/base/envs/dff -DPython3_INCLUDE_DIR:PATH=/opt/homebrew/Caskroom/miniconda/base/envs/dff/include/python3.11 -DPython3_FIND_REGISTRY:STRING=NEVER -DENABLE_TESTS:BOOL=OFF -DENABLE_COVERAGE:BOOL=OFF -DBUILD_TOOLS:BOOL=OFF -DPEDANTIC:BOOL=OFF -DPYGEODIFFVERSION=2.0.1 -DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=12.0 -DCMAKE_OSX_ARCHITECTURES:STRING=arm64
      
      -- The C compiler identification is AppleClang 13.1.6.13160021
      -- The CXX compiler identification is AppleClang 13.1.6.13160021
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- The geodiff is built using scikit-build for pygeodiff Python package
      -- Found PythonInterp: /opt/homebrew/Caskroom/miniconda/base/envs/dff/bin/python (found version "3.11")
      -- Found PythonLibs: /opt/homebrew/Caskroom/miniconda/base/envs/dff/lib/libpython3.11.dylib (found version "3.11.0")
      -- Found SQLite3: /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include (found version "3.37.0")
      -- SQLite3 version: 3.37.0
      -- SQLite3 include dirs: /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include
      -- SQLite3 library: /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/lib/libsqlite3.tbd
      -- Determining available thread-local storage mechanism
      -- Checking if __thread is supported
      -- Performing Test TLS_USE_THREAD
      -- Performing Test TLS_USE_THREAD - Success
      -- Checking if __thread is supported - yes
      -- Determining available thread-local storage mechanism - Done
      -- Determining available locale mechanism
      -- Checking if _create_locale is supported
      -- Looking for locale.h
      -- Looking for locale.h - found
      -- Performing Test LOCALE_USE__CREATE_LOCALE
      -- Performing Test LOCALE_USE__CREATE_LOCALE - Failed
      -- Checking if _create_locale is supported - no
      -- Checking if newlocale is supported
      -- Looking for xlocale.h
      -- Looking for xlocale.h - found
      -- Performing Test LOCALE_USE_NEWLOCALE
      -- Performing Test LOCALE_USE_NEWLOCALE - Success
      -- Checking if newlocale is supported - yes
      -- Determining available locale mechanism - Done
      -- Configuring done
      -- Generating done
      CMake Warning:
        Manually-specified variables were not used by the project:
      
          Python3_EXECUTABLE
          Python3_FIND_REGISTRY
          Python3_INCLUDE_DIR
          Python3_ROOT_DIR
          Python_EXECUTABLE
          Python_FIND_REGISTRY
          Python_INCLUDE_DIR
          Python_ROOT_DIR
      
      
      -- Build files have been written to: /private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-install-6f9b5ket/pygeodiff_993321479ce24f789896c0f1effce627/_skbuild/macosx-12.0-arm64-3.11/cmake-build
      [  3%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/geodiff.cpp.o
      [  6%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/geodiffutils.cpp.o
      [  9%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/geodiffrebase.cpp.o
      [ 12%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/geodifflogger.cpp.o
      [ 16%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/geodiffcontext.cpp.o
      [ 19%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/changesetconcat.cpp.o
      [ 22%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/changesetreader.cpp.o
      [ 25%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/changesetutils.cpp.o
      [ 29%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/changesetwriter.cpp.o
      [ 32%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/driver.cpp.o
      [ 35%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/tableschema.cpp.o
      [ 38%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/drivers/sqlitedriver.cpp.o
      [ 41%] Building CXX object CMakeFiles/pygeodiff-2.0.1-python.dir/src/drivers/sqliteutils.cpp.o
      /private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-install-6f9b5ket/pygeodiff_993321479ce24f789896c0f1effce627/geodiff/src/drivers/sqliteutils.cpp:244:12: error: use of undeclared identifier 'sqlite3_enable_load_extension'
        int rc = sqlite3_enable_load_extension( db->get(), 1 );
                 ^
      1 error generated.
      make[2]: *** [CMakeFiles/pygeodiff-2.0.1-python.dir/src/drivers/sqliteutils.cpp.o] Error 1
      make[1]: *** [CMakeFiles/pygeodiff-2.0.1-python.dir/all] Error 2
      make: *** [all] Error 2
      Traceback (most recent call last):
        File "/private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-build-env-oksyersm/overlay/lib/python3.11/site-packages/skbuild/setuptools_wrap.py", line 649, in setup
          cmkr.make(make_args, install_target=cmake_install_target, env=env)
        File "/private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-build-env-oksyersm/overlay/lib/python3.11/site-packages/skbuild/cmaker.py", line 686, in make
          self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
        File "/private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-build-env-oksyersm/overlay/lib/python3.11/site-packages/skbuild/cmaker.py", line 717, in make_impl
          raise SKBuildError(
      
      An error occurred while building with CMake.
        Command:
          /private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-build-env-oksyersm/overlay/lib/python3.11/site-packages/cmake/data/bin/cmake --build . --target install --config Release --
        Install target:
          install
        Source directory:
          /private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-install-6f9b5ket/pygeodiff_993321479ce24f789896c0f1effce627
[environment.txt](https://github.com/MerginMaps/mergin-py-client/files/10694512/environment.txt)

        Working directory:
          /private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/pip-install-6f9b5ket/pygeodiff_993321479ce24f789896c0f1effce627/_skbuild/macosx-12.0-arm64-3.11/cmake-build
      Please check the install target is valid and see CMake's output for more information.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pygeodiff
ERROR: Could not build wheels for pygeodiff, which is required to install pyproject.toml-based projects
| 
failed

[NewGraphEnvironment]

thought I attached the environement.yml file but don't see it so here it is again

environment.txt

[wonder-sk]

So the error is that sqlite3_enable_load_extension function is not available - it looks like it is picking sqlite3 from macOS, which does not include it. I am not very familiar with macOS, the correct way would be that the build would pick sqlite3 from conda (or homebrew?).

Not sure if this is something that can be overridden somehow during the install, or needs to be adjusted in the package... maybe @PeterPetrik would know?

[NewGraphEnvironment]

tried adding sqlite to environment.yml file with no joy. What seems odd is that running pip3 install mergin-client from within a fresh environment gives the same result (see below for which python and which sqlite info. Is there something hard coded in the build about where it looks for sqlite perhaps? sorry for all the hassles.

It is obviously not using the sqlite specified in the conda env.

This is within the conda env

(dff) 
airvine at Allans-MacBook-Pro in ~/Projects/repo/dff-2022 on master [!]
$ python
Python 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:26:40) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.40.0'
>>> 



(dff) 
airvine at Allans-MacBook-Pro in ~/Projects/repo/dff-2022 on master [!]
$ which sqlite3
/opt/homebrew/Caskroom/miniconda/base/envs/dff/bin/sqlite3

[PeterPetrik]

pip install usually takes binary packages, so you do not need to compile anything locally on your computer. However, we do not provide apple M1 binaries yet, since GitHub does not have runners for CI for this yet. Therefore as part of the installation, pip tries to compile geodiff locally.

For compilation of geodiff , you need sqlite library with sqlite3_enable_load_extension, which is not the case for apple-provided sqlite on system. Bear in mind that anaconda may not have such sqlite too, I think that conda-forge sqlite3 has extensions, but you need to check.

I suggest to use brew sqlite similarly to our CI (https://github.com/MerginMaps/geodiff/blob/master/.github/workflows/python_packages.yml#L165).

brew install sqlite3
pip3 install setuptools scikit-build wheel cmake
pip3 install mergin-client

if this one doesn't work or you need some custom sqlite, you need to force CMake to pick the sqlite you like. check https://github.com/MerginMaps/geodiff/blob/eab7829144e81782b827e0abb929dcc21fc22f6a/geodiff/cmake/FindSQLite3.and https://github.com/MerginMaps/geodiff/blob/master/setup.py#L13 for how we picks sqlite.

[NewGraphEnvironment]

oh jeez. Seems like I got a bit closer but issue unresolved. Can't be conceivable that there is noone on a M1 or M2 mac is yet using the mergin-client tool? Its too juicy

Tried:

1. removing and reinstalling CommandLineTools
2. brew update, brew install sqlite3. Then within conda virt env pip3 install setuptools scikit-build wheel cmake (separately of course). Creeped out by installing a pile of packages within the main system environment so am trying to do this within virt env. Guessing this could be a barrier.
3. tried to force CMake to pick the sqlite by nesting the calls to pip install for mergin-client inside a test virt env with manually specified variables for SQLite3_INCLUDE_DIRS, SQLite3_LIBRARIES, SQLite3_VERSION and SQLite3_FOUND with hopes of overriding the cmake lookup in here but that (perhaps unsurprisingly) could not stop cmake for finding the CommandLineTools version of SQLite3
4. Actually quite the gorb when it comes to python but tried to enable extension loading in the CommandLineTools sqlite as well as within the brew intalled version with a call to sqlite3_enable_load_extension( onoff==1 ) from the python terminals but NameError: name sqlite3_enable_load_extension was not defined in either case. Not sure if this is because my syntax was rubbish or if the call was simply not available but thinking it is a trash issue.

Do you think it is worth a try with agit clone option? Again, python gorb but guessing I would try to run through the Development section of the main py client repo page.

Sorry for the lost puppy.

Attached the environment.yml file as.txt file again on the way off chance it is helpful for anything besides clutter.
environment.txt

[PeterPetrik]

Hi

2. the approach I proposed is outside conda environment, if you want to use conda, you need to add conda-forge, install sqlite that is compiled with extensions (i believe one in conda-forge is, but it is good to google and check)
3. these are cmake variables, not environment variables, so they are not picked as you setup them. We would need to modify FindSqlite3.cmake in geodiff to use environment variables
4. it is not runtime flag, but compilation flag. So if sqlite is compiled without extension you cannot switch them off afterwards. Therefore you need to pick the correct sqlite

@SrNetoChan do you know if sqlite3 in conda-forge is compiled with sqlite3_enable_load_extension ?

[NewGraphEnvironment]

Peter, Martin and Alexandre. Thank you so much for your time. The suite of tools from Lutra is looking incredible and to diff the gpkg and pg is obviously not a small deal. What a pain to massage all the OS bullcrapery. Your help is much appreciated.

Now tested outside of conda. As you mentioned, it seems perhaps as though the key issue is how Cmake chooses the sqlite3. Now have brew installed sqlite3 on path and can see the load_extension is on However, on build of mergin-client with all listed requirements the developer tools version of sqlite3 is still chosen. As mentioned, the answer may be in some sort of mod toFindSqlite3.cmake. Here is the workflow for testing outside of conda.

$brew install sqlite

Add brew sqlite3 to path

$ echo 'export PATH="/opt/homebrew/Cellar/sqlite/3.40.1/bin:$PATH"' >> ~/.bash_profile

close and reopen terminal and test with call to .dbconfig

$ which sqlite3
/opt/homebrew/Cellar/sqlite/3.40.1/bin/sqlite3

$ sqlite3
SQLite version 3.40.1 2022-12-28 14:03:47
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .dbconfig
          defensive off
            dqs_ddl on
            dqs_dml on
        enable_fkey off
        enable_qpsg off
     enable_trigger on
        enable_view on
     fts3_tokenizer off
 legacy_alter_table off
 legacy_file_format off
     load_extension on
   no_ckpt_on_close off
     reset_database off
        trigger_eqp off
     trusted_schema on
    writable_schema off
sqlite>

Ensure dependencies are installed

$ pip3 install setuptools scikit-build wheel cmake
Requirement already satisfied: setuptools in /opt/homebrew/lib/python3.11/site-packages (65.6.3)
Collecting scikit-build
  Using cached scikit_build-0.16.6-py3-none-any.whl (79 kB)
Requirement already satisfied: wheel in /opt/homebrew/lib/python3.11/site-packages (0.38.4)
Collecting cmake
  Using cached cmake-3.25.2-py2.py3-none-macosx_10_10_universal2.macosx_10_10_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl (45.1 MB)
Collecting distro
  Using cached distro-1.8.0-py3-none-any.whl (20 kB)
Collecting packaging
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Installing collected packages: cmake, packaging, distro, scikit-build
Successfully installed cmake-3.25.2 distro-1.8.0 packaging-23.0 scikit-build-0.16.6

Try to install mergin-client

$ pip3 install mergin-client

mergin-client_build_error.txt file attached with resulting output to terminal. Cmake is still looking for love in not the right place. Dang

mergin-client_build_error.txt

[SrNetoChan]

@PeterPetrik said :

@SrNetoChan do you know if sqlite3 in conda-forge is compiled with sqlite3_enable_load_extension ?

Not sure where that would be, but I don't see it anywhere in the building script.

should be around here, right?

https://github.com/conda-forge/sqlite-feedstock/blob/main/recipe/build.sh#L36

We can try to include it.

[PeterPetrik]

@SrNetoChan I think it is this one https://github.com/qgis/QGIS-Mac-Packager/blob/346e9c89a1671b7fab164613987e05a67d7063a3/qgis_deps/recipes/sqlite/recipe.sh#L69

--enable-dynamic-extensions