install fails due to pyjq

[jcpoyser]

CloudMapper install on OSX High Sierra 10.13.6 fails. I successfully:

• cloned the repo,
• used brew to install all the prerequisites, including jq & pyjq

Then when using pipenv to install - i consistently get errors regarding "pyjq"

Here is the output log of the failed install

$ pipenv install --skip-lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing dependencies from Pipfile…
An error occurred while installing pyjq==2.1.0! Will try again.
An error occurred while installing boto3==1.9.70! Will try again.
An error occurred while installing pandas==0.23.4! Will try again.
An error occurred while installing seaborn==0.9.0! Will try again.
An error occurred while installing matplotlib==2.2.2! Will try again.
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 11/11 — 00:00:12
Installing initially failed dependencies…
☤ ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:52
[pipenv.exceptions.InstallError]: File "/usr/local/Cellar/pipenv/2018.11.26_1/libexec/lib/python3.7/site-packages/pipenv/cli/command.py", line 254, in install
[pipenv.exceptions.InstallError]: editable_packages=state.installstate.editables,
[pipenv.exceptions.InstallError]: File "/usr/local/Cellar/pipenv/2018.11.26_1/libexec/lib/python3.7/site-packages/pipenv/core.py", line 1874, in do_install
[pipenv.exceptions.InstallError]: keep_outdated=keep_outdated
[pipenv.exceptions.InstallError]: File "/usr/local/Cellar/pipenv/2018.11.26_1/libexec/lib/python3.7/site-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]: pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]: File "/usr/local/Cellar/pipenv/2018.11.26_1/libexec/lib/python3.7/site-packages/pipenv/core.py", line 862, in do_install_dependencies
[pipenv.exceptions.InstallError]: _cleanup_procs(procs, False, failed_deps_queue, retry=False)
[pipenv.exceptions.InstallError]: File "/usr/local/Cellar/pipenv/2018.11.26_1/libexec/lib/python3.7/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]: raise exceptions.InstallError(c.dep.name, extra=err_lines)

[pipenv.exceptions.InstallError]: ['Collecting pyjq==2.1.0',
Using cached https://files.pythonhosted.org/packages/ec/c1/b5fcba48cb715b88b5ad619d1769204411383e2a275dfddda2d2fa8f99dd/pyjq-2.1.0.tar.gz',
Requirement already satisfied, skipping upgrade: six in /Users//venv/lib/python3.7/site-packages (from pyjq==2.1.0) (1.10.0)',
Installing collected packages: pyjq'
Running setup.py install for pyjq: started',
Running setup.py install for pyjq: finished with status 'error'",
Complete output from command /Users//venv/bin/python3 -u -c "import setuptools, tokenize;file='/private/var/folders/_2/nrg1pnsx2z583v436t5ch1hc0000gr/T/pip-install-c4p4gc1i/pyjq/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /private/var/folders/_2/nrg1pnsx2z583v436t5ch1hc0000gr/T/pip-record-egf1koti/install-record.txt --single-version-externally-managed --compile --install-headers /Users//venv/include/site/python3.7/pyjq:'
running install'
running build'
running build_py'
creating build'
creating build/lib.macosx-10.13-x86_64-3.7'
copying pyjq.py -> build/lib.macosx-10.13-x86_64-3.7'
warning: build_py: byte-compiling is disabled, skipping.
running build_ext'
checking for a BSD-compatible install... /usr/bin/install -c'
checking whether build environment is sane... yes'
checking for a thread-safe mkdir -p... ./install-sh -c -d'
checking for gawk... gawk'
checking whether make sets $(MAKE)... yes'
checking for gcc... gcc'
checking whether the C compiler works... yes'
checking for C compiler default output file name... a.out'
checking for suffix of executables...'
checking whether we are cross compiling... no'
checking for suffix of object files... o'
checking whether we are using the GNU C compiler... yes'
checking whether gcc accepts -g... yes'
checking for gcc option to accept ISO C89... none needed'
checking for style of include used by make... GNU'
checking dependency style of gcc... gcc3'
checking build system type... i386-apple-darwin17.7.0'
checking host system type... i386-apple-darwin17.7.0'
checking how to print strings... printf'
checking for a sed that does not truncate output... /usr/bin/sed'
checking for grep that handles long lines and -e... /usr/bin/grep'
checking for egrep... /usr/bin/grep -E'
checking for fgrep... /usr/bin/grep -F'
checking for ld used by gcc... /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld'
checking if the linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) is GNU ld... no'
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B'
checking the name lister (/usr/bin/nm -B) interface... BSD nm'
checking whether ln -s works... yes'
checking the maximum length of command line arguments... 196608'
checking whether the shell understands some XSI constructs... yes'
checking whether the shell understands "+="... yes'
checking how to convert i386-apple-darwin17.7.0 file names to i386-apple-darwin17.7.0 format... func_convert_file_noop'
checking how to convert i386-apple-darwin17.7.0 file names to toolchain format... func_convert_file_noop'
checking for /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld option to reload object files... -r'
checking for objdump... objdump'
checking how to recognize dependent libraries... pass_all'
checking for dlltool... no'
checking how to associate runtime and link libraries... printf %s\n',
checking for ar... ar'
checking for archiver @file support... no'
checking for strip... strip'
checking for ranlib... ranlib'
checking command to parse /usr/bin/nm -B output from gcc object... ok'
checking for sysroot... no'
checking for mt... no'
checking if : is a manifest tool... no'
checking for dsymutil... dsymutil'
checking for nmedit... nmedit'
checking for lipo... lipo'
checking for otool... otool'
checking for otool64... no'
checking for -single_module linker flag... yes'
checking for -exported_symbols_list linker flag... yes'
checking for -force_load linker flag... yes'
checking how to run the C preprocessor... gcc -E',
checking for ANSI C header files... yes'
checking for sys/types.h... yes'
checking for sys/stat.h... yes'
checking for stdlib.h... yes'
checking for string.h... yes'
checking for memory.h... yes'
checking for strings.h... yes'
checking for inttypes.h... yes'
checking for stdint.h... yes'
checking for unistd.h... yes'
checking for dlfcn.h... yes'
checking for objdir... .libs'
checking if gcc supports -fno-rtti -fno-exceptions... yes'
checking for gcc option to produce PIC... -fno-common -DPIC'
checking if gcc PIC flag -fno-common -DPIC works... yes'
checking if gcc static flag -static works... no'
checking if gcc supports -c -o file.o... yes'
checking if gcc supports -c -o file.o... (cached) yes'
checking whether the gcc linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) supports shared libraries... yes'
checking dynamic linker characteristics... dyld: Library not loaded: /usr/local/opt/readline/lib/libreadline.7.dylib'
Referenced from: /usr/local/bin/awk'
Reason: image not found'
dyld: Library not loaded: /usr/local/opt/readline/lib/libreadline.7.dylib'
Referenced from: /usr/local/bin/awk'
Reason: image not found'
darwin17.7.0 dyld'
checking how to hardcode library paths into programs... immediate'
checking whether stripping libraries is possible... yes'
checking if libtool supports shared libraries... yes'
checking whether to build shared libraries... no'
checking whether to build static libraries... yes'
checking whether make sets $(MAKE)... (cached) yes'
checking for ANSI C header files... (cached) yes'
checking for stdlib.h... (cached) yes'
checking for string.h... (cached) yes'
checking for strings.h... (cached) yes'
checking sys/time.h usability... yes'
checking sys/time.h presence... yes'
checking for sys/time.h... yes'
checking for unistd.h... (cached) yes'
checking sys/times.h usability... yes'
checking sys/times.h presence... yes'
checking for sys/times.h... yes'
checking size of int... 4'
checking size of short... 2'
checking size of long... 8'
checking for an ANSI C-conforming const... yes'
checking whether time.h and sys/time.h may both be included... yes'
checking for size_t... yes'
checking for working alloca.h... yes'
checking for alloca... yes'
checking for working memcmp... yes'
checking for prototypes... yes'
checking for variable length prototypes and stdarg.h... yes',
configure: creating ./config.status'
config.status: creating Makefile'
dyld: Library not loaded: /usr/local/opt/readline/lib/libreadline.7.dylib',
Referenced from: /usr/local/bin/gawk',
Reason: image not found'
./config.status: line 1184: 43841 Done(141) eval sed \"\$ac_sed_extra\" "$ac_file_inputs"'
43842 Abort trap: 6 | $AWK -f "$ac_tmp/subs.awk" > $ac_tmp/out', config.status: error: could not create Makefile'
Traceback (most recent call last):'
File "", line 1, in '
File "/private/var/folders/_2/nrg1pnsx2z583v436t5ch1hc0000gr/T/pip-install-c4p4gc1i/pyjq/setup.py", line 116, in ', " 'Programming Language :: JavaScript',"
File "/Users//venv/lib/python3.7/site-packages/setuptools/init.py", line 143, in setup'
return distutils.core.setup(attrs)'
File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/core.py", line 148, in setup'
dist.run_commands()'
File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 966, in run_commands',
self.run_command(cmd)'
File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command'
cmd_obj.run()'
File "/Users//venv/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run'
return orig.install.run(self)'
File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/command/install.py", line 545, in run'
" self.run_command('build')"
File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/cmd.py", line 313, in run_command'
self.distribution.run_command(command)'
File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command'
cmd_obj.run()'
File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/command/build.py", line 135, in run'
self.run_command(cmd_name)'
File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/cmd.py", line 313, in run_command'
self.distribution.run_command(command)'
File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command'
cmd_obj.run()'
File "/private/var/folders/_2/nrg1pnsx2z583v436t5ch1hc0000gr/T/pip-install-c4p4gc1i/pyjq/setup.py", line 29, in run'
self._build_oniguruma()'
File "/private/var/folders/_2/nrg1pnsx2z583v436t5ch1hc0000gr/T/pip-install-c4p4gc1i/pyjq/setup.py", line 43, in _build_oniguruma'
["make", "install"],'
File "/private/var/folders/_2/nrg1pnsx2z583v436t5ch1hc0000gr/T/pip-install-c4p4gc1i/pyjq/setup.py", line 70, in _build_lib'
subprocess.check_call(command, cwd=lib_dir)'
File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 347, in check_call'
raise CalledProcessError(retcode, cmd)', " subprocess.CalledProcessError: Command '['./configure'
CFLAGS=-fPIC'
--disable-shared'
--prefix'
/private/var/folders/_2/nrg1pnsx2z583v436t5ch1hc0000gr/T/pip-install-c4p4gc1i/pyjq/dependencies/onig_install']' returned non-zero exit status 1."
'
----------------------------------------']
[pipenv.exceptions.InstallError]:
'Command "/Users//venv/bin/python3 -u -c "import setuptools, tokenize;file='/private/var/folders/_2/nrg1pnsx2z583v436t5ch1hc0000gr/T/pip-install-c4p4gc1i/pyjq/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))"
install --record /private/var/folders/_2/nrg1pnsx2z583v436t5ch1hc0000gr/T/pip-record-egf1koti/install-record.txt --single-version-externally-managed --compile --install-headers /Users/*/venv/include/site/python3.7/pyjq"
failed with error code 1 in /private/var/folders/_2/nrg1pnsx2z583v436t5ch1hc0000gr/T/pip-install-c4p4gc1i/pyjq/']
ERROR: ERROR: Package installation failed...

[0xdabbad00]

Hmm, nothing jumps out at me as the cause for this error. This is more reason for me to Dockerize for issue #287

[jcpoyser]

potential solution for me was to clone the jq repo and go through the steps to make & install it manually instead of relying on the Homebrew install of jq. Once I did that and redid the cloudmapper install steps, it still paused on installing the dependencies from Pipfile - but this time it did not error out on the pyjq install... i agree that Dockerize-ing may solve this by providing the complete set of libraries needed at the correct version.

[zachzeid]

I've also gotten this error, and my workaround is to ensure that I do a pipenv shell first after installation.

[nanookusa]

Definitely have had all of these installation problems on Mac Sierra.
I've worked through the dependency hell of pyjq and can offer the following advice.

• During installation attempts if a Lockfile has been created, make sure to remove it before attempting to pipenv install again.

• Installing with the --sequential can be helpful when working through dependency and installation issues https://docs.pipenv.org/en/latest/advanced/

• If you still experience a failure when resolving dependencies and installing, trying installing pyjq separately with pipenv install pyqi --sequential

[dariushazimi]

Had to do python pip install pyjq
But still pyjq fails.

OS: Linux Ubuntu 18.x
Python 3.6.x

(envpy3) ➜  cloudmapper git:(master) ✗ pip install pyjq                                                   
Collecting pyjq
  Downloading https://files.pythonhosted.org/packages/66/74/7f5c7789c894a943088040a0c148f7ac5daddf4096cf820e3ff15ae75ea0/pyjq-2.3.1.tar.gz (2.1MB)
    100% |████████████████████████████████| 2.1MB 753kB/s 
Collecting six (from pyjq)
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Building wheels for collected packages: pyjq
  Running setup.py bdist_wheel for pyjq ... error
  Complete output from command /home/dariush/project/aws/securityauditor/cloudmapper/envpy3/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-f4x0rzd6/pyjq/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpfcpmiu8upip-wheel- --python-tag cp36:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
  error: invalid command 'bdist_wheel'
  
  ----------------------------------------
  Failed building wheel for pyjq

[0xdabbad00]

I've rerun the install steps on a mac without issue.

git clone git@github.com:duo-labs/cloudmapper.git
cd cloudmapper/
pipenv install --skip-lock
pipenv shell
mv config.json.demo config.json
python cloudmapper.py prepare --account demo

For @dariushazimi, it looks like you're install pyjq 2.3.1 when it should be 2.1.0 as that is what is set in the Pipfile (https://github.com/duo-labs/cloudmapper/blob/master/Pipfile#L7)

Without more info for the other comments in this issue, I'm closing this due to staleness under the assumption that the other issues were due to the install instructions not being followed correctly, or possibly other issues on the host system.

[linuxdevops-34]

I am facing same issue in ubuntu 18.04 OS.

Command "/home/ubuntu/cloudmapper/venv/bin/python3 -u -c "import setuptools, tok enize;file='/tmp/pip-build-l0pynqe8/pyjq/setup.py';f=getattr(tokenize, 'open ', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(co de, file, 'exec'))" install --record /tmp/pip-_p8fg38t-record/install-record .txt --single-version-externally-managed --compile --install-headers /home/ubunt u/cloudmapper/venv/include/site/python3.6/pyjq" failed with error code 1 in /tmp /pip-build-l0pynqe8/pyjq/

[yunw]

i hit the same issue with Amazon Linux. had tried downgrade the version to 2.1.0 but still no luck.

[kayue]

I am running into issue on Mac.

Building wheels for collected packages: pyjq
  Building wheel for pyjq (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/kayue/Sites/cloudmapper/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py'"'"'; __file__='"'"'/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-wheel-fjgz138y
       cwd: /private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/
  Complete output (54 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.14-x86_64-3.8
  copying pyjq.py -> build/lib.macosx-10.14-x86_64-3.8
  running build_ext
  checking for a BSD-compatible install... /usr/bin/install -c
  checking whether build environment is sane... yes
  checking for a thread-safe mkdir -p... ./install-sh -c -d
  checking for gawk... no
  checking for mawk... no
  checking for nawk... no
  checking for awk... awk
  checking whether make sets $(MAKE)... no
  checking whether make supports nested variables... no
  checking for gcc... gcc
  checking whether the C compiler works... no
  configure: error: in `/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/dependencies/onig-6.9.0':
  configure: error: C compiler cannot create executables
  See `config.log' for more details
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py", line 93, in <module>
      setup(
    File "/Users/kayue/Sites/cloudmapper/venv/lib/python3.8/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/Users/kayue/Sites/cloudmapper/venv/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 290, in run
      self.run_command('build')
    File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py", line 30, in run
      self._build_oniguruma()
    File "/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py", line 39, in _build_oniguruma
      self._build_lib(
    File "/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py", line 70, in _build_lib
      subprocess.check_call(command, cwd=lib_dir)
    File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['./configure', 'CFLAGS=-fPIC', '--disable-shared', '--prefix', '/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/dependencies/onig_install']' returned non-zero exit status 77.
  ----------------------------------------
  ERROR: Failed building wheel for pyjq
  Running setup.py clean for pyjq
Failed to build pyjq
Installing collected packages: pyjq, toml, pylint, requests, scipy, seaborn, typed-ast
    Running setup.py install for pyjq ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/kayue/Sites/cloudmapper/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py'"'"'; __file__='"'"'/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-record-35kzmlbz/install-record.txt --single-version-externally-managed --compile --install-headers /Users/kayue/Sites/cloudmapper/venv/include/site/python3.8/pyjq
         cwd: /private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/
    Complete output (56 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.14-x86_64-3.8
    copying pyjq.py -> build/lib.macosx-10.14-x86_64-3.8
    running build_ext
    checking for a BSD-compatible install... /usr/bin/install -c
    checking whether build environment is sane... yes
    checking for a thread-safe mkdir -p... ./install-sh -c -d
    checking for gawk... no
    checking for mawk... no
    checking for nawk... no
    checking for awk... awk
    checking whether make sets $(MAKE)... no
    checking whether make supports nested variables... no
    checking for gcc... gcc
    checking whether the C compiler works... no
    configure: error: in `/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/dependencies/onig-6.9.0':
    configure: error: C compiler cannot create executables
    See `config.log' for more details
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py", line 93, in <module>
        setup(
      File "/Users/kayue/Sites/cloudmapper/venv/lib/python3.8/site-packages/setuptools/__init__.py", line 144, in setup
        return distutils.core.setup(**attrs)
      File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/Users/kayue/Sites/cloudmapper/venv/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py", line 30, in run
        self._build_oniguruma()
      File "/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py", line 39, in _build_oniguruma
        self._build_lib(
      File "/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py", line 70, in _build_lib
        subprocess.check_call(command, cwd=lib_dir)
      File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['./configure', 'CFLAGS=-fPIC', '--disable-shared', '--prefix', '/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/dependencies/onig_install']' returned non-zero exit status 77.
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/kayue/Sites/cloudmapper/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py'"'"'; __file__='"'"'/private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-install-b8te63qo/pyjq/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/l2/s5ng1l0s5ggcf0sfwmmdxhp80000gn/T/pip-record-35kzmlbz/install-record.txt --single-version-externally-managed --compile --install-headers /Users/kayue/Sites/cloudmapper/venv/include/site/python3.8/pyjq Check the logs for full command output.
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/kayue/Sites/cloudmapper/venv/bin/python3 -m pip install --upgrade pip' command.

[yunw]

i hit the same issue with Amazon Linux. had tried downgrade the version to 2.1.0 but still no luck.

got it fixed with reinstalling the libs: yum install autoconf automake libtool python3-devel.x86_64 python3-tkinter python-pip jq awscli

[kbroughton]

The pyjq dependency seems a bit finicky. In general, I just run cloudmapper docker and there are no problems. Recently I was forced to run cloudmapper on windows on AWS Workspace which doesn't support nested virtualization/docker. I could install jq via conda or github releases but I think pyjq needs the headers so it still failed.
At first I tried to replace pyjq with jmespath. Unfortunately, jmespath doesn't support all the pyjq syntax. In the end I came up with a hack that allows me to run collect.py natively on windows using subprocess to call out to native jq. I wouldn't want to see this merged into master, it's so hacky, but for extreme workarounds, it might be helpful. https://github.com/kbroughton/cloudmapper/tree/collect_jq.
Probably a more robust solution would be to build conda packages for jq with headers + pyjq across all platforms if native installs are required by some users.