Dependency errors for sphinx, pyocd, junit2html, cmsis-pack-manager

[makslikesalo]

Hello, I tried to build an image for https://github.com/zephyrproject-rtos/docker-image, SHA is d761a83.

The following non-fatal errors occurred during the installation process:

...
ERROR: sphinx 4.5.0 has requirement docutils<0.18,>=0.14, but you'll have docutils 0.18.1 which is incompatible.
ERROR: sphinx-rtd-theme 1.0.0 has requirement docutils<0.18, but you'll have docutils 0.18.1 which is incompatible.
ERROR: sphinx-tabs 3.3.1 has requirement docutils~=0.17.0, but you'll have docutils 0.18.1 which is incompatible.
ERROR: pyocd 0.33.1 has requirement six<2.0,>=1.15.0, but you'll have six 1.14.0 which is incompatible.
ERROR: junit2html 30.1.3 has requirement jinja2>=3.0, but you'll have jinja2 2.10.1 which is incompatible.
...
ERROR: sphinx-tabs 3.3.1 has requirement docutils~=0.17.0, but you'll have docutils 0.16 which is incompatible.
ERROR: pyocd 0.33.1 has requirement pyyaml<7.0,>=6.0, but you'll have pyyaml 5.4.1 which is incompatible.
ERROR: pyocd 0.33.1 has requirement six<2.0,>=1.15.0, but you'll have six 1.14.0 which is incompatible.
ERROR: cmsis-pack-manager 0.4.0 has requirement pyyaml<7.0,>=6.0, but you'll have pyyaml 5.4.1 which is incompatible.

Although the image is built successfully in the end, it prevents later build of the Latex PDF documentation - the process fails.

Could you tell me what is the reason and how to solve it? Perhaps there are any workarounds?

Thanks in advance.

[cjgraydon]

I've run into similar errors, and I believe this is more an issue with pip3 and the zephyr/scripts/requirements*.txt files than the docker image, per se. The fix for me has been to basically pin the versions that the pip3 install is complaining about.

What I have done to resolve, both inside docker image and in native linux/mac OS environment is:

• general package update. i.e. apt-get update && apt-get upgrade
• Run "pip3 install -r zephyr/scripts/requirements.txt" and capture everything it complains about, as you listed your comment above.
• Pin the versions of things it complains about, like docutils. I generally just add these to one of the requirements*.txt files and go again. i.e I added docutils<0.16,>=0.10 to requirements-doc.txt. Do that for each complaining dependency then run "pip3 install -r zephyr/scripts/requirements.txt" again.

That generally does the trick for me.

Alternatively, you could possibly edit the Dockerfile to do that in the initial pip3 requirements install. Maybe right before the requirements run. I haven't actually tried this yet, since I am not seeing the issue at the moment, but I see no reason offhand why that wouldn't work.
i.e.

RUN pip3 install --upgrade pip
pip3 install wheel pip -U &&\
---> MAYBE HERE <----
pip3 install -r https://raw.githubusercontent.com/zephyrproject-rtos/zephyr/master/scripts/requirements.txt &&
pip3 install -r https://raw.githubusercontent.com/zephyrproject-rtos/mcuboot/master/scripts/requirements.txt &&
pip3 install west &&\
pip3 install sh &&\
pip3 install awscli PyGithub junitparser pylint
statistics numpy
imgtool
protobuf
GitPython

You could add it at the end, but you'd need to run the install against the requirements again to pick up the failed packages from the first pass.

As to WHY this happens, I have no real idea. My best guess is that there are package version changes that happen after Zephyr releases and docker image updates, and it breaks. And should a version not be pinned or have max/min in requirements*.txt, boom. Again, just my wild guess.

Maintainers here or on https://github.com/zephyrproject-rtos/zephyr may have a legit answer to that question.

Anyway, hope that helps with working around it.

[cjgraydon]

Incidentally, I ran into this exact same set of dependency issues just now:

Error: sphinx 4.5.0 has requirement docutils<0.18,>=0.14, but you'll have docutils 0.18.1 which is incompatible.
ERROR: sphinx-rtd-theme 1.0.0 has requirement docutils<0.18, but you'll have docutils 0.18.1 which is incompatible.
ERROR: sphinx-tabs 3.3.1 has requirement docutils~=0.17.0, but you'll have docutils 0.18.1 which is incompatible.
ERROR: pyocd 0.34.0 has requirement six<2.0,>=1.15.0, but you'll have six 1.14.0 which is incompatible.
ERROR: junit2html 30.1.3 has requirement jinja2>=3.0, but you'll have jinja2 2.10.1 which is incompatible.

ERROR: sphinx-tabs 3.3.1 has requirement docutils~=0.17.0, but you'll have docutils 0.16 which is incompatible.
ERROR: pyocd 0.34.0 has requirement pyyaml<7.0,>=6.0, but you'll have pyyaml 5.4.1 which is incompatible.
ERROR: pyocd 0.34.0 has requirement six<2.0,>=1.15.0, but you'll have six 1.14.0 which is incompatible.
ERROR: cmsis-pack-manager 0.4.0 has requirement pyyaml<7.0,>=6.0, but you'll have pyyaml 5.4.1 which is incompatible.

[cjgraydon]

Here's what I did to fix the dependency issues:

In the Dockerfile:

• Install Python dependencies
• Awscli requires docutils<0.17 (0.16) which breaks all kinds of things, so remove that
• protobuf pinned to unbreak samples/modules/nanopb .proto stuff
• Default six package is not high enough version for pyocd and others, so bump
  RUN pip3 install wheel pip -U &&
  pip3 install -r https://raw.githubusercontent.com/zephyrproject-rtos/zephyr/master/scripts/requirements.txt &&
  pip3 install -r https://raw.githubusercontent.com/zephyrproject-rtos/mcuboot/master/scripts/requirements.txt &&
  pip3 install west &&
  pip3 install sh &&
  pip3 install PyGithub junitparser pylint
  statistics numpy
  imgtool
  protobuf
  GitPython &&
  pip3 install --upgrade "protobuf<=3.19.0" "docutils==0.17" "six>=1.5.0" "jinja2>=3.0"

The main culprit is Awscli, which demand docutils<0.17. Docutils less than that breaks all the sphinx stuff. Versions above 0.18 breaks sphinx-tabs. So I pinned at 0.17. Default versions of six and jinja on this version of Ubuntu are too old for junit2html 30.1.3, pyocd and smsis-pack-manager. Iirc, awscli was also knocking down the PyYaml to 5.4.1. Once I got rid of awscli, that issue vanished.

If you need awscli, that's pretty ugly, but one way to hack around that would be to only install the requirements you need for awscli and not install everything. Just a thought there. For my needs, it was easiest just to nuke that.

[cjgraydon]

P.S. the protobuf pin was to fix a build issue with samples/modules/nanopb. It wasn't related to the other issues.

[makslikesalo]

Dear Connor Graydon ,
thank you very much for your response, help in finding a direction to solve the problem!

Your suggestion is good enough, I adapted it a bit to avoid problems at all. And this works for me:

# Install Python dependencies
RUN pip3 install wheel pip -U &&\
	pip3 install docutils==0.17 &&\
	pip3 install six==1.15 &&\
	pip3 install Jinja2==3.0 &&\
	pip3 install sphinx~=4.0  &&\
	pip3 install sphinx-tabs==3.2.0 &&\
	pip3 install -r https://raw.githubusercontent.com/zephyrproject-rtos/zephyr/zephyr-v3.1.0/scripts/requirements.txt && \
	pip3 install -r https://raw.githubusercontent.com/zephyrproject-rtos/mcuboot/master/scripts/requirements.txt && \
	pip3 install west &&\
	pip3 install sh &&\
	pip3 install PyGithub junitparser pylint \
		     statistics numpy \
		     imgtool \
		     protobuf \
		     GitPython

1. As you rightly point out, getting rid of awscli (I don't need it) solved part of the problem.
2. Then, to resolve the remaining tangle of dependencies, before installing Zephyr's requirements, I preinstalled packages with the required versions.
3. And finally, in order for the dependencies to be correct for Zephyr v.3.1.0, I specified the url to the requirements for this particular tag https://raw.githubusercontent.com/zephyrproject-rtos/zephyr/zephyr-v3.1.0/scripts/requirements.txt. Because by default it takes from master, and in it it doesn't like sphinx~=5.0.

In general, this workaround works for me, sphinx/latexpdf documentation builds successfully after that.

Once again, many thanks for help!

[cjgraydon]

@makslikesalo Your solution is prettier than mine. lol Glad the info helped!