BUG: `nextstrain update conda` doesn't seem to update to latest available conda-base

[corneliusroemer]

Context

I should have gotten the newest conda-base as available, but I don't seem to. I still have augur --version returning 21.0.1, see log later

This is the version of augur in the metapackage:

name: augur
  version: "==21.0.1" => "==21.1.0"
  build: "pyhdfd78af_1" => "pyhdfd78af_0"

❯ nextstrain update conda 
Checking for newer versions of Nextstrain CLI…

nextstrain-cli is up to date!

Updating conda runtime…
Updating Conda packages…

                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/

conda-forge/osx-64                                          Using cache
conda-forge/noarch                                          Using cache
bioconda/osx-64                                             Using cache
bioconda/noarch                                             Using cache
nextstrain/osx-64                                             No change
nextstrain/noarch                                             No change
Transaction

  Prefix: /Users/corneliusromer/.nextstrain/runtimes/conda/env

  All requested packages already installed


Transaction starting
Transaction finished

Runtime updated!

nextclade_data_workflows/sars-cov-2 on  feat/placement-priors [$!?] via 🅒 nextstrain took 12s 
❯ nextstrain shell --conda .
Entering the Nextstrain runtime

Run the command "exit" to leave the runtime.

 Nextstrain  ~/code/nextclade_data_workflows/sars-cov-2 $ mamba list
bash: mamba: command not found
 Nextstrain  ~/code/nextclade_data_workflows/sars-cov-2 $ conda list
bash: conda: command not found
 Nextstrain  ~/code/nextclade_data_workflows/sars-cov-2 $ augur --version
augur 21.0.1
 Nextstrain  ~/code/nextclade_data_workflows/sars-cov-2 $ 

[tsibley]

@corneliusroemer What's the output of

nextstrain version --verbose

[tsibley]

My guess is you don't actually have the latest version of the nextstrain-base meta-package.

[tsibley]

When I run

nextstrain update conda

locally I get the same

All requested packages already installed

message and nothing updated, but my current nextstrain-base version is back at 20230208T180335Z (hb0f4dca_1_locked).

This implies to me that the channel index (generated and served by anaconda.org) maybe isn't up-to-date. Or that something else is going wrong with the underlying

micromamba update nextstrain-base

[tsibley]

The latest nextstrain-base version, 20230314T174809Z, is definitely in the channel index:

curl -fsSL https://conda.anaconda.org/nextstrain/linux-64/repodata.json \
  | jq '.["packages.conda"]["nextstrain-base-20230314T174809Z-hb0f4dca_1_locked.conda"]'

so now I'll look into Micromamba's local caching of the index, since both @corneliusroemer's output and mine (with s/osx/linux/) report:

nextstrain/osx-64                                             No change
nextstrain/noarch                                             No change

and observing the network requests indicates Micromamba is making a conditional request with:

If-Modified-Since: Tue, 14 Mar 2023 18:02:59 GMT

First thing is probably to see where that date is coming from…

[tsibley]

That date (for me) is coming from:

$ jq -r ._mod ~/.nextstrain/runtimes/conda/micromamba/pkgs/cache/0d65795e.json
Tue, 14 Mar 2023 18:02:59 GMT

(or maybe also from the sibling file (a binary blob) 0d65795e.solv, if that's derived from the .json).

Notably, that index file is correct and up-to-date and contains the latest package version:

jq '.["packages.conda"]["nextstrain-base-20230314T174809Z-hb0f4dca_1_locked.conda"]' \
  ~/.nextstrain/runtimes/conda/micromamba/pkgs/cache/0d65795e.json

So I guess the issue is in Micromamba's solution for the requested update. Perhaps that's represented in the .solv file, but I'm not familiar with the binary format and would have to dig into Micromamba's source.

An easier tack is probably to request more verbose output from the underlying micromamba update that nextstrain update conda is running.

[tsibley]

Ah, from a quick grep for solv in the Micromamba source, I see it uses libsolv. Working on a hunch, I look for debugging tools for libsolv via apt and find libsolv-tools, which contains dumpsolv to read a libsolv file.

[tsibley]

A quick check of that shows that it does know about the latest nextstrain-base package:

…
solvable 13 (14):
solvable:name: nextstrain-base
solvable:evr: 20230314T174809Z
solvable:provides:
  nextstrain-base = 20230314T174809Z
solvable:requires:
  …

[tsibley]

@corneliusroemer I'm not sure yet why Micromamba doesn't think an update is necessary/possible, but I suspect you can work around this if you want with:

NEXTSTRAIN_CONDA_BASE_PACKAGE="nextstrain-base==20230314T174809Z" \
  nextstrain update conda

[tsibley]

Interestingly, when I use:

NEXTSTRAIN_CONDA_BASE_PACKAGE="nextstrain-base >20230208T180335Z" 

to try and force an upgrade (i.e. update to a version greater than the current version I have), Micromamba's solution picks 20230303T182321Z. This indeed satisfies the requirement, but it's still not the latest version! It's only the next version several versions up from the lower bound I gave.

I'm starting to think maybe something changed in our packaging after 20230208T180335Z that causes Micromamba to be reluctant to upgrade across the boundary unless forced. But then why does it also stop at 20230303T182321Z?

[Edit: misread 0208 as 0228.]

[tsibley]

Notably, a fresh Conda runtime setup does get the latest package version, e.g. when I do:

NEXTSTRAIN_RUNTIMES=/tmp/runtimes \
  nextstrain setup conda

[tsibley]

When I force a new setup with 20230208T180335Z and then run an unrestricted update on it, I also get 20230303T182321Z. That is, it still stops short of 20230314T174809Z. Puzzling.

[tsibley]

Currently, my best summary of observations so far is: micromamba update nextstrain-base (+ a bunch of flags) doesn't always pick the latest version of nextstrain-base it knows about.