[question] Workspace and overrides

[kevinpoulet]

What is your question?

Hello,

Let's say I have a workspace with app1 (prod1) -> libA (prod1) -> libB (prod2) -> libC (prod2).

The product between parentheses also denotates which team is responsible for the majority of the development.

We use a floating version (our CI overwrite it in our AF on push) we call develop within a product so that we always point to the latest development (develop can be seen as a kind of beta/RC). But a team doesn’t want unexpected changes from another team, so prod1 requires fixed versions of prod2. So, within prod1 we require other prod1 with /develop and prod2 with /1.2.3.

Our problem is that prod1 team often contributes the features they need into prod2. But since they are developing in prod2, what they cloned is requiring the /develop of other prod2. When inside the workspace, it creates a conflict because.

This kind of conflict also arises when we want to try using a new version of libC for instance in a quick build to check for breaking changes/improvements. In that case we would need to either update all the recipes or have or CI create a new full-fledged release just for testing purposes which we cannot realistically do.

A possible way to express the conflict resolution would be to allow only override=True requirements in the workspace recipe.

Another solution would be to checkout the sources based on the versions in the workspace. I haven’t dug too deep into the workspace open concept, but for reference, the repositories are cloned following a fixed structure and are outside of the workspace. Other tools and scripts rely on the clone location and it feels like workspace open only clones into the workspace.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[AbrilRBS]

Hi @kevinpoulet thanks a lot for your question.

As the workspace feature is still a feature we're incubating, some of its behaviour can still be a bit unclear, and the docs around it might be missing important information, sorry that you found one of the holes!

We'll get back to you in a few days once we discuss this use-case and how workspaces fit it, thanks a lot for your patience :)

[memsharded]

Hi @kevinpoulet

I'd like to have some better understanding of this.
Because when defining a workspace, the versions are really decided by the user that is setting up the workspace. Because the workspace is a "source" approach, basically it requires to checkout those 4 specific repos and do the checkout of the branch/tag/commit that want to develop on top.

So, if I understood correctly, developers will checkout a develop branch of libB, that will happen to have a version = "develop" definition?
Then, the developer will need to go to the libA recipe and change the requires("libb/develop") to use it?

I'd also like to understand why not using some versioning mechanism, because you commented (develop can be seen as a kind of beta/RC), but beta and release candidates are also versioned, like a pre-release can be libb/2.0.0-pre.1.2.3, and have the pre.1.2.3 also increasing?

I am in general a bit confused about this 😅
Because:

But a team doesn’t want unexpected changes from another team, so prod1 requires fixed versions of prod2. So, within prod1 we require other prod1 with /develop and prod2 with /1.2.3.

So the team is not expecting changes from the other team, but the other team is actually cloning the prod2 repos adding them to the workspace to do changes on them? The separation of versions develop and 1.2.3 seems like a self-inflicted constraint that you are actively trying to workaround?

[kevinpoulet]

Hello,

Thanks for your answer. Here are “some” more elements ^^

Concerning the source approach, in a sense I’m trying to have it go further: ignore the version requirement/version listed. As long as the recipe name matches the name in the cloned recipes, it should be used.

Even if my consumer says it wants libB/1.0.0 and I checkout 2.0.0 in the libB repo, I’m looking for a way to force it to build anyway. Especially since I’d also have to fix all conflicts to libC, because it could also be explicitly required by libA or app1 in addition to libB.

So, if I understood correctly, developers will checkout a develop branch of libB, that will happen to have a version = "develop" definition?
Then, the developer will need to go to the libA recipe and change the requires("libb/develop") to use it?

This sounds like what we are doing yes.

Like you said, the goal is to checkout whatever sources I want to use, and not bother with the required versions in the recipes. Freezing the versions is something we do closer to the release.

I’m not advocating for this to be the default, having Conan tell you that the repository is not checked out to the correct version is also a useful mode. The goal is having an option/method that would solve basically the same problems force/override=True solves for requirements(): temporary quick fixes, quick testing using newer versions, one of demos/POC/prototypes where a couple of recipes are changed, …

Concerning the use case/workflow we have, the prod2 team is basically developing an SDK that is used by our other products as well as our clients directly. So it is a product in its own right with its own evolutions, but at the same time the basis for our other products.

The separation “1.2.3 vs develop” is indeed a choice we made back in V1 that seemed to solve most of our problems. The goal is that our evergreen “develop” version is basically our nightly, or master branch build in some organizations. That way, within each product (let’s take prod1 here as an example), when a development is merged back in the develop branch (which is our master, and has version = “develop” in the recipe), other developers in prod1 can do a conan install … --update and check that their developments are compatible with the latest state of prod1. Meanwhile, prod2 works on its “develop” version in preparation of the next release.

Now, where the cookie crumbles is that prod2 can’t handle all the fixes and developments on their own. So in the end, when major features are added in prod1, they end up requiring things to be added/edited in prod2 (which is our SDK). But the developers with the best understanding of what needs to happen (and available time) are often those of prod1, and they want to directly make sure that the changes they make in prod2 (the SDK), solve their issue. When this change is being developed, the prod2 base branch must be develop to reduce merge conflicts down the line (hence the recipe having version=”develop” when checked out on disk). When the prod1 developer has finished its update to the prod2 (SDK), it submits a Merge Request to the prod2 (SDK) develop branch. It is now part of the evergreen develop version of prod2 (SDK) where the prod2 can be tested for regressions, conflicts, … Later, the prod2 team publishes a new “freezed” version (with a fixed number) once validated. The prod1 developer then updates its branches to use the newer release of prod2. And after a prod1 Merge Request, prod1 develop version now has the new feature and uses the new version of prod2.

We do not use prod2 (SDK) develop version as it will start development for two releases later from prod1 point of view. Prod2 releases first and might even make intermediate releases with large changes because we have prod3 and so on that also need prod2 but with different schedules to spread out the development.

I made it sound like there are not that many recipes, but we are now nearing the 300 recipes across our main products. We are also looking at workspaces to enable monolith builds/everything in the same IDE window. But since developing “across product” is something that happens regularly, we can’t really use workspaces currently. It is also why we are not just using editables. (Slightly off topic but adjacent: we had the same “problem” with editables, trying to set the editable to a given version that was not the one in the recipe required to edit the recipe. Which felt even weirder because once added as an editable, you could put back the version to not matching the editable one and it would still work.)

We also started introducing editables before workspaces and got mixed feedbacks: some user prefer to continue commit and export as it feels more intuitive to them. That is why the workspace is a separate repository in our case: those who wish to not have to deal with exports or prefer monolith project in their IDE would use the workspace and not have to bother setting editables. Meanwhile, those who prefer exports/feel uncomfortable with editables can continue using individual recipes, or use the recipe used by the installer to gather the product in its entirety.

Hoping this makes our situation somewhat clearer.

[memsharded]

Thanks for the feedback, it helps to understand a bit better.

A couple of quick comments:

• It is not planned or expected to have the workspace feature to automatically ignore or default the versions and the resolution of the versions to a single default version. Because workspaces intend to support the use case of having multiple different versions of the same dependency in the workspace. Not for the super-build case with conan workspace super-install, but having several different apps depending on different versions of the same thing checked out in different folders of the workspace is something that the "orchestrated" commands such as conan workspace build support.
• But the automatic definition of package versions might be achieved with the Workspace packages() method (see https://docs.conan.io/2/reference/workspace_files.html#conanws-py first example). That could automatically determine different versions, irrespective of what the recipe defines. The recipe could have a version=1.2.3, but if the packages() method returns, it is version develop, it will be version develop

Have you explored the Workspace packages() possibility?

If I understood it correctly, the packages() might help with automatically defining the versions, but the issue of existing requires() that use the other version will happen?
If that is the case, would it be much different the developer UX if the solution is either changing the requires("lib/1.2.3") to something like requires("lib/1.2.3", override=True) or changing it to requires("lib/develop")? It sounds that the dev UX would be mostly the same?

I think I start to understand the issue a bit better, it sounds like it could be better something in the line of [replace_requires] in the profile (but maybe automatically added by the workspace?) than having to modify the recipes, even if it is to add an override=True. I need to further analyze the development flows.

[kevinpoulet]

Thank you for taking the time.

It is not planned or expected to have the workspace feature to automatically ignore or default the versions [...]

I agree. As I said, I don't think this should be the default.

Have you explored the Workspace packages() possibility?

As you mentioned, this does not help with the requirements conflicts. I had done some tests and dismissed it because of this. But I'll keep it in mind if we want to ignore the version="" in th recipes.
We currently use the default conanws.yaml, but we might override the packages() method to allow for an optional version: field in the packages entries to override the one in the cloned recipes.

It sounds that the dev UX would be mostly the same?

The solutions you mentioned indeed sounds to be the same.
My inital thought was to allow to specify the overrides in the conanws.py recipe.
That way, they are centralized and easy to add/edit/remove. Something like either a dedicated method or a requirements() that would only allow for calls of requires("..." , override=true).

I think I start to understand the issue a bit better, it sounds like it could be better something in the line of [replace_requires] in the profile (but maybe automatically added by the workspace?) than having to modify the recipes, even if it is to add an override=True. I need to further analyze the development flows.

I didn't know of [replace_requires], it looks way more flexible that what I initially thought of. Some system like this at the workspace level would be great indeed.

Maybe allow for a replace_requires() method in either class (the Workspace one seems more logical to me) of the conanws.py that would return a tuple of tuples ((("zlib/*", "zlibng/*"), ("boost/*", "boost/1.86.0")))?
That way end users can make it flexible: read it from a separate file, integrate it in their conanws.yaml and even allow for some logic (for instance, only enable replacement if an environment variable is set to reduce the risk of this activating in CI).
The method could even receive the existing [replace_requires] from the computed profile as an argument to allow to either mix as needed or discard entirely.

I guess whatever is used should also support [replace_tool_requires] and the rest of the family.

[memsharded]

I am doing a PR in #19339, with a test.

That test implement the desired custom logic using the [replace_requires] in the profile.

As profiles can be composed, it is mostly a matter of adding -pr=myprofile -pr=myreplaces, and it seems to work pretty smoothly.

It sounds like this would be the recommended way.

The possible UX improvement is if we wanted to be able to change the default profiles or something like that for the current workspace. Recall that with a .conanrc is easy to setup a local home folder for the current workspace only, and that local home can change also its defaults and configuration too.

[memsharded]

Closed by #19339.

Please open new tickets for any further question or issue. Thanks for the feedback"