Change compilation layout #1676

bobot · 2018-12-15T14:18:50Z

In order to keep all the .cmx visible, we are only separating the .cmi of the private module:

one directory for all cmx
one directory for all cmo, cmi, cmt, cmti
one directory for the public cmi, cmt, cmti

rgrinberg · 2018-12-17T12:54:08Z

Is it possible to split the first 2 commits, and the 3rd commit to separate PR's? I've had a brief glance and they look good.

bobot · 2018-12-20T09:39:19Z

Ok, it is done in #1696 and #1697 .

rgrinberg · 2018-12-21T01:59:43Z

Thanks. Those PR's have now been merged, so this needs a rebase.

bobot · 2018-12-23T12:21:08Z

Rebased on accepted PR and now based on #1709 .

rgrinberg · 2019-01-01T17:35:31Z

Looks like this PR adds a warning in the JS tests:

-      ocamlopt lib/.x.objs/x__.{cmx,o}
-      ocamlopt lib/.x.objs/x__Y.{cmx,o}
-      ocamlopt lib/.x.objs/x.{cmx,o}
+      ocamlopt lib/.x.objs/.native_objs/x__.{cmx,o}
+      ocamlopt lib/.x.objs/.native_objs/x__Y.{cmx,o}
+      ocamlopt lib/.x.objs/.native_objs/x.{cmx,o}
+  File "_none_", line 1:
+  Warning 58: no cmx file was found in path for module X__Y, and its interface was not compiled with -opaque

(also reflected in the commit I just pushed that promoted this result).

rgrinberg · 2019-01-01T17:47:25Z

What do you think about removing the _objs suffix from the object directories? Names like .foo.objs/.native_objs seem a bit redundant, and windows users don't appreciate the longer paths

rgrinberg · 2019-01-02T07:13:24Z

The issue above is fixed, but there's still an issue with this PR in how it handles Lib_file_deps.

rgrinberg · 2019-01-03T05:24:01Z

I've refactored the Lib_file_deps module, and have added private module .cmx into the cmx glob. I'm pretty sure those .cmx files should be in there. @bobot please review that commit.

bobot · 2019-01-03T08:33:47Z

What do you think about removing the _objs suffix from the object directories? Names like .foo.objs/.native_objs seem a bit redundant, and windows users don't appreciate the longer paths

I agree it is redundant.

src/lib_file_deps.ml

rgrinberg · 2019-01-03T18:10:02Z

Okay, I cleaned up the search history and removed the unnecessary stuff. I'm convinced this work and will review later today after giving this a final read.

rgrinberg · 2019-01-03T20:00:09Z

I've renamed the paths as discussed and also did a few interface tweaks. Obj_dir.t is now abstract. I think it was a bit confusing to have both a public_cmi_dir accessor function and a field with the same name.

- all the cmx are all at the same place - all the cmi and cmo are in another directory - the public directory contains the public cmi Signed-off-by: François Bobot <[email protected]> Signed-off-by: Rudi Grinberg <[email protected]>

Always make sure that .cmx files are available in the include paths Signed-off-by: Rudi Grinberg <[email protected]>

Signed-off-by: Rudi Grinberg <[email protected]>

* Move setting up glob aliases to Lib_file_deps * Allow only fixed number of extensions as we only setup a few of them * Do not setup extraneous dependencies for cmo's Signed-off-by: Rudi Grinberg <[email protected]>

@diml

CHANGES: - Second step of the deprecation of jbuilder: the `jbuilder` binary now emits a warning on every startup and both `jbuilder` and `dune` emit warnings when encountering `jbuild` files (ocaml/dune#1752, @diml) - Change the layout of build artifacts inside _build. The new layout enables optimizations that depend on the presence of `.cmx` files of private modules (ocaml/dune#1676, @bobot) - Fix merlin handling of private module visibility (ocaml/dune#1653 @bobot) - unstable-fmt: use boxes to wrap some lists (ocaml/dune#1608, fix ocaml/dune#1153, @emillon, thanks to @rgrinberg) - skip directories when looking up programs in the PATH (ocaml/dune#1628, fixes ocaml/dune#1616, @diml) - Use `lsof` on macOS to implement `--stats` (ocaml/dune#1636, fixes ocaml/dune#1634, @xclerc) - Generate `dune-package` files for every package. These files are installed and read instead of `META` files whenever they are available (ocaml/dune#1329, @rgrinberg) - Fix preprocessing for libraries with `(include_subdirs ..)` (ocaml/dune#1624, fix ocaml/dune#1626, @nojb, @rgrinberg) - Do not generate targets for archive that don't match the `modes` field. (ocaml/dune#1632, fix ocaml/dune#1617, @rgrinberg) - When executing actions, open files lazily and close them as soon as possible in order to reduce the maximum number of file descriptors opened by Dune (ocaml/dune#1635, ocaml/dune#1643, fixes ocaml/dune#1633, @jonludlam, @rgrinberg, @diml) - Reimplement the core of Dune using a new generic memoization system (ocaml/dune#1489, @rudihorn, @diml) - Replace the broken cycle detection algorithm by a state of the art one from [this paper](https://doi.org/10.1145/2756553) (ocaml/dune#1489, @rudihorn) - Get the correct environment node for multi project workspaces (ocaml/dune#1648, @rgrinberg) - Add `dune compute` to call internal memoized functions (ocaml/dune#1528, @rudihorn, @diml) - Add `--trace-file` option to trace dune internals (ocaml/dune#1639, fix ocaml/dune#1180, @emillon) - Add `--no-print-directory` (borrowed from GNU make) to suppress `Entering directory` messages. (ocaml/dune#1668, @dra27) - Remove `--stats` and track fd usage in `--trace-file` (ocaml/dune#1667, @emillon) - Add virtual libraries feature and enable it by default (ocaml/dune#1430 fixes ocaml/dune#921, @rgrinberg) - Fix handling of Control+C in watch mode (ocaml/dune#1678, fixes ocaml/dune#1671, @diml) - Look for jsoo runtime in the same dir as the `js_of_ocaml` binary when the ocamlfind package is not available (ocaml/dune#1467, @nojb) - Make the `seq` package available for OCaml >= 4.07 (ocaml/dune#1714, @rgrinberg) - Add locations to error messages where a rule fails to generate targets and rules that require files outside the build/source directory. (ocaml/dune#1708, fixes ocaml/dune#848, @rgrinberg) - Let `Configurator` handle `sizeof` (in addition to negative numbers). (ocaml/dune#1726, fixes ocaml/dune#1723, @Chris00) - Fix an issue causing menhir generated parsers to fail to build in some cases. The fix is to systematically use `-short-paths` when calling `ocamlc -i` (ocaml/dune#1743, fix ocaml/dune#1504, @diml) - Never raise when printing located errors. The code that would print the location excerpts was prone to raising. (ocaml/dune#1744, fix ocaml/dune#1736, @rgrinberg) - Add a `dune upgrade` command for upgrading jbuilder projects to Dune (ocaml/dune#1749, @diml) - When automatically creating a `dune-project` file, insert the detected name in it (ocaml/dune#1749, @diml) - Add `(implicit_transitive_deps <bool>)` mode to dune projects. When this mode is turned off, transitive dependencies are not accessible. Only listed dependencies are directly accessible. (ocaml/dune#1734, ocaml/dune#430, @rgrinberg, @hnrgrgr) - Add `toplevel` stanza. This stanza is used to define toplevels with libraries already preloaded. (ocaml/dune#1713, @rgrinberg) - Generate `.merlin` files that account for normal preprocessors defined using a subset of the `action` language. (ocaml/dune#1768, @rgrinberg) - Emit `(orig_src_dir <path>)` metadata in `dune-package` for dune packages built with `--store-orig-source-dir` command line flag (also controlled by `DUNE_STORE_ORIG_SOURCE_DIR` env variable). This is later used to generate `.merlin` with `S`-directives pointed to original source locations and thus allowing merlin to see those. (ocaml/dune#1750, @andreypopp) - Improve the behavior of `dune promote` when the files to be promoted have been deleted. (ocaml/dune#1775, fixes ocaml/dune#1772, @diml) - unstable-fmt: preserve comments (ocaml/dune#1766, @emillon) - Pass flags correctly when using `staged_pps` (ocaml/dune#1779, fixes ocaml/dune#1774, @diml) - Fix an issue with the use of `(mode promote)` in the menhir stanza. It was previously causing intermediate *mock* files to be promoted (ocaml/dune#1783, fixes ocaml/dune#1781, @diml) - unstable-fmt: ignore files using OCaml syntax (ocaml/dune#1784, @emillon) - Configurator: Add `which` function to replace the `which` command line utility in a cross platform way. (ocaml/dune#1773, fixes ocaml/dune#1705, @Chris00) - Make configurator append paths to `$PKG_CONFIG_PATH` on macOS. Previously it was prepending paths and thus `$PKG_CONFIG_PATH` set by users could have been overridden by homebrew installed libraries (ocaml/dune#1785, @andreypopp) - Disallow c/cxx sources that share an object file in the same stubs archive. This means that `foo.c` and `foo.cpp` can no longer exist in the same library. (ocaml/dune#1788, @rgrinberg) - Forbid use of `%{targets}` (or `${@}` in jbuild files) inside preprocessing actions (ocaml/dune#1812, fixes ocaml/dune#1811, @diml) - Add `DUNE_PROFILE` environment variable to easily set the profile. (ocaml/dune#1806, @rgrinberg) - Deprecate the undocumented `(no_keep_locs)` field. It was only necessary until virtual libraries were supported (ocaml/dune#1822, fix ocaml/dune#1816, @diml) - Rename `unstable-fmt` to `format-dune-file` and remove its `--inplace` option. (ocaml/dune#1821, @emillon). - Autoformatting: `(using fmt 1.1)` will also format dune files (ocaml/dune#1821, @emillon). - Autoformatting: record dependencies on `.ocamlformat-ignore` files (ocaml/dune#1824, fixes ocaml/dune#1793, @emillon)

Summary: Dune is updated from 1.6.3 to 1.10.0. One thing to keep in mind is that dune 1.7.0 changed the compilation layout compared to 1.6.3 (see ocaml/dune#1676). Reviewed By: ngorogiannis Differential Revision: D16121954 fbshipit-source-id: 8ceb8429f

bobot requested a review from trefis as a code owner December 15, 2018 14:18

bobot force-pushed the change_compilation_layout branch 3 times, most recently from 5ba818c to 01fc922 Compare December 15, 2018 18:11

bobot mentioned this pull request Dec 17, 2018

Private modules & External Virtual libraries do not work #1686

Closed

bobot force-pushed the change_compilation_layout branch from 01fc922 to 2c290bf Compare December 23, 2018 12:12

bobot requested a review from a user December 23, 2018 12:12

bobot requested review from emillon, fpottier and rgrinberg as code owners December 23, 2018 12:12

bobot changed the title ~~[WIP] Change compilation layout~~ Change compilation layout Dec 23, 2018

bobot force-pushed the change_compilation_layout branch from 2c290bf to 9814464 Compare December 23, 2018 12:23

rgrinberg force-pushed the change_compilation_layout branch 2 times, most recently from ff1ee4e to 1c76a7f Compare January 1, 2019 17:34

rgrinberg force-pushed the change_compilation_layout branch from b9a756d to 3683b24 Compare January 3, 2019 04:30

bobot commented Jan 3, 2019

View reviewed changes

src/lib_file_deps.ml Outdated Show resolved Hide resolved

rgrinberg force-pushed the change_compilation_layout branch from 3683b24 to 79051c2 Compare January 3, 2019 18:05

bobot and others added 2 commits January 3, 2019 15:22

Change compilation layout

37c76b2

- all the cmx are all at the same place - all the cmi and cmo are in another directory - the public directory contains the public cmi Signed-off-by: François Bobot <[email protected]> Signed-off-by: Rudi Grinberg <[email protected]>

Fix module compilation

e3217ec

Always make sure that .cmx files are available in the include paths Signed-off-by: Rudi Grinberg <[email protected]>

rgrinberg added 2 commits January 3, 2019 15:26

Move setting up file deps to Lib_file_deps

2e96c6b

Signed-off-by: Rudi Grinberg <[email protected]>

Refactor Lib_file_deps handling

bb95755

* Move setting up glob aliases to Lib_file_deps * Allow only fixed number of extensions as we only setup a few of them * Do not setup extraneous dependencies for cmo's Signed-off-by: Rudi Grinberg <[email protected]>

rgrinberg force-pushed the change_compilation_layout branch from 1e4c5f2 to bb95755 Compare January 3, 2019 20:26

rgrinberg merged commit 86926eb into ocaml:master Jan 3, 2019

rgrinberg mentioned this pull request Feb 12, 2019

[new release] dune (1.7.0) ocaml/opam-repository#13444

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Change compilation layout #1676

Change compilation layout #1676

Uh oh!

bobot commented Dec 15, 2018

Uh oh!

rgrinberg commented Dec 17, 2018

Uh oh!

bobot commented Dec 20, 2018

Uh oh!

rgrinberg commented Dec 21, 2018

Uh oh!

bobot commented Dec 23, 2018

Uh oh!

rgrinberg commented Jan 1, 2019

Uh oh!

rgrinberg commented Jan 1, 2019

Uh oh!

rgrinberg commented Jan 2, 2019

Uh oh!

rgrinberg commented Jan 3, 2019

Uh oh!

bobot commented Jan 3, 2019

Uh oh!

Uh oh!

rgrinberg commented Jan 3, 2019

Uh oh!

rgrinberg commented Jan 3, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Change compilation layout #1676

Change compilation layout #1676

Uh oh!

Conversation

bobot commented Dec 15, 2018

Uh oh!

rgrinberg commented Dec 17, 2018

Uh oh!

bobot commented Dec 20, 2018

Uh oh!

rgrinberg commented Dec 21, 2018

Uh oh!

bobot commented Dec 23, 2018

Uh oh!

rgrinberg commented Jan 1, 2019

Uh oh!

rgrinberg commented Jan 1, 2019

Uh oh!

rgrinberg commented Jan 2, 2019

Uh oh!

rgrinberg commented Jan 3, 2019

Uh oh!

bobot commented Jan 3, 2019

Uh oh!

Uh oh!

rgrinberg commented Jan 3, 2019

Uh oh!

rgrinberg commented Jan 3, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants