Skip to content

Add notes for package maintainers #25912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
simonbyrne merged 14 commits into from
Feb 27, 2018
Merged

Add notes for package maintainers #25912

Changes from 8 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
5be21a9
Add notes for package maintainers
simonbyrne Feb 6, 2018
a5c1d79
typo and link
simonbyrne Feb 6, 2018
bc026a0
more typos
simonbyrne Feb 6, 2018
ea24f56
rephrase sentence
simonbyrne Feb 6, 2018
7b0d032
gentoo link
simonbyrne Feb 6, 2018
501bff0
Add comment re: newer LLVM versions
simonbyrne Feb 6, 2018
0e7526d
Add mention of LLVM makefile
simonbyrne Feb 6, 2018
9534d93
Add link to awk
simonbyrne Feb 6, 2018
3e10b43
Update based on @staticfloat's comments
simonbyrne Feb 6, 2018
a3f39c6
Tweak text
simonbyrne Feb 6, 2018
665035a
Update based on @eli-schwartz's comments
simonbyrne Feb 7, 2018
0d1846e
Add notes on libuv
simonbyrne Feb 7, 2018
5fa9587
minor tweak to text
simonbyrne Feb 7, 2018
59c26e7
link to packages.html
simonbyrne Feb 10, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 34 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,15 +294,15 @@ Building Julia requires that the following software be installed:

Julia uses the following external libraries, which are automatically downloaded (or in a few cases, included in the Julia source repository) and then compiled from source the first time you run `make`:

- **[LLVM]** (3.9) — compiler infrastructure.
- **[LLVM]** (3.9 + patches) — compiler infrastructure (see [note below](#llvm)).
- **[FemtoLisp]** — packaged with Julia source, and used to implement the compiler front-end.
- **[libuv]** — portable, high-performance event-based I/O library.
- **[libuv]** (custom fork) — portable, high-performance event-based I/O library.
- **[OpenLibm]** — portable libm library containing elementary math functions.
- **[DSFMT]** — fast Mersenne Twister pseudorandom number generator library.
- **[OpenBLAS]** — fast, open, and maintained [basic linear algebra subprograms (BLAS)](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) library, based on [Kazushige Goto's](https://en.wikipedia.org/wiki/Kazushige_Goto) famous [GotoBLAS](https://www.tacc.utexas.edu/research-development/tacc-software/gotoblas2).
- **[OpenBLAS]** — fast, open, and maintained [basic linear algebra subprograms (BLAS)](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) library, based on [Kazushige Goto's](https://en.wikipedia.org/wiki/Kazushige_Goto) famous [GotoBLAS](https://www.tacc.utexas.edu/research-development/tacc-software/gotoblas2) (see [note below](#blas-and-lapack)).
- **[LAPACK]** (>= 3.5) — library of linear algebra routines for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems.
- **[MKL]** (optional) – OpenBLAS and LAPACK may be replaced by Intel's MKL library.
- **[SuiteSparse]** (>= 4.1) — library of linear algebra routines for sparse matrices.
- **[SuiteSparse]** (>= 4.1) — library of linear algebra routines for sparse matrices (see [note below](#suitesparse)).
- **[ARPACK]** — collection of subroutines designed to solve large, sparse eigenvalue problems.
- **[PCRE]** (>= 10.00) — Perl-compatible regular expressions library.
- **[GMP]** (>= 5.0) — GNU multiple precision arithmetic library, needed for `BigInt` support.
Expand All @@ -318,6 +318,7 @@ Julia uses the following external libraries, which are automatically downloaded
[patch]: http://www.gnu.org/software/patch
[wget]: http://www.gnu.org/software/wget
[m4]: http://www.gnu.org/software/m4
[awk]: http://www.gnu.org/software/gawk
[gcc]: http://gcc.gnu.org
[clang]: http://clang.llvm.org
[python]: https://www.python.org/
Expand Down Expand Up @@ -347,12 +348,41 @@ Julia uses the following external libraries, which are automatically downloaded
[mbedtls]: https://tls.mbed.org/
[pkg-config]: https://www.freedesktop.org/wiki/Software/pkg-config/

### Notes for distribution package maintainers

Package maintaners will typically want to make use of system libraries where possible. Please refer to the above version requirements and additional notes below.
Copy link
Contributor

@eli-schwartz eli-schwartz Feb 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maintaners ==> maintainers


Currently community maintained packages are:
Copy link
Contributor

@tkelman tkelman Feb 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these were all removed as unsupported in #23484 (comment)

Copy link
Member Author

@simonbyrne simonbyrne Feb 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see.

My argument for having this is so that maintainers can look to see what others have done. Perhaps a permalink to somewhere julialang.org or a discourse page would be more useful.

Copy link
Member

@StefanKarpinski StefanKarpinski Feb 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about describing them as no-longer-maintained and still mentioning them here?

Copy link
Member

@nalimilan nalimilan Feb 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are maintained, just not by us. OTC a different bullet below mentions packages which are unmaintained.

We could move instructions for packagers to a separate file so that casual users don't find them. But honestly we're talking about README.md, not the main download page, so I'm not too concerned. Anyway people are going to use these packages, so refusing to mention them is a bit silly IMHO. It's even useful to distinguish actively maintained packages from totally outdated ones to prevent surprises.

Copy link
Contributor

@tkelman tkelman Feb 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the stance of the upstream project, that people should be using them or that how they're built can't be vouched for and may introduce problems that are strictly due to packaging? Maintaining a list of what's maintained is bound to get out of date and doesn't sound so useful. https://julialang.org/downloads/platform.html says "It is strongly recommended that the official generic linux binaries from the downloads page be used to install Julia" and "The following distribution-specific packages are community contributed. They may not use the right versions of Julia dependencies or include important patches that the official binaries ship with. In general, bug reports will only be accepted if they are reproducible on the generic linux binaries on the downloads page." so why should the readme have redundant or contradictory statements about them?

Copy link
Member Author

@simonbyrne simonbyrne Feb 8, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree we should probably move them, but for different reasons: the README is a snapshot: it isn't that useful to know which packages were maintained with 0.6 when looking at the 0.7 README. I think better to move them to a maintainable place, such as a discourse wiki page, or even a subpage of https://julialang.org/downloads/, and mention that in the README.

Copy link
Member Author

@simonbyrne simonbyrne Feb 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which of these would people prefer?

Copy link
Member Author

@simonbyrne simonbyrne Feb 8, 2018 •
edited by nalimilan
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I didn't see https://julialang.org/downloads/platform.html, we can just use that.

Copy link
Member

@nalimilan nalimilan Feb 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, let's put that information on that page, with all warnings we may want to add, especially regarding the fact that distribution packages generally lag behind upstream since they only provide the lastest version which was available when the distro was released. Better acknowledge that these packages exist and inform users than ignore them and leave users on their own discovering the potential issues.

- Fedora: [official](https://src.fedoraproject.org/rpms/julia), [Copr (for backports)](https://copr.fedorainfracloud.org/coprs/nalimilan/julia/)
- [Arch](https://www.archlinux.org/packages/community/x86_64/julia/)
- [Gentoo](https://packages.gentoo.org/packages/dev-lang/julia)
- [Raspbian](https://github.com/JuliaBerry/julia-raspbian)

Additionally, the following unmaintained packages may still be useful:
- [Homebrew](https://github.com/staticfloat/homebrew-julia)
- [Ubuntu PPA](https://launchpad.net/~staticfloat/+archive/ubuntu/juliareleases) (0.5)
- [Debian](https://packages.debian.org/sid/julia) (0.4)

### System Provided Libraries

If you already have one or more of these packages installed on your system, you can prevent Julia from compiling duplicates of these libraries by passing `USE_SYSTEM_...=1` to `make` or adding the line to `Make.user`. The complete list of possible flags can be found in `Make.inc`.

Please be aware that this procedure is not officially supported, as it introduces additional variability into the installation and versioning of the dependencies, and is recommended only for system package maintainers. Unexpected compile errors may result, as the build system will do no further checking to ensure the proper packages are installed.

### LLVM

The most complicated dependency is LLVM, for which we require version 3.9 with some additional patches (LLVM is not backward compatible). We recommend either:
Copy link
Member

@nalimilan nalimilan Feb 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to mention that these are upstream patches? Packagers really appreciate this kind of information.

Copy link
Contributor

@mschauer mschauer Feb 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ref line 379

- adding the patches to the LLVM 3.9 package of the distribution (note: these are backported upstream bug fixes, _not_ Julia-specific patches: all have been contributed into upstream LLVM), or
Copy link
Member

@vchuravy vchuravy Feb 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a note the the list of patches is available at deps/llvm.mk?

Maybe

julia/deps/llvm.mk

Lines 517 to 519 in 251a501

# Independent to the llvm version add a JL prefix to the version map
# Depends on `llvm-D31524-sovers_4.0` for LLVM_VER==3.9
$(eval $(call LLVM_PATCH,llvm-symver-jlprefix)) # DO NOT REMOVE
also needs special mentione. E.g. the Julia specific version of llvm is modified so that it can be used safely with other versions of llvm (4.0 or a system 3.9). This is only true for Julia 0.7+, before that collisions between the Julia version and the system version are likely.

Copy link
Member Author

@simonbyrne simonbyrne Feb 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe that note would be better if it appeared in the Makefile itself? (it does appear apart from the others)

Copy link
Member Author

@simonbyrne simonbyrne Feb 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vchuravy Would you mind adding something to that effect (since I don't quite understand these issues)?

Copy link
Member

@staticfloat staticfloat Feb 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What Valentine just said seems to conflict with the text; that sure seems like a julia-specific patch. (But does it matter if we're using distribution-packaged LLVM? We don't care about conflicts then ight?)

Copy link
Member

@nalimilan nalimilan Feb 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what this patch does exactly, but we should leave packagers deal with the issue of parallel installation of multiple LLVM versions. It's kind of complex, with some hacking and lots of options to set, and it's far beyond the scope of Julia-specific patches. If that's useful, you can point at how Fedora handles this: https://src.fedoraproject.org/rpms/llvm3.9/blob/master/f/llvm3.9.spec

- bundling a Julia-only LLVM library inside the Julia package.

A complete list of patches is available in `deps/llvm.mk`, and the patches themselves are in `deps/patches/

Using an unpatched or different version of LLVM will result in errors and/or poor performance. There are flags in the Makefiles for newer LLVM versions, but support for this should be regarded as experimental and not suitable for packaging.
Copy link
Member

@staticfloat staticfloat Feb 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand what this means; are you saying that there are flags I can set within the Makefile to say "I'm building with LLVM 3.9 without julia patches"?


### BLAS and LAPACK

As a high-performance numerical language, Julia should be linked to a multi-threaded BLAS and LAPACK, such as OpenBLAS, and _not_ the reference implementations which are the default on some systems.
Copy link
Member

@staticfloat staticfloat Feb 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Including an example here would be nice. Something like "For example, packages such as ATLAS or OpenBLAS will provide much higher performance than the default reference libblas".


### SuiteSparse

SuiteSparse is a special case, since it is typically only installed as a static library, while `USE_SYSTEM_SUITESPARSE=1` requires that it is a shared library. Running the script `contrib/repackage_system_suitesparse4.make` will copy your static system SuiteSparse installation into the shared library format required by Julia. `make USE_SYSTEM_SUITESPARSE=1` will then use the SuiteSparse that has been copied into Julia's directory, but will not build a new SuiteSparse library from scratch.
Expand Down