Upgrade to GAP 4.10

[nthiery]

GAP 4.10 comes with a completely rewritten build system that will simplify
our packaging. In particular, libGAP no longer needs to be a separate package.

What the branch does:

• Remove the libgap spkg

• Update the gap spkg to the new build system and build and install libgap

• Replace gap.shi.patch by a plain gap startup script for Sage

  Rationale: GAP used to provide a startup shell script. The GAP devs
  are in the process of getting rid of it and provide a very minimal
  one. They recommend to just write our own rather than patching it.

• Update a few doctests w.r.t. changes of output of some GAP functions

• Reworks how gap is installed in $SAGE_LOCAL: Rather than installing the entire source tree we just install the gap and libgap binaries to standard locations, and add a $SAGE_LOCAL/share/gap containing the GAP_ROOT_DIR which is stripped down to the minimum needed for GAP to work (standard libs and packages, docs, as well as the source code for introspection of kernel functions, but all build artifacts are carefully omitted).

  • Some of how this is done is still broken w.r.t. building compiled packages; need to work on it a bit more.
• • Possibly controversial: The new libgap currently 'does not come' with symbol rewriting (foobar -> libGAP_foobar). This avoids messing around with GAP's sources; in particular opening the door for using a stock GAP from the OS distribution. However there always is a risk of name conflicts. And indeed, GAP's enums T_INT, T_FLOAT, ... conflict with Python's constants defined in structmember.h. This is hopefully not actually a problem in practice due to the way how Cython orders includes.

  Something similar was started by Volker at Unprefixed libgap #19915.

• Removes configure.patch: it was patching configure for better GMP
  detection under Cygwin (New Gap spkg (>=4.5) does not build with shared only GMP/MPIR #13954). This should not be needed anymore
  with the new build system and use of --with-gmp. If it is, upstream
  asked for it to be reported and they will fix it.

• Removes additional patches for now--we would like to focus on being
  able to work with a system GAP as much as possible.

• Revert GAP: use newer version of config.guess #19726 (not needed anymore)

Status

Currently broken - crashes deep inside GAP error handling system after few simple commands.

Basic tests on libgap:

sage: libgap.eval("GAPInfo.Version")
sage: libgap.DihedralGroup(10).CharacterTable()
CharacterTable( <pc group of size 10 with 2 generators> )
sage: libgap.Group(libgap.eval("[(1,2,3),(1,2)]")).Size()
6

Running most relevant tests:

sage -tp 8 sage/groups sage/libs/gap

Current status: lots of errors

• Still have some miscellaneous segfaults and other weird crashes

• Still have work to do on improving error handling; replacing the built-in ErrorInner function might help here.

• SIGINT handling by Python is broken by GAP_Intialize; need to work around this.

Testing packages with dynamic loading (e.g. IO):

Install IO:

cd $SAGE_LOCAL/share/gap/pkg
wget https://www.gap-system.org/pub/gap/gap4/tar.gz/packages/io-4.5.4.tar.gz
tar xvf io-4.5.4.tar.gz
cd io-4.5.4
./configure --with-gaproot=../..
make

Test it locally:

cd $SAGE_ROOT
gap
gap> LoadPackage("IO");
true

This does not yet work:

#W dlopen() error: /home/embray/src/sagemath/sage/local/share/gap/pkg/io-4.5.4/bin/x86_64-pc-linux-gnu-default64/io.so: undefine\
d symbol: ChangedBags
Error, module '/home/embray/src/sagemath/sage/local/share/gap/pkg/io-4.5.4/bin/x86_64-pc-linux-gnu-default64/io.so' not found
Error, was not in any namespace
Syntax warning: Unbound global variable in /home/embray/src/sagemath/sage/local/share/gap/lib/init.g:803
    ColorPrompt( UserPreference( "UseColorPrompt" ) );
               ^
Error, SetGasmanMessageStatus: function is not yet defined
true

This should be fixed once GAP's gap binary is built on top of libgap.
See: markuspf/gap#1 I believe this is fixed, but there are still some problems with the way this ticket is "installing" GAP for $SAGE_LOCAL.

Note:

• Max Horn reviewed the list of GAP symbols we use in Sage; some have already changed in 4.9. See this pad for notes.

Upstream PRs

A few pull requests we made to improve use of GAP as a library:

• Add support for writing to ERROR_OUTPUT from kernel code gap-system/gap#3043
• Allow disabling the GAP kernel's signal handling when using the API gap-system/gap#3072
• Prototype for GAP_Enter/Leave macros gap-system/gap#3096
• prevent infinite recursions in echoandcheck and SyWriteandcheck gap-system/gap#3102

Other open issues that don't have PRs yet:

• libgap API: Problems with GASMAN and StackBottom gap-system/gap#3089
• libgap API: How much (if any) of GASMAN to expose? gap-system/gap#3030
• Possible infinite recursion in PRINT_OR_APPEND_TO_FILE_OR_STREAM gap-system/gap#3028

Some other PRs useful to this effort:

• Backport libgap api to 4.10 gap-system/gap#3065
• Add command line option --bare to start GAP without even needed packages (developer tool) gap-system/gap#2952

The update of gap_packages and database_gap is on #26856

Tarball: https://www.gap-system.org/pub/gap/gap-4.10/tar.gz/gap-4.10.0.tar.gz

Depends on #26874

CC: @alex-konovalov @dimpase @embray @kiwifb @antonio-rojas @sebasguts @jpflori @sagetrac-markuspf @nthiery @slel @vbraun @williamstein @timokau

Component: interfaces

Keywords: days85, libgap

Stopgaps: error handling in libgap, documentation display

Author: Nicolas M. Thiéry, Dima Pasechnik, Erik Bray, Jeroen Demeyer

Branch: b446ebb

Reviewer: Erik Bray, Dima Pasechnik, Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/22626

[nthiery]

Description changed:

--- 
+++ 
@@ -3,9 +3,34 @@
 so that we can get rid of our separate libgap package.
 
 The branch to be attached to this ticket is a preliminary experiment
-from a fork of Markus to be merged in the devel version of GAP.
+from a fork by Markus to be merged in the devel version of GAP.
 
-Fetching the Markus's GAP sources:
+
+What the branch does:
+- Remove the libgap spkg
+
+- Update the gap spkg to the new build system and build and install libgap
+
+- Replace gap.shi.patch by a plain gap startup script for Sage
+
+  Rationale: GAP used to provide a startup shell script. The GAP devs
+  are in the process of getting rid of it and provide a very minimal
+  one. They recommend to just write our own rather than patching it.
+
+- Update a few doctests w.r.t. changes of output of some GAP functions
+
+- ???
+
+TODO:
+
+- Automatic handling of headers (see below for how to do it by hand).
+  GAP's build system will eventuall provide a rule to install headers
+  which will make this trivial.
+
+- Use GAP's own `make install` when it will be implemented.
+
+
+Fetching Markus's GAP sources:
 
 ```
     git clone git@github.com:markuspf/gap.git $LIBGAP
@@ -42,9 +67,10 @@
     sage -f gap                      # -s
 ```
 
-Copy GAP's header files, as well as gen/config.h to $SAGE/local/include
-
-Fix them to adapt the include path: #include <src/...> -> #include <gap/...>
+Header files:
+- Copy GAP's header files, as well as gen/config.h to $SAGE/local/include
+- Fix them to adapt the include path: #include <src/...> -> #include <gap/...>
+- **Replace T_INT by 0 in TNUM_OBJ, around line 414 of objects.h**
 
 Run:
 
@@ -52,3 +78,52 @@
     sage -b
 ```
 
+Basic tests on libgap:
+
+```
+    sage: libgap.eval("GAPInfo.Version")
+    sage: libgap.DihedralGroup(10).CharacterTable()
+    CharacterTable( <pc group of size 10 with 2 generators> )
+    sage: libgap.Group(libgap.eval("[(1,2,3),(1,2)]")).Size()
+    6
+```
+
+Running most relevant tests:
+
+```
+    sage -tp 8 sage/groups sage/libs/gap
+```
+Current status: all tests pass!
+
+Testing packages with dynamic loading (e.g. IO):
+
+Install IO:
+
+```
+    cd $SAGE/local/gap/latest/pkg
+    wget http://www.gap-system.org/pub/gap/gap4/tar.gz/packages/io-4.4.6.tar.gz
+    tar xvf /tmp/io-4.4.6.tar.gz
+    mv io-4.4.6 io
+    cd io
+    ./configure
+    make
+```
+
+Test it locally:
+
+```
+    cd ../..
+    ./gap -l .
+    gap> LoadPackage("IO");
+    true
+```
+
+This does not yet work:
+
+```
+    sage: libgap.LoadPackage("IO")
+    ValueError: libGAP: Error, module '/opt/sage-git/local/gap/latest/pkg/io/bin/x86_64-pc-linux-gnu-gcc-default64/io.so' not found
+```
+This should be fixed once GAP's gap binary is built on top of libgap.
+See: https://github.com/markuspf/gap/issues/1.
+

[nthiery]

Description changed:

--- 
+++ 
@@ -18,6 +18,16 @@
   one. They recommend to just write our own rather than patching it.
 
 - Update a few doctests w.r.t. changes of output of some GAP functions
+
+- **Possibly controversial:** The new libgap currently *does not come*
+  with symbol rewriting (Foo -> libGAP_Foo). This avoids messing
+  around with GAP's sources; in particular opening the door for using
+  a stock GAP from the OS distribution. However there always is a risk
+  of name conflict. And indeed, GAP's constants (actually cpp macros)
+  T_INT, T_FLOAT, ... conflict with Python's constants. This is worked
+  around by duplicating and forcing their value in gap_include.pxd.
+  Another conflict in objects.h currently needs to be workedaround by
+  hand (see below).
 
 - ???
 

[nthiery]

Description changed:

--- 
+++ 
@@ -26,7 +26,7 @@
   of name conflict. And indeed, GAP's constants (actually cpp macros)
   T_INT, T_FLOAT, ... conflict with Python's constants. This is worked
   around by duplicating and forcing their value in gap_include.pxd.
-  Another conflict in objects.h currently needs to be workedaround by
+  Another conflict in objects.h currently needs to be worked around by
   hand (see below).
 
 - ???

[nthiery]

Description changed:

--- 
+++ 
@@ -39,6 +39,8 @@
 
 - Use GAP's own `make install` when it will be implemented.
 
+- Update the documentation in sage.libs.gap.libgap.pyx to not mention
+  the `libgap_` prefix
 
 Fetching Markus's GAP sources:
 

[nthiery]

Description changed:

--- 
+++ 
@@ -28,6 +28,11 @@
   around by duplicating and forcing their value in gap_include.pxd.
   Another conflict in objects.h currently needs to be worked around by
   hand (see below).
+
+- Removes configure.patch: it was patching configure for better GMP
+  detection under Cygwin (#13954). This should not be needed anymore
+  with the new build system and use of --with-gmp. If it is, upstream
+  asked for it to be reported and they will fix it.
 
 - ???
 

[nthiery]

Branch: u/nthiery/upgrade_to_gap_4_9

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

1c645e0	#22626: remove GAP's symbol prefixing in libgap: libGAP_Foo -> Foo
1ecc67b	#22626: doctest update w.r.t. minor changes of output in GAP
e8ebbca	#22626: GMP detection patch for cygwin should not be needed anymore
fd65b06	#22626: Remove libgap spkg
9511733	#22626: replace patch for GAP's startup script template in favor of a custom script
550625e	#22626: remove GAP's symbol prefixing in libgap: libGAP_Foo -> Foo, and workaround GAP <-> Python symbol conflict
a9c859c	#22626: updated gap spkg w.r.t. GAP's devel version and its new build system; also include compilation and installation of libgap
3011ac0	Merge branch 'develop' into t/22626/upgrade_to_gap_4_9

[sagetrac-git]

Commit: 3011ac0

[nthiery]

Description changed:

--- 
+++ 
@@ -1,9 +1,12 @@
 GAP 4.9 will come with a completely rewritten build system that will
-simplify our packaging. It will also enable building GAP as a library,
-so that we can get rid of our separate libgap package.
+simplify our packaging. In fact, it may well enable Sage to use a vanilla GAP installation as provided by the distribution. It will also enable building GAP as a library, so that we can get rid of our separate libgap package.
 
-The branch to be attached to this ticket is a preliminary experiment
-from a fork by Markus to be merged in the devel version of GAP.
+The branch attached to this ticket updates Sage to run on top of
+[a branch of GAP](https://github.com/markuspf/gap/tree/hpc-merge-libgap)
+by Markus Pfeiffer that adds libgap compilation and will merged soon
+in the devel version of GAP.
+
+Tentatively, all tests pass!
 
 
 What the branch does:

[nthiery]

Description changed:

--- 
+++ 
@@ -3,7 +3,7 @@
 
 The branch attached to this ticket updates Sage to run on top of
 [a branch of GAP](https://github.com/markuspf/gap/tree/hpc-merge-libgap)
-by Markus Pfeiffer that adds libgap compilation and will merged soon
+by Markus Pfeiffer that adds libgap compilation and will be merged soon
 in the devel version of GAP.
 
 Tentatively, all tests pass!

[nthiery]

Description changed:

--- 
+++ 
@@ -3,7 +3,7 @@
 
 The branch attached to this ticket updates Sage to run on top of
 [a branch of GAP](https://github.com/markuspf/gap/tree/hpc-merge-libgap)
-by Markus Pfeiffer that adds libgap compilation and will be merged soon
+by Markus Pfeiffer that adds libgap compilation and [|will be merged](https://github.com/fingolfin/gap/pull/64) soon
 in the devel version of GAP.
 
 Tentatively, all tests pass!