g.extension: doesn't return list of available extensions in the official GRASS GIS Addons repository

[tmszi]

Describe the bug
g.extension doesn't return list of available extensions in the official GRASS GIS Addons repository.

To Reproduce
Steps to reproduce the behavior:

1. Launch g.extension -l
2. Doesn't return list of available extensions

Expected behavior
g.extension should return a list of available extensions

System description (please complete the following information):

• GRASS GIS version dev 8.0

Additional context
g.extension tries find module.xml file at this URL https://grass.osgeo.org/addons/grass8/modules.xml, but grass8 directory doesn't exists on the server, only grass6 and grass7 exists. Please check them using this URL https://grass.osgeo.org/addons/.

[neteler]

Additional context
g.extension tries find module.xml file at this URL https://grass.osgeo.org/addons/grass8/modules.xml, but grass8 directory doesn't exists on the server, only grass6 and grass7 exists. Please check them using this URL https://grass.osgeo.org/addons/.

This is stuck at OSGeo/grass-addons#613 which we need to implement asap.

[veroandreo]

Related to #1960

[tmszi]

Sorry I closed this issue prematurely, my mistake.

[gregNS]

This report relates to grass 7.8.6 but it seems related to this issue for v8.0

The works-for-me solution was to edit g.extension.py based on a posting found here. I do not have sufficient understanding of proper usage of SSL and python networking to suggest this is a good solution, only that it worked.

# gds: added import ssl
import ssl

def urlopen(url, *args, **kwargs):
    """Wrapper around urlopen. Same function as 'urlopen', but with the
    ability to define headers.
    """
    request = urlrequest.Request(url, headers=HEADERS)
    # gds edit: added ssl._create_default_https_context = ssl._create_unverified_context
    ssl._create_default_https_context = ssl._create_unverified_context
    return urlrequest.urlopen(request, *args, **kwargs)

Details:

I recently installed a completely new and updated osgeo4w bundle on my Windows 10 laptop.
Attempting to install extensions in grass failed with an error message indicating the package url could not be opened. i.e:

ERROR: Cannot open URL:
http://wingrass.fsv.cvut.cz/grass78/x86_64/addons/grass-7.8.6/r.in.pdal.zip

I'm not experienced with python networking code so added a bunch of grass.message(...) statements to identify the failure was occurring with return urlrequest.urlopen(request, *args, **kwargs) called from the urlopen in g.extension.py.

Running python from the grass command window and trying to retrieve the url directly using urllib in the interpreter produced an error related to an SSL failure...

>>> import urllib
>>> import urllib.request
>>> urllib.request.urlopen('http://wingrass.fsv.cvut.cz/grass78/x86_64/addons/grass-7.8.6/r.in.pdal.zip')
Traceback (most recent call last):
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 1253, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 1299, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 1248, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 1008, in _send_output
    self.send(msg)
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 948, in send
    self.connect()
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 1422, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "C:\OSGeo4W\apps\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\OSGeo4W\apps\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\OSGeo4W\apps\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: A failure in the SSL library occurred (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 523, in open
    response = meth(req, response)
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 632, in http_response
    response = self.parent.error(
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 555, in error
    result = self._call_chain(*args)
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 747, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 517, in open
    response = self._open(req, data)
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error A failure in the SSL library occurred (_ssl.c:1129)>

[hellik]

ERROR: Cannot open URL:
http://wingrass.fsv.cvut.cz/grass78/x86_64/addons/grass-7.8.6/r.in.pdal.zip

IIRC, modules/addons linking to pdal aren't available in winGRASS due to C++ compiling issues on incompatible build environment.

[gregNS]

Sorry if I distracted you...
g.extension -l failed to return any result and no extension would install when named on the command line.

After the previous noted change for the SSL failure I am able to have g.extension return the list and all extensions I attempted to install were successful. (including r.in.pdal)

[veroandreo]

I just updated the main branch and when running g.extension -l, I get:

List of available extensions (modules):
Fetching list of extensions from GRASS-Addons SVN repository (be patient)...
Traceback (most recent call last):
  File "/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 681, in list_available_modules
    tree = etree_fromurl(file_url)
  File "/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 383, in etree_fromurl
    file_ = urlopen(url)
  File "/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 235, in urlopen
    return urlrequest.urlopen(request, *args, **kwargs)
  File "/usr/lib64/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib64/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib64/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 2570, in <module>
    sys.exit(main())
  File "/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 2517, in main
    list_available_extensions(xmlurl)
  File "/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 556, in list_available_extensions
    list_available_modules(url)
  File "/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 693, in list_available_modules
    list_available_extensions_svn(url)
  File "/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 761, in list_available_extensions_svn
    sline = pattern.search(line)
TypeError: cannot use a string pattern on a bytes-like object

When I try to do the same in the GUI, i.e., go to Settings-> Addons extensions-> Install extension, I get this in the GUI console tab:

Exception in thread
Thread-4
:
Traceback (most recent call last):
  File "/usr/lib64/python3.9/threading.py", line 973, in
_bootstrap_inner

self.run()
  File
"/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-
gnu/gui/wxpython/core/gthread.py", line 154, in __run

self.__run_backup()
  File
"/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-
gnu/gui/wxpython/core/gthread.py", line 117, in run

ret = vars()["callable"](*args, **kwds)
  File
"/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-
gnu/gui/wxpython/modules/extensions.py", line 369, in Load

raise GException(_("Unable to load extensions. %s") % msg)
core.gcmd
.
GException
:
Unable to load extensions. Flag 'g' ignored, addons metadata
file not available

Installing extensions from the terminal works just fine though.

System Info:

GRASS version: 8.0.dev                                                          
Code revision: 563797cd3                                                        
Build date: 2021-12-27                                                          
Build platform: x86_64-pc-linux-gnu                                             
GDAL: 3.2.2                                                                     
PROJ: 7.2.1                                                                     
GEOS: 3.9.0                                                                     
SQLite: 3.34.1                                                                  
Python: 3.9.7                                                                   
wxPython: 4.0.7                                                                 
Platform: Linux-5.15.6-100.fc34.x86_64-x86_64-with-glibc2.33

[tmszi]

It seems, that problem is missing build GRASS GIS 8 version addons on CTU server, missing modules.xml and toolboxes.xml file, see this URL directory https://grass.osgeo.org/addons/grass8/. g.extension -l command point to this URL server.

In this line code is missing version 8, same here and here and etc., it seems that building GRASS GIS Addons (matching addons xml files) on CTU server doesn't count with version 8.

[neteler]

It seems, that problem is missing build GRASS GIS 8 version addons on CTU server, missing modules.xml and toolboxes.xml file, see this URL directory https://grass.osgeo.org/addons/grass8/. g.extension -l command point to this URL server.

Could we generate these (missing) files on grass.osgeo.org itself to not be dependent on remote files?

In this line code is missing version 8, same here and here and etc., it seems that building GRASS GIS Addons (matching addons xml files) on CTU server doesn't count with version 8.

Good catch! Maybe we need to introduce a config file which defines GRASS_VERSION_MIN and GRASS_VERSION_MAX or so, for better loops and a central place to configure it (with ´source config.txt` in the beginning of the respective shell files).

[tmszi]

Could we generate these (missing) files on grass.osgeo.org itself to not be dependent on remote files?

Yes we could. We will have to complete this PR and at the same time fix this issue #1960.

Good catch! Maybe we need to introduce a config file which defines GRASS_VERSION_MIN and GRASS_VERSION_MAX or so, for better loops and a central place to configure it (with ´source config.txt` in the beginning of the respective shell files).

Yes I agree a good idea.

[neteler]

Maybe we need to introduce a config file which defines GRASS_VERSION_MIN and GRASS_VERSION_MAX or so, for better loops and a central place to configure it (with ´source config.txt` in the beginning of the respective shell files).

Since I didn't find a sh/bash/... compatible way to parse a config.cfg file without defining an absolute path to it, for now I changed the scripts to use variables and define them on top of the scripts. See #655.