Create from SCP Receive.vi: "Memory Allocation Error - Unable to allocate memory, most likely the system is out of memory."

[ngblume]

When using the "Create from SCP Receive.vi" (for example within the SCP Download example), I get a "Memory Allocation Error" (-8102) at the Call Library Node (Function: lv_libssh2_scp_receive).
The problem is that this occurs only with some servers, not all and not at all times.
Working locally with an RPi works 100% of the time without problems.
Working with 2 other servers, I get this error 8 out of 10 times (roughly).

1. How can I fix this issue?
2. Is this a problem with the memory on my client machine or on the server? My understanding would be server-side since it oocurs with some machines, but not others.. But the help of the "Create from SCP Receive.vi" sounds as if this happens locally on the client..

Thanks !

Cheers
Niels

[volks73]

The Create from SCP Receive.vi is a "wrapper" around the lv_libssh2_scp_receive function from the libssh2lv project. When I look at the source code for the lv_libssh2_scp_receive function, it is a "thin wrapper" around the libssh2_scp_recv2 function and there are lines 91 to 95:

LIBSSH2_CHANNEL* inner = libssh2_scp_recv2(session->inner, path, file_info->inner);
if (inner == NULL) {
    return LV_LIBSSH2_STATUS_ERROR_MALLOC;
}

However, according to the documentation for the libssh2_scp_recv2 function:

Pointer to a newly allocated LIBSSH2_CHANNEL instance, or NULL on errors.

and there are three possible errors: LIBSSH2_ERROR_ALLOC, LIBSSH2_ERROR_SCP_PROTOCOL, and LIBSSH2_ERROR_EAGAIN.

The above C wrapper code in the lv_libssh2_scp_receive function is "masking" all errors from the libssh2_scp_recv2 function on a NULL value as a memory allocation error, when it could be a memory allocation error, a SCP Protocol error from the server, and a warning for non-blocking IO but the call would block (The LIBSSH2_ERROR_EAGAIN is not actually an error, it is used with non-blocking IO). This is a bug in the error handling within the libssh2lv C code. If the libssh2_scp_recv2 function returns NULL, then the error code should be checked and the appropriate error should be returned.

I will add an issue to the libssh2lv project to update the error handling. However, this is not the cause of your (@ngblume) issue but a complicating factor which is indicating a memory allocation error but really it is most likely a SCP Protocol error based on your description of conditions when running with two other servers. Unfortunately, the HTML documentation for the libssh2_scp_recv2() function for the LIBSSH2_ERROR_SCP_PROTOCOL error is empty and does provide any details on what could cause this error.

If it is in fact a LIBSSH2_ERROR_SCP_PROTOCOL error, then this is most likely something with the servers and not the client. I have had problems with newer SSH servers, but I believe that was related to supported cryptography algorithms, not with SCP protocol. The SCP protocol has been around for a long time and it does not change often or frequently.

It is also possible that the error is a LIBSSH2_ERROR_EAGAIN warning/error. Are you using non-blocking IO (Write Mode VI in the Channel API)?

Some additional debugging:

1. Are you able to use the scp command from the command line, external to LabVIEW and this package?
2. You mentioned a Raspberry Pi. Can you try the Read-Execute-Print-Loop with a Raspberry Pi example?
3. Can you try the Read-Execute-Print-Loop with a Raspberry Pi example but connect to the two other servers? It should work, there is nothing really special about the LabVIEW code that makes it only work with a Raspberry Pi. I just get a lot of requests to interface with a Raspberry Pi via SSH with LabVIEW using this package.
4. What type (OpenSSH?) and version of SSH server is running on the 2 problematic servers? Is this the same as the Raspberry Pi? If not, then there is something different between the SSH servers that is causing the SCP protocol error.
5. Can you install or use the server and version on the Raspberry Pi on the other servers?
6. Can you try the scp -vvv command from the command line with the three different servers and observe the output? The output might indicate where the three servers (Raspberry Pi and the other two) might differ.

[ngblume]

The above C wrapper code in the lv_libssh2_scp_receive function is "masking" all errors from the libssh2_scp_recv2 function on a NULL value as a memory allocation error, when it could be a memory allocation error, a SCP Protocol error from the server, and a warning for non-blocking IO but the call would block (The LIBSSH2_ERROR_EAGAIN is not actually an error, it is used with non-blocking IO). This is a bug in the error handling within the libssh2lv C code. If the libssh2_scp_recv2 function returns NULL, then the error code should be checked and the appropriate error should be returned.

If I understand you correctly, I'm not necessarily chasing a memory allocation issue, but it might be one of three options, right?

1. memory allocation error,
2. a SCP Protocol error from the server
3. a warning for non-blocking IO

Would a memory error nevertheless be on server or client-side?

Assuming, and I think your assumption here is justified, that my error is actual a "LIBSSH2_ERROR_SCP_PROTOCOL" error, I will try looking into the SSH logs on the servers in more detail. As of now, I was only able to find the connection, authentification and disconnect log entries (Doing disconnect even if transfer fails might be something good for examples as well, because otherwise the server needs to cleanup the left open sessions > reported as new issue #39 for improvement).

Regarding error "LIBSSH2_ERROR_EAGAIN": I just used the example, which would be a no for non-blocking IO...

Regarding your questions:

1. yes, for example with WinSCP
2. The RPi is the one working perfectly every time... (I'll see to try the exampe as soon as possible..)
3. Not sure, what you mean here
4. Server Versions

• RPi: "Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Raspbian-10+deb9u7 // debug1: match: OpenSSH_7.4p1 Raspbian-10+deb9u7 pat OpenSSH* compat 0x04000000" (based on "ssh -v localhost")
• server 1: "Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 // debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000" (based on "ssh -v localhost")
• server 2: "Remote protocol version 2.0, remote software version OpenSSH_7.4 // debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000" (based on "ssh -v localhost")

5. Need to check with IT
6. CMD on rpi is not recognized / accepted.. will try other server next..

Thanks for the help!
Will report with more debugging soon...

Cheers
Niels Göran

[ngblume]

Debugging-Addition:

• Server 3: "Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.10 // debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.10 pat OpenSSH*"
  Similiar error -8102 but this time without noticable waiting time (other 2 servers, the VI hangs for ~20 seconds)

• only the local server worked so far..
  Other were remote or via VPN
  Could this have someting to do with the issue ?

[volks73]

3. Can you try the Read-Execute-Print-Loop with a Raspberry Pi example but connect to the two other servers? It should work, there is nothing really special about the LabVIEW code that makes it only work with a Raspberry Pi. I just get a lot of requests to interface with a Raspberry Pi via SSH with LabVIEW using this package.

3. Not sure, what you mean here

There is the Read-Execute-Print-Loop with a Raspberry Pi example. It may not be available in the current release on VIPM.io, but you should be able to clone this repository's main branch, i.e. git clone https://github.com/fieldrndservices/libssh2-labview.git, to get the example. While the title/name of the example is "...with a Raspberry Pi", the REPL can be used with any remote SSH server. This would test and determine if the issue is specific to the SCP-related functions/VIs.

Thank you for the version output. To summarizes and confirm:

Remote Host	SSH Server Version	Connectivity	Works? (%)
RPi	OpenSSH v7.4	LAN	100
Server 1	OpenSSH v8.2	VPN	20
Server 2	OpenSSH v7.4	VPN	20
Server 3	OpenSSH v5.9	VPN	0

What is the client host, a desktop/laptop, RPi, PXI controller, or cRIO?

• only the local server worked so far..
  Other were remote or via VPN
  Could this have someting to do with the issue ?

By "local" do you mean a Local Area Network (LAN), i.e. sitting next to the client host connected either by a (managed) switch or cross-over cable, and by "remote" you mean a Wide Area Network (WAN), possibly even in a different office, building, city, country, etc., and must go through your organization's IT-managed network to the Internet?

I am trying to narrow down if your issue is specific to the SCP command, the LabVIEW code, the libssh2 C library, or a network issue with your servers. While this package is far from perfect and I greatly appreciate the bug notifications and feature request from you, I am leaning towards a network/connectivity issue with your servers and the VPN might have something to with it. The difficulty with remote/VPN servers and the delays all indicate some network configuration issue rather than something amiss with the code. Are you using IP addresses for the host addresses or hostnames or Fully-Qualified Domain Names (FQDN)? For example, if connecting from the client with the command line tools,

# IP addresses
ssh -vvv niels@192.168.1.5

# Hostnames/mDNS
ssh -vvv niels@raspberrypi.local
# or
ssh -vvv niels@raspberrypi

# FQDNs
ssh -vvv niels@your.organization.com

Is IPv6 enabled and being used instead of IPv4? Sometimes I have had issues if IPv6 is enabled with some embedded hardware and network connectivity.

There is an overall trend to move away from SCP and use SFTP instead. See (i) What's the difference between SFTP, SCP, and FISH protocols and (ii) OpenSSH 8.0 Release Notes. For your use case, can you use SFTP instead of SCP? The LIBSSH2-LabVIEW package supports the entire SFTP API implemented in the libssh2 library. There are examples for transferring files using SFTP:

1. Simple SFTP File Download
2. Simple SFTP File Upload

I believe SFTP uses the same port as SSH and the OpenSSH server does support the SFTP protocol. You should not have to change anything on the remote servers or enable through your IT department. If the SFTP-related Examples and/or VIs work with the remote servers, then it is something with the SCP-related implementation. However, if the SFTP-related Examples and/or VIs yield similar results, then this would be more evidence that it is something with the connectivity/networking/VPN.

[ngblume]

1. Can you try the Read-Execute-Print-Loop with a Raspberry Pi example but connect to the two other servers? It should work, there is nothing really special about the LabVIEW code that makes it only work with a Raspberry Pi. I just get a lot of requests to interface with a Raspberry Pi via SSH with LabVIEW using this package.

1. Not sure, what you mean here

There is the Read-Execute-Print-Loop with a Raspberry Pi example. It may not be available in the current release on VIPM.io, but you should be able to clone this repository's main branch, i.e. git clone https://github.com/fieldrndservices/libssh2-labview.git, to get the example. While the title/name of the example is "...with a Raspberry Pi", the REPL can be used with any remote SSH server. This would test and determine if the issue is specific to the SCP-related functions/VIs.

I will try to look at it more closely, but it seems, that the example pretty much does what I included in my test VI already regarding commands / buffers passed back and forth.

Thank you for the version output. To summarizes and confirm:

Remote Host SSH Server Version Connectivity Works? (%)
RPi OpenSSH v7.4 LAN 100
Server 1 OpenSSH v8.2 VPN 20
Server 2 OpenSSH v7.4 VPN 20
Server 3 OpenSSH v5.9 VPN 0
What is the client host, a desktop/laptop, RPi, PXI controller, or cRIO?

The client is a Windows host, the targets are some servers (1 RPi, rest: rack server)...

• only the local server worked so far..
  Other were remote or via VPN
  Could this have someting to do with the issue ?

By "local" do you mean a Local Area Network (LAN), i.e. sitting next to the client host connected either by a (managed) switch or cross-over cable, and by "remote" you mean a Wide Area Network (WAN), possibly even in a different office, building, city, country, etc., and must go through your organization's IT-managed network to the Internet?

yes, local means LAN across my WiFi router...
The rest were servers in the internet (private server) or in the company network via VPN

I am trying to narrow down if your issue is specific to the SCP command, the LabVIEW code, the libssh2 C library, or a network issue with your servers. While this package is far from perfect and I greatly appreciate the bug notifications and feature request from you, I am leaning towards a network/connectivity issue with your servers and the VPN might have something to with it. The difficulty with remote/VPN servers and the delays all indicate some network configuration issue rather than something amiss with the code. Are you using IP addresses for the host addresses or hostnames or Fully-Qualified Domain Names (FQDN)? For example, if connecting from the client with the command line tools,

I tried FQDN, as well as IPs..
I am trying to do the same by testing almost every combination...
I have a similar feeling regarding the connectivity.. but I somehow considered the various servers with identical issues somewhat as a pointer towards the lib in combination with connectivity...

Is IPv6 enabled and being used instead of IPv4? Sometimes I have had issues if IPv6 is enabled with some embedded hardware and network connectivity.

not sure, need to check... but I think the LAN is IPv6 (mostly) and WAN ist mostly IPv4..

I believe SFTP uses the same port as SSH and the OpenSSH server does support the SFTP protocol. You should not have to change anything on the remote servers or enable through your IT department. If the SFTP-related Examples and/or VIs work with the remote servers, then it is something with the SCP-related implementation. However, if the SFTP-related Examples and/or VIs yield similar results, then this would be more evidence that it is something with the connectivity/networking/VPN.

Sounds like a good idea..
Will try SFTP and see what happens...

Update:
Trying SFTP right now, and I'm missing the SFTP -Read All.vi.
It is in the newest commit on GitHub, but I can't seem to find it in the package installed via VI Package Manager (V1.1.1.25).
Is it missing there?

Is there a way to change the underlying lib to a more "chatty" one that might give a more detailled view as what to might be wrong?
I'm yet not able to find proper logs for SCP on any of my servers... > any hint here as to what might be worth looking at?

Thanks !

Cheers
Niels

[volks73]

It is in the newest commit on GitHub, but I can't seem to find it in the package installed via VI Package Manager (V1.1.1.25).
Is it missing there?

It is missing from v1.1.1, but it is available in the main branch. Looks like it was added as of afa6722, but that commit has not been released. I really should make a new release.

Is there a way to change the underlying lib to a more "chatty" one that might give a more detailled view as what to might be wrong? I'm yet not able to find proper logs for SCP on any of my servers... > any hint here as to what might be worth looking at?

There is the libssh2_trace() in the libssh2.org API, but it appears I have not wrapped it and exposed it for the libssh2lv project. I also don't think the libssh2lv library is built with tracing enabled. It is disabled by default. I have created an issue in the libssh2lv project to add the functionality, which can then be added to the LabVIEW API.