Memory usage

[avdi]

I have a program that's pretty simple: it downloads a list of remote video files; for each one, it downloads the file locally, then uploads it with a POST to a different host.

Running in a single process, this loop steadily increases the OS process RAM usage. The lasy time I killed it off it was at around 2.5GB resident / 4.5GB virtual.

I don't know anything about memory profiling in Erlang, and I don't know for sure that this is related to hackney. But since downloading and uploading is the only thing this program does involving large amounts of data, I was wondering if it was possible that hackney might hang on to data under some circumstances.

[benoitc]

can you show the code you're using?

[avdi]

I'll share the repo with with you privately. Warning: it's a huge mess.

[avdi]

Actually, for some reason Github makes it a huge pain to share my private repo with one person.

[avdi]

FWIW, I'm using get followed by a stream_body loop for the download, and I'm using a single request with the multipart file uploading support you just added for the upload.

I'm curious... should the final stream_body be followed by any kind of "close" action to get hackney to release resources?

I'll see if I can write a small script to reproduce this, but I don't know how soon I'll have time.

[benoitc]

no it shouldn't if the request is a keepalive. If then, the request is put in the pool.

Can you simply share the way you send the body? Are you using {file, Path} or do you put the body as a binary (and then ope it in memory?)

[behrad]

@benoitc I have facing a memory leakage in my application which is caused by hackney_manager ets table!
This is my memory dump which shows that 3.4G of 4.5G is taken by ets which also is not being freed after GC:

[{total,4514425040}, {processes,187347156}, {processes_used,187028896}, {system,4327077884}, {atom,867249}, {atom_used,850796}, {binary,790772584}, {code,17628238}, {ets,3443264344}]

And the ets dump shows:
hackney_manager hackney_manager set 2387668 422719141 hackney_manager hackney_pool hackney_pool set 2 314 hackney_manager

As you can see all ets memory is taken up by hackkey_manager. verifiable via 422719141 * erlang:system_info(wordsize)

As I understand, not all request ets records are being deleted!
My application is using couchbeam to connect to CouchDB.

@avdi You can also test if the same happens in your case.

[benoitc]

@behrad thanks it's a useful report!. Are you just using hackney:request or hackney:connect ? I'm on a thing right now but will work on hackney on the night.

[behrad]

@benoitc Hope it helps and I can patch it up my production server soon.
Actually none of them! I'm using couchbeam API within my code, and found the memory leakage after half a day in a production system. Debugged it and found that this is hackney eating ets memory by adding a record for each request! ( I have also reported benoitc/couchbeam#102 before this)

[brigadier]

As far as I can see, there's no memleak when you do hackney:request() + hackney:body(Client), ie read to the end.

There is a memleak when you do hackney:request() and either don't read the body altogether or use hackney:stream_body and do not read the body to the end, even if request was simple 'head'. In this case you must do hackney:close(), this call will remove ref from ets table.

If you do something like {ok, 200, _, Client} = hackney:request() you will always have a memory leak when :request() returns HTTP code other than 200, as an exception will be raised and Client ref will remain in ets table forever.

I didn't test the case when request is performed via proxy and something fail in between successful proxy:connect and failed actual request.

[benoitc]

I don't reproduce it either. But will add more guards asap

[benoitc]

f46a9d4 makes sure that the socket is closed.

[behrad]

@benoitc this is not also helping....
Any updates on this!? we have memory problems in production.

[benoitc]

@behrad any code that could help me to reproduce?

[behrad]

This is run in my gen_server init:

couchbeam:start(),
hackney_pool:set_max_connections( default, 2000 ),
hackney_pool:start_pool( opxi2_atm_db_pool, [{timeout,150000},{pool_size,2000}] ),

and this is all I do for any incoming request:

Server = couchbeam:server_connection(Url, []),
{ok, Db} = couchbeam:open_db(Server, DbName, []),
couchbeam:lookup_doc_rev(Db, DocId)

%%  in some occasions call this (call an update handler):
couchbeam_httpc:db_request(Method, UpdateUrl, Headers, JsonObj, [], [200,201])
JsonProp = couchbeam_httpc:json_body(Ref),
{Resp} = couchbeam_doc:set_value(<<"last_rev">>, LastRev, JsonProp),

%% in some other call this:
couchbeam_httpc:db_request(Method, UpdateUrl, Headers, JsonObj, [{pool,opxi2_atm_db_pool}], [200,201])

%% and call this at the end:
couchbeam_view:fold(...);

This is all on my usage of couchbeam. Should I call any dispose/shutdown/close anywhere !?

[benoitc]

are you reading the full body with couchbeam_httpc:db_request ?