Memory Leak in History (and Active Flag does not to be working)

[marcushill]

My team has noticed a significant memory leak in version: seneca@3.4.3. We have a rather long running seneca action in which we make a large number (on the order of a few hundred thousand) calls to other seneca actions. Memory utilization would spike as expected, but the corresponding drop off after those results should have been freed from memory did not occur.

We took a number of heap dumps and it became clear that the Seneca History object contained the message (and response?) of every Seneca action we called. I believe this behavior is intended, but it doesn't ever clean up after itself over the course of the couple of hours that the job runs. I believe that this function has a bug in it that is causing prune not to delete messages as often as it should.

Also, setting history.active to false in the config does not fully disable history functionality:

{
   history: {
     active: false
   }
}

[AkaBrace]

I noticed the same issue with the history feature in Seneca when I snapped some memory heaps roughly a month ago. I didn't have the time to go hunt it down precisely though. Glad I wasn't the only one to notice this.

[jeromevalentin]

I agree the history is source of a major memory leak even when not activated!

@rjrodger The fix is really simple, you just have to patch seneca.js at line 1269 by adding one 'if' before adding data into the history. Like this:

  data.id = data.meta.id
  data.result = []
  data.timelimit = Date.now() + data.meta.timeout

  // The following test is missing
  if (opts.$.history.active) private$.history.add(data)

  if (opts.$.legacy.meta) {
    data.msg.meta$ = meta
  }

With that patch, the memory leak is fixed when setting history: { active : false } in seneca options

On the other hand, it seems that meta_trace and parent_meta_trace are also source of memory leaks, I had to patch them like this in my code to free all memory:

  import Seneca from 'seneca'

  Seneca.intern.meta_trace = () => {}
  Seneca.intern.parent_meta_trace = () => {}

Hope this helps

[marcushill]

I'm glad someone put the time into actually it. We solved our initial error and then priority shifted to other work.

For anyone who finds this before it is fixed:

The history holds to a full copy of every message until the message would have timed out, even after it has received a response. We were able to fix this problem for ourselves by shortening the timeout on requests made from our long running Seneca action.

[rjrodger]

Great catch! Thank you - fixed history disable

[suyuan1203]

@jeromevalentin "parent_meta_trace memory leak" could be a problem of Seneca plugin "this"

module.exports = function demo_plugin() {
  this.add('role: service1, cmd: get_name',  (msg, reply) => {
    this.act('role: service2, cmd: get_name', { name: msg.name }, function (err, result) {
      if (err) return reply(err);
      reply(null, {
        name: result.name
      });
    });
  });
}

Be careful, the second "this" is a reference of demo_plugin instance, instead of a new Seneca instance

Seneca creates a new Seneca instance for each action request. And it will be released after the request is finished. But a Seneca plugin will stay alive as soon as the service is alive.

function outward_trace(ctxt, data) {
  var private$ = ctxt.seneca.private$

  var meta = data.meta
  var reply_meta = data.reply_meta

  if (meta && reply_meta) {
    meta.trace = meta.trace || []
    meta.trace.push({
      desc: Common.make_trace_desc(reply_meta),
      trace: reply_meta.trace || []
    })
  }

  var parent_meta = private$.act && private$.act.parent
  if (parent_meta) {
    parent_meta.trace = parent_meta.trace || []
    parent_meta.trace.push({
      desc: Common.make_trace_desc(meta),
      trace: meta.trace || []
    })
  }
}

Here the parent of the inner action is always the plugin instance, so the "trace" array of the plugin will increase infinitely

It is by design that Seneca plugin is also a Seneca instance. And it just works even I reference the wrong "this".

@rjrodger Is there a way we can prevent bugs like this? Like a plugin's act method can only be called inside certain context?

[suyuan1203]

Since every act has a uniq id, it seems useless to keep a history of act object.

function inward_act_cache(ctxt, data) {
  var so = ctxt.options
  var meta = data.meta

  var actid = meta.id
  var private$ = ctxt.seneca.private$

  if (actid != null && so.history.active) {
    var actdetails = private$.history.get(actid)
    if (actdetails) {
    }
  }
}

if (actdetails) could always be false.

[tswaters]

We had a massive memory leak in prod - resulting in OOMKILL every day or so for certain services, far more frequent deaths when the system was under load.... tracked it down to plugins that used arrow functions instead of function and called act on the plugin's instance. This should definitely be in a readme somewhere.