ES6 in view functions

[diachedelic]

I recently upgraded from CouchDB 1.6.1 to 3.2.2, and took a moment to clean up my view functions. In the following view function, all I did was change the var to a let:

function (doc) {
    let start;
    if (
        doc._id.indexOf("geometries_") === 0 &&
        doc.features.length > 0 &&
        doc.parentID
    ) {
        start = doc.features[0].geometry.coordinates;
        if (typeof start[0] !== "number") {
            start = start[0];
        }
        emit([0, start[0]], doc.parentID);
        emit([1, start[1]], doc.parentID);
    }
}

As a result, GET requests to this view began responding with an error:

{
  error: 'killed',
  reason: '{gen_server,call,\n' +
    '    [couch_index_server,\n' +
    '     {get_index,\n' +
    '         {couch_mrview_index,\n' +
    '             {mrst,\n' +
    '                 <<174,230,212,124,70,180,183,172,250,90,95,225,147,32,152,1>>,\n' +
    '                 nil,undefined,\n' +
    '                 <<"shards/00000000-7fffffff/events_near_me_c3sc8-7r.1663034046">>,\n' +
    '                 <<"_design/geometries">>,<<"javascript">>,[],false,\n' +
    '                 {[]},\n' +
    '                 [{mrview,0,0,0,\n' +
    '                      [<<"by_parent">>],\n' +
    '                      [],\n' +
    '                      <<"function (doc) {\\n                if (doc._id.indexOf(\\"geometries_\\") === 0) {\\n                    if (doc.parentID) {\\n                        emit(doc.parentID);\\n                    }\\n                }\\n            }">>,\n' +
    '                      nil,[]},\n' +
    '                  {mrview,1,0,0,\n' +
    '                      [<<"location">>],\n' +
    '                      [],\n' +
    '                      <<"function (doc) {\\n                let start;\\n                if (\\n                    doc._id.indexOf(\\"geometries_\\") === 0 &&\\n                    doc.features.length > 0 &&\\n                    doc.parentID\\n                ) {\\n                    start = doc.features[0].geometry.coordinates;\\n                    if (typeof start[0] !== \\"number\\") {\\n                        start = start[0];\\n                    }\\n                    emit([0, start[0]], doc.parentID);\\n                    emit([1, start[1]], doc.parentID);\\n                }\\n            }">>,\n' +
    '                      nil,[]}],\n' +
    '                 nil,0,0,undefined,undefined,undefined,undefined,undefined,\n' +
    '                 nil},\n' +
    '             <<"shards/00000000-7fffffff/events_near_me_c3sc8-7r.1663034046">>,\n' +
    '             <<174,230,212,124,70,180,183,172,250,90,95,225,147,32,152,1>>}},\n' +
    '     infinity]}',
  ref: 2175788010
}

Reverting to the var solves the issue. I have tested this both with a native MacOS installation and the official Docker image, with identical results.

Calling GET /_node/_local/_versions gives:

{
  "javascript_engine": {
    "version": "78",
    "name": "spidermonkey"
  },
  ....
}

From what I could discern, SpiderMonkey v78 was used in Firefox v78, which certainly supported the let statement. So shouldn't let be supported in view functions? Have I missed something?

[nickva]

@diachedelic I think it could be related to the transformation that happens in https://github.com/apache/couchdb/blob/main/share/server/60/rewrite_fun.js

There, we parse js functions into an AST, patch them to turn "function(...) {...}" into a proper JS syntax (CouchDB global "function(...) {...}" it turns out are not valid JS), and then back to a js source string. It could be that the parsers we use esprima and escodegen don't handle ES6 syntax. That's just a guess so far, I haven't validated it yet.

A related issue perhaps as well: #2372

[diachedelic]

Sorry, I have mischaracterised this error. It appears to actually be a load-sensitive, intermittent failure that happened to correspond perfectly with me switching var and let a few times. I still don't understand why the error is occurring, but I don't think it is caused solely by using ES6 features in view functions.

[nickva]

Ah no worries, thanks for reaching out.

There was an issue related to performance regression with custom reduces with later (> 1.8.5. spidermonkey) #3517. Something to be aware when upgrading from older CouchDB instances.

[diachedelic]

I was not using custom reducers, but the error certainly occurred more often when I used let instead of var. Perhaps that was just because let made SpiderMonkey work harder.

[nickva]

That could be, it's hard to saw without more details metrics. That parser/emitter piece of code might have to work harder some newer code constructs like let.

[diachedelic]

Is there a way to profile CouchDB's execution of JavaScript? It would be interesting to know where the bottleneck is.

[nickva]

couchjs processes are run as a pool so it maybe possible to compare the CPU usage of those couchjs processes before and after.

Or run a benchmark building a view on both setups and see which takes longer.

With the 2.x/3.x versions, setting the Q sharding factor could help parallelize view builds, so with Q=16 if you have a large database, all 16 shard ranges (and 3 copies if you have a 3 node cluster) will build in parallel.

[diachedelic]

Thanks, that's good to know. If I run into unavoidable performance problems I will give it a shot.

Something else I would like to know is why the view functions need to be transpiled for newer SpiderMonkeys, but not for 1.8.5. Is there a discussion anywhere I can read more about this?

[diachedelic]

Oh, I found #2345, the PR for upgrading to SpiderMonkey 60 from 1.8.5. Something I still don't understand is why a view function like function(doc) { ... } ceased to be valid. It's just a function expression, why would that stop working with a newer version of SpiderMonkey? There is mention of a "new style" of view function, like (function(doc) { ... }); but that makes even less sense to me, because it is an expression statement which does nothing.

[diachedelic]

Oh, I found https://issues.apache.org/jira/browse/COUCHDB-1643.