module initialization error

[pputhran]

Hello,

I am trying to use AWS lambda function to read off Kinesis stream and insert into Google Big Query.
It appears to work fine locally (on my mac) but when I deploy this to AWS and execute as AWS lambda function, I run into initialisation error

error stack:

module initialization error: Error at Error (native)
  at Object.Module._extensions..node (module.js:434:18)
  at Module.load (module.js:343:32)
  at Function.Module._load (module.js:300:12)
  at Module.require (module.js:353:17)
  at require (internal/module.js:12:17)
  at Object.<anonymous> (/var/task/node_modules/gcloud/node_modules/grpc/src/node/src/grpc_extension.js:40:15)
  at Module._compile (module.js:409:26)
  at Object.Module._extensions..js (module.js:416:10)
  at Module.load (module.js:343:32)

Appears to be failing to invoke this
gcloud/node_modules/grpc/src/node/extension_binary/grpc_node.node

Below is the sample code

Sample code:

var gcloud = require('gcloud');

module.exports.handler = function(event, context, cb) {

 var bigquery = gcloud.bigquery({
    projectId: process.env.GCLOUD_PROJECT_ID,
    keyFilename: './keys.json'
  });

  event.Records.forEach(function(record) {
    // Kinesis data is base64 encoded so decode here
    var payload = new Buffer(record.kinesis.data, 'base64').toString('ascii');
    console.log('Decoded payload:', payload);
  });
  cb(null, "message");
};

[jgeewax]

This very well could be an issue with gRPC not being able to run on Lambda (it uses a pre-compiled binary and needs some fancy support).

@stephenplusplus : Is there a way to wrap the gRPC requirement with a try/except since it really should be an optional module?

Seems we should fail gracefully if gRPC doesn't import correctly (ie, start the change here https://github.com/GoogleCloudPlatform/gcloud-node/blob/master/lib/pubsub/index.js#L32) ?

[stephenplusplus]

Is there a way to wrap the gRPC requirement with a try/except since it really should be an optional module?

We haven't designed it as an optional dependency thus far, i.e. we don't fallback to HTTP via the request module. Once we added support for gRPC to an API, we removed the old stuff. If that's how it should be, we would declare gRPC as an optionalDependency, so that if it fails to install / be require'd (try/catch like you said), we would fall back to request. Should we go in that direction or try to make gRPC compatible with the Lambda environment?

[murgatroid99]

From the Lambda documentation, it looks like it shouldn't be too hard to get Node gRPC working on Lambda. In fact, I'm not really sure why it doesn't already work. I can look into this, but maybe not immediately.

[stephenplusplus]

@pputhran thanks for reporting. You can track this over in grpc/grpc#6443.

@jgeewax if you'd like us to have a fallback in place if gRPC cannot be installed, please open a new issue.