I am using AWS EMR and I can't seem to get hadoop streaming to run using the emr-s3-io input formatters. My custom output formats work fine, but input format seems to blow up in Configuration.setClass where it tests "isAssignableFrom". I have included the emr-s3-io jar in my customer-profiles-1.0.jar which includes all my custom extensions.
I am running:
AMI version:2.4.6
Hadoop distribution: 1.0.3
I think there must be something wrong with the way I am running the job as I can't seem to specify any inputformat class from the command line? Can you help me? Even the TextInputFormat - which should be the default, blows up in the same place when specified via -inputformat.
Here is the command I am running.
hadoop jar /home/hadoop/contrib/streaming/hadoop-streaming.jar -libjars /home/hadoop/customer-profiles-1.0.jar -input 's3:////////' -output s3:/// -inputformat com.atlantbh.hadoop.s3.io.S3ObjectSummaryInputFormat -mapper s3n:///map_keys.rb -reducer aggregate
Gives this exception.
Exception in thread "main" java.lang.RuntimeException: class com.atlantbh.hadoop.s3.io.S3ObjectSummaryInputFormat not org.apache.hadoop.mapred.InputFormat
at org.apache.hadoop.conf.Configuration.setClass(Configuration.java:982)
at org.apache.hadoop.mapred.JobConf.setInputFormat(JobConf.java:600)
at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:798)
at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:122)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
I am using AWS EMR and I can't seem to get hadoop streaming to run using the emr-s3-io input formatters. My custom output formats work fine, but input format seems to blow up in Configuration.setClass where it tests "isAssignableFrom". I have included the emr-s3-io jar in my customer-profiles-1.0.jar which includes all my custom extensions.
I am running:
AMI version:2.4.6
Hadoop distribution: 1.0.3
I think there must be something wrong with the way I am running the job as I can't seem to specify any inputformat class from the command line? Can you help me? Even the TextInputFormat - which should be the default, blows up in the same place when specified via -inputformat.
Here is the command I am running.
hadoop jar /home/hadoop/contrib/streaming/hadoop-streaming.jar -libjars /home/hadoop/customer-profiles-1.0.jar -input 's3:////////' -output s3:/// -inputformat com.atlantbh.hadoop.s3.io.S3ObjectSummaryInputFormat -mapper s3n:///map_keys.rb -reducer aggregate
Gives this exception.
Exception in thread "main" java.lang.RuntimeException: class com.atlantbh.hadoop.s3.io.S3ObjectSummaryInputFormat not org.apache.hadoop.mapred.InputFormat
at org.apache.hadoop.conf.Configuration.setClass(Configuration.java:982)
at org.apache.hadoop.mapred.JobConf.setInputFormat(JobConf.java:600)
at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:798)
at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:122)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:187)