The given sequenceToken is invalid errors

[vaijab]

I am getting errors like one below:

Oct 12 13:37:08 ip-10-50-0-244.eu-west-1.compute.internal docker[4880]: The given sequenceToken is invalid. The next expected sequenceToken is: 49545729770787039968577718761852552028493222699855781714 {:level=>:warn, :exception=>Aws::CloudWatchLogs::Errors::InvalidSequenceTokenException, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.1.14/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.1.14/lib/aws-sdk-core/plugins/param_converter.rb:21:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.1.14/lib/seahorse/client/plugins/response_target.rb:21:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.1.14/lib/seahorse/client/request.rb:70:in `send_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.1.14/lib/seahorse/client/base.rb:207:in `put_log_events'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:159:in `put_log_events'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:139:in `flush'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:138:in `flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:102:in `register'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:338:in `deq'", "org/jruby/RubyKernel.java:1511:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:333:in `deq'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:101:in `register'"]}

I have multiple instances running logstash and pushing to the same LOG_GROUP_NAME/LOG_STREAM_NAME. Would running multiple logstash instances cause an error like the one above?

[wanghq]

Hi @vaijab, yes, that could cause above error. A log stream is sequence of log events from a single emitter of logs. You can read more about CloudWatch Logs concepts at http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatchLogs.html#CloudWatchLogsConcepts. Pointing multiple logstash instances to the same log stream could cause data duplicates and reduce the ingestion throughput.

Here is a simple example. If you have several apache servers, each generating access log and error log. Then you could have two log groups (e.g. AccessLog and ErrorLog) and use instance id as log stream name, and you end up with below log streams.

#log group/log stream

AccessLog/i-instance123
AccessLog/i-instance456
AccessLog/i-instance789

ErrorLog/i-instance123
ErrorLog/i-instance456
ErrorLog/i-instance789

[vaijab]

@wanghq thanks for the explanation. I had a feeling that was the case. I'd love to be able to create dynamic streams based on types of events, but see #1 why I cannot do that unfortunately.

Today, in containerized world instances don't mean much. Containers move around and can be short-lived.

[teebu]

i only have 1 output, and im seeing these errors.