cpd visitor: IllegalArgumentException: 208 is not a valid line offset for pointer

[benrsx]

Hi,

I run sonar cxx 0.9.7 analysis with sonar 5.6.1 and sonnar-scanner 2.7 and got a problem when parsing multi-lines string.

ERROR: Error during SonarQube Scanner execution
org.sonar.squidbridge.api.AnalysisException: Unable to analyze file: /var/projects/sonar/x/x/x/x/x.cxx
at org.sonar.squidbridge.AstScanner.scanFiles(AstScanner.java:131)
at org.sonar.plugins.cxx.squid.CxxSquidSensor.execute(CxxSquidSensor.java:128)
at org.sonar.batch.sensor.SensorWrapper.analyse(SensorWrapper.java:57)
at org.sonar.batch.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:58)
at org.sonar.batch.phases.SensorsExecutor.execute(SensorsExecutor.java:50)
at org.sonar.batch.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:83)
at org.sonar.batch.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:192)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)
at org.sonar.batch.scan.ProjectScanContainer.scan(ProjectScanContainer.java:241)
at org.sonar.batch.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:236)
at org.sonar.batch.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:226)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)
at org.sonar.batch.task.ScanTask.execute(ScanTask.java:47)
at org.sonar.batch.task.TaskContainer.doAfterStart(TaskContainer.java:86)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)
at org.sonar.batch.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:106)
at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:119)
at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
at com.sun.proxy.$Proxy0.execute(Unknown Source)
at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:244)
at org.sonarsource.scanner.api.EmbeddedScanner.runAnalysis(EmbeddedScanner.java:154)
at org.sonarsource.scanner.cli.Main.runAnalysis(Main.java:110)
at org.sonarsource.scanner.cli.Main.execute(Main.java:72)
at org.sonarsource.scanner.cli.Main.main(Main.java:60)
Caused by: java.lang.IllegalArgumentException: 208 is not a valid line offset for pointer. File [moduleKey=x:master, relative=x/x/x/x.cxx, basedir=/var/projects/sonar/x] has 72 character(s) at line 1089
at org.sonar.api.internal.google.common.base.Preconditions.checkArgument(Preconditions.java:148)
at org.sonar.api.batch.fs.internal.DefaultInputFile.checkValid(DefaultInputFile.java:218)
at org.sonar.api.batch.fs.internal.DefaultInputFile.newPointer(DefaultInputFile.java:209)
at org.sonar.api.batch.fs.internal.DefaultInputFile.newRange(DefaultInputFile.java:240)
at org.sonar.plugins.cxx.cpd.CxxCpdVisitor.visitToken(CxxCpdVisitor.java:78)
at com.sonar.sslr.impl.ast.AstWalker.visitToken(AstWalker.java:107)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:86)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.walkAndVisit(AstWalker.java:69)
at org.sonar.squidbridge.AstScanner.scanFiles(AstScanner.java:110)
... 31 more

The string in th cxx file is
sscanf( ligne, "%s%3s%d%d%lf%d%d%lf%lf%lf%lf%lf%lf%lf%lf%lf%lf"
"%d%d%d/%d/%d%d/%d/%d%d%d%d%d/%d/%d%d/%d/%d%d%d%d%d/%d/%d%d/%d/%d%d"
"%d%d%d/%d/%d%d/%d/%d%d%d%d%d/%d/%d%d/%d/%d%d%d%d%d/%d/%d%d/%d/%d%d%d\n", ...

The problem is that the string token is joined in one line and his length differ from the source line length.

If I remove JoinStringsPreprocessor.java this error disappear

[guwirth]

@robenj please try again with latest snapshot. #945 should catch such an exception.

[scmbuildguy]

Hi @guwirth,

I get the same error with the latest code.

This was working in v0.9.6 and now it doesn't seem to handle multiple line strings without some addtional syntax. Seems as though commit 3693569 could be an issue, however, that is only an assumption.

Example:

Original test code:
string str = "hello"
"world"

I fixed it by modifying:
string str = "hello"
<< "world"

I have a lot of code with the original format, so I am hoping this can be fixed.

Best Regards,

--Mark

[benrsx]

@guwirth I tried with the latest snapshot, I can see the exception for highligthing

Highligthing error in file '/var/projects/sonar/x/x/x/x/x.cxx' at line:1089, column:31

but my problem come just after

[guwirth]

Hi @robenj did a short test guwirth@3f3a591. It's working?

[benrsx]

I did a sonar scan on your test file after a clone and a build of the branch refactor/highlighting and I get the same error :

DEBUG: Highligthing error in file '/home/webadm/sonar-cxx-refactor-highlighting/sonar-cxx-plugin/src/test/resources/org/sonar/plugins/cxx/highlighter.cc' at line:79, column:21
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
INFO: Total time: 5.722s
INFO: Final Memory: 40M/116M
INFO: ------------------------------------------------------------------------
ERROR: Error during SonarQube Scanner execution
org.sonar.squidbridge.api.AnalysisException: Unable to analyze file: /home/webadm/sonar-cxx-refactor-highlighting/sonar-cxx-plugin/src/test/resources/org/sonar/plugins/cxx/highlighter.cc
at org.sonar.squidbridge.AstScanner.scanFiles(AstScanner.java:131)
at org.sonar.plugins.cxx.squid.CxxSquidSensor.execute(CxxSquidSensor.java:128)
at org.sonar.batch.sensor.SensorWrapper.analyse(SensorWrapper.java:57)
at org.sonar.batch.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:58)
at org.sonar.batch.phases.SensorsExecutor.execute(SensorsExecutor.java:50)
at org.sonar.batch.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:83)
at org.sonar.batch.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:192)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)
at org.sonar.batch.scan.ProjectScanContainer.scan(ProjectScanContainer.java:241)
at org.sonar.batch.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:236)
at org.sonar.batch.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:226)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)
at org.sonar.batch.task.ScanTask.execute(ScanTask.java:47)
at org.sonar.batch.task.TaskContainer.doAfterStart(TaskContainer.java:86)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)
at org.sonar.batch.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:106)
at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:119)
at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
at com.sun.proxy.$Proxy0.execute(Unknown Source)
at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:244)
at org.sonarsource.scanner.api.EmbeddedScanner.runAnalysis(EmbeddedScanner.java:154)
at org.sonarsource.scanner.cli.Main.runAnalysis(Main.java:110)
at org.sonarsource.scanner.cli.Main.execute(Main.java:72)
at org.sonarsource.scanner.cli.Main.main(Main.java:60)
Caused by: java.lang.IllegalArgumentException: 35 is not a valid line offset for pointer. File [moduleKey=PLUGIN, relative=resources/org/sonar/plugins/cxx/highlighter.cc, basedir=/home/webadm/sonar-cxx-refactor-highlighting/sonar-cxx-plugin/src/test] has 29 character(s) at line 79
at org.sonar.api.internal.google.common.base.Preconditions.checkArgument(Preconditions.java:148)
at org.sonar.api.batch.fs.internal.DefaultInputFile.checkValid(DefaultInputFile.java:218)
at org.sonar.api.batch.fs.internal.DefaultInputFile.newPointer(DefaultInputFile.java:209)
at org.sonar.api.batch.fs.internal.DefaultInputFile.newRange(DefaultInputFile.java:240)
at org.sonar.plugins.cxx.cpd.CxxCpdVisitor.visitToken(CxxCpdVisitor.java:82)
at com.sonar.sslr.impl.ast.AstWalker.visitToken(AstWalker.java:107)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:86)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.walkAndVisit(AstWalker.java:69)
at org.sonar.squidbridge.AstScanner.scanFiles(AstScanner.java:110)
... 31 more

[guwirth]

Hi there is

DEBUG: Highligthing error in file '/home/webadm/sonar-cxx-refactor-highlighting/sonar-cxx-plugin/src/test/resources/org/sonar/plugins/cxx/highlighter.cc' at line:79, column:21

which is in code https://github.com/SonarOpenCommunity/sonar-cxx/blob/master/sonar-cxx-plugin/src/main/java/org/sonar/plugins/cxx/highlighter/CxxHighlighter.java#L153. So exception is handled and only a debug log file item. Don't know why scanner is saying failure?

[benrsx]

The problem come from the token length in the file CxxCpdVisitor.java

After adding a try/catch the scan run without errors :

  try {
      TextRange range = inputFile.newRange(token.getLine(), token.getColumn(), token.getLine(), token.getColumn() + token.getValue().length());
      cpdTokens.addToken(range, text);
  }catch (Exception e){
      LOG.debug("Token error at line: {}, columns: {}, {} is not a valid line offset for pointer",token.getLine(),token.getColumn(),token.getColumn() + token.getValue().length());
  }

[guwirth]

Hi @robenj txh for the hint. Sorry my fault: mixed up the cpd and the highlighting fix - but rout cause is the same. Error is here: https://github.com/SonarOpenCommunity/sonar-cxx/blob/master/sonar-cxx-plugin/src/main/java/org/sonar/plugins/cxx/cpd/CxxCpdVisitor.java#L82. Will add a try/catch to avoid this.

Better solution would be to handle this inside of preprocessor. Point is only that current preprocessor is deprecated and I'm not sure if it is worth to spend the effort.