"Incompatible expression types" or verification errors when using ternary expressions with null in one branch

[JoshRosen]

Attempting to compile

Integer result = false ? null : 1;

with ScriptEvaluator results in

org.codehaus.commons.compiler.CompileException: Line 1, Column 24: Incompatible expression types "void" and "int"
  org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11877)
  org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6912)
  org.codehaus.janino.UnitCompiler.access$15600(UnitCompiler.java:218)
  org.codehaus.janino.UnitCompiler$22$2.visitConditionalExpression(UnitCompiler.java:6438)
  org.codehaus.janino.UnitCompiler$22$2.visitConditionalExpression(UnitCompiler.java:6414)
  org.codehaus.janino.Java$ConditionalExpression.accept(Java.java:4570)
  org.codehaus.janino.UnitCompiler$22.visitRvalue(UnitCompiler.java:6414)
  org.codehaus.janino.UnitCompiler$22.visitRvalue(UnitCompiler.java:6393)
  org.codehaus.janino.Java$Rvalue.accept(Java.java:4171)
  org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6393)
  org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5585)
  org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2581)
  org.codehaus.janino.UnitCompiler.access$2700(UnitCompiler.java:218)
  org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:1506)
  org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:1490)
  org.codehaus.janino.Java$LocalVariableDeclarationStatement.accept(Java.java:3577)
  org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1490)
  org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1570)
  org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3398)
  org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1360)
  org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1333)
  org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:825)
  org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:435)
  org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:218)
  org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:414)
  org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:409)
  org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1417)
  org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:409)
  org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:381)
  org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)
  org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:466)
  org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:313)
  org.codehaus.janino.ScriptEvaluator.cook2(ScriptEvaluator.java:725)
  org.codehaus.janino.ScriptEvaluator.cook(ScriptEvaluator.java:714)
  org.codehaus.janino.ScriptEvaluator.cook(ScriptEvaluator.java:661)
  org.codehaus.janino.ScriptEvaluator.cook(ScriptEvaluator.java:629)
  org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
  org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)
  org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75)

with Janino 3.0.12.

Also, compiling

boolean cond;
Integer result = cond ? null : 1;

generates bytecode that triggers a verification error:

java.lang.VerifyError: Bad local variable type
Exception Details:
  Location:
    SC.eval0()V @0: iload_0
  Reason:
    Type top (current frame, locals[0]) is not assignable to integer
  Current Frame:
    bci: @0
    flags: { }
    locals: { }
    stack: { }
  Bytecode:
    0x0000000: 1a99 0007 01a7 0004 044c b1

  java.lang.Class.getDeclaredMethods0(Native Method)
  java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
  java.lang.Class.getDeclaredMethods(Class.java:1975)
  org.codehaus.janino.ScriptEvaluator.cook2(ScriptEvaluator.java:746)
  org.codehaus.janino.ScriptEvaluator.cook(ScriptEvaluator.java:714)
  org.codehaus.janino.ScriptEvaluator.cook(ScriptEvaluator.java:661)
  org.codehaus.janino.ScriptEvaluator.cook(ScriptEvaluator.java:629)
  org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
  org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)
  org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75)

This is a simplified version of a problem that we encountered when generating code in Apache Spark: apache/spark#24636 (comment)

[aunkrig]

Thanks for your precise bug description! Actually "b ? 7 : null" is a case that is not defined in JLS7!? Now I implemented it the same way as JAVAC.

Please test!

[JoshRosen]

Hi @aunkrig,

Thanks for the quick turnaround on this fix. I tried this out with the original Spark code and have run into a new problem:

Caused by: java.lang.VerifyError: Expecting a stackmap frame at branch target 20
Exception Details:
  Location:
    org/apache/spark/sql/catalyst/expressions/GeneratedClass$GeneratedIteratorForCodegenStage1.processNext()V @4: ifne
  Reason:
    Expected stackmap frame at this location.
  Bytecode:
    0x0000000: 2ab4 007d 9a00 102a 04b5 007d 2a2a b400
    0x0000010: 0eb8 007f 2ab4 005b 2ab4 0067 949a 0042
    0x0000020: 2ab4 0071 1400 8094 9e00 1614 0080 402a
    0x0000030: 59b4 0071 1400 8065 b500 71a7 0018 2ab4
    0x0000040: 0071 402a 0385 b500 711f 0385 949a 0006
    0x0000050: a701 1b2a 59b4 0067 1f0a 6961 b500 672a
    0x0000060: b400 672a b400 5b65 0a6d 883e 0336 04a7
    0x0000070: 00cd 1504 850a 692a b400 5b61 3705 0336
    0x0000080: 0702 3608 1605 8836 0815 0799 0007 01a7
    0x0000090: 0005 1508 3a09 013a 0a2a b400 8506 32c0
    0x00000a0: 0087 1909 b900 8b02 00c0 008d 3a0a a700
    0x00000b0: 183a 0bbb 0091 592a b400 8505 32c0 0093
    0x00000c0: 190b b700 96bf 190a c600 0703 a700 0404
    0x00000d0: 360c 0236 0d15 0c9a 000a 190a b600 9a36
    0x00000e0: 0d2a b400 2e04 32b6 009f 2ab4 002e 0432
    0x00000f0: b600 a22a b400 2e04 3203 150d b600 a52a
    0x0000100: 2ab4 002e 0432 b600 a9b6 00ad 2ab6 00b1
    0x0000110: 9900 292a 1605 0a61 b500 5b2a b400 8503
    0x0000120: 32c0 00b3 1504 0460 85b6 00b6 2ab4 002a
    0x0000130: 1504 0460 85b6 00bb b184 0401 1504 1da1
    0x0000140: ff33 2a2a b400 67b5 005b 2ab4 0085 0332
    0x0000150: c000 b31d 85b6 00b6 2ab4 002a 1d85 b600
    0x0000160: bb2a b400 1cb6 00be a7fe acb1
  Exception Handler Table:
    bci [153, 174] => handler: 177

You can find the generated Java code in this gist: https://gist.github.com/JoshRosen/416eac9d35bd027475cc787100b0bfe5

Line 100 is where the ternary expression fix was needed: https://gist.github.com/JoshRosen/416eac9d35bd027475cc787100b0bfe5#file-janino_90-java-L100

[aunkrig]

That's weird... according to the JVMS, the StackMapTable attribute is optional for class files with major version <= 50 (Java 6).
Could you please run java -version and copy the output here?
Also, could you provide the full command line of your JVM please?

[JoshRosen]

Here's java -version output:

$ java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

Maybe it's a problem with how I did a local mvn install of Janino? That local build was done using the above JDK and not a Java 6 JDK.

Instead, maybe I should set -source and -target explicitly in the Maven compiler plugin?

[aunkrig]

You may (experimentally) run your JVM with -noverify (or -Xverify:none) and see if it works then...

        // Must not set these to "..._1_7", because then the JVM insists on a StackMapTable (JVMS9 4.7.4), which
        // JANINO currently does not produce:
        //
        //    java.lang.VerifyError: Expecting a stackmap frame at branch target 13
        //
        // ("jre -noverify" removes that requirement, but we cannot force our users to run their JVMs with that
        // option.)
        this.majorVersion  = ClassFile.MAJOR_VERSION_JDK_1_6;
        this.minorVersion  = ClassFile.MINOR_VERSION_JDK_1_6;

[aunkrig]

Ping.

[JoshRosen]

Spark recently bumped its Janino version to 3.0.13, which contains this fix, but unfortunately it doesn't look like this resolves my issue. I now get a new exception like

[info] ScalaUDFSuite:
[info] - basic *** FAILED *** (1 second, 371 milliseconds)
[info]   Code generation of UDF(1) failed:
[info]   java.lang.VerifyError: Bad type on operand stack
[info]   Exception Details:
[info]     Location:
[info]       org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificMutableProjection.apply(Ljava/lang/Object;)Ljava/lang/Object; @6: astore_3
[info]     Reason:
[info]       Type integer (current frame, stack[0]) is not assignable to reference type
[info]     Current Frame:
[info]       bci: @6
[info]       flags: { }
[info]       locals: { 'org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificMutableProjection', 'java/lang/Object', 'org/apache/spark/sql/catalyst/InternalRow' }
[info]       stack: { integer }
[info]     Bytecode:
[info]       0x0000000: 2bc0 0018 4d04 4e01 3a04 2ab4 0022 0332
[info]       0x0000010: c000 2404 322a b400 2205 32c0 0026 2db9
[info]       0x0000020: 0028 0200 b900 2802 00c0 001a 3a04 a700
[info]       0x0000030: 183a 05bb 002c 592a b400 2204 32c0 002e
[info]       0x0000040: 1905 b700 32bf 1904 c600 0703 a700 0404
[info]       0x0000050: 3605 0236 0615 059a 000a 1904 b600 3636
[info]       0x0000060: 062a 1505 b500 3a2a 1506 b500 3e2a b400
[info]       0x0000070: 3a9a 0012 2ab4 0012 032a b400 3eb6 0042
[info]       0x0000080: a700 0b2a b400 1203 b600 452a b400 12b0
[info]       0x0000090:
[info]     Exception Handler Table:
[info]       bci [10, 46] => handler: 49

This happens when I change the code generator to

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
index 3274b66e98..27d08ead7c 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
@@ -1007,10 +1007,8 @@ case class ScalaUDF(
       case ((eval, i), dt) =>
         val argTerm = ctx.freshName("arg")
         val initArg = if (CatalystTypeConverters.isPrimitive(dt)) {
-          val convertedTerm = ctx.freshName("conv")
           s"""
-             |${CodeGenerator.boxedType(dt)} $convertedTerm = ${eval.value};
-             |Object $argTerm = ${eval.isNull} ? null : $convertedTerm;
+             |Object $argTerm = ${eval.isNull} ? null : ${eval.value};
            """.stripMargin
         } else {
           s"Object $argTerm = ${eval.isNull} ? null : $convertersTerm[$i].apply(${eval.value});"

[aunkrig]

Would you please set a system property -DUnitCompiler.disassembleClassFilesToStdout=true and retry? This should print disassemblies of all generated .class files to STDOUT, so we can analyze why this causes a VerifyError.