Skip to content

[Java] Update Scala for JDK 11 compatibility #2279

Description

@asfimport

When trying to build the parquet-mr code on OSX Mojave with OpenJDK 10 and 9, the build fails for me in parquet-scala with:

[INFO] --- maven-scala-plugin:2.15.2:compile (default) @ parquet-scala_2.10 ---
[INFO] Checking for multiple versions of scala
[INFO] includes = [**/*.java,**/*.scala,]
[INFO] excludes = []
[INFO] /Users/uwe/tmp/apache-parquet-1.11.0/parquet-scala/src/main/scala:-1: info: compiling
[INFO] Compiling 1 source files to /Users/uwe/tmp/apache-parquet-1.11.0/parquet-scala/target/classes at 1547922718010
[ERROR] error: error while loading package, Missing dependency 'object java.lang.Object in compiler mirror', required by /Users/uwe/.m2/repository/org/scala-lang/scala-library/2.10.6/scala-library-2.10.6.jar(scala/package.class)
[ERROR] error: error while loading package, Missing dependency 'object java.lang.Object in compiler mirror', required by /Users/uwe/.m2/repository/org/scala-lang/scala-library/2.10.6/scala-library-2.10.6.jar(scala/runtime/package.class)
[ERROR] error: scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.
[ERROR] at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
[ERROR] at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.getClassByName(Mirrors.scala:99)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.getRequiredClass(Mirrors.scala:102)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:264)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:264)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.AnyRefClass$lzycompute(Definitions.scala:263)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.AnyRefClass(Definitions.scala:263)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.specialPolyClass(Definitions.scala:1120)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.RepeatedParamClass$lzycompute(Definitions.scala:407)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.RepeatedParamClass(Definitions.scala:407)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses$lzycompute(Definitions.scala:1154)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses(Definitions.scala:1152)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode$lzycompute(Definitions.scala:1196)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode(Definitions.scala:1196)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1261)
[INFO] at scala.tools.nsc.Global$Run.<init>(Global.scala:1290)
[INFO] at scala.tools.nsc.Driver.doCompile(Driver.scala:32)
[INFO] at scala.tools.nsc.Main$.doCompile(Main.scala:79)
[INFO] at scala.tools.nsc.Driver.process(Driver.scala:54)
[INFO] at scala.tools.nsc.Driver.main(Driver.scala:67)
[INFO] at scala.tools.nsc.Main.main(Main.scala)
[INFO] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[INFO] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[INFO] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[INFO] at java.base/java.lang.reflect.Method.invoke(Method.java:564)
[INFO] at org_scala_tools_maven_executions.MainHelper.runMain(MainHelper.java:161)
[INFO] at org_scala_tools_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)

This is because the referenced JARs were made for a Scala version not supporting JDK 11, we need to update to 2.12.

Reporter: Uwe Korn / @xhochy
Assignee: Uwe Korn / @xhochy

Related issues:

PRs and other links:

Note: This issue was originally created as PARQUET-1496. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Mar 26, 2019

    @asfimport
    Author

    Zoltan Ivanfi / @zivanfi:
    There seems to be an unresolvable circular incompatibility issue here:

    • Java 11 is incompatible with Scala 2.10, needs newer version, like Scala 2.12
    • Scala 2.12 is incompatible with Scrooge 4, needs newer version, like Scrooge 19.
    • Scrooge 19 is incompatible with our parquet.thrift file for two reasons:
      • It doesn't handle one of our empty structs correctly. Update: This turned out to be due to using a javadoc-style comment in the empty struct.
      • It doesn't handle the String logical type correctly, because in the code it generates it does not use fully qualified names. Since the name of this logical type shadows the stock String type, this leads to a compilation failure in the generated LogicalType.scala file.
  2. asfimport commented on Mar 27, 2019

    @asfimport
    Author

    Zoltan Ivanfi / @zivanfi:
    According to https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html, Scala 2.10.7 also supports JDK 11, which may provide a resolution for this problem. Update: No, it doesn't. Parquet-scrooge doesn't compile with Scala 2.10.7 either.

  3. asfimport commented on Mar 29, 2019

    @asfimport
    Author

    Zoltan Ivanfi / @zivanfi:
    I have opened Scrooge#303 and Scrooge#304 about these issues.

  4. asfimport commented on May 3, 2019

    @asfimport
    Author

    Nándor Kollár / @nandorKollar:
    Scrooge recently release 19.4.0 with a fix for Scrooge#304, we can now upgrade to this version. The other problem (Scrooge#303) is fixed by replacing the problematic comment in parquet-format, I'm afraid we'll need a release from format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions