Uploaded image for project: 'SW'
  1. SW-683

H2OContext can't be initalized on Databricks cloud

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.23, 2.1.22, 2.2.8
    • Component/s: None
    • Labels:
      None
    • CustomerVisible:
      No

      Description

      In databricks environment, create cluster, add library - sparkling-water-assembly jar and create notebook with

      val hc = H2OContext.getOrCreate(spark).
      

      This will fail on class not found exception: org.spark_project.jetty.servlet...

      The reason for this is that Spark normally uses shaded jetty in package org.spark_project.jetty and we use it as well https://github.com/h2oai/sparkling-water/blob/81d36ce63794ae5aa60912f6a1533da235f1becf/core/src/main/scala/org/apache/spark/h2o/ui/SparklingWaterInfoPage.scala#L29

      However Databricks does not use this shaded jetty an uses regular jetty, that is why we see this exception.

      The first idea won't work - change shaded jetty to regular jetty in the class above as we are relocating H2O jetty ot a new package and this class would be relocated as well. We might try to forbid relocation for this particular class as here, we need to use just spark jetty.

      The current workaround is to disable SParkling Water UI Tab as

      val conf = new H2OConf(spark).set("spark.ui.enabled", "false")
      
      val hc = H2OContext.getOrCreate(spark, conf)
      

        Attachments

          Activity

            People

            • Assignee:
              Kuba Jakub Hava
              Reporter:
              Kuba Jakub Hava
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: