Uploaded image for project: 'SW'
  1. SW-347

Cannot start Sparkling Water at HDP Yarn cluster

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.9, 2.0.6, 2.1.1
    • Component/s: None
    • Labels:
      None
    • CustomerVisible:
      No

      Description

      michal@mr-0xd1:~/tmp/sparkling-water-2.1.0$ bin/sparkling-shell --master yarn-client --conf hadoop.yarn.timeline-service.enabled=false
      
      -----
        Spark master (MASTER)     : yarn-client
        Spark home   (SPARK_HOME) : /home/michal/tmp/spark-2.1.0-bin-hadoop2.6
        H2O build version         : 3.10.3.2 (tverberg)
        Spark build version       : 2.1.0
        Scala version             : 2.11
      ----
      
      Warning: Ignoring non-spark config property: hadoop.yarn.timeline-service.enabled=false
      Warning: Master yarn-client is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
      Setting default log level to "WARN".
      To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
      17/03/07 18:03:43 WARN spark.SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0
      17/03/07 18:03:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      17/03/07 18:03:45 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
      java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
        at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:45)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:163)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:151)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
        at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
        ... 47 elided
      Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.config.ClientConfig
      

        Attachments

          Activity

            People

            • Assignee:
              michal Michal Malohlava
              Reporter:
              michal Michal Malohlava
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: