Uploaded image for project: 'SW'
  1. SW-276

ClassLoading issue when running code using SparkSubmit

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.3, 1.6.9
    • Component/s: None
    • Labels:
      None
    • CustomerVisible:
      No

      Description

      When running the app using spark-submit the application can't find certain classes. It is caused by this code

          if (Main.interp != null) {
            //application has been started using SparkShell script, reuse the classloader
            interpreterClassloader = new InterpreterClassLoader(Main.interp.intp.classLoader)
          } else {
            //application hasn't been started using SparkShell
            interpreterClassloader = new InterpreterClassLoader()
          }
      
      /**
        * Interpreter classloader which allows multiple interpreters to coexist
        */
      class InterpreterClassLoader(val mainClassLoader: Option[ClassLoader]) extends ClassLoader {
        def this(mainClassLoader: ClassLoader) = this(Some(mainClassLoader))
      
        def this() = this(None)
      
        override def loadClass(name: String): Class[_] = {
          if (name.startsWith("intp_id")) {
            val intp_id = new Scanner(name).useDelimiter("\\D+").nextInt()
            H2OIMain.existingInterpreters(intp_id).classLoader.loadClass(name)
          } else if (mainClassLoader.isDefined) {
            mainClassLoader.get.loadClass(name)
          } else {
            super.loadClass(name)
          }
        }
      }
      

      When running the app using spark submit, we don't set the fallback class-loader and parent class-loader is used to load classes which are not defined in our custom repl. The parent class-loader is however system class loader which can't see the classes.

      The fix is simple - we just need to pass context classloader to the interpreterClassloader

      This is fix to this issue as well https://0xdata.atlassian.net/projects/SW/issues/SW-152?filter=allopenissues

        Attachments

          Activity

            People

            • Assignee:
              Kuba Jakub Hava
              Reporter:
              Kuba Jakub Hava
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: