Uploaded image for project: 'SW'
  1. SW-386

SW REPL cannot be used in combination with Spark Dataset

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.3, 2.0.7
    • Component/s: None
    • Labels:
      None
    • CustomerVisible:
      No
    • Sprint:

      Description

      {format}
      case class Person(id: Long)
      val ds = Seq(Person(0), Person(1)).toDS
      val count = ds.count{format}

      executed from Flow REPL will throw an exception:

      defined class Person
      org.apache.spark.sql.AnalysisException: Unable to generate an encoder for inner class `Person` without access to the scope that this class was defined in.
      Try moving this class out of its parent class.;
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNewInstance$$anonfun$apply$33$$anonfun$applyOrElse$13.applyOrElse(Analyzer.scala:2104)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNewInstance$$anonfun$apply$33$$anonfun$applyOrElse$13.applyOrElse(Analyzer.scala:2100)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:288)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:288)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:287)
        at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionDown$1(QueryPlan.scala:248)
        at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:258)
        at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$6.apply(QueryPlan.scala:267)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188)
        at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsDown(QueryPlan.scala:267)
        at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressions(QueryPlan.scala:236)
      

      The problem is that Spark OuterScopes has hardcoded structure of generated REPL classes:

      private[this] val REPLClass = """^(\$line(?:\d+)\.\$read)(?:\$\$iw)+$""".r
      

      More readings:

        Attachments

          Activity

            People

            • Assignee:
              michal Michal Malohlava
              Reporter:
              michal Michal Malohlava
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: