Decimal(2,1) not compatible in h2o frame

Description

Hi, I have a dataframe in spark with a column created with:
,IF(r.status = 'ok', 0.0,1.0) AS cancelled

When I move the dataframe over to h2o with .as_h2o_frame(), I get:

Py4JJavaError: An error occurred while calling o69.asH2OFrame.
: java.util.NoSuchElementException: key not found: DecimalType(2,1)
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:59)
at scala.collection.MapLike$class.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:59)
at org.apache.spark.h2o.utils.ReflectionUtils$.supportedTypeOf(ReflectionUtils.scala:110)
at org.apache.spark.h2o.converters.SparkDataFrameConverter$$anonfun$4.apply(SparkDataFrameConverter.scala:70)
at org.apache.spark.h2o.converters.SparkDataFrameConverter$$anonfun$4.apply(SparkDataFrameConverter.scala:70)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.h2o.converters.SparkDataFrameConverter$.toH2OFrame(SparkDataFrameConverter.scala:70)
at org.apache.spark.h2o.H2OContext.asH2OFrame(H2OContext.scala:132)
at org.apache.spark.h2o.H2OContext.asH2OFrame(H2OContext.scala:133)
at org.apache.spark.h2o.JavaH2OContext.asH2OFrame(JavaH2OContext.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)

When I change the sql to:
,IF(r.status = 'ok', 0,1) AS cancelled
It works.

So it's fine for me to do it as the above of course. Actually, the 0.0 / 1.0 was due to Spark ML needing a double instead of an Int. But anyway, it is good for h2o not to crash on this subtleties.

If it is duplicate please mark it DUP.

Assignee

Michal Malohlava

Reporter

Avkash Chauhan

Labels

None

CustomerVisible

No

testcase 1

None

testcase 2

None

testcase 3

None

h2ostream link

None

Affected Spark version

None

AffectedContact

None

AffectedCustomers

AffectedPilots

None

AffectedOpenSource

None

Support Assessment

Platform Issue

Customer Request Type

Support Incident

Support ticket URL

None

End date

None

Baseline start date

None

Baseline end date

None

Task progress

None

Task mode

None

ReleaseNotesHidden

None

Fix versions

Priority

Major
Configure