Intermittent script test issue on external backend

Description

Script Tests / water.sparkling.scripts.ScriptPipelineHamOrSpamDL.hamOrSpamDeepLearning.script.scala

Error Message
java.lang.AssertionError: assertion failed: Value of term "answer2" should be true
Stacktrace
java.lang.AssertionError: assertion failed: Value of term "answer2" should be true
at scala.Predef$.assert(Predef.scala:170)
at water.sparkling.scripts.HamOrSpamTester$.test(ScriptsTestSuite.scala:246)
at water.sparkling.scripts.ScriptPipelineHamOrSpamDL$$anonfun$15.apply$mcV$sp(ScriptsTestSuite.scala:230)
at water.sparkling.scripts.ScriptPipelineHamOrSpamDL$$anonfun$15.apply(ScriptsTestSuite.scala:230)
at water.sparkling.scripts.ScriptPipelineHamOrSpamDL$$anonfun$15.apply(ScriptsTestSuite.scala:230)
at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)
at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
at org.scalatest.Suite$class.run(Suite.scala:1424)
at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
at water.sparkling.scripts.ScriptPipelineHamOrSpamDL.org$scalatest$BeforeAndAfterAll$$super$run(ScriptsTestSuite.scala:221)
at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
at water.sparkling.scripts.ScriptPipelineHamOrSpamDL.run(ScriptsTestSuite.scala:221)
at org.scalatest.junit.JUnitRunner.run(JUnitRunner.scala:99)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy1.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:146)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:128)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
at java.lang.Thread.run(Thread.java:745)
Standard Output

Interpreter Response:
import org.apache.spark.SparkFiles
import org.apache.spark.ml.PipelineModel
import org.apache.spark.ml.feature._
import org.apache.spark.ml.h2o.features.ColumnPruner
import org.apache.spark.ml.h2o.algos.H2OGBM
import org.apache.spark.sql.types.{StringType, StructField, StructType}
import org.apache.spark.sql.{DataFrame, Row, SQLContext}
import water.support.SparkContextSupport
import water.fvec.H2OFrame
import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.h2o.algos.H2ODeepLearning
smsDataFileName: String = smsData.txt
smsDataFilePath: String = examples/smalldata/smsData.txt
load: (dataFile: String)(implicit sqlContext: org.apache.spark.sql.SQLContext)org.apache.spark.sql.DataFrame
import org.apache.spark.h2o._
h2oContext: org.apache.spark.h2o.H2OContext =

Sparkling Water Context:

  • H2O name: sparkling-water-scripts-tests-1706185573

  • cluster size: 2

  • list of used nodes:
    (executorId, host, port)
    ------------------------
    (0,172.16.2.182,54323)
    (1,172.16.2.186,54323)
    ------------------------

Open H2O Flow in browser: http://172.16.2.184:54323 (CMD + click in Mac OSX)

  • Yarn App ID of external H2O cluster: application_1513873501160_9479

sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@7446115c
tokenizer: org.apache.spark.ml.feature.RegexTokenizer = regexTok_e0ec7cfbaff9
stopWordsRemover: org.apache.spark.ml.feature.StopWordsRemover = stopWords_397988ef9205
hashingTF: org.apache.spark.ml.feature.HashingTF = hashingTF_f6015df94ca7
idf: org.apache.spark.ml.feature.IDF = idf_a13318886981
dl: org.apache.spark.ml.h2o.algos.H2ODeepLearning = dl_fc9eadeaa70d
colPruner: org.apache.spark.ml.h2o.features.ColumnPruner = h2oColRemover_f7f9255362ec
pipeline: org.apache.spark.ml.Pipeline = pipeline_a8ab901f4cb6
data: org.apache.spark.sql.DataFrame = [label: string, text: string]
model: org.apache.spark.ml.PipelineModel = pipeline_a8ab901f4cb6
isSpam: (smsText: String, model: org.apache.spark.ml.PipelineModel, hamThreshold: Double)Boolean

Printed output:
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
Determining driver host interface for mapper->driver callback...
[Possible callback IP address: 172.16.2.184]
[Possible callback IP address: 127.0.0.1]
Using mapper->driver callback IP address and port: 172.16.2.184:56191
(You can override these with -driverif and -driverport/-driverportrange.)
Memory Settings:
mapreduce.map.java.opts: -Xms6g -Xmx6g -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dlog4j.defaultInitOverride=true
Extra memory percent: 10
mapreduce.map.memory.mb: 6758
18/01/29 08:52:50 INFO impl.TimelineClientImpl: Timeline service address: http://mr-0xd10.0xdata.loc:8188/ws/v1/timeline/
18/01/29 08:52:50 INFO client.RMProxy: Connecting to ResourceManager at mr-0xd7.0xdata.loc/172.16.2.187:8050
18/01/29 08:52:51 INFO mapreduce.JobSubmitter: number of splits:2
18/01/29 08:52:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1513873501160_9479
18/01/29 08:52:52 INFO mapred.H2OYARNRunner: Setting application type to H2O
18/01/29 08:52:52 INFO impl.YarnClientImpl: Submitted application application_1513873501160_9479
18/01/29 08:52:52 INFO mapreduce.Job: The url to track the job: http://mr-0xd7.0xdata.loc:8088/proxy/application_1513873501160_9479/
Job name 'sparkling-water-scripts-tests-1706185573' submitted
JobTracker job ID is 'job_1513873501160_9479'
For YARN users, logs command is 'yarn logs -applicationId application_1513873501160_9479'
Waiting for H2O cluster to come up...
H2O node 172.16.2.182:54323 requested flatfile
H2O node 172.16.2.186:54323 requested flatfile
Sending flatfiles to nodes...
[Sending flatfile to node 172.16.2.182:54323]
[Sending flatfile to node 172.16.2.186:54323]
H2O node 172.16.2.186:54323 reports H2O cluster size 1 [leader is 172.16.2.186:54323]
H2O node 172.16.2.182:54323 reports H2O cluster size 1 [leader is 172.16.2.182:54323]
H2O node 172.16.2.186:54323 reports H2O cluster size 2 [leader is 172.16.2.182:54323]
H2O node 172.16.2.182:54323 reports H2O cluster size 2 [leader is 172.16.2.182:54323]
Cluster notification file (notify_sparkling-water-scripts-tests-1706185573) created.
H2O cluster (2 nodes) is up
Open H2O Flow in your web browser: http://172.16.2.182:54323
Disowning cluster and exiting.

For YARN users, logs command is 'yarn logs -applicationId application_1513873501160_9479'

false
false

Standard Error
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).

Status

Assignee

Jakub Hava

Reporter

Jakub Hava

Labels

None

CustomerVisible

No

testcase 1

None

testcase 2

None

testcase 3

None

h2ostream link

None

Affected Spark version

None

AffectedContact

None

AffectedCustomers

None

AffectedPilots

None

AffectedOpenSource

None

Support Assessment

None

Customer Request Type

None

Support ticket URL

None

End date

None

Baseline start date

None

Baseline end date

None

Task progress

None

Task mode

None

Fix versions

Priority

Major
Configure