PySparkling 3.26.2 in Zeppelin environment cannot find AddURL

Description

1 2 3 4 5 6 %pyspark # start h2o cluster inside spark cluster from pysparkling import * hc = H2OContext.getOrCreate(spark)

reports:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Fail to execute line 4: from pysparkling import * # NOQA Traceback (most recent call last): File "/tmp/zeppelin_pyspark-4834646886817757880.py", line 375, in <module> exec(code, _zcUserQueryNameSpace) File "<stdin>", line 4, in <module> File "<frozen importlib._bootstrap>", line 971, in _find_and_load File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 656, in _load_unlocked File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible File "/nfs/h2o/artifacts/sparkling_water/sparkling-water-3.26.2-2.4/py/build/dist/h2o_pysparkling_2.4-3.26.2-2.4.zip/pysparkling/__init__.py", line 55, in <module> Initializer.load_sparkling_jar(sc) File "/nfs/h2o/artifacts/sparkling_water/sparkling-water-3.26.2-2.4/py/build/dist/h2o_pysparkling_2.4-3.26.2-2.4.zip/pysparkling/initializer.py", line 34, in load_sparkling_jar Initializer.__add_sparkling_jar_to_spark(sc) File "/nfs/h2o/artifacts/sparkling_water/sparkling-water-3.26.2-2.4/py/build/dist/h2o_pysparkling_2.4-3.26.2-2.4.zip/pysparkling/initializer.py", line 49, in __add_sparkling_jar_to_spark Initializer.__add_url_to_classloader(jvm, url) File "/nfs/h2o/artifacts/sparkling_water/sparkling-water-3.26.2-2.4/py/build/dist/h2o_pysparkling_2.4-3.26.2-2.4.zip/pysparkling/initializer.py", line 121, in __add_url_to_classloader cl.addURL(url) File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__ answer, self.gateway_client, self.target_id, self.name) File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco return f(*a, **kw) File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 332, in get_return_value format(target_id, ".", name, value)) py4j.protocol.Py4JError: An error occurred while calling o233.addURL. Trace: py4j.Py4JException: Method addURL([class java.net.URL]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748)

spark version: 2.4.3

Environment

None

Status

Assignee

Unassigned

Reporter

Hongzhao Zhu

Labels

Release Priority

None

CustomerVisible

No

testcase 1

None

testcase 2

None

testcase 3

None

h2ostream link

None

Affected Spark version

None

AffectedContact

None

AffectedCustomers

AffectedPilots

None

AffectedOpenSource

None

Support Assessment

None

Customer Request Type

None

Support ticket URL

None

End date

None

Baseline start date

None

Baseline end date

None

Task progress

None

Task mode

None

Fix versions

Affects versions

3.26.2
2.4.13
2.4.12

Priority

Major
Configure