Parse: Parquet file fails to parse when h2o cloud comprises of a lot of nodes with less memory

Description

ref: https://support.h2o.ai/helpdesk/tickets/90923
a test parquet of size 35GB in hdfs fails to parse on a cloud of 40nodes 10gb each.
the issue also happens when the file only has 200 partitions.

Assignee

Michal Kurka

Fix versions

None

Reporter

Nidhi Mehta

Support ticket URL

None

Labels

None

Affected Spark version

None

Customer Request Type

None

Task progress

None

ReleaseNotesHidden

None

CustomerVisible

No

AffectedCustomers

Components

Priority

Major
Configure