Create an R/Python Function that calculates chunk size

Description

Create an R/Python function that calculates chunk size based on:

  • raw size of the data

  • number of cpu cores

  • number of nodes

This function can be used to aid in reproducibility for users reproducing a model trained on different hardware with different number of cpu cores.

Assignee

Veronika Maurerov√°

Fix versions

Reporter

Megan Kurka

Support ticket URL

None

Labels

None

Affected Spark version

None

Customer Request Type

None

Task progress

None

ReleaseNotesHidden

None

CustomerVisible

No

Priority

Major
Configure