H2O stops accessing HDFS after Kerberos ticket is renewed

Description

In Kerberized clusters, a non-expired ticket is required to access HDFS.
This ticket has a limited lifetime.
After a ticket is renewed (it has to be renewed before it expires), H2O service has problems accessing HDFS:

01-06 09:28:12.728 10.20.37.31:53011 30029 #s/exists ERRR: Caught exception: HDFS IO Failure:
01-06 09:28:12.728 10.20.37.31:53011 30029 #s/exists ERRR: accessed URI : hdfs://dcoabhdfs01.some.domain:8020/user/svc_h2oqa/h2oflows/environment/clips
01-06 09:28:12.728 10.20.37.31:53011 30029 #s/exists ERRR: configuration: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, /var/run/cloudera-scm-agent/process/1696-yarn-NODEMANAGER/core-site.xml
01-06 09:28:12.728 10.20.37.31:53011 30029 #s/exists ERRR: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 529 for svc_h2oqa) can't be found in cache; Stacktrace: [water.persist.PersistHdfs.exists(PersistHdfs.java:438), water.persist.PersistManager.exists(PersistManager.java:317), water.init.NodePersistentStorage.exists(NodePersistentStorage.java:94), water.api.NodePersistentStorageHandler.exists(NodePersistentStorageHandler.java:21), sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method), sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57), sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.lang.reflect.Method.invoke(Method.java:606), water.api.Handler.handle(Handler.java:64), water.api.RequestServer.handle(RequestServer.java:644), water.api.RequestServer.serve(RequestServer.java:585), water.JettyHTTPD$H2oDefaultServlet.doGeneric(JettyHTTPD.java:617), water.JettyHTTPD$H2oDefaultServlet.doGet(JettyHTTPD.java:559), javax.servlet.http.HttpServlet.service(HttpServlet.java:707), javax.servlet.http.HttpServlet.service(HttpServlet.java:820), org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)]

klist shows new ticket as refreshed and not expired.
Does h2o service has to somehow renew its token for the renewed Kerberos ticket? See error above around "token (HDFS_DELEGATION_TOKEN token 529 for svc_h2oqa) can't be found in cache"?

Assignee

New H2O Bugs

Fix versions

None

Reporter

Ruslan Dautkhanov

Support ticket URL

None

Labels

None

Affected Spark version

None

Customer Request Type

None

Task progress

None

CustomerVisible

No

AffectedOpenSource

Components

Priority

Critical
Configure