Uploaded image for project: 'Public H2O 3'
  1. Public H2O 3
  2. PUBDEV-3970

StackedEnsemble: Support saving & loading binary models

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.10.5.1
    • Component/s: StackedEnsemble
    • Labels:
      None
    • CustomerVisible:
      No

      Description

      We should add support for saving and loading binary Stacked Ensemble models.

      Currently, you can save a model using the standard `h2o.saveModel(ensemble, path = "my_ensemble")` functionality, but there are some errors when you load it back in and try to make predictions (after stopping and restarting the H2O cluster).

      This works:

      > # Train a stacked ensemble using the GBM and RF above
      > ensemble <- h2o.stackedEnsemble(x = x,
      +                                 y = y,
      +                                 training_frame = train,
      +                                 model_id = "my_ensemble_binomial",
      +                                 base_models = list(my_gbm@model_id, my_rf@model_id))
        |=======================================================================================================| 100%
      > 
      > h2o.saveModel(ensemble, path = "ensembletest2")
      [1] "/Users/me/ensembletest2/my_ensemble_binomial"
      > model <- h2o.loadModel("/Users/me/ensembletest2/my_ensemble_binomial")
      > pred <- h2o.predict(ensemble, newdata = test)
        |=======================================================================================================| 100%
      > pred2 <- h2o.predict(model, newdata = test)
        |=======================================================================================================| 100%
      

      But if you kill the cluster, load the ensemble and try to make a prediction on a frame, you get something like this:

      > model <- h2o.loadModel("/Users/me/testensemble/my_ensemble_binomial")
      > # Import a sample binary outcome train/test set into H2O
      > train <- h2o.importFile("https://s3.amazonaws.com/erin-data/higgs/higgs_train_10k.csv")
        |=============================================================| 100%
      > test <- h2o.importFile("https://s3.amazonaws.com/erin-data/higgs/higgs_test_5k.csv")
        |=============================================================| 100%
      > 
      > # Identify predictors and response
      > y <- "response"
      > x <- setdiff(names(train), y)
      > 
      > # For binary classification, response should be a factor
      > train[,y] <- as.factor(train[,y])
      > test[,y] <- as.factor(test[,y])
      
      > perf <- h2o.performance(model = model, newdata = test)
      
      ERROR: Unexpected HTTP Status code: 500 Server Error (url = http://localhost:54321/3/ModelMetrics/models/my_ensemble_binomial/frames/RTMP_sid_9882_7)
      
      java.lang.NullPointerException
       [1] "java.lang.NullPointerException"                                                                              
       [2] "    hex.StackedEnsembleModel.predictScoreImpl(StackedEnsembleModel.java:93)"                                 
       [3] "    hex.Model.score(Model.java:964)"                                                                         
       [4] "    hex.Model.score(Model.java:949)"                                                                         
       [5] "    water.api.ModelMetricsHandler.score(ModelMetricsHandler.java:229)"                                       
       [6] "    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)"                                             
       [7] "    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)"                           
       [8] "    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)"                   
       [9] "    java.lang.reflect.Method.invoke(Method.java:497)"                                                        
      [10] "    water.api.Handler.handle(Handler.java:61)"                                                               
      [11] "    water.api.RequestServer.serve(RequestServer.java:429)"                                                   
      [12] "    water.api.RequestServer.doGeneric(RequestServer.java:280)"                                               
      [13] "    water.api.RequestServer.doPost(RequestServer.java:217)"                                                  
      [14] "    javax.servlet.http.HttpServlet.service(HttpServlet.java:755)"                                            
      [15] "    javax.servlet.http.HttpServlet.service(HttpServlet.java:848)"                                            
      [16] "    org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)"                                  
      [17] "    org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)"                              
      [18] "    org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)"                           
      [19] "    org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)"                             
      [20] "    org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)"                       
      [21] "    org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)"                      
      [22] "    org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)"                               
      [23] "    org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)"                        
      [24] "    org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)"                       
      [25] "    org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)"                           
      [26] "    org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)"                   
      [27] "    org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)"                         
      [28] "    org.eclipse.jetty.server.Server.handle(Server.java:370)"                                                 
      [29] "    org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)"          
      [30] "    org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)"           
      [31] "    org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)"                
      [32] "    org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)"
      [33] "    org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)"                                        
      [34] "    org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)"                                   
      [35] "    org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)"                  
      [36] "    org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)"            
      [37] "    org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)"                        
      [38] "    org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)"                         
      [39] "    java.lang.Thread.run(Thread.java:745)"                                                                   
      
      Error in .h2o.doSafeREST(h2oRestApiVersion = h2oRestApiVersion, urlSuffix = page,  : 
        
      
      ERROR MESSAGE:
      
      Caught exception: java.lang.NullPointerException
      

      Related (since we may just make a generic `h2o.saveModels()` function that works on Grid, AutoML and Stacked Ensembles):

        Attachments

          Issue links

            Activity

              People

              • Assignee:
                navdeep Navdeep
                Reporter:
                erin Erin LeDell
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: