I noticed that the threshold is always set by the* max f1* value. However, it does not always work for very imbalanced data so the accuracy of the minority will be quite low.
However, I do find H2O actually calculate the sensitivity table in H2O Flow and has different threshold by different metric. For example, in my case, I found the threshold associated with max of min_per_class_accuracy works the best, because it made sure the minority will be predicted well too.
In my case, I need to first find out the threshold using find_threshold_by_max_metric() function, then I manually set the label based on the p1 vs the threshold. It would be great if there is a hyperparameter to specify the metrics that will be optimized for during the training or prediction, then the model will just use the threshold for that metrics during the prediction.
Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.9366837607680464:
0 1 Error Rate
0 61.0 226.0 0.7875 (226.0/287.0)
1 50.0 733484.0 0.0001 (50.0/733534.0)
Total 111.0 733710.0 0.0004 (276.0/733821.0)
threshold = 0.99
Predicted 0 1
0 224 63
1 1292 732242