Defining Goals for the Machine Learning Models

Business Statement: False negatives are five times more costly than false positives. Therefore, we should optimise for recall over precision to satisfy this when designing our model.

Precision: What proportion of positive identifications was actually correct? A model that produces no false positives has a precision of 1.0.
Recall: What proportion of actual positives was identified correctly? A model that produces no false negatives has a recall of 1.0.

If you increase precision, it will reduce recall and vice versa. This is called the precision/recall tradeoff.

So, we need to find a balance between the product team’s goal of optimising for precision and data scientist’s goal of minimising the model’s loss.

Solution: Instead of calling the classifier’s predict() method, you can call its decision_function() method, which returns a score for each instance, and then make predictions based on those scores using any threshold we want.

Decision Points for Technical Business Owners: When defining the goals for our model, it’s important to consider the needs of different teams across an organisation, and how each team’s needs relate back to the model. By analysing what each team is optimising for before building out our solution, we can find areas of compromise in order to optimally balance these multiple objectives.