Information Gain of a Decision-Tree Split

Reported in interviews at Bridgei2i DRW IMC Trading Lt Financial Services Microsoft Mindtree

Problem statement

A decision-tree node has 10 positive and 10 negative examples. A split sends $(8\text{ pos}, 2\text{ neg})$ left and $(2\text{ pos}, 8\text{ neg})$ right. Define information gain using Shannon entropy, then compute it in bits.

Show hint

How do you measure how mixed a node's labels are, and how would you compare that before and after the split?

Relevant roles: ML Engineer, Data Scientist and Applied Scientist.

Related free problems