Decision Tree ID3: A Complete Solved Numerical Example

Try Decision Tree Solver →Read Decision Tree Theory →

Scenario: Bank Loan Approval

The Objective: Predict whether a loan application should be approved based on applicant profile.

Step 1: The Training Data

Before calculating recursive splits, we must define the dataset. The ID3 algorithm evaluates each feature to predict the target class (Approved).

Data Point	Credit_History	Employment	Income_Level	Owns_Property	Approved
P1	Good	Stable	High	Yes	Yes
P2	Good	Stable	Medium	No	Yes
P3	Poor	Unstable	Low	No	No
P4	Poor	Stable	Medium	Yes	No
P5	Good	Unstable	High	Yes	Yes
P6	Poor	Unstable	High	No	No
P7	Good	Stable	Low	No	Yes
P8	Poor	Stable	Low	Yes	No
P9	Good	Unstable	Low	No	No
Target	Good	Unstable	Medium	No	?

Want to edit this data in the live solver?

Step 2: Recursive Splitting (Entropy & Gain)

To build the tree, we recursively calculate the Entropy of the system and find the Information Gain for every available feature. The feature with the highest gain becomes the splitting node.

Iteration 1

Context: Root

1. Entropy of Target Class, Entropy(S)

Formula:

\text{Entropy}(S) = - \dfrac{P}{P+N} \log_2\left(\dfrac{P}{P+N}\right) - \dfrac{N}{P+N} \log_2\left(\dfrac{N}{P+N}\right)

Positives (P) for 'Yes' = 4

Negatives (N) for 'No' = 5

$\text{Entropy}(S) = 0.9911$

2. Subset Information Required

Formula:

\text{Entropy}(P_i, N_i) = - \dfrac{P_i}{P_i+N_i} \log_2\left(\dfrac{P_i}{P_i+N_i}\right) - \dfrac{N_i}{P_i+N_i} \log_2\left(\dfrac{N_i}{P_i+N_i}\right)

Evaluating Feature: Credit_History

Value	Pi	Ni	I (Pi, Ni)
Good	4	1	0.7219
Poor	0	4	0.0000

Evaluating Feature: Employment

Value	Pi	Ni	I (Pi, Ni)
Stable	3	2	0.9710
Unstable	1	3	0.8113

Evaluating Feature: Income_Level

Value	Pi	Ni	I (Pi, Ni)
High	2	1	0.9183
Medium	1	1	1.0000
Low	1	3	0.8113

Evaluating Feature: Owns_Property

Value	Pi	Ni	I (Pi, Ni)
Yes	2	2	1.0000
No	2	3	0.9710

3. Weighted Feature Entropy

Formula:

\text{Entropy}(A) = \sum \left[ \dfrac{p_i + n_i}{P + N} \right] \times \text{Entropy}(P_i, N_i)

Credit_HistoryEntropy = 0.4011

EmploymentEntropy = 0.9000

Income_LevelEntropy = 0.8889

Owns_PropertyEntropy = 0.9839

4. Feature Information Gain

Formula:

\text{Gain}(S, A) = \text{Entropy}(S) - \text{Entropy}(A)

Credit_History0.9911 - 0.4011 =Gain:0.5900

Employment0.9911 - 0.9000 =Gain:0.0911

Income_Level0.9911 - 0.8889 =Gain:0.1022

Owns_Property0.9911 - 0.9839 =Gain:0.0072

5. Feature Selection Decision

Credit_History generated the highest Information Gain (0.5900).
It is selected as the optimal splitting node for this subset.

Resulting Split

Credit_History ?

Good

Class: ?

Poor

Class: No

Iteration 2

Context: Credit_History = Good

Current Data: Filtered by Credit_History = Good5 Rows

Data Point	Credit_History	Employment	Income_Level	Owns_Property	Approved
P1	Good	Stable	High	Yes	Yes
P2	Good	Stable	Medium	No	Yes
P5	Good	Unstable	High	Yes	Yes
P7	Good	Stable	Low	No	Yes
P9	Good	Unstable	Low	No	No

1. Entropy of Target Class, Entropy(S)

Formula:

\text{Entropy}(S) = - \dfrac{P}{P+N} \log_2\left(\dfrac{P}{P+N}\right) - \dfrac{N}{P+N} \log_2\left(\dfrac{N}{P+N}\right)

Positives (P) for 'Yes' = 4

Negatives (N) for 'No' = 1

$\text{Entropy}(S) = 0.7219$

2. Subset Information Required

Formula:

\text{Entropy}(P_i, N_i) = - \dfrac{P_i}{P_i+N_i} \log_2\left(\dfrac{P_i}{P_i+N_i}\right) - \dfrac{N_i}{P_i+N_i} \log_2\left(\dfrac{N_i}{P_i+N_i}\right)

Evaluating Feature: Employment

Value	Pi	Ni	I (Pi, Ni)
Stable	3	0	0.0000
Unstable	1	1	1.0000

Evaluating Feature: Income_Level

Value	Pi	Ni	I (Pi, Ni)
High	2	0	0.0000
Medium	1	0	0.0000
Low	1	1	1.0000

Evaluating Feature: Owns_Property

Value	Pi	Ni	I (Pi, Ni)
Yes	2	0	0.0000
No	2	1	0.9183

3. Weighted Feature Entropy

Formula:

\text{Entropy}(A) = \sum \left[ \dfrac{p_i + n_i}{P + N} \right] \times \text{Entropy}(P_i, N_i)

EmploymentEntropy = 0.4000

Income_LevelEntropy = 0.4000

Owns_PropertyEntropy = 0.5510

4. Feature Information Gain

Formula:

\text{Gain}(S, A) = \text{Entropy}(S) - \text{Entropy}(A)

Employment0.7219 - 0.4000 =Gain:0.3219

Income_Level0.7219 - 0.4000 =Gain:0.3219

Owns_Property0.7219 - 0.5510 =Gain:0.1710

5. Feature Selection Decision

Employment generated the highest Information Gain (0.3219).
It is selected as the optimal splitting node for this subset.

Resulting Split

Employment ?

Stable

Class: Yes

Unstable

Class: ?

Iteration 3

Context: Employment = Unstable

Current Data: Filtered by Employment = Unstable2 Rows

Data Point	Credit_History	Employment	Income_Level	Owns_Property	Approved
P5	Good	Unstable	High	Yes	Yes
P9	Good	Unstable	Low	No	No

1. Entropy of Target Class, Entropy(S)

Formula:

\text{Entropy}(S) = - \dfrac{P}{P+N} \log_2\left(\dfrac{P}{P+N}\right) - \dfrac{N}{P+N} \log_2\left(\dfrac{N}{P+N}\right)

Positives (P) for 'Yes' = 1

Negatives (N) for 'No' = 1

$\text{Entropy}(S) = 1.0000$

2. Subset Information Required

Formula:

\text{Entropy}(P_i, N_i) = - \dfrac{P_i}{P_i+N_i} \log_2\left(\dfrac{P_i}{P_i+N_i}\right) - \dfrac{N_i}{P_i+N_i} \log_2\left(\dfrac{N_i}{P_i+N_i}\right)

Evaluating Feature: Income_Level

Value	Pi	Ni	I (Pi, Ni)
High	1	0	0.0000
Low	0	1	0.0000

Evaluating Feature: Owns_Property

Value	Pi	Ni	I (Pi, Ni)
Yes	1	0	0.0000
No	0	1	0.0000

3. Weighted Feature Entropy

Formula:

\text{Entropy}(A) = \sum \left[ \dfrac{p_i + n_i}{P + N} \right] \times \text{Entropy}(P_i, N_i)

Income_LevelEntropy = 0.0000

Owns_PropertyEntropy = 0.0000

4. Feature Information Gain

Formula:

\text{Gain}(S, A) = \text{Entropy}(S) - \text{Entropy}(A)

Income_Level1.0000 - 0.0000 =Gain:1.0000

Owns_Property1.0000 - 0.0000 =Gain:1.0000

5. Feature Selection Decision

Income_Level generated the highest Information Gain (1.0000).
It is selected as the optimal splitting node for this subset.

Resulting Split

Income_Level ?

High

Class: Yes

Low

Class: No

Step 3: Final Computed Decision Tree

Combining all the recursive splits from Step 2 yields the final classification tree.

Credit_History ?

Good

Employment ?

Stable

Class: Yes

Unstable

Income_Level ?

High

Class: Yes

Low

Class: No

Poor

Class: No

Final Takeaway

This tree mathematically proves which variables have the highest predictive power for determining loan approvals based on historical data.