Can bailiffs only take goods from my principle home if I own three houses?

Yes

The enforcement agent may only take control of the debtor's goods where the debtor usually lives or carries on a trade or business.^[1]

If the enforcement agent wants to take control of the debtor's goods at an address or premises at which the debtor neither lives nor carries on a trade or business, then he must apply for that power separately.^[2] He must make an application to court.^[3]

^{[1] Paragraph 14(6) of Schedule 12 to the Tribunals, Courts and Enforcement Act 2007
[2] Paragraph 15 of Schedule 12 to the Tribunals, Courts and Enforcement Act 2007
[3] Civil Procedure Rule 84.9}