the point of the castle is to garrison ranged units inside so they can shoot from relative safety. u r a ranged unit and r inside.if it collaspes u dont die, but r then exposed to all the enemy units and u dont have a lot of health thus u will die pretty soon. the same goes for them, except they dont have a leader like you so all that happens is they cant garrison anymore and the people inside will probably die. the only reason its not important is because they dont have a ranged boss in it like you do
 "Bowmaster Prelude"){kind=icon}
 "Tunnel Rider"){kind=icon}
 "Bunni: How we first met"){kind=icon}
 "Rotatix"){kind=icon}
 "Balloon in a Wasteland"){kind=icon}
 "Protector"){kind=icon}