My suggestion would be this.
Rather than shortening the time it takes for the energy bar to be fully refilled, I suggest removing the time system for energy entirely from view, and allowing the same number of clicks/button mashing to be used no matter what house you have.
For example, when you have house 1, 1000 clicks of energy can be used and refilled in 5 hours and 45 minutes. House 2 gets you the same amount of clicks, but in 5h and 15 minutes, and so on, rather than just...cutting the amount of time and clicks a person can use.