THE MAN NEVER SLEEPS. This is amusing, but not really a problem. I enjoy the mental image of working his fingers to the bone for pittance. I am his lord, he is my serf! Well, okay, eventually he may get promoted to vassal.
I'm also amused by the thought of the occasional random event that shuts down a field, like "plague of locusts" or "blight" which requires the player to pop in every once in a while to clean up the field. Richer compost attracts more bugs, time to remedy balanced out by quality of tools, rarity/cost/length of tilling a single crop raises the chances, etc. Making it a little more than "just" an idle game, but still fun & requiring a little more thought and strategy.
 "Idle Farmer"){kind=icon}
I like the idea, but as you say that will ruin the "idle" style of play. And after all, this isn't more that a little idle game...