If you’re a professional, and you’ve raced more than 40 days in the last 9 months, then yes, 3 weeks or so off the bike makes sense. Your body needs a chance to recover completely before beginning to prepare it for your new season.
If you’re like the rest of us, and you have to plan your training around a full time job and family commitments, then it’s unlikely you’ve raced so many days and trained so hard that you need sustained time off the bike. In fact, taking more than a few days off the bike will required extra effort to recover the fitness you lose.