My wife and I went last year with our two young boys. Like you we didn't want the early morning fights for both the breakfast buffet and sun lounges so we wanted somewhere a bit quieter. We found the perfect resort -
El Gouna (near, but NOT in, Hurghada on the Red Sea). We stayed at the Three Corners Resort Rhianna which we found to be a nice hotel. The food was a bit bland but we ate out mostly. Loads of excursions you could go on, not that many kids about and one of the pools was virtually empty each day (we went in June).
Totally recommend it.