In a similar vein, if the baby falls asleep around company, allow him to stay there in lieu of moving him to some quiet room.
Be inventive - come up with a few other ideas in which you consciously ascertain that Daddy and baby spend some time together.