Moderation email not being sent

We keep seeing moderation emails not being sent.  We have the site set up to be 1 abusive flag and it immediately gets pulled to review.  We have the notification section set to 1 with no delay.  Am I missing something?