Once these changes are made, a system administrator will need to go and delete any of the activities/reports that appeared after the close date. A report of Overdue tasks can be found under Case Tasks>Due Soon/Past Due
To identify which Activities and Reports have these boxes checked, navigate to Home Activities or Home Reports (Under Organization Settings), or Case Activities or Case Reports (Under Program). In the column marked Options search for the words PROMPTDELETE. This will indicate the boxes have been checked. You can compare this against the Column marked Repeat to see Activities and Reports with repeating due dates.