Over the course of time you might add nodes that you later delete, and end up with taxonomy terms that have no nodes. These empty terms add overhead to your site, and add empty taxonomy listing pages. It's best to remove the empty terms.
I said "Overhead".. I don't know all the instances but have seen a couple places where all terms are loaded into memory. If your vocabulary includes empty terms (no nodes for the term) each empty vocabulary term makes for a bigger array that is loaded into memory.
The following are for Drupal 6.. haven't looked at how to do this with Drupal 7.