[ampaterry]

Hunter, you are seeing genetic selection in action.
Killing a disproportionate number of the biggest bucks will, over time, cause the average size of the deer to decrease.
We use this to our advantage when, as breeders, we select for a given trait and breed ONLY those animals that display that trait. My wife bred Doberman's for many years, and over that time developed a line of dogs that were beautiful, healthy, and had a good temperment.
We do the opposite when, in the field, we harvest the biggest game - which leaves only the smaller ones to reproduce.

The fix?
Convince everyone that the smaller the deer, the better the meat.