The reason I am mostly certain we have not been visited is my, admittedly, limited knowledge of physics. The cost in time, power and energy to make such a trip dwarfs the imagination of the average person.
If an intelligent race were that motivated to expend that much in resources to come here, they most certainly wouldn't hang out in backwoods swamps.
And I feel that it is our humongous human ego to think that earth is even worth visiting in the first place. If another race is so advanced as to be able to get here, they are hardly going to need anything we have to offer.
All of the above, of course, assumes that the speed of light is indeed inviolate.....