Please read this piece about ads on your page, it makes a good point that the ads are part of your brand. And that you should pick them with care, not accept anything a banner farm throws at your site.

Mule Design Studio's Blog: Ads are Content: "The ads you allow on your site are part of your brand."

In my opinion this also is true about the speed you want to give your visitors. Still too many banner farms ( I won't name and shame, but you know how you are. ) and ad agencys don't do anything about speed. Using for instance extremely slow javascript solutions to serve the banners. At one of my clients I tried to convince them to just give us some xml or json and let us place the banners from the link provided by them, but they didn't wanted to do that. Hiding behind tracking the visitor 'cause it was important to them, the continued to worsen the experience we could provide for our visitors.

In fact it was so bad the the load time was multiplied by 5 with banners. FIVE times as long, all because they wanted to track users to better their experience...