Yes there are trade-offs as with most any technology.
Much (but not all) of the slower speed has to do with how many filters you use, etc... encryption also slows down the traffic speed.

Here is a Microsoft Webcast that goes into some more details.

TechNet Support WebCast: How to isolate servers by using Internet Protocol security