I run Exchange 2010 virtualized on an HP ProLiant DL380 G6 with a quad-core Xeon, 24GB of RAM, 4 NIC ports, and plenty of storage for 150 users. I have all 4 major roles-- MBX, CA, HT, and UM-- plus Forefront Protection for Exchange (limited to one process for each realtime scanner), and I've never had any major issues.
I have about 30 ActiveSync mobile connections, 10 OWA users, and 20 RPC over HTTPS connections and the rest make up desktops using Outlook 2007 or 2010. My CPU usage generally sits about 60% during the day, and drops to 5-10% during the evening hours. My RAM load is about 19GB, and paging is about 200MB (so very minimal). Network traffic is load-balanced on 4 1Gbps ports, so again during the day it can spike a bit but I rarely see it using more than 1-2 Gbps.
I'm not saying it's ideal for everyone, but from what I'm reading in your post, it generally seems reasonable that as long as you monitor CPU/RAM/Disk/Network loads, and as long as you're taking suitable recovery needs for your environment, then running a multi-role server is a very efficient, cost-effective way of consolidating your servers.