The easiest solution for achieving your stated goal of allowing multiple people to sell on your site while exercising fine-grained control of who can and cannot do so is perhaps using WPMU's MarketPress in tandem with BuddyPress or WordPress Multisite. I'm not a die-hard fan of WordPress, per se, but that might be an expedient way for you to get to a minimal viable product and to validate your idea before shelling out the time and/or cash to custom build it from the ground up, and/or labor ad nauseam with tweaking an existing framework. MarketPress is a good plug-in that'll give you many of the features of a full-fledged e-commerce framework... BuddyPress, of course, will allow you to set up individual vendor's with their own sites under your brand. The two work together. More on MarketPress at:
Another alternative is Jimdo's PagePartners. I haven't used it, but it looks intriguing. I like their design sensibilities, and their stated business ethos. This might be a viable option, too. The caveat being: it's not white label. More info about Jimdo's PagePartners here:
Finally, another interesting CMS to explore is SetSeed. I think it'll allow you to launch multiple sites for each vendor via a central hub you control, and will allow you to maintain your branding within each. How, the,n any sort of renumeration would flow back to you for setting up an individual vendor's store would be up to you to figure out... This is a fairly new CMS and it looks like it's evolving smartly and rapidly. If you require some customization of it, to approach more specifically what you ask for, now might be a good time to reach out to the developer...but you might be able to think of an effective way to adapt it for your use right out of the box.
Unfortunately, none of the above is open-source--but, again, the ease by which you could get to a functional site approximating your idea may off-set that drawback. Jimdo is an open-source contributor, however. So, maybe even an e-mail to them might be a fruitful dialogue to begin. If anything, check out each of the above, and it may influence how you search for other solutions, and will at least provide some models in your own thinking or with other developers. The shopping cart is an integrated feature, I believe, in all of the above cases. With regard to giving your vendors the capacity to deliver digital goods (e-books, mp3s, etc.), check out Fetchapp.com. Very cool app. Very easy to set-up...could probably be rolled into one of the above frameworks. The frameworks would handle the issue of individual vendor profiles and/or sub-domains.