Managing 500000 users or more online

Does anyone here know how web based email providers manage such large
amounts of data and users, and where do they host and store all this
data in?

I'm asking because i'm building an online diary (sort of) that allows
users to store their friends contact details like birthdays etc.
online. Each user will have one table in the database which stores
all his data. Then comes other tables for login data etc.

Like this:

Database
Logindata
User1Table
User2Table
User3Table

I'm thinking of using discountasp.net to host it and use a MySql
database.

If I end up having, say, 500000 (there'll be more) users, does this
mean there'll be 500000 tables created in the database.

Will MySQL permit so many tables? I just tried inserting 20000 email
addresses and it allowed me only 1000 (that's here on my local
machine)

How does this work actually? I somehow dont feel its the right way to
do this. I'm sure there'll be maximum number of tables per database
and Maximum Number of databases and Sizes and number of users
connecting etc.

So how do i handle so many users?.

How do google or yahoo do this. Where do they host such things and
what databases do they rely on.

Also where's a good place to host such a thing if not on a shared
account (recommendations welcome).Any place that'll let me install my
own database software (plus other softwares) and create as many
databases and have total control over the server.

Jason

-- Jason Menezes, September 9, 2005

Answers

There is no reason to use a relational database at all, given your stated requirements. If you wanted to do queries across multiple users, e.g., "show me people who are under the age of 35 and have at least 4 contacts in South Dakota", that would be a good reason to use an RDBMS and have information from multiple users in the same table. An industrial-strength database management system such as Oracle or Microsoft SQL Server could probably be made to do it (check the online docs at Oracle.com for "limits" (a whole separate book)) but you would be paying for services that you aren't using. A company such as Google that wants to hold a lot of email for a lot of customers doesn't bother with an RDBMS. They just have a custom file format and lots of different computers.

-- Philip Greenspun, September 13, 2005

The fact that you have ambitions to serve half a million or more users, yet are planning to use MySQL, suggests that you haven't done enough research. Also, it might be useful to you to ask such questions on the OpenACS Q&A forum.

-- Andrew Piskorski, September 10, 2005

So which DB should I use? I'm using MySQL cause its free, and i've heard that a number of sites use it. Including yahoo. And finally where do I host such a thing. BTW thanx for the openacs link.

-- Jason Menezes, September 10, 2005

Use MS SQL Server 2000. It is available at various price points by most ISPs. I use the following SQL Server Hosting companies. I can say that both service providers offer reasonable prices and reliable, expert services.

www.discountasp.net

www.hostmysite.com

-- Steve Tout, September 10, 2005

I should have asked this question 2 weeks ago. I had Windows 2000 server then. I formatted and installed Windows 2000 Professional. MS Sql will not install the server on Win 2000 Pro. BTW the last time I saw someone trying to do something with MS Sql he kept ending up with "not a trusted user" (thats all i remember).

Yesterday I inserted an email address (jasonmzs@yahoo.com) 7152342 times into My Sql using ASP. It took about 23 minutes on my 566 Celeron machine with 128 MB RAM. Size of the table now is: 129MB. Any idea how big the table would be in MS SQL?

Jason

-- Jason Menezes, September 11, 2005