Business and enterprise software, and other applications that help a business run

Intrigued by a report this morning in the online version of InfoWorld, I started looking into Aster Data Systems. I have been designing and building databases, including some good sized data warehouses, for many years. Aster's claims to be building what InfoWorld refers to as an “internet-scale cluster database.” My response was, “Yikes, that's big.”

I have worked on databases with millions and millions of records, and more. No problem. But what Aster is aiming at is a little bigger. They have just signed up MySpace as a customer. So it is going to have to be big. I have seen the database in which one of the major credit card issuers stores transactions, and worked on data warehouses to answer questions about worldwide telephone traffic. Those do not hold a candle to the processing required by something like MySpace.

If you're thinking about a killer Web startup, and think you can hit it big, you need to start thinking about data architecture. Clustered databases: massively parallel processing to be exact. Symmetric processing. Process-oriented programming. Or, if you're in marketing, you can just think “Cutting edge everything.”

Aster apparently is going one step further. They are dividing the processing into discrete parts, forming clusters which operate as specialized processes. Some of these specialized clusters load data, some insert new rows, some do edits, and so on. These are in turn controlled by what Aster calls “queen nodes” (which I think is a beehive reference) which coordinate the overall processing and provide system-wide services, such as on-the-fly query optimization.

All of this means that Aster is trying to take the database to a higher level. Data grows proportionally as the population and societal complexity increase. We are creating a LOT of data. I'm glad there are people thinking about how we are going to handle it.