Unlike a cluster in which only a few nodes are interconnected, massively parallel processing (MPP) systems allow a large number of nodes to be interconnected. There are MPP systems with more than a thousand nodes. Each of these nodes is a separate system that has its own internal CPU, disks, controllers, memory, and system buses to form a “loosely coupled” architecture that shares nothing. All of these nodes are connected through a high-speed, high-bandwidth interconnection. Each node has a separate copy of the operating system. In the MPP architecture, Oracle is installed in RAC mode.
Each Oracle instance on each node is responsible for all the resources it contains, has a view of the entire database, and can find out which node has a lock anywhere in the database. If you need something that is locked by another node, it will require a ping between instances, where the other instance would have to write all the changes it has made to disk. A typical database query would go against a node; the data would be collected from the memory or disks of any appropriate node and would travel across the interconnect to return the data to the requesting node. Each node controls its own set of disks and can take over another set if a node fails. Therefore, all nodes can be configured to be primary or a combination of primary and secondary. The database files are placed on the primary nodes, while the secondary nodes provide the redundancy necessary to take over and replace the primary nodes, should the latter fail.
The biggest downside to MPP architectures is not the architecture itself, but the application design flaws that exist in a typical deployment. Most application designers are well versed in implementing applications on SMP machines. However, MPP architectures require a total paradigm shift in terms of analyzing what data should be placed on which node to reduce data exchange and ping between instances between nodes. Since each node is highly independent in a share-nothing architecture, any situation that causes nodes to exchange resources on a large scale due to data exchange generates high traffic on the interconnect. This situation is causing severe performance problems for Oracle RAC databases. The performance of the Oracle RAC database is mainly dependent on an incorrect hardware architecture configuration. You must consider the hardware architecture and the operating system implementation before processing the Oracle Real Application Cluster configuration. Otherwise, you would get a lot of trouble after deployment and changing the hardware configuration requires downtime of your highly available databases.
