In the previous blog in this mini-series, I arrived into the project and was asked to review the external vendor’s initial High Level Design (HLD).
As stated previously, there are a total of five Oracle databases in the system architecture. All will be running on 10g which is good news because external vendors often don’t support the latest Oracle releases without a fight and that limits the functionality we can use. It subsequently turned out that one of the vendors insists on using 10.1 for one of the databases, rather than 10.2. Discussion about that is on-going 😉
Whenever I review a system design, these are some of the main factors I keep in mind. (Oh, I could write a book about this stuff, so don’t take this as the whole story)
- What are the business requirements? Until I know how much data we* can afford to lose and how much time we have to get it back, I can’t comment on the suitability of any technical solution.
- Given the business requirements and the application’s demands on the databases, what is the worst that can happen? A vivid imagination helps here, as does extensive prior experience. I would liken my approach to recovery design to the approach security professionals use – be as rigorous, negative and miserable as possible 😉 My attitude is simple. If the business decides that a risk is very unlikely or not likely enough to warrant the additional expenditure to protect against it, then that’s fine. However, they need to have all of the information at their disposal to make that decision and it’s not my job to decide what risks they want to take – just identify the possible problems and offer solutions to them.
- How complex is the approach and how well does it fit our own internal system management standards? Although a different approach might be extremely smart, it will need to be supported by the existing DBA team who already have dozens or hundreds of databases to support. Using a set of standard approaches means less to learn, for both existing and future team-mates, and reduces the likelihood of human error. Not only should human intervention be as simple as possible but should be eliminated where possible. Time after time I’ve seen expensive recovery strategies that looked good during design meetings fall apart on the rocks of human factors.
To give some brief project-specific answers to those three areas
- As I mentioned in the previous blog, this is a system that our customers will connect to directly, not an internal business system, so any failure would be noticeable, impact our customers and our business reputation. However, three of the five databases are only used internally and the business is prepared to accept longer recovery times for these. In fact, it was quite a while in to the project before the Non-Functional Requirements (or Service Level Agreements) were defined. They’re still being refined weeks later.
- What’s the worst that could happen? Oh, the usual things – a server fails, someone deletes a database file, the data centre catches fire, an workman drills through a cable. (I’ve seen that one take down quite a few unbreakable systems!) I’ll talk a little more about the risks in the next blog.
- The initial design consisted of the following
- Two node RAC cluster supporting two seperate databases
- Two node Veritas VCS cluster supporting two seperate databases
- Two node Veritas VCS cluster support one databases
All the databases will be stored on Veritas filesystems sitting on RAID 1+0 in EMC arrays using Brocade SAN and all of the expensive and cool stuff that people like Kevin Closson know about (If you’re interested in this system design stuff, you simply have to read his blog) and businesses like my current site can afford 😉
Everything in the initial design was familiar technology at my site, particularly the VCS clusters, as that’s our standard deployment model for production systems. If one server fails, the storage, database instances and any application resources are failed over to the node that’s still operating. There’s a noticeable failover time but it’s normally acceptable to the business for most systems.
Next time I’ll focus on the most critical couple of databases, the solution the vendor suggested and the flaws in that solution.
* Note my use of the words ‘our’ and ‘we’ throughout the
blog. I’m a contractor and have no intention of becoming an employee of the client but,
when I’m designing a system, it’s my responsibility and when I’m providing
services to a client, the client’s customers are my customers