Decisions

 

I got a good laugh out of this one and thought a lot of engineers can relate to Dilbert in this situation.

Original Dilbert page here:
http://www.dilbert.com/2012-05-26/

Installation Planning - Part 1

During the course of any project there will be a time when the system actually gets installed. This is a step in the overall roll out of a solution that generally lasts from a few days to a few months depending on the type of a system and number of locations being installed. By this time all requirements gathering, planning, system designs have been completed, socialized and approved. To be more specific, an engineering team is scheduled to go onsite to actually install and configure the software. 

A common mistake I see is the lack of documentation for this phase; usually it is a line item on a project plan with some high level milestones under it. From a project management perspective, that is all that is required to maintain visibility, however this lack of documentation can significantly hinder the project depending on the circumstances.

The solution is an installation planning document. I usually build these documents in excel and they have at least 6 columns. Task ID, Task Name, Task Notes, Assignee, Status and Time Spent. This turns into a list of steps that the system installer will need to perform to get the system installed. It needs to be as granular as you feel is necessary, I prefer to document each configuration file change, installation locations, ports, host names and paths for temp directories. I also like to add snippets of configuration files in the Task Notes column. Generating this document can be time consuming; however the content is re-usable and provides a great data set for estimating the schedule for future installations. 

Spending time generating these installation checklists are well worth the effort. There are some very practical engineering benefits. For example, in complex systems by generating the configuration files ahead of time, you may be able to spot certain issues with the configuration. These issues are much easier addressed if they are spotted prior to system installation.  

If you have multiple engineers installing the system, the checklist is a coordination lifesaver. Each engineer will know what they will need to do and having configuration files ahead of time ensures that all settings will match up and the system will operate when you turn it on.

There are advantages from the management standpoint as well. A key one is a peer review process, the installation checklist can be disseminated for review to senior engineers, architects and the system manufacturer. They may be able to spot issues with the configuration ahead of time or offer some improvement suggestions. This truly takes advantage of corporate reach back that so many large vendors claim they have.

Furthermore, this can actually decrease costs of implementation since the configurations can be developed off site by architects and senior engineers and the onsite installation can be performed by lower level systems engineers. Since all hard design work was completed ahead of time, the lower level systems engineers will not need to be as experienced. In this scenario the onsite installer can still call and reach a higher level engineer if trouble arises, but this frees up your architects and other corporate talent to work on more projects.

If the system is being deployed in multiple locations the installation checklist will ensure that the system installation methodology and configuration is the same. This will assist greatly if the maintenance of the system has to be transitioned to a different support group within an organization. Operations and Maintenance (O&M) will be a nightmare if each system is configured differently and planning upgrades will not be easy.

These are just some of the benefits of spending additional time planning the installation. In the next post I’ll share some techniques and tips on creating an effective system installation checklist.

Five CPE Credits for ISC2 - InfoSecurity

If you are an ISC2 certified security guru, you are required to maintain education levels through accumulation of CPE credits. One of the ways to obtain CPE credits is through magazine subscriptions.

InfoSecurity magazine is free and offers 5 CPE credits for the magazine subscription.

Apply For Free Subscription Here

Scalability

I think scalability is where true engineering takes place. Each vendor will have scalability documentation on how much load their product can support. These are usually conservative amounts so that they don’t promise too much. These need to be examined very carefully, since they will almost never provide you ALL the information you need, but are just a part of the puzzle.

Imagine that you are deploying a system for around 10,000 users. Out of these, there are perhaps 2000 concurrent users doing various tasks. The system will already have some data in it and new data will be coming in hourly. So you look at the vendor scalability documentation and it provides a case scenario of a 2000 user system, with 100 concurrent users doing one or two common tasks and data volume much lower than yours. Therefore, multiplying the environment in the scalability documentation would not provide you with the performance you are looking for.

This is mainly because, depending on the user actions, the bottle neck may occur in different system areas. Here is where the knowledge of backend technologies plays a crucial role. You will need to speed up the bottle neck areas and the only way to figure out how is through theory and experience.

For example, let’s say that during operating hours, a back end database like SQL is hard-hit and is causing a bottle neck. Instead of the common shotgun approach of upgrading hardware and moving to faster storage whip out performance monitor and start logging metrics. Windows Performance Monitor is a hugely underrated tool. It can provide valuable insight into many system performance issues. However, in this scenario, it would not be enough to determine which hardware component is causing the bottle neck. You would need to monitor specific SQL performance metrics to determine why the hardware component is causing the bottle neck, and then address the issue.

Going on with the example, you can run PerfMon to determine that hard drives are causing the bottle neck; it’s pretty common, since storage is usually the slowest part of the system. Drilling deeper, you can determine that the issue is due to constant reads from disks, at which point you can dig even deeper to see what the cache hit ratio is. If it is low, you can enhance the cache hit ratio to increase performance of SQL without necessarily upgrading hardware. You may also notice that some storage mediums are being heavily read, while another storage array is underutilized. In that case, moving the database or a few heavily read tables to that medium would also enhance performance.

Remember, various storage architectures are better for reading or for writing. It’s not uncommon to have different arrays for the same database server for different databases or even tables in the database. Impact of other components should be considered as well.

Furthermore, we can seek optimization outside of the bottle neck component. For example, let’s say that front end web servers are being heavily taxed. If the sessions are encrypted, it could be because of the large overhead for SSL and authentication. In this scenario, you can look into offloading SSL authentication at a different front end component such as an application delivery controller.

Don’t underestimate simple tools for purposes of scalability. A simple performance monitor and Microsoft Excel with the analytics plug-in is all you need in most cases.

Creatively scaling solutions is a dying talent because most of the time, people think it’s easier to just throw more hardware at the issue.

Solutions Engineering

So what is the difference between deploying a solution and a system? Sure, a system can be a solution to something, but deploying a solution has an entirely different scope of work. Deploying a system generally consists of having a few engineering sessions, going onsite to install it, doing some configurations, maybe migrating some data, training the customer and calling it done.

Deploying a solution is an entirely different ballgame and requires a different mindset from the client and the vendor standpoint. A solution will naturally have a system deployment component, but ultimately there are a lot of other moving pieces that contribute to program success.

Basically, there are a lot of other tasks involved in solution deployment to give the users a usable system that meets their needs. A prime example of this is training. In a solution deployment, you would not train users on the system; you would train them on the new business processing using the new system. Prior to training on the new business process using the new system, these processes would actually need to be re-defined, designed, developed, tested, deployed, vetted through user acceptance testing, re-designed and re-deployed and re-tested and re-vetted through users, etc… there is a lot of work involved.

Let’s go through a few differences on common deliverables between deploying systems and solutions. As a head up, I understand that all projects are different. It is certainly possible that some items from solution column bleed over to the system column. However, it’s important to acknowledge that when providing a solution you need to provide ALL of the pieces that make it complete. In other words, just because a system implementation has the project artifacts from the solutions column does not guarantee success. 

Deliverable	System	Solution
Technical Architecture	·         Hardware/Software Requirements ·         A few high level architecture drawings ·         Hardware is usually provided by the client	·         Hardware is usually included in the initial design ·         Environmental equipment such as load balancers, storage, networking, servers, KVM, UPSs are included in the design
System Documentation	·         System Manuals ·         Technical Support Number ·         CDs and License Keys	·         Installation Documentation ·         Configuration Documentation ·         Operations, Monitoring and Sustainment Plan ·         Test Plan ·         Migration Plan ·         Cut-Over Plan ·         Change Management Plan ·         Security Plan Project Documentation include: ·         Risk Management Plan ·         Training Plan ·         Communications Plan ·         Project Plan
Configuration	Only basic, essential configuration is implemented.	Specific to the client’s requirements.
Testing	Ensures the system and all components function as desired.	Tests the system under specific configuration of the client, usually involves load/stress testing, user acceptance testing, DR/HA testing, etc…
Change Management (Operations)	n/a	Perform change management to redefine existing business process to take advantage of new technology.
Change Management (IT)	Client makes any changes they want to the system.	Formal process for requesting, documenting, approving and implementing changes to the design.

Now that you’ve examined the list, I would ask you to reflect on your past experience with IT deployments and team members and ask yourself two questions: 

1)  Do you think some projects issues that you witnessed on previous projects could have been avoided by thinking in terms of a solution instead of a system?
2)  Who on the team is most suited to writing all of this documentation?

Most of this is Project Management 101; however, it’s very seldom implemented. Teams know that they should do it, but they choose not to, or the value of these artifacts is not stressed upon by the management. There is a lot of planning that needs to be done ahead of time for large solutions. This planning needs to be documented prior to system implementation. This does a few things, but one of the most important is that it creates the same expectation of the final product. This essentially defines

Change Management

In this context, the change management is not referring to user outreach, but change to project requirements, scope, system design and other project artifacts. Change management has a few faces depending on what phase of the project you are in. The further into the project you are, the more crucial it becomes. Basically, I see it as having a few main aspects 1) evaluate impact of change 2) notify everyone of the change so that they can plan accordingly.

To evaluate the impact of the change solid baseline for what the solution is needs to be defined through project and system design documentation.

·         Impact analysis - Without a formal process, you will not know the full impact of a design change. This is why a lot of projects finish late with costs that are higher than expected. The changes are incorporated into the project without proper evaluation.

·         Change control between groups – When multiple teams are working on pieces of solution, they all need to be in sync.

·         Requirements Changes – As requirements are added and/or change (and they will), additional time and money may be required to meet them. The impact of new requirements needs to be evaluated against the baseline to gauge the additional time and money required.

Vendor Selection

The difference between the almost right word and the right word is really a large matter.
Mark Twain, Author

I am a big fan of COTS products that allow for extensive product configuration. I think that custom development has its place, but with the toolsets available to engineers now there is little need to have an entirely custom solution. I see the value in building specific modules. However, I believe that custom development solutions are less effective once the entire software lifecycle; long-term retention and updating are factored into the cost.

The COTS route also prevents a scenario where the mission critical system in your organization is 15 years old, runs on a legacy Novell server through IPX/SPX with a lone wolf system administrator who sits in the basement and is the only one who knows how it works. Here are a few observations on vendor selection.

Often, after all requirements have been gathered, documented and validated (a process that can take anywhere from a few weeks to months), the first stages of system design start to take shape. It’s a beautiful experience. All the engineering powerhouse brains start brainstorming and some vague representations of what it may look like in the end start appearing on airport bar napkins.

Regardless of the industry, the requirements are somehow communicated to third parties and something is expected in return, regarding how the requirements will be met and how much it will approximately cost. If a requirements matrix is sent out, it is a safe bet that any vagueness in requirement interpretation will allow each vendor to decisively declare that they meet that requirement and put a check mark next to it.

Since we are still in very early stages of solution design, a vendor bake off or a proof of concept may not be practical or may not fully validate how the requirements are met. As such, I have found it necessary to grill vendors extensively on how the product works, in order to find any deficiencies. Sometimes, I wish I had a lie detector that I could hook sales people up to, but since I don’t, I have to rely on my technical prowess.

Almost every software package has certain core competencies. That is, what the product does very well. These core competencies are usually the main focus of the company when it is first started, and are what the product is used for most of the time. As a product matures and existing clients make software change requests, the vendor tacks on additional functionality to meet those specific requirements. Most of the time, the added functionality is put in place with little afterthought to how it will be implemented. In addition, due to the competitive nature of business, vendors may tack on yet another layer of capability so that they can add check marks next to items of requirements matrixes. This is called sales functionality and is generally not suitable or flexible enough for most implementations.  In a nutshell, a product usually does about half of the advertised functionality well.

Usually, the adequate and sales claims functionality is actually present. However, the specific way in which the functionality is implemented may make it impractical for use, or may not scale very well to an enterprise-level system.

A practical scenario is a project where my team had to implement both a case management and business process management solution. One of the requirements was that cases and documents had to be routed via business logic to a user for approval. Naturally, most of the product suites we evaluated had this functionality. However, the way that some vendors implemented the routing decision was poor and did not allow for dynamic token selection. As such, their products would have required large data sets of If… Then… Else statements to decide which person the case had to go to. This was impractical for a system with thousands of users where dynamic routing decisions where necessary. Also, dynamic routing decision requirement could have been interpreted in so many ways that even If… Then… Else statements would have met the criteria on a requirements matrix.

Make sure that high priority requirements are in the core competency of the product.

Thinking about the requirements

Users don't know what they want until you show it to them.

-Unknown

As good requirements are trickling in, it is important to think about their implications and how all the requirements are interrelated. After all, we are putting together an entire solution that will meet the requirements in a seamless manner. Two or more requirements in combination can actually generate a third requirement. Let’s examine this over-simplified example.

Requirement 1: The system will be able to receive alerts via CAP 1.1 and CAP 1.2.

Requirement 2: The system will allow the user to distribute incoming messages via CAP 1.1 and CAP 1.2 through a variety of channels to include CCTV video displays, speaker arrays, telephone speakerphones, computers and SMS.

So this is pretty straight forward; we receive alerts via a specific protocol and the user can send the message out via a wide variety of systems. Let’s see how it would play out in real world.

An alert comes in and the user sees the alert. The user goes into the CCTV system to publish it to displays. Then the user taps into the speaker array system and does Text to Voice to sound off the alarm, then the user taps into the phone system and transmits the recording to the phone system, then the user goes into another system to send the alert to PC and finally to the last system to send out the notice through SMS. In the end, the process took much longer than expected and the user is not happy and the distributed message may not be relevant anymore.

A derived requirement in this scenario could be:

Requirement 3: The system will provide a standard interface for transmission of messages from Requirement 1 through channels in Requirement 2.

In this scenario the user would receive the message, go through some checkbox list of systems that the message needs to go out of, click a button and voila!

Business users will not be able to proactively see the solution in the skeptical light that engineers can. As such, an experienced project team should develop additional requirements to improve end-user experience and usability. Naturally, these “good ideas” need to be vetted through the customer prior to being included in the design requirements.

There is much more to properly gathering and documenting requirements. However, I keep running into this scenario and as such wanted to address it.  

Requirements

Alice: Oh, no, no. I was just wondering if you could help me find my way.
Cheshire Cat: Well that depends on where you want to get to.
Alice: Oh, it really doesn't matter, as long as...
Cheshire Cat: Then it really doesn't matter which way you go.

The requirements are the start of it all. If there weren’t any we would all be in the unemployment line. Without solid requirements, the solution will never meet expectations. It’s as simple as that. Most of the information here is common sense and industry’s best practice; however it still gets ignored most of the time. You would be surprised at how many systems start being designed without requirements. There are a few things that I would like to add on top of the regular “requirements gathering” curriculum.

There are plenty of resources on requirements gathering and I will not attempt to explain the entire process, however here are some common pitfalls that I have seen:

●      Requirements gathering and validation is a very politically and fiscally challenging process. Few people realize that there is a very direct relationship between requirements and money. Simply said, the more you ask for, the more you will need to pay. If this is not acknowledged, the project team needs to help the customer manage expectations.

●      Another common mistake is for incorrect requirements, your requirement can’t be to buy product XYZ. The requirement needs to be for a system that provides functionality A, B and C. Based on this the project team should conduct market research and select the best product or a combination of products to meet the requirements.

●      The requirements shouldn’t be blindly accepted by the project team. If the information you need is not there, you need to take a step back and get the right information. This is sometimes difficult.

●      Nothing kills morale more than working for weeks in one direction only to find that the requirements weren’t accurately documented and having to shift gears.

●      Requirements are complete when all requirements have been documented and mapped to validation criteria which in turn map to validation scenarios. Documentation of requirements is essential because it communicates the requirements to all parties involved. Language in the document needs to be plain, clear and specific.