Lessons Learned from a Failed Deployment

Last week we were scheduled to replace a critical component in a complex, mission-critical hospital system. About two-thirds of the way through the deployment, it became clear that I had missed something during the preparatory work for the change (security, always check security). Additional work would be needed before we could complete the upgrade, and it was very likely that we wouldn’t finish the deployment on time…

Lessons from Previous Implementation Projects

Given the critical nature of this change, experience told us that we needed to do things “properly”. Previous experience suggested that we needed to:

  1. Test the new solution thoroughly (we put 2,000,000 transactions through the new component and compared the results to the old solution).
  2. Write a sufficiently detailed implementation plan
    1. Include prep-work required prior to implementation
    2. Include enough detail so you don’t have to think during implementation. This helps under pressure, and ensures that energy is available to tackle the unexpected.
    3. Outline post-implementation work required
  3. Test the implementation plan (This was not possible for us due to differences between our test and live environments. Rectifying this would cost £ hundreds of thousands).
  4. Write a sufficiently detailed roll-back plan
  5. Test the roll-back plan (Again, not possible).
  6. Keep users and stakeholders informed… allowing plenty of time for them to make necessary arrangements for down-time.
  7. Define a change window
    1. When you’ll cause least disruption during the change
    2. When failure of the new component will cause least chaos
    3. When you have enough support from others
    4. When you’ll have enough time for post-implementation testing
  8. Get approval from stakeholders… in writing
    1. Explain the purpose of the thing you’re changing
    2. Explain why you’re making the change
    3. Say how things are at the moment
    4. Say how things will be in the future
    5. Explain how you will monitor the new solution
    6. Prepare your implementation and roll-back plans in advance
  9. Check the state of the system before changing it (so we could be sure that any faults were due to our changes and not existing faults)

Well, we had done all that, but I had made a minor mistake during prep, so things were going badly.
So, my team leader made the call: to roll back.

Lesson 1: Be Prepared to Roll Back

I don’t just mean having a written plan, although that was extremely useful. I mean psychologically. It is sometimes hard to admit defeat. However, it is better to roll-back than either (1) upset customers by breaching the change window and (2) making mistakes whilst working under pressure. It just isn’t worth it.

Lesson 2: A Successful Roll-Back Is Not a Failure

… it is a tactical retreat. As we had a good roll-back plan we were able to revert to the old module without loss of data, and to do so within the change window. We had maintained the status quo.

Lesson 3: Roll Back Completely

You really don’t want to leave a system in an indeterminate state. As it was, we left some things in place ready for our next roll-out attempt. This was a mistake, as it caused (1) some minor confusion, and (2) if we had forgotten and done more testing, we could have caused corruption of live data.

Lesson 4: Communicate

We explained the reasons for the roll-back and the steps we had taken to make sure that we wouldn’t experience the same difficulties again. This was trust-building, and others were supportive of our action.

Lesson 5: Rally and Retry

Once the roll-back was verified the others went home. I stayed late to fix the problem that had caused the implementation issues.

Conclusion

This was a great learning experience for me. Today we did the implementation again. But this time, it went smoothly.

Calling Constructors in C# Using Reflection

Reflection can be used in C# to invoke a class’ constructor. This is useful when you don’t know the type of the class that you want to instantiate until runtime.
As usual, there are various options open to you depending on your requirements.

Invoking the Default (Parameterless) Constructor

When You Can Pass the Required Type as a Generic Parameter

Under these circumstances, you don’t need to use reflection at all:
[code language=”C#”]
T Instantiate() where T : new()
{
return new T();
}
[/code]
Alternatively, you can call the generic version of Activator.CreateInstance:
[code language=”C#”]
ObjectType instance = Activator.CreateInstance();
[/code]

When You Know the Required Type

You can instantiate using either the type itself:
[code language=”C#”]
ObjectType instance = (ObjectType)Activator.CreateInstance(objectType);
[/code]
Or the type’s name:
[code language=”C#”]
ObjectType instance = (ObjectType)Activator.CreateInstance(“MyNamespace.ObjectType, MyAssembly”);
[/code]

When You Need a Reference to the Default Constructor

This is useful if you want to call the constructor later.
[code language=”C#”]
var constructor = type.GetConstructor(Type.EmptyTypes);

ObjectType instance = (ObjectType)constructor.Invoke(null);
[/code]

Invoking a Parameterized Constructor

If you want to get a reference to a constructor that has, for example, two parameters, the first of which is a long and the second an int:
[code language=”C#”]
// We need the constructor with the following signature:
var argTypes = new[]
{
typeof (long),
typeof (int)
};
ConstructorInfo constructor = ObjectType.GetConstructor(argTypes);
[/code]
This constructor can then be invoked like this:
[code language=”C#”]
var argValues = new object[] { lngFirstParameter, intSecondParameter };
ObjectType instance = (ObjectType) constructor .Invoke(argValues);
[/code]

Getting Information About Constructor Parameters

The following method will write out the parameters of type ObjectType:
[code language=”C#”]
public void PrintParameters()
{
var ctors = typeof(ObjectType).GetConstructors();
// Assuming class ObjectType has only one constructor:
var ctor = ctors[0];
foreach (var param in ctor.GetParameters())
{
Console.WriteLine(string.Format(“Parameter {0} is named {1} and is of type {2}”,param.Position, param.Name, param.ParameterType));
}
}
[/code]

References

Colour Your Environment

Have you ever been about to type a command and stopped to double-check if you’re on a test server or a production box? Or even executed a command and then had your heart stop because you weren’t sure which environment you’re logged in to?
One of the most effective ways to make sure you always know which environment you’re in is simply to colour-code. Use different coloured backgrounds on your desktops or change the colours of your menu bars.
When choosing colours I suggest the following:

  • Be consistent in your colouring. E.g. Production environments should always be the same colour. Never any other colour.
  • Use a simple colour scheme. Mark important differences rather than unimportant ones. For example, a 3 colour system: production, test and development environments.
  • Use a memorable colour scheme. For example, red for production to indicate “danger” or “stop”, because you need to be careful with production. Orange for “test” because you can be a little more relaxed but still need to be aware of other tests that are running. Green for your personal development environment, because you should be able to do anything on your own box.
  • Once you’ve picked a colour scheme, use it everywhere: paper folders, physical servers,  whiteboards, everywhere.

Hopefully, if you colour your environment, you’ll never have that “Production? Please don’t be production” feeling again.
 

WordPress on Windows with IIS7 Error 500 Problem Solved

There are many reasons why an application might send an error 500 page. This post describes a common one for WordPress installations, and describes the simple solution.
There are a variety of circumstances where WordPress can’t finish processing a request and so it calls an internal function called “wp_die”. This is supposed to display a friendly error message so that the user can take corrective action. Unfortunately, however, IIS7 intercepts the error and displays a generic “Error 500″ page instead.
An example is where use the “Add Site” function in a multi-site installation. If you enter an invalid site name, WordPress will try to tell you that there is a problem. Unfortunately, IIS7 traps the error and sends back the default 500 error page instead of the helpful message. This leaves you with no way to know what the real problem could be.
The simple solution is to add the following code to your web.config file:
[sourcecode language=”HTML”]

… all your other stuff

[/sourcecode]
This tells IIS to pass the exception through to the end user without the interception, and results in you seeing the friendly exception details instead of the default error page.

Happy Workers Result in Happy Investors

A recent paper has demonstrated that companies with happy workers are worth than those with unhappy workers.
The 2.4-3.7% difference cited by the study doesn’t sound like much, but over the years that difference accumulates. If you invested £100 in a happy-worker company, you’d get between £180 and £240 more money back in 25 years time than if you’d invested in a sad-worker company. That’s about twice your original investment!
According to the original research paper, this implies:

  • Happy workers result in happy investors
  • An increased emphasis on looking after your people can increase the value of your company
  • Investors need to pay more attention to human factors rather than just just financial ones

Reset Microsoft Word

I recently had problems with Word style getting messed up, so I figured I’d just zap the Normal.dot file and everything would be OK. It wasn’t, so I took more drastic measures.
It turns out you can use the registry editor to blow away Word’s settings, and Word will restore the defaults when you start it next. The key that holds the settings varies according to the version of Word that you’re running.
Warning: If you think this is a safe thing to do, don’t do it.

Word 97 HKEY_CURRENT_USER/Software/Microsoft/Office/8.0/Word
Word 2000 HKEY_CURRENT_USER/Software/Microsoft/Office/9.0/Word
Word 2002 HKEY_CURRENT_USER/Software/Microsoft/Office/10.0/Word
Word 2003 HKEY_CURRENT_USER/Software/Microsoft/Office/11.0/Word
Word 2007 HKEY_CURRENT_USER/Software/Microsoft/Office/12.0/Word

The PRINCE2 Business Case Theme

The Business Case is one of 7 PRINCE2 Themes (things that must be attended to throughout the life of a project).

The Purpose for the Business Case

A basic principle of PRINCE2 is “Continuous Justification”, which asks the question, “is the project (still) worth investing in?”. The project’s Business Case documents the answer to this question.
Different projects will measure success is different ways, for example ROI or legislative compliance.  Nevertheless, all projects must be evaluated to see if the right  approach is being taken to achieve benefits with available resource. In order to do so, the Business Case provides the means to evaluate the project on whether it is:

  • Desirable = Is the project worth investing in, given its anticipated costs, benefits and risks (CB&R)?
  • Viable = Is the project possible? Will it produce the intended products?
  • Achievable = Will the products of the project provide the intended benefits?

Delivering a project’s products doesn’t mean benefits of these products will be realised. For this reason, it is important to distinguish between the project’s products (or outputs), outcomes and  benefits.

  • Output = a specialist product of the project, e.g. new web site
  • Outcome = result of using project’s outputs , e.g. more sales
  • Benefit = measurable change that is advantageous to stakeholders, e.g. increased profits

In  contrast to a project’s products, its outcomes (and therefore benefits)  may not occur until some time after the project closes. To avoid the  danger of loosing sight of benefits and focusing on products, the  Business Case must make the relationship of products, outcomes and benefits explicit.

The Contents of the Business Case

The Business Case must contain sufficient information to determine if project is DVA i.e. will give a ROI (return on investment). A typical Business Case will contain:

  1. An executive summary
  2. Reasons for undertaking the project, including options that have been considered
  3. The anticipated benefits and and dis-benefits
  4. Estimates of costs and time-scales
  5. A risk profile
  6. An investment appraisal

The Reasons for the Project and the Options Considered

This section outlines the reasons that the project is required, given corporate strategies and objectives. These reasons may be derived from the Project Mandate, but clarification should be sought if that is not the case.
The options considered should be included in sufficient detail for stakeholders to determine which gives the best ROI for business. It will include estimated CB&R (costs, benefits and risks for each option).
PRINCE2 recommends that options should include:

  • Doing nothing (default)
  • Doing the bare minimum
  • Doing something

As the project progresses, the Business Case will be continually updated with regard to CB&R.

Expected Benefits and Dis-benefits

The project’s products are determined by the benefits desired from the project. The business case should list the tangible and intangible benefits anticipated from the selected option. (Intangible benefits might be “increased employee morale”)
Details of each benefit should include:

  • How the benefit fits with corporate objectives and strategy
  • How It relates to the projects outputs and outcomes
  • How much benefit is expected (with tolerances)
  • How the benefit will be measured (even for intangible benefits)
  • Who benefits

To enable continuous justification of the project:

  • The relative importance of benefits should be recorded
  • Benefits should be objectively quantifiable

In addition dis-benefits should be included in the Business Case. Dis-benefits are those perceived as negative outcome by at least one stakeholder. The difference between risks and dis-benefits is that risks are uncertain whereas dis-benefits are expected.

Time-scales and Costs

Information about time-scales will include:

  • When resources will be required
  • When benefits will be realised and verified
  • The earliest / latest feasible project start / end dates

Information on costs will include:

  • Project costs based on the Project Plan
  • Ongoing operational and maintenance costs
  • The basis of costings
  • Funding arrangements

The time-scales and costs in the Business Case are used as basis of tolerances as the project progresses.

Risk Profile

Projects are uncertain, and therefore risky. Risks can impact both benefits and resources both positively and negatively. The justification needs to take major sources of risk into account, so a summary of risks will be included in the Business Case, and major risks will be. The recording of risk should  focus on benefits, and should therefore includes both project and ongoing operational / maintenance risks.

Investment Appraisal

An Investment Appraisal is a comparison of costs and benefits over a period of time (a fixed period, usually in years, or life of product). It will includes both project costs and ongoing costs, and is intended to demonstrate the value of the project to stakeholders.

The Life of the Business Case

There are 4 stages in the life of a Business Case:

  1. Development
  2. Review
  3. Maintenance
  4. Confirmation of Benefits

Developing the Business Case

Initially, the information in the Business Case is based on approximations. These estimates should be iteratively refined as more is learned, and this may lead to project replanning.

Process Development of the Business Case
SU An Outline Business Case is drawn up based on the Project Mandate. The Outline Business Case forms part of the Project Brief.
IP The detailed Business Case id based on the Outline Business Case and Project Plan, and may include information from the Project Brief and Risk Register. The Business Case forms part of the PID.

Reviewing of the Business Case

At various times throughout the project, the Business Case will need to be revised and re-evaluated to see if the project is still justified. This will occur:

When Why Who
At the end of the SU process To ensure project is worth investing in Project Board
At the end of the IP process To authorise the project Project Board
As a result of changes to the project’s anticipated costs, benefits or risks To assess the impact of the revision on the project PM
Following an exception To approve an Exception Plan Project Board
After each stage To authorise the next stage Project Board
During final stage To ensure project will provide expected benefits PM
As part of the benefits review To ensure that benefits are realised Corporate / programme management

Maintaining the Business Case

The Business Case is continually updated regarding CB&R. This is especially important at the end of each stage before the next is authorised. Updates will include benefits already realised as the project progresses.

Confirming the Benefits

Most benefits are realised after the project has closed, so a mechanism is required to confirm them after the end of the project. This is the purpose of the Benefits Review Plan, which is based on the Business Case. The Business Review Plan will include:

  1. Expectations regarding the scope, timing and responsibilities associated with review.
  2.  Information required to confirm that the benefits of the project:
  • What benefits are expected
  • What (objective) measures will be used to confirm these measurements
  • What baseline measures will be taken
  • How measures will be collected

The Business Review Plan may be updated during the project if benefits are realised before it is closed.

Responsibilities

Corporate or Programme Management

  • Provides the Project Mandate, upon which the Business Case is based
  • Ensures that the Benefits Review takes place

Executive

  • Ensures the Business Case and Benefits Review Plan are written and approved (doesn’t necessarily write them)
  • Secures funding
  • Ensures that the benefits specified by the Senior User represent ROI, are aligned to corporate strategy and that they are realizable.

Senior User

  • Specifies the benefits
  • Realizes benefits by using products
  • Confirms their realization in the Benefits Review
  • Individual beneficiaries may be recruited to support the Senior User

Project Manager

  • Assists in writing the Business Case and Benefits Review Plan
  • Performs an Impact Analysis following changes in CB&R
  • Assesses the Business Case at the end of each Stage
  • Reporting benefits realised during project in the End Stage Report

Project Assurance

  • Assist in developing the Business Case
  • Monitor the Business Case to ensure that events outside the project have no impact upon it
  • Verifies and monitors the Benefits Review Plan

The Seven PRINCE2 Principles

Introduction

PRINCE2 is founded upon 7 principles (made explicit in the 2009 refresh) that underpin best practice in project management. These principles are therefore fundamental to PRINCE2, and should be used as a guide to applying of the PRINCE2 methodology. A project cannot be said to be a PRINCE2 project if any of these principles is neglected. They provide the basis for decision making throughout the project’s life-cycle.

The 7 Principles (JERSEPT)

  1. Continued business justification
  2. Learn from experience
  3. Defined roles and responsibilities
  4. Manage by stages
  5. Manage by exception
  6. Focus on products
  7. Tailor to suit the project environment

Continued Business Justification (2.1)

There should be a clear ROI (return on investment) for any project, both at the start and throughout its life-cycle. Even compulsory (legislated) projects require justification of the approach taken to solve them. The justification may change as the project progresses, but it should remain valid. If a project becomes unjustified it should be stopped.
The justification is documented in the Business Case, which includes the resources required and the expected benefits for the project. The business case should, of course, be aligned with corporate strategy.

Learn from Experience (2.2)

Throughout the project, all participants should seek out and document learning opportunities. A lesson that doesn’t result in change is not a lesson learned!
At the start of a project, the project team should:

  • Consider seeking external expertise
  • Review similar past projects

As the project progresses, they should:

  • Include lessons learned in all reviews

And at the close:

  • Pass on learning at the end of the project

Defined Roles and Responsibilities (2.3)

For a project to be successful, it is essential that:
1. the right people are involved in the project
2. these people know what is expected of them
3. what they can expect from others
The project must have a clearly defined management team who represent the interests of:

  • The business – to ensure that the project is justified in terms of ROI
  • The users – to ensure that the project produces the intended benefits
  • Suppliers – to ensure that the project is feasible from a resourcing perspective

Manage by Stages (2.4)

Every project has a “Planning Horizon”, which is the limit beyond which meaningful planning cannot be undertaken.
To tackle problems associated with this, PRINCE2 divides work into discrete Management Stages. Detailed planning can be undertaken for current stage, whereas plans for the whole project can remain in outline. A significant advantage of this approach is that learning from earlier stages can be carried forward into later stages.
Between each stage are control points where senior management (i.e. the Project Board) undertakes to:

  • assess the previous stage
  • verify Business Case
  • review plans for next stage
  • decides whether or not to continue with the project

Longer stages therefore provide less control management control, whereas shorter stages require more work from management.
A PRINCE2 project as a minimum of two stages:
1. The Initiation Stage
2. One additional Management Stage, which includes project closure.

Manage by Exception (2.5)

It is essential that decisions are made at the right level in the organisation, and that each level of authority:

  • Has delegated authority to manage at that level
  • Is accountable to higher levels of authority
  • Can delegate to lower levels in order to use time effectively.

To achieve this, PRINCE2 projects define tolerances at each management level for:

  • Time
  • Cost
  • Quality
  • Scope
  • Risk
  • Benefit

Changes inside tolerance are dealt with at that management level without bothering higher levels. Issues outside tolerance are Exceptions, and are reported to level above. For example, the Project Board normally only hears from the Project Manager in regular reports, unless there is an Exception in which case they are alerted immediately.

Focus on Products (2.6)

A project should focus on outcomes rather than activities. Without clearly defined outcomes, there are no shared expectations. As a result, the project risks:

  • rework
  • scope creep
  • acceptance disputes
  • customer dissatisfaction

To mitigate these risks, PRINCE2 Product Descriptions detail outcomes (esp. for quality) and acceptance criteria. Product Descriptions should define such things as the product’s purpose, composition, derivation, format, quality criteria and quality method.
Product Descriptions define the scope of project, and provide the basis for planning resource requirements, dependencies and schedules.

Tailor to Suit the Project Environment (2.7)

PRINCE2 is applicable to all project types, but approach and effort must be tailored as appropriate the project’s needs: environment (e.g.. existing management structure), size, complexity, importance, capability and risk (e.g.. reporting frequency, emphasis on risk).
The Project’s Project Initiation Document (PID) should describe how PRINCE2 will be tailored for that project.

Metaphors for Programming

In the opening chapter of “The Developer’s Code“, author Ka Wai Cheung describes the book as “a collection of lessons, observations, and missteps I’ve gathered, first-hand, in our industry”.

The following is my summary of Chapter 2: Metaphor. Whilst I’ve tried to do justice to the author’s intentions, the interpretation is my own. I’ve included my comments in italics.

Follow metaphors with care

In other professions (cooking, music) people’s experience tells them if something is good or bad. However, programmers must resort to metaphors like “software architect” because people have little experience with code. Whilst metaphors are useful, they can distort our priorities if relied upon too heavily.

Several examples follow:

Plan enough then build

The architecture model over-emphasises planning over writing code and testing it. Agile development unbinds us from the metaphor.
In construction, the product requires detailed specifications:

  1. re-work is expensive or impossible
  2. task scheduling is complex
  3. raw materials are limited
  4. errors during production are expensive

Whereas in software production:

  1. the construction remains malleable
  2. the product is testable in (near) real situations
  3. errors during production needn’t equate to issues in use
  4. the product can be duplicated at no cost

Launch is just the first release

Buildings are built and then opened. In contrast, software can mature and develop over time, and users are increasingly accustomed to change.
Launch is a milestone, but not the end of a software products life-cycle. Get the important stuff right and fix the small stuff over time.

The “ivory tower” architect is a myth

Real architects don’t hammer in nails, but the division between designing and building software is arbitrary. In reality, software architects need the insights that are only available to coders. Software architects must, therefore, find time to do at least some coding.

Throw away your old code

Unlike real-world materials that are often reused, we don’t need to keep old code because:

  • Unlike real-world materials (bricks, metal and so on) the raw elements of code are inexpensive and easily replaced
  • Commented-out code is distracting
  • We rarely uncomment code that we comment out
  • Resurrecting old code is often more expensive than rewriting it

An important exception to this rule is where you want to keep a record of your thinking rather than the code itself.

Diversification over specialization

Specialised roles in building occur in different physical places, whereas our tasks are all on the same screen. For this reason, there is no reason we can’t be good at a range of jobs.

The logic of this argument is flawed, and the conclusion questionable. While over-specialising can be dangerous, trying to be a jack-of-all-trades can be equally hazardous.

My guess is that this is just a weak example of a powerful idea: the author is trying to illustrate the idea of architecture as a limiting metaphor rather than provide a justification for becoming a generalist.

Metaphors Hide Better Ways of Working

Metaphors both liberate and constrain our thinking.
Where they are a problem:

  1. Learn to see where metaphors are being used and their limitations
  2. Try changing metaphors

The C# Null Coalescing Operator

Introduction

A the ?? operator is a binary operator that is used to simplify checking for null values.

If the left operand is not null, it returns the left operand, otherwise it returns the right operand.

For example:

This means that:

Is equivalent to:

And is even shorter than:

The ?? operator is right-associative, which means that it is evaluated from right to left. This means that:

is evaluated as:

Rules

  • Arguments can be reference types or nullable types.
  • If both arguments are constant, the result is not considered to be a constant.
  • Both arguments must be of the same type.

Uses

Assigning a Non-Nullable Type from a Nullable Type

int result;
int a? = null;
result = a ?? 1; // Assigns 1 to result.

Lazily Instantiated Properties

Avoiding Double Evaluation
Chaining

References