Introduction to Reflecting (Actual vs. Plan)

and the PEP Tool

Lynn Robert Carter


Macro Lifecycle Model

At the heart of the Personal Engineering Process (PEP) is a belief that lessons from the past can be leveraged to produce better results on future projects. When people are open to the patterns of the work they do, it is easy for them to see recurring activities in their projects. These patterns of work are often called lifecycles. The most fundamental lifecycle in PEP consists of four major recurring activities:
  1. Plan
  2. Perform
  3. Reflect on actual versus plan
  4. Reflect on quality

Assessing Deviations, Impacts, and Probability of Recurrence

A plan is successful when it helps you and your team produce software faster than you would have without it. A plan can also add value when it helps you understand how you accomplished the planning activities in order to improve them.

People are not born knowing how to plan effectively. Most things people call "plans" are little more than statements of desire, as there is no basis in reality. Planning is a skill that is mastered by doing and can not be mastered by reading books or attending lectures. As with most skill-based activities, mastery comes from a sequence of steps:

  1. Contemplate what is required and how it can be done
  2. Perform the activity in alignment with the contemplation
  3. Compare actual performance with contemplated performance looking for deviations
  4. Evaluate the deviations to determine where or not they are significant
  5. For significant deviations, decide whether or not the impact of the deviation and/or the probability of recurrence warrants further attention
  6. For significant deviations with a high enough impact and a great enough probability of recurrence, assess the root cause of the deviation and how the activity might be done differently to correct the situation in the future
  7. Establish mechanisms likely to support changing behavior to align with the new insight

The PEP System supports each of these steps. The Plan and Perform activities described in the earlier sections address the first two steps. This section addresses steps three through six. (The seventh step is also addressed by the planning and performing PEP activities, but it is discussed in the Leveraging Lessons Learned section.)

The Reflection (Actual versus Plan) Main Page is similar to the previous PEP System Main Pages. It provides feedback about where you are in the process guides you through the process steps.

Before serious reflection can begin, you need to capture information about what happened. The actual complexity for each of the deliverables is required. This is accomplished by clicking on the link indicated below.

Figure 1

The figure below shows a page very similar to a page you completed during planning, only this time, you know how many lifecycle tasks were actually performed and how many deliverables you actually produced. (Please assume that you will be producing the Lessons Learned deliverable and count it as an actual, as that is what you are doing now.)

It is important for you to be as accurate as you can be about each of these figures, as they will be used to determine your performance baseline. (A baseline is set of data that describe your performance.) As the system gathers more information from you about more assignments, this baseline will become more and more useful predicting effort based on reasonable complexity estimates.

The effort log specifies the amount of effort, to which deliverable that effort should be charged, and to which lifecycle task that effort should be charged. Please don't guess on the number of lines. Count the lines by hand, use an editor or word counting utility, or write and run your own program. Information from the effort log and the actual information provided in the page below provides all that is needed to begin the reflection. To specify the actual complexity of each deliverable, click on the link, as shown below, for each of the deliverables.

Figure 2

If you made mistakes during planning, now is the time to correct the situation. For example, you might not have understood that "NT + ND" meant the number of tasks in the lifecycle provided to you by your professor (7 for this class) plus the number of deliverables required by the project (3 is the appropriate number for the first programming assignment; a project plan, a programmed solution, and a lessons learned). If you didn't plan to do one of the lifecycle tasks and didn't actually perform it, enter the actual number you did perform (plus the number of deliverables).

Please do not insert 10 just because this document tells you that this is the expected answer. Regardless of what value you specify, be prepared to explain the value in the reflection.

Figure 3

After you've records the actual figures for each deliverable, click on the Save link as shown below.

Figure 4

With the actual complexity captured, the system is now ready to walk you through a process of reflection on your performance and the deviations that might exist between your actual performance and the plan.

Figure 5

You need to ask yourself a question: "How close was my actual figure to my plan?" It is reasonable to expect that your actual may be quite different, and that's okay. I would suggest that anything that deviates more than 20% up or down is significant.

If any of the deliverables are significantly different from the plan, you need to assess the impact of this deviation and the probability it will happen again. The nature of complexity deviations usually mean they have major impact as they are used to drive so much of the planning process. While you are mastering the use of the PEP System, deviations are likely to recur without some concrete action on your part. Therefore, you are strongly encouraged to classify most deviations over 20% as:

Should you classify the deviation as insignificant, less than a major impact, or less than likely probability of recurring, PEP will suggest you skip the reflection step. For this class, don't do it! If any of the reflections is found to be blank, no points will be awarded for that element of the reflection.

The figure below shows that the actual code produced was much more than 20% below the estimate. The assessment values are set according to these recommendations and the Save button is being pressed.

Figure 6

Since there was an assessment of some significant event with at least a major impact that is at least likely to recur, the system suggests a reflection is needed.

Regardless of the system's advice, ECEN4553 students are required to comment on each and every element of the Actual versus Plan Reflection. If the figures warrant comment on the root cause for the deviation, provide it. If the figures are close enough, provide a description of what you did that worked so well and how you expect to keep doing it.

Figure 7

In Figure 8, the student saw that his assumption about the size of the solution was off the mark. Just because a student produces a program does not mean it should be dramatically larger than the professor's solution. A student may require a great deal more time, but the size of the solution may not really be that different. This will be particularly true where the solution is little more than an adjustment to existing code.

When you have finished writing your reflection on your experience, be sure to click on the Save button as shown in Figure 8.

Figure 8

The sequence of reflection activities is the same as the sequence during planning. Once you have finished reflecting on complexity, the next issue is the estimation of the effort to produce each of the deliverables. Remember that the amount of effort to produce a deliverable is computed by multiplying a complexity figure by a production rate.

Figure 9

Even though the system indicated that the production rate was off by only 18% in Figure 10, the student realized there was a much more significant issue here. The planned production rate was set to be three times the professor's rate. Experience has shown, for the first assignment, that a figure of four would be better.

When you've finished your assessment, click on the Save button as shown.

Figure 10

Based on the assessment, the PEP System advises you to perform the reflection. (Notice that the assessment step has been marked "Done" and the reflection step has been marked "Active".) To perform the reflection, click as shown in Figure 11. (Just because something is marked "Done" does not mean you can't go back and change things. The Finalize button is different. Once you click on that button, the system changes state and you will no longer be allowed to make changes to these pages. If you should accidentally click the finalize button, your professor can reset the system state for you.)

Figure 11

The student correctly saw that the -18% error in the effort estimate masked a 22% error in the production estimate. Had the student estimated the complexity to match the actual number of lines of code for the assignment, the error would have been +22%.

The student focussed on the production rate for the "Programmed solution" with a reasoning that much more time will be spent doing the implementation than filling out the PEP forms. Therefore the impact of a smaller error on the programmed solution would be more likely to cause more pain in the future. Secondly, the student believed that the time required filling out the planning tool pages was really a lack of understanding about the tool. In hindsight, the process did not really appear to be that difficult and should not take anywhere near as much time in the future.

This focus and the related assumptions are reasonable, but this reflection would not be awarded full credit. The assumption that the 75% error was a first time fluke may not be remembered. Capture all of your thinking as you reflect. While these thoughts may seem so obvious now, there is a real chance that you will be clueless about them several days or weeks from now. A few moments spent capturing your thoughts now could save you significant pain in the future.

When you've finished capturing your reflection, be sure to click the Save button.

Figure 12

With the completion of the reflection on production rates and effort, the next step is to reflect on how you allocated the effort to the various lifecycle tasks. This is accomplished by clicking as shown in Figure 13.

Figure 13

The PEP System walks you through each of the lifecycle tasks from the planning and performing phases of the project. You are to assess whether any of the deviations between your planned allocation and your actual allocation:

Click, in turn, on each of the lifecycle tasks with active links and assess the deviations. The example below shows the student clicking to assess the planning deviation.

Figure 14

The PEP System wants you to reflect on all of the steps you planned to perform to accomplish the lifecycle task. You need to assess the deviation in terms of the impact the deviation had on your ability to accomplish your work in a timely manner with a high-quality result. Nailing the estimate is meaningless if you rushed to meet the goal and generated defects. Logging effort that differs from reality in order to have the numbers appear close is a waste of your time. Your grade is not influenced by how close these reports indicate your actual was to your planned efforts. You are required to spend just as much time in this reflection in either case and the quality of your writing about your experience (positive or negative) will determine your grade.

Notice that the actual effort spent planning and the planned effort for planning are almost identical. How can this be if the effort estimated to produce the project plan had a 75% error? This is where you must take time to think things through. Reflection is not an instant gratification activity. To find critical information, you must understand things that do not appear to make sense.

If the effort to produce the project plan was off by 75% and yet the actual versus plan time allocated to planning are off by less than 2%, what's going on? Think back to the process you used to allocate time to the planning activity. You allocated 100% of the effort to create the "Project plan" to planning, but you also allocated some percentage of the time estimated to produce the "Programmed solution" to planning as well. (The effort estimate for the "Project plan" is the amount of effort to perform the seven PEP Planning Steps. The effort estimated for the "Programmed solution" was spread over all of the lifecycle tasks and the percentage of that allocated to planning was the time you spend understanding the problem so you would know what to plan.)

In this example, the student significantly underestimated the amount of effort to fill out the PEP System web pages and yet the total time spent planning was nearly the same as the plan. The only possible interpretation is that the percentage of effort allocated to the planning activity from the "Programmed solution" deliverable was significantly over estimated. While the data has been captured to reveal this, the current implementation of the PEP System does not produce a report to show this to you.

There wasn't a great deal to understand in this first programming assignment, assuming that you did a careful job during the question and answer assignment that preceded it. It wouldn't be too surprising for a student to overestimate effort required to understand the problem and to develop an idea on how to accomplish it. (If you're going to make a planning error, it is better to overestimate here than to underestimate.) Now the question is what will you do differently on the next assignment? A reasonable position would be to split the difference between the actual and the plan. In the case of the "Project plan" deliverable, a production rate of 8 might be reasonable (roughly an average of the actual production rate of 10.5 and the planned production rate of 6. With the allocation of the effort to the planning task from the "Programmed solution", determining the adjustment is a bit more difficult.

From the project plan, this student estimated 325 minutes needed to produce the "Programmed solution". This was produced by multiplying the complexity estimate of 325 Lines Of Code (LOC) by the production rate of 1 minute per LOC. The student allocated 45% of the effort to produce this deliverable to the planning lifecycle step as the time needed to understand the problem, understand the tools to solve the problem, and to develop a high level design. 45% of 325 minutes is 140.25 minutes of effort to produce the "Programmed solution" to be allocated to the planning task (understanding steps). Had the estimate for the "Project plan" been precisely what was needed, the effort allocated from the "Programmed solution" would have been 41 minutes too high. (Take the 45 minute overestimate of the effort for the "Project plan" (computed from 105 - 60 as shown in figure 12) and subtract the 4 minutes the plan underestimated the actual effort needed (computed from 206 - 210 as shown in figure 14).) The allocation from the "Programmed solution" should have been roughly 99 minutes. Since the total effort estimate for the "Programmed solution" was in error as well, 99 minutes of a total of 225 minutes should have been allocated. Therefore an allocation of 44.9% of the total estimated effort for the deliverable would have been required. In this case, it appears the guess the student made was right on the mark.

Please understand that this is a made-up example. You need to go through similar thinking. When it comes time to capture material in the "Record root causes..." text block, producing information such as what you've just read would be far more useful than a statement asserting that a change in the allocation of effort might been needed. This example shows that what first might appear to be an allocation problem actually turns out to be no problem at all.

Figure 15

In Figure 15 above, the student indicates that the allocation of effort for planning does not really need to change. If the logic were little more than saying 206 is not too different from 210, the student has made a mistake. If the assessment was made based on analysis as provided above, the student would be correct.

After working through each of the allocations (the rest are much easier as there is only one deliverable contributing to the effort for the performing lifecycle tasks), the page would be similar to the one shown in Figure 16. Be sure to click on the Save.

Figure 16

Due to the fact that the student did not consider any of these deviations to be significant or high enough in impact or great enough in probability of recurrence, the PEP System suggests skipping the related reflection activity. (See the figure below.) Don't do it. Regardless of your assessment, you are required to reflect on this experience. When there are deviations, analysis as provided above needs to be performed. Even when there appears to be no major deviations, you need to consider whether or not your experience in this assignment will be typical and whether or not the allocation you used will be useful in the future.

If you skip the reflection step as this student has, you will receive no credit for this part of the reflection.

Figure 17

The reflection on the elaboration of tasks is intended to assist you reflect on the kinds of activities you should be performing during the performing phase of your project. The lifecycle tasks provided by your instructor are just too high level to really help you during the implementation. Everyone knows such a lifecycle and having it appear during the logging activity does little to assist you. By allowing you to elaborate the lifecycle task into a sequence of one or more steps, the PEP System allows you to record what things you believe you need to do during the planning activity, and then forces you to look at that list as you log your time. It is hoped that by looking at the list, you will be reminded that you had planned on doing all of these steps. Once you are reminded, maybe you will remember why you thought they were important and actually perform them.

Now that the performing phase is over, which of these steps were useful? Which of these steps were not useful? What steps should have been on the list? As with previous reflections, you must work your way through each of the lifecycle tasks. In this case, you only need to address the performing phase tasks as those are the only ones PEP allows you to elaborate. Use the links as shown below to look at each elaboration.

Figure 18

As with the other reflection activities, you begin by performing an assessment. Once you have finished all of the assessments, the PEP System will determine whether or not any of the assessments warrant proceeding to the capture of reflection data. Again, ignore the PEP System suggestion that you can skip the reflection step for any of these items.

Figure 19

In this example, the student ran into problems due to the lack of good unit tests. This led the student to believe that he should have had a separate step in this task to think about unit testing and to produce a set of unit tests. The figure above shows this assessment and also shows the student clicking on the Save button.

In the figure below, the student has completed all of the assessments for all of the lifecycle tasks and is about to click on the Save button to move forward.

Figure 20

Since at least one of the assessments satisfied the PEP System requirement to recommend that a reflection be performed, the following page shows the student about to click on that link. Once again, if the system suggests that you can skip this reflection, please ignore it and do the reflection.

Figure 21

In this figure, the PEP System lists those lifecycle tasks for which you asserted something significant with high enough impact and great enough probability of recurrence. If you didn't rate any of them that way, the table body will be empty.

Capture your insights from your experience. Be sure to consider what kinds of lifecycle tasks steps might be useful for all of the following assignments. Spending a lot of time reflecting on how you could have done the previous assignment better is not as useful as reflecting on what you can learn from that experience that could help you with future assignments. Reflections that conclude with how you will do things better in the future will receive more credit than ones that only address how the past could have been better. What do you think of this student's reflection using the criteria provided here?

As always, be sure you click on the Save button when you've finished your reflection.

Figure 22

The reflection captured in the previous figure is reasonable and would collect full marks.

When you are finished with the reflection and save the results, the PEP System will present a page similar to the one that follows.

Figure 23

A common mistake is to use the Effort Adjustment feature of the PEP System when changes should have been made to the complexity estimates, the production rate estimates, or the effort allocation.

The reason to use the Effort Adjustment feature of the PEP System is to make final adjustments due to something unusual that you expect or know will occur. (e.g. If one of your team mates will require an unusual amount of help from you for an assignment and you expect it will not be a common occurence, use the Effort Adjustment feature.

If you find that more than one or two lifecycle tasks have adjustments, some of the adjustments are large (less than 50% or more than 200%), or multiple adjustments are all less than 100% or greater than 100%, you are probability not performing the earlier planning steps properly.

When you make adjustments, be very specific about why you are making an adjustment as opposed to changing previous planning values. Also be very specific on how you concluded that an 80% or a 120% adjustment was needed. Without such specific information, it makes it very difficult to do this portion of the reflection in a useful way.

Figure 24

In this example, the student did not feel that there was any event that occurred during the performing or planning phase that warrants an adjustment. Do not do as this student has done and ignore the capture of a reflection. Even if you believe no adjustments were needed, you must capture why you believe your recent experience will be a good predictor of future situations. If you skip this, you will not receive points for this element of the reflection.

Figure 25

Assessing how well you did in scheduling your effort requires you to compare when you planned to perform the various lifecycle tasks with when you actually performed them. In the following figure, you will see that the PEP System has displayed two values for each task for each date in the form n:m. The "n" value is the actual amount of time you spent on that task (as captured in the effort log). The "m" value is the planned amount of time from your project plan.

Think about how the project unfolded. Were there surprises? Did you have access to everything you needed in a timely manner? Did you do work early enough so you could take advantage of your professor's office hours to validate the correctness of your answers before it was too late to make changes. (Too many students fail to take advantage of the Professor's office hours. While the professor is unlikely to tell you exactly how to do the work or how to fix a problem, it is reasonable for him to tell you that you have a problem with your proposed solution and to suggest what you might need to reread in order to appreciate the issue. It is far better to have him find problems with your work before you turn it in than it is to have him find problems with your work once you assert you've finished it.)

Don't worry about the deviation between actual and planned effort. You've already addressed that issue in previous elements of the reflection. This element of the reflection should be spent thinking about what transpired that made it difficult or easy for you to make progress. If you had scheduled more time with your team mates earlier, would that have made things easier? If you hadn't scheduled all of the planning effort or other lifecycle tasks, for the day the assignment was due (or the weekend before), would you have been able to peer review your work with other team mates or the professor and had things go more smoothly?

Figure 26

One final time, do not skip the reflection capture as this student has done. Regardless of how close your planned schedule appears to have matched your actual effort, the questions listed above need to be considered in the light of your future assignments.

Figure 27

Capturing the amount of effort and insight from your reflection experience is the last step in the Reflection (Actual versus Plan) phase of the PEP process. If you have done this activity in several blocks of time, please use the system to create several log entries, one for each block of time. (Do this by clicking on the Effort log link to the right side of the page. From the effort log, you will be able to create a number of effort log entries. If you click on the final link as shown above, you will only have the opportunity to create one effort log entry.)

Figure 28

When the effort log is completed, click on the Update button as shown above. The PEP System will update the log and return you to the Course Main Page as shown below. At this point, you are now waiting for customer feedback so you can begin your reflection on defects.

Figure 29


lrc@sei.cmu.edu
Revision 1.2 (1999/09/11 22:25:00)