One of the classic papers that I remember most from this course is The paper by David Parnas and Paul Clements entitled A Rational Design Process: How and why to fake it. This is one of the first classic software engineering papers (before agile programming, extreme programming, or whatever the latest buzz-word fad is) where the authors seemed to acknowledge that a formal software design processes will always be an idealisation. We had spent much of the semester reading about the idealized process, including the aforementioned On the Criteria... paper, and this paper really brought out some interesting thoughts during our discussions.
Among many of the examples Parnas and Clements give as to why the process is an idealization, is that in most cases the people that request the software system in the first place do not know exactly what they want and are unable to communicate everything that they do know. I think most software engineers with any experience trying to use a formal development processes can attest to this observation.
The authors also note that even if the engineers knew all of the requirements up front, other details needed to implement a system that satisfies the requirements do not become known until the implementation is already underway. Had the engineers known this information up front, it is likely that the design would be different. One of the most important points made is that even if we did know all of the facts up front, human beings are unable to fully understand all of the details that must be taken into account in order to design and build a correct (and ideal) system.
The process of designing software, according to Parnas and Clements is a process by which we attempt to separate concerns so that we are working with manageable amounts of information at one time. However, the catch-22 is that until we have separated concerns we are bound to make errors due to the complexity of the system. Furthermore, as long as there are humans involved there will be human error, even after the concerns have been separated by the design process.
Parnas and Clements make several other points including the fact that even the most trivial projects are subject to change due to external reasons, and these changes can invalidate design decisions or make decisions non-ideal given the new circumstances.
Sometimes we use a projec to try out a new idea - an idea that may not be derived from the requirements in a rational process.
However,despite the fact that the rational process is unrealistic, it is still useful. It can be used to drive the project, but it should be expected that there will be deviations and backtracking. In the end, the authors argue that we should "fake it" and produce quality documentation written as if the final design of the system was designed following the ideal rational process. We identify what documents would be produced had we followed the idea process, and attempt to produce those documents in the proper order. Any time information is not known, we make a note of it in the documentation where that information would have gone and move on as if that information is expected to change. Any time a design error is found the all of the documentation (including documents made in "previous steps" in the process) must be corrected. No matter how many times we back track, or some external influence changes the requirements, the documentation is updated and in the end the final documents are rational and accurate.
Documentation plays a major roll in the process, yet many times we don't seem to get it right.
Most programmers regard documentation as a necessary evil, written as an afterthought only because some bureaucrat requires it. They don't expect it to be useful.
This is a self fulfilling prophecy; documentation that has not been used before it is published, documentation that is not important to its author, will always be poor documentation.
Most of that documentation is incomplete and inaccurate but those are not the main problems. If those were the main problems, the documents could be easily corrected by adding or correcting information. In fact, there are underlying organisational problems that lead to incompleteness and incorrectness and those problems ... are not easily repaired [1]
I think this classic paper is a must-read by anyone trying to implement a strict waterfall process (or trying to impose such a process on their underlings). Despite being written over 20 years ago it seems like there are still lessons to be learned.
[1] Parnas, D. L. and Clements, P. C. 1986. A rational design process: How and why to fake it. IEEE Trans. Softw. Eng. 12, 2 (Feb. 1986), 251-257.
