The Long View
Macintosh Common Lisp
In the early 1980's, two advanced computer languages were each widely promoted as the language of the future. One of them, Smalltalk-80, was new, as indicated by the number in its name. The other language was an old-timer. Lisp was invented by John McCarthy in the 1950's, and was well known to the world since about 1960. It had a very deeply committed following within the Artificial Intelligence community, but had not yet gained widespread commercial use. It was old, but it was considered to have been invented ahead of its time, and to have not yet realized its full potential.
Although they were often mentioned together, these two computer languages were opposites in a number of ways. Smalltalk was designed on, and intended to be used on, graphical displays that were not yet commonly available. The inventor of Smalltalk-80 pictured his language eventually being used by children for fun and education. Lisp was invented in a time when computer programs were written on card punch machines and ran in glass rooms with air conditioning built into the floor. Lisp was not designed as an graphical language, or for use by children. It was designed for the study of human reason, language and intelligence. Its development was funded by the military, who wanted to use it for military purposes. Culturally, Lisp and Smalltalk-80 were worlds apart.
What Lisp is Like
I am only a lisp-tourist, not a lisp-evangelist and I will not try to convince you about Lisp one way or the other. There has been a lot written about it already. Its syntax is different from other languages, and this turns a lot of people away from it. Okay, here it is. If I want to add 2 and 2 in Lisp I have to write it like this: (+ 2 2). The parentheses are obligatory. Does that seem repulsive to you? Many people seem to think it is, but seriously, is this so different from your usual way of writing it? Probably, you like to write it as 2 + 2, or if you are a Forth or Postscript programmer, 2 2 +. Those three pretty much cover the possibilities. Is one of them that much more natural, or just more familiar? Lisp has its reasons. It does it this way because + is a function, 2 and 2 are its arguments, and it expects all function calls to be lists in this form. A lot of people I respect consider Lisp to be the greatest of all computer languages. For me, that's enough reason to get past the parentheses and give it a closer look. But if the parentheses and putting the plus operator first are intolerable for you and you can't get past it, well then it is just over between you and Lisp, and there's nothing more to say.
There is a slightly deeper problem you might have had with Lisp. Not so much an issue with the language, but rather with the way people write about it. To get a feel for the difference between Smalltalk and Lisp, you could compare the lovely, playfully illustrated books on Smalltalk by Adele Goldberg, published In 1983 and the official book about Lisp by Guy Steele, published the following year. Common Lisp, by Guy Steele is actually a great book. A copy of its second edition came with my copy of Macintosh Common Lisp, a long time ago. I still have it, and I like it a lot now. But the first time I opened it, it freaked me out. It is a serious all-business specification for the language, complete with numerous statements about changes being made by the standards committee as the book was going to print. Of course, this is a language reference, not an introduction. But books for beginners are not much easier going. Lisp is a language designed by computer language theoreticians for their own purposes. They can't write about it without discussing the pros and cons of dynamic versus lexical scoping or the difference between applicative and normal order of evaluation. Eventually, all programmers need to understand the theoretical ideas that underlie the design of the language they use. But maybe it is better to get some practical experience first. If you get a book on how to program in C it will start you out in a light-hearted way, teaching you to write "hello world". When that is done, you haven't learned much about programming or about C. All you have learned is that programming in C it isn't so hard. You are on your way, even though you don't really know a thing. A beginners book on Lisp will start you out comparing code for evaluating the factorial using head recursion versus tail recursion. If you were starting to learn programming because you needed to fulfill some particular practical need, this may not be exactly what you are looking for. On the other hand, if you have some experience programming and now you want to understand what it is really about, you would be wise pick up a copy of the Steele (or some other) book and learn Lisp and the theory that goes with it. Moreover, Lisp, like Smalltalk, is interpreted (or at least seems that way when you use it) and so can be very interactive with the programmer. This makes it a great learning tool. Lisp has a certain style not seen everywhere. Programming in Lisp is like writing. You write it in short complete sentences. Each one can be tried and tested by itself, and you can can make each sentence work to your satisfaction before you go on to the next. Any piece of your program can be executed at any time. And like Smalltalk, Lisp has debugging facilities built right into the language.
Lisp on PCs (was hard to do)
Lisp implementations had always been starved for resources. Lisp uses a lot of memory, and memory was not plentiful on any kind of computer in the early days. In the early 1980's, the thawing of AI winter had released a lot of new funding for artificial intelligence applications, and a lot of that went toward buying expensive specialized computer equipment designed to speed up execution of Lisp. These specialized computers, called Lisp Machines, were made by a handful of commercial vendors with an eye on industry or government research operations. Serious Lisp users usually tried to get access to these (if they could), or otherwise used time-sharing mainframe systems with lisp implementations. A few companies were creating Lisp implementations for general purpose minicomputers, and increasingly also for the workstation market that was putting minicomputer-strength equipment in (relatively) small boxes on the desks of scientists and engineers. Prominent among these Lisp software companies were Lucid and Franz.
With the dramatic success of the IBM PC in the early 1980's, it was obvious that there would someday be lisp implementations for personal computers. But the limitations of the early PC's 16 bit processor and its hobbled memory addressing scheme meant that a Lisp running on the PC would be little more than a toy. Lisp did not take off on the PC until the x386 computers with a 32 bit flat addressing space became plentiful in the late 1980’s . But the Macintosh, despite its well-known limitations, used the same Motorola 68000 CPU used in many engineering workstations. The original Macintosh had only 128k bytes of memory, but it this was more than most PC's at the time, and a number of third party memory expansion kits were available in almost immediately. Apple announced their own memory-enhanced Macintosh within a few months, and it was available in the fall of 1984. Seen this way, the Macintosh was not a more powerful PC, but rather a small inexpensive workstation. There was reason to think that it could be used as a Lisp platform.
Lisp on the Mac
The availability of Lisp environments on the Macintosh was tracked in the pages of MacTutor magazine by Contributing Editor Andy Cohen of Hughes Aircraft. In Issue 6 of MacTutor, in June, 1985, Andy wrote the first of his Lisp Listener columns, describing the prospects for Lisp computing on the Macintosh. At that point there was a lot of work being done, but no finished products actually available. David Belz was offering his XLisp, still under development, unfinished, and admittedly not yet fully functional. XLisp is finished now, is still around, and runs on a variety of platforms, including Mac OS X.
In the next issue of MacTutor, Andy described a pre-release version of the first commercial version of Lisp called ExperLisp. Expertelligence, the company behind ExperLisp, belonged to Denison Bollay. It would continue to offer groundbreaking Macintosh Lisp products through the 1990's. From the beginning, ExperLisp was a uniquely Macintosh product. The first version had three kinds of windows: a listener window, text editor windows, and a Graphics window in which programs could draw graphics using a 3-dimensional version of Turtle graphics. Experlisp cost about $500, which seemed like a lot so expectations were high. Coming to market so quickly, it was natural that it was still a little buggy, and also that there would be a rapid succession of versions sporting different features. Before it would be over, here would be a myriad of product names: ExperLisp, ExperLogo, ExperCommonLisp, ExperProlog. But the implementations were criticized as being incomplete versions of the language. Common Lisp is a large language with a lot of features. The first Macintosh version might not be expected to implement the entire thing, but users porting their working programs to the Macintosh expected them to just work. Eventually, Expertelligence would obtain a distribution license for Procyon Lisp, which was a very complete Common Lisp. But that basic system cost $650 (full system $2000). And by that time, Apple's purchase of Coral's Allegro Common Lisp was overshadowing everyone else. In the 1990’s, Expertelligence offered a number of amazing Lisp products, including a great class browser and GUI generator (Action!).
MacScheme was the second commercial Lisp for Macintosh, written by Will Clinger, John Ulrich, Liz Heller, and Eric Ost, and offered by Semantic MicroSystems starting in August 1985. Scheme is a modernized and simplified version of Lisp. It was better suited for implementation on a small computer and MacScheme was well received at the time. It was a surprisingly functional version of Lisp that ran on early (128k and 512k) Macintosh computers despite their limitations. MacScheme was well suited to educational purposes and for a while, Semantic MicroSystems offered a student version for only $70. It was widely used in undergraduate computer science courses.
The most desirable features for any Macintosh programming system in those days were direct access to the ROM toolbox, and the ability to make stand-alone (double-clickable) applications. ExperLisp offered direct toolbox access early on, and wrapped the toolbox interface in Lisp so that it looked very much like the way things were described in Inside Macintosh (e.g. (paintoval '(top left bottom right))). MacScheme offered a version called MacScheme+Toolbox, which also offered Lisp wrappers for practically all calls into the Macintosh ROM. With the +Toolbox version, it was possible to make a complete Macintosh program using MacScheme. Making a stand-alone application was a tougher problem. You could store a snapshot of the heap, that could be started by double-clicking it, but the Lisp interpreter and runtime were essential. The solution to that was to basically distribute a stripped down version of the Lisp runtime, embedded into the application. All Lisp systems eventually offered this facility. But few commercial programs were actually created this way.
Coral Allegro Common Lisp
In 1987, Coral Software of Cambridge Massachusetts introduced a complete Common Lisp implementation for the Macintosh. Very soon afterward, they entered into a joint development agreement with Franz, and their product was renamed Macintosh Allegro Common Lisp, to match the branding with the rest of the Franz Lisp line. The Macintosh product was not an implementation of the Franz Allegro Common Lisp product, which ran on minicomputers, but was an independently developed Macintosh-only implementation. Promoted as "Lisp for the CDR of us", it was unique in that it claimed to implement every single Common Lisp feature described in the Guy Steele book. It cost $400. Another important feature was its inclusion of an object-oriented language extension, and its use of that to implement support for the Macintosh toolbox. Instead of just providing a set of Lisp syntax wrappers for Macintosh Toolbox calls, user interface features like windows and menus were implemented as objects. Coral eventually expanded their product line to three programs, the full-featured Allegro Common Lisp, an inexpensive stripped down version called Pearl Lisp, and Object Logo. In 1989, Apple acquired Coral, and rebranded Allegro Common Lisp as Apple Macintosh Common Lisp. Apple included Pearl Lisp on "Phil and Dave's Excellent CD", one of the first in the Apple Developers' CD-ROM series, and that was the last that was seen of it. Macintosh Common Lisp lived on, went through a series of changes, and its descendants are still available today.
Lisp Objects
Being able to access the toolbox and make real Macintosh programs in Lisp was great, but it wasn’t what you really wanted. You really wanted Lisp to make it easier to write a Macintosh program. Everybody knew it was hard to write user interface code in procedural languages. The solution, discovered at Xerox PARC, was to use an object oriented language with objects representing the user interface elements. Now we wouldn’t do it any other way. Lisp was not originally an object-oriented language. When the advantages of objects became clear in the 1980s, several competing object-oriented extensions appeared. The first version of Macintosh Common Lisp included one of these, called Object Lisp, designed by Lisp Machines (a maker of specialized computers for running Lisp). Two other object systems were in play at that time, one called New Flavors, developed at Symbolics (a corporate relative and competitor of Lisp Machines), and one called CommonLoops from Xerox PARC. In 1987, meetings among proponents of New Flavors and CommonLoops led to a release of a specification for a new system, called the Common Lisp Object System (CLOS). Seeing the writing on the wall, The MCL team wisely replaced Object Lisp with the Common Lisp Object System in version 2 of Macintosh Common Lisp.
Lisp is a different kind of language, and it will not come as a surprise that the way objects are implemented in Lisp is different from Smalltalk-80, or in object-oriented extensions of Pascal or C. Unlike those languages, CLOS does not employ message passing between objects. Instead, functions that can operate on objects (called generic functions) look and behave just like other functions in Lisp, except that they can have objects as parameters. In addition to making manipulation of objects look like the rest of Lisp, this means that generic functions can be passed as arguments to other functions. This ability is central to the philosophy and practice of Lisp programming. But because generic functions do not live exclusively within the namespace of classes, they do not have exclusive access to an object's instance or class variables, which are called slots. In fact, there are no private instance variables. Slot values can be accessed freely by anyone using a generic function called slot-value. The result is that an object is more like a specialized public data type, and generic functions are able to do equivalent things to data of all types. Still, the implementation of a generic function must be be class-specific, and so classes need to have their own implementations of all generic functions that can be used with them. Class-specific versions of generic functions (called methods, as in Smalltalk) are part of the class specification. Methods do not have special access to class and instance variables implicitly, or through a ‘self’ keyword, but they can get access to the environment of their class using the macro with-slots. Within the context created by with-slots, the slot variables may be addressed simply by name. Like Smalltalk, there are both class and instance variables, and the difference is declared at the time the class is defined.
Subclasses may override the methods or slots of their superclasses. When this happens, the subclass is said to have shadowed the superclass version of the method. In Smalltalk, it is common for an instance method to explicitly execute the code for the corresponding (overridden) method in the superclass. This is especially common in instance initialization, for which the entire chain of inheritance may have some responsibility. In CLOS, this is a little more complex and subclasses have less responsibility to explicitly call methods in their superclasses, but a method can do so using the function call-next-method. It is different, but CLOS is a full featured sophisticated class and object system. In 1987, it was a very advanced way to do Macintosh user interface programming.
A Lisper's IDE
Like Pascal and C, Lisp was invented in a time before before integrated development environments. The kind of IDE that we normally use now, like Microsoft's Visual Studio or Apple's XCode, was invented by THINK Technologies for their Pascal and C programming systems in the early days of the Macintosh. Before that (except for some experimental systems) programmers developed interactive tools that relied on character, not graphics, displays. Interactive programming meant using a text editor to write programs and a command line to type commands to compile and execute them. Programs usually took files as input, and wrote their output to files, so a text editor was also the tool you needed to evaluate the outcome of your program’s execution. The emacs text editor was developed by Richard Stallman and Guy Steele, and became the standard tool for Lisp programmers. Emacs was a text editor with a built-in command shell, so there was no need to ever leave the editor while programming. Emacs was written (at least partly) in Lisp, so the Lisp programmer could alter the behavior of the editor and shell, as well as writing other programs. Changes you made took effect immediately, without ever leaving Emacs. Although far from what we are used to now, it was an integrated environment for programming. Actually, a lot of Lisp programmers use Emacs as their IDE, even today.
Lisp development relies on two programs, a customizable text editor with a command shell, and the Lisp interpreter which runs in the command shell. The process used to execute Lisp expressions and to which results and error messages were printed is traditionally called Listener. Lisp programmers developed sophisticated character-based tools to help analyze and debug their programs, and all these ran inside the Listener. For example, trace is a Lisp function that prints (to the Listener) the sequence of calls made during execution of a function.
Some other Lisp programmer's tools were designed to return information about Lisp functions or variables. When creating a Lisp function or variable, a programmer can provide documentation on its use as a string embedded inline within the code. The Documentation function will return that string (in the Listener). Another similar function, apropos, prints the call syntax for a function, or the data type and value of a variable. Describe and Inspect print information about variables or data structures. None of this stuff required a graphical display.
Early Macintosh Lisp implementations all looked basically the same as the character-based systems, and were not fundamentally different from Emacs. They just put the text editor and Listener into different windows. In Macintosh Common Lisp, the text editor was called FRED (Fred Resembles Emacs Deliberately). Like Emacs, FRED is customizable, although within limits. Lisp expressions typed into FRED windows could be executed, in the style of Smalltalk or MPW windows, by selecting it and selecting Eval Selection from the Eval menu. The listener window was likewise designed to look familiar to the old-school Lisper. The authors of all the Macintosh Lisp implementations were mostly alumni of MIT or Stanford Computer Science departments, loved the Lisp programming environments they were used to, and wanted to reproduce the standard look and feel on the Macintosh. But unlike ExperLisp or MacScheme, Macintosh Common Lisp moved some of the Listener’s multiple uses into separate windows. The informative functions were ideal for this, and the Inspect and Apropos functions were made available as menu items with their output written to special windows. Likewise, Macintosh Common Lisp put many of the debugger functions, including the stack backtrace into their own windows. Inspect was made to open a special inspect window, but a similar function describe typed its output to Listener as usual. These things were a far cry from the Smalltalk code browsers or THINK-style debuggers, but Lisp programmers were accustomed to doing things that way. To me it seems a little sparse, but the debugging facilities Lisp programmers had available in the pre-graphical era were pretty good, and I'm sure it seemed to them like a working motorcycle.
The programming environment in Macintosh Common Lisp did not change much over versions, and continues to be relatively the same in its Mac OS X offspring, RMCL and Clozure Common Lisp. In fact, the user interface for Macintosh Common Lisp was so popular that it was separated from the rest of the program and developed for use with a wide range of Lisp implementations, in a project called MCLIDE. This seems a little strange to me. Although it must have seemed great in 1987, I can’t understand why this environment is considered enough today. Lisp is a large language with many built-in functions. Programmers write a lot more functions for their applications. It seems to me that it would be desirable to have much more interactive tools than apropos or documentation to look up the name of a vaguely-remembered function, or the difference between the equality predicates eq, eql, equal, etc. It also seems to me that Lisp programmers would want class browsers for the classes they use and make in CLOS. But there are few things of this sort. Lisp programmers are certainly capable of making tools like this for themselves if they wanted them. An incredibly great set of browsers and analysis tools were built into the Macintosh version of the Lisp-derived language Dylan, by the same Apple Cambridge group that built and maintained MCL in the 1980’s and 1990’s. It can’t be a coincidence that these things are not part of Lisp IDEs, but I can’t figure out why not.
User Interface Tools
Macintosh Common Lisp included a Lisp implementation of a simple GUI builder. It was primitive, but it showed how this kind of thing should be done. An earlier version called SOS Interface, was written using ExperLisp by Jean-Marie Hullot, who worked at the National Institute for Research in Computer Science in Paris. Hullot went to work at Expertelligence in 1985 to continue work on his Interface Builder, and while there he got a chance to demonstrate it to Steve Jobs, who was at work creating the NeXT computer. Jobs hired Hullot, who built a similar program, also called Interface Builder, for the objective-c programming environment on the NeXT, released in 1990. It was probably no coincidence that Interface Builder was first built on a Macintosh using Lisp. The lightweight GUI builder that came with MCL illustrates the reason why. In the usual way of programming the Macintosh, the connections of user interface items like buttons and menus with the code that they executed were not present in the source code. They were made after the program was running. 
The data that defined the user interface elements was stored in resources by identification number. There was nothing in the resource that could directly be used to execute any part of the program. The programmer would code the program to look for a resource with a particular identification number, and draw it on the screen. When the user clicked a button or selected a menu, or scrolled with a scroll bar, the code had to look up the ID number of that resource and jump to the correct subroutine to execute the command. This was even true in MacApp, except that MacApp provided the programmer an easy and automatic way to look up resource ID numbers. In Interface Builder for ExperLisp, and in the MCL GUI builder, the source code that should be executed to take action on an interface item was included directly in the data for that item. Because Lisp can read, compile and execute code on the fly, its code is data, and can be stored with the other data describing a user interface item. The user could drag a user interface item into a window from a palette, like you did in ResEdit, but during the execution of the program, rather than off-line. This process resembled that made famous by the GUI builder included in HyperCard, which was also released in 1987. 
The GUI could be used as is, as soon as it was constructed, without even restarting the program. The GUI builder included with MCL could do this as well, and could also generate Lisp code to recreate the user interface items. It could make windows, menus, and some simple controls including buttons and selectable lists or tables. Future versions added pop-up menus. Most noticeably missing was a way to add scroll bars and scrolling to windows. Scrolling was something programmers didn’t think they should have to implement themselves, and the old Macintosh way of doing it was a drag.
MCL Use at Apple
When Apple acquired Coral Common Lisp they also acquired its creators, who became the Cambridge campus of the Apple Advanced Technology Group. Among these were Gary Byers, Jeremy Jones, Andrew Shalit, Bill St. Clair, and Gail Zacharias. They continued to develop Macintosh Common Lisp for Apple, and there were several versions developed over about 10 years. During that time, MCL was used in a number of projects, both inside and outside Apple, but was not used to create a lot of commercial programs. To see how much it was used in scientific and academic circles, you can just search for Macintosh Common Lisp in Amazon.com, under books. I did it, and found 78 books that credit MCL or include MCL code for topics ranging from computer sound synthesis to geographic information systems. These books are mostly the work of academic users, among whom MCL was very popular. It is ironic, given the history of Lisp and Lisp Machines, that the little Macintosh became one of the most popular platforms for Lisp in academic circles.
Apple also used Macintosh Common Lisp as a programming platform for several of their experimental programs. The most famous of these are two other programming environments, Dylan and SK8. Dylan is an advanced object-oriented Lisp-like language with an advanced IDE. It was the Lisp I would expect Lispers would make for themselves, dressed out with elaborate browsers and code analysis tools. SK8 was an Applescript-like language and IDE for its use in program prototyping and multimedia authoring. The Dylan team was primarily located at the Cambridge campus, and overlapped with the MCL development team. SK8 was developed in California.
MCL after Apple
In 1994, Apple introduced their first computers with the PowerPC processor. Macintosh Common Lisp was a 68k program, and part of its excellent performance was due to its having been programmed close to the metal and containing many 68k-dependent hacks. It did not run in the 68k emulator. The job of porting the program was turned over to Digitool, which so far as I can find out consisted of pretty much the same people as Coral and Apple Cambridge. When Apple closed down the Advanced Technology Group, including the Cambridge campus, in 1997, Digitool acquired Macintosh Common Lisp, which they continued to offer as a commercial product. In 1996, MCL 3.1 (for 68k) and 4.0 (for PPC) were sold as a package for $725 ($580 academic price). A redistribution license to distribute the runtime version for programs written with MCL cost an additional $500 ($400 academic). In 2003, Digitool released the first OS X native (carbon) version of MCL, version 5, for the PowerPC processor. Version 5.1, the last commercial version of MCL, was released in 2005. Version 5.2 was open-sourced in 2007, and it is still available at Google Code as RMCL. It runs on PPC macintoshes in OS X, and under Rosetta on intel Macs through Snow Leopard. The most remarkable thing about this is its strong resemblance to the original MCL product, more than 10 years later. This is not a bad thing. MCL is a great program, and the fact that it lived so many lives over the years is a tribute to its excellent design.
MCL does not run in the Rosetta-free versions of Mac OS X, Lion and beyond. But the lineage of Coral Common Lisp continues in the form of Clozure Common Lisp. Clozure consists of many of the same people who were Coral, Apple Cambridge, and Digitool. That probably explains why Clozure Common Lisp looks so much like MCL. It has many of the same tools, including Backtrace and Apropos in separate windows, but at least right now it lacks most of the rest of the IDE. The good new is that it is a completely rewritten cocoa program, and has 32 and 64 bit versions. CCL can make stand alone programs and user interfaces can be built for it using a really great GUI generator. In a move that has to satisfy any Lisp fan’s love of recursion, Clozure chose to use a program inspired by Hullot’s original SOS, Apple’s Interface Builder. Interfaces are built in Apple’s IB, and saved as nib files. CCL can read those at runtime and establish the links between interface items and actions, in a way similar to the way it happens in objective-c programs. Closure directly creates and manipulates cocoa objects via a straightforward set of lisp wrappers for objective-c. MCL lives on in Clozure Common Lisp.
Sunday, February 17, 2013
