The Long View
The Long View
Resources and Resource Editors
Resources were a new thing for most Macintosh users and programmers. Like lots of Macintosh innovations, the idea for resources came from Smalltalk, by way of Bruce Horn (see "The Grand Unified Model (1) - Resources" on folklore.org). In a deeply object-oriented system like Smalltalk, all memory-resident stuff (both data and code) consists of objects allocated on a heap. In the mid 1970‘s, Ted Kaehler and Dan Ingalls at PARC had written an object-based virtual memory system called OOZE. Unlike virtual memory systems used in most of today’s operating systems, this one did not swap large, constant sized pages of memory to disk, but rather individual objects. Both data and code in objects were written out to disk storage and purged from memory when not in use, to make room in memory for objects that were needed. The key to this system was keeping track of which objects were in memory and which on disk, and if on the disk, where they were stored. The whole point was to save memory, so the database used for the virtual memory system had to be frugal with memory. The key to that was sorting objects by class and length, so there was no need to maintain a complete list of object locations on disk.
The Purpose of Resources
The ambitions of the Macintosh software designers were in some ways modest, but still far exceeded the memory available on the hardware. Any opportunity to purge any unneeded items from memory was welcome. The Macintosh operating system wasn't object-oriented, so OOZE could not be used in its details, but a lot of important data were kept in data structures stored on the heap. These data structures were the closest Macintosh equivalent to objects, and they represented windows, menus, files, and all other kinds of Macintosh things. They might not have been stored on the heap if the Macintosh Toolbox was originally designed to be programmed in C (see discussion by Bob Denny, in A New World, MacTutor vol1). But it was designed with Pascal in mind. Pascal data structures, called Records, didn't have to be dynamically allocated on the heap, but it was more efficient if they were. In Pascal, it was impossible to specify the initial values of a statically-allocated Record when it was declared. Instead, all fields of a Record were initialized with zeros, and would have to be initialized explicitly with initialization code that executes at runtime. This was memory-wasteful, because the initialization code would run only once and then would hang around, memory resident, for the life of the session. This kind of thing happened a little bit anyway, and the initialization code was usually put in a data initialization segment that could be unloaded after the program was started. But even then, to make any changes to the initial values of those structures the programmer would have to hunt up the initialization code, make changes, and recompile and re-link the program. Bruce Horn designed a relatively simple OOZE-like system for serializing data structures to disk. With it, he could save code space in memory, simplify maintenance and changes in the initial values of data structures, and make it possible to purge data structures from memory when they weren't needed, all in one go. He wrote a small database for keeping track of data structures to be handled in this way, reading them in from a disk file when needed, writing modified versions to disk, and purging them from memory when space was short. He called the Pascal records stored on disk and memory Resources, and the API for using it was called the Resource Manager. Bruce Horn intended to create Resources for use by the Finder, but the Resource Manager caught on among Macintosh Toolbox designers and was incorporated into nearly every part of the Macintosh system. The data that specified practically every user interface item in the system, and a variety of other things, came to be stored in Resources. The Resource Manager became so pervasive that segment loader even used it to store executable code, with it being stored in a sequence of resources, one for each segment.
The Resource Fork
A virtual memory system usually relies on the presence of a fixed disk that you can depend on being there all the time. The Macintosh didn’t have a fixed disk. Floppy disks could be inserted and ejected. The System File and the running application were the only files you could depend on being there. It was obviously best for programs to store their resources in the application file. The user could easily understand that the diskette containing the program had to be available to the computer during execution, so using it as a swap file for resources made the best sense.
I don’t know how it happened that resources and non-resource data came to able to be stored in a parallel fashion in single files. Executable files normally contained only resources, so the distinction was not needed for them. But in other kinds of files, it might be desirable for resources and non-resource data to reside in separate parts of the same file. This gave rise to the resource and data forks. The Resource Manager could only read resources from a file, and it wrote a directory of resources for itself within the file so that it could easily determine if a file had a resource fork. Anything on the file that was not a resource was invisible to the Resource Manager. On the other hand, ordinary file system calls, like the c standard library functions open, close, read and write, were blind to the resource fork. They accessed files at the beginning of the data fork, which was not necessarily the beginning of the file, but they couldn’t tell that. This seems to me to have been a special stroke of genius. Programmers came up with a wealth of great uses for the resource fork. For example, my favorite word processor of the pre-OS X era was NisusWriter. It kept your document as plain text in the data fork, and kept all style information in the resource fork. If you wanted to read a NisusWriter document but you didn’t have NisusWriter, it looked like plain text to any editor. In NisusWriter, the same document could contain all matter of style information, pictures, equations, etc, all stored in the resource fork. MPW famously kept information about file marks in the resource forks of its otherwise vanilla text files.
Localization
An tremendous side benefit of the Resource Manager was the ability of programmers to store their program’s text in resources, which made it easy to localize them for use in various languages. This was evident to Bruce Horn and other Apple Engineers at the time, and was appreciated as a major motivation for using resources. According to Horn,
The Resource Manager was a solution to several problems: managing dynamic data for the Finder; factoring out localizable information (strings, icons, and so on) from applications, and finally, managing memory use as frugally as possible.
-- www.folklore.org, Grand Unified Model (1) - Resources
The Practice of Making Resources
Apple programmer education guru Scott Knaster famously told Macintosh programmers “Everything you know is wrong”. His point was just that there were a lot of new ways of doing things and the old ways would not suffice. But how was a programmer supposed to create the resources needed in a program? On the Lisa, there was a resource editor called RMaker that could read plain text files containing the fields of the data structures for essential resources, such as windows, menus, and strings. It could compile them into the binary format needed in the executable file, and package them into the final product. The arcane syntax for RMaker was explained to programmers in the Software Supplements in a preliminary chapter of Inside Macintosh called "Putting Together a Macintosh Application", written by Caroline Rose. This was written during 1983 and 1984, and distributed with the loose-leaf pre-publication version of Inside Macintosh that came with the Software Supplement mailings to members of the Developer’s Program. The last version of it was included in the July 1984 mailing. Programmers using the Lisa Workshop, like those at Apple, had this information and were okay. But this chapter never made it into the final published version of Inside Macintosh that was used by most later programmers. A short fragment with an example of a very small resource source file was included in the Promotional (“Telephone Book”) Edition of Inside Macintosh that was sent to Software Supplement subscribers at the end of the program (also sent to purchasers of the Macintosh Development System). The final (hardback) version of Inside Macintosh did not have even that. A lot of important stuff was in that missing chapter. Without it, there was really no way to put together a Macintosh application. Why was it omitted? It’s because some of the information in there was specific to the Lisa Workshop, and Apple knew by that time that most developers would not be using that system. They might have rewritten it for the Macintosh 68000 Development System, but they didn’t. I don’t know why, but it left a large hole in the documentation. And things got worse.
The Software Supplements also included a series of graphical resource editors, that allowed resources to be made in a WYSIWYG fashion. Some of them were crazy, like an icon editor that saved icons in a format that couldn’t be used by any other program. Others were brilliant. The coolest programs in the early mailings were REdit and DialogCreator. REdit was specifically designed for doing localization of text, so it could create and modify resources that contain textual elements, like menus, windows and strings. It looked like the beginning of a general purpose resource editor. DialogCreator allowed you to lay our a dialog box, including regular Macintosh dialog items like buttons and strings, but also user-defined items like pictures. Both of them saved the resources created that way as text files that could be read by the Lisa resource editor.
The Macintosh Development System included a Macintosh resource editor similar to the one from on the Lisa Workshop. It was called RMaker, which was the same name as the Lisa program, and was written by Andy Hertzfeld (who also wrote the Lisa version). The Macintosh version of RMaker is an enigma. Because it had the same name and author as the Lisa version, it was reasonable to hope that it would use the same syntax. The syntax for Macintosh RMaker files was never documented or distributed by Apple, so far as I know. Maybe it was assumed that we would get that information in Inside Macintosh (but of course we didn’t). And it turns out it was similar to the documented Lisa version, but it was just enough different that it could not read the files created by REdit or DialogCreator. It could read some Lisa examples in the Software Supplements, but not others. The syntax of some resources was the same in both programs, and some were different. There were a couple of working RMaker examples included in the MDS example programs, but it wasn’t enough to make all of a real program’s resources. RMaker did not offer much in the way of helpful error messages when it encountered what it thought was an error in the input file. It was just a mystery for programmers using a Mac, and it was not addressed in any Technical Note, which seemed to be a statement that there was nothing wrong. Bill Duvall (author of MDS and Consulair Mac C thought that it was better to create resources using the assembler anyway, and David Smith had explained the method of doing that very well and early in the first volume of MacTutor magazine. Much later, the whole thing was sorted out by David Smith, in an 1986 article in MacTutor. It was good to know why RMaker had not worked all that time, but by 1986 I didn’t care anymore, because of ResEdit.
ResEdit
The first version of ResEdit that was distributed to developers was version 0.5. It was shipped with the May 1985 Software Supplement, which also included a short but helpful and accurate manual for ResEdit. ResEdit was able to create and edit a lot of different resource types, and even user-created resources (although the latter were rendered without formatting). It was unique in that it saved resources in compiled format -- not in RMaker (either Lisa or Macintosh RMaker) format, but in the binary format used in the executable file. It could also add resources to a pre-existing file, using cut and paste. That meant that resources created in ResEdit could be simply pasted into an executable file created by the compiler and linker. The early version (0.5) showed most resources simply as data in a dialog box, but even that version had a graphical icon editor. This first version could read resources from the ROM, and had one un-closable window representing the ROM. Starting with version 1.0 (from the December 1985 Software Supplement), the ROM resources disappeared, but some resources, like dialogs, were displayed graphically in the style of REdit and DialogCreator. From that point on, there was no reason for making resources any other way. This was months before David Smith explained the mystery of RMaker.
ResEdit had a very productive run, being available for every version of the Mac OS. All along the way, Apple introduced new kinds of resources, for new data structures that accompanied new functions. This would result in a new version of ResEdit. The ResEdit authors were conservative in their version numbering system, and despite numerous upgrades, the last final release was version 2.1.3. There was even a preliminary version that could edit the new resources being introduced in Copland . It was not only popular with programmers, but also with users who wanted to customize applications in various (sometimes funny) ways. Later versions also allowed for ResEdit to be customized by including editors for custom resources.
Resorcerer
ResEdit set the bar pretty high for resource editors, and there were few third party competitors. But it was outdone by Resorcerer. It was written by Douglas McKenna and published by Mathemæsthetics Inc, in Boulder, Colorado. Resorcerer had the greatest user interface I’ve ever seen for this kind of thing. It also could do things that ResEdit couldn’t do, like disassemble and edit code segments. Resorcerer is the best way there is to modify executable code, as we had to do to fix Canvas. It continues to be the best way to create or modify Applescript Dictionaries (aete resources). I still maintain some Carbon programs for use in OS X, and their resources are maintained using Resorcerer version 2.2, running on my Lombard, or in SheepShaver. I don’t know what I did with the manual for Resorcerer. I don’t think I ever needed it.
The Resource Manager is Not a Database
One of the first Macintosh Technical Notes I ever saw was entitled “The Resource Manager is Not a Database”, or something to that effect. I don’t know the story behind this note. I’m sure it would all make sense if I knew about it. But at the time, it seemed wrong in so many ways. First off, the Resource Manager most obviously was a database, so how about that? Why do people say things that everybody can see are just wrong (e.g. daylight savings time)? I suppose they were trying to say that it was not a good database. That is, it didn’t have a fast algorithm for lookup and could not handle a really large number of resources and so would fail down if you tried to make it handle a big database. But mostly I just could not understand why, with all the things that I needed to know about the Resource Manager, that was the one they chose to share with me. How about a technical note explaining the damn REdit syntax? Later I came to love the Technical Notes.
The Resource Manager is not a GUI Generator
The Resource Manager was not envisioned as a GUI-generator. A lot of resources had to do with user interface items like dialog boxes, but it was much more general than that. But you couldn’t help thinking about GUI-generation when working with resources for dialogs and menus. If you were programming a dialog box on the Macintosh, you had to write a lot of code that addressed the individual items by number, and put values into fields one at a time. When the user dismissed the dialog, you had to read every value there and put them each into itss own variable. A GUI-generator would have been the next natural step. In your dreams, you would created a dialog box in ResEdit or Resorcerer, and specify the variable that would be associated with each field, and then somehow the code to keep that dialog item synchronized with that variable would be created and installed in your program for you. GUI-generators of this kind appeared pretty early as third party products, but ResEdit never evolved into that kind of thing. That would have connected it too much to some particular programming language. ResEdit was agnostic about the development system you were using, and GUI-generation is necessarily done in some particular language. GUI-generators are especially natural for object oriented languages. Apple’s own object-oriented framework, MacApp, used some resources for a simple and clever kind of GUI-generation. For example, it had its own special kind of menu resource that included a command number, and command objects given that number would automatically be created and executed when the menu was selected. Special window resources called views handled scrolling and screen updating automatically. To create these resources, MacApp used its own special resource editors. This kind of thing took off increasingly toward the end of the pre-OS X era, as frameworks like Mac App and PowerPlant became standard.
But in some ways, using resources in a GUI-generator just confused things. Resources were not only for GUI items. The method used in MacApp left programmers using more than one resource editor. ResEdit or Resorcerer were used for all resources not related to GUI items, whereas a GUI-editor was used for the rest. This tension arose from the fact that the Resource Manager was not designed as a part of a code-generating system.
Cocoa Does Not Use Resources
Everything is different in Cocoa. For one thing, it discourages the use of the resource fork. The Resource Manager is still there in Carbon, and the HFS+ file system still supports the resource fork. But Cocoa doesn’t offer any access to it, and it’s clear that Resources are not in Apple’s future. Why not? Because it wasn’t part of the NeXT operating system, that’s why.
The resource fork caused Macintosh files to look strange to other operating systems. This didn’t matter for executable files, because they weren’t portable anyway. But ordinary text files made by programs like MPW, that put a couple of resources in every text file, looked crazy if viewed in another operating system. Generally, this problem could be solved by stripping the resource fork off of any file when it was moved to another (less advanced) operating system. Apple also developed a way of creating a special file for the resource fork when files were moved to unix file systems on the mac (like A/UX or MachTen) or elsewhere. Of course, files moved onto a Macintosh from a fork-less file system like Windows or unix would be fine on the Macintosh. The problem only arose when going from the richer 2-fork file system to a more primitive 1-fork file system. You might have thought that was a problem for the other guys to solve. But in the 1990’s, when the Macintosh lost most of its coolness, arguments like that lost strength. When Steve Jobs founded NeXT, they did not incorporate any of the advanced features of the Macintosh file system. They adopted the decades-old 3-character file extension system for for specifying file types, and made files with no resource fork. When NeXT was bought and its operating system adopted by Apple, the resource fork was devalued. But at least it was not destroyed, and it continues to be used by lots of Carbon programs. I guess HFS+ will not last forever, but let’s savor it as long as we can.
For executable files, the same purpose is fulfilled in Mac OS X by Packages. These are directories disguised as files. Resources can be stored as files within the directory. This works okay for executable files, but it is clumsy for data files. To me the real problem with Packages is that they are an easily-found-out lie, and so a weakness in the user-illusion. They claim to be files, but they are actually directories. They blur the distinction between the two, making the file system harder, not easier to understand. If it was up to me, I’d keep the resource fork if I could. But what’s done is done.
Nib files are Different
In Cocoa, GUI items and all other resources are kept separate and treated differently. Nib files and the interface builder that makes them are GUI-generators and are unrelated to other kinds of resources. Nib files are stored in programs’ Packages, and read in when programs are loaded into memory. They specify the relationship between user interface items and the data they represent. Separation of resources like pictures and strings from GUI items is a good thing, because it lets them be themselves. But GUI-independent resources are not handled in the uniform and consistent way they used to be in the old Mac OS. The stuff that used to be handled all in ResEdit now have to be in a variety of different kinds of forms, plists, icon files, string lists, and who knows what else. Some of them are pretty easy to work with (like icons) and others are crazy awful XML files (you know who you are... plists). And the Interface Builder continues to change, as Apple searches for a good interface for building GUIs. All I can say is that they haven’t found it yet. Every time I try to use Interface Builder I have to read the documentation to remember how it works.
-- BG (basalgangster@macGUI.com)
Saturday, May 1, 2010