What is Domino / Lotus Notes ?
I explain elsewhere, How the original product, Lotus Notes, became Domino.
In the following material, I use the term "Domino" for the product and "Lotus Notes" for the underlying architecture and data model.

It is my aim to uncover some of the concepts that make Domino such a unique product.
This information is not just for developers new to Domino.
As I wrote this page I discovered many new consequences of what I thought I "knew" about the Lotus Notes architecture. The following ideas can fundamentally shift the way you approach Domino development. I hope you find them illuminating !

Domino is a new tool. It is not a simple email system. Nor is it a traditional database.
There is little to be gained in a comparison between Domino and these traditional systems.

Domino development will be far less frustrating if you understand the Lotus Notes data model from the ground up. There are many good introductory "how to" books and it is not my intention to duplicate their content. What I cover is more "fundamental". When you understand the ideas behind Lotus Notes, the "How tos" will be easier to implement.

If any of the following is unclear, please email me. I will try to improve it !

Lotus Notes and Domino
The Notes Data Model - The Documents (Notes)
The Forms
Notes Views
Response Documents

What, then, is Domino ?

The Domino product is an integrated messaging and Web application software platform.
It provides a robust environment for creating Web applications that support workflow, RBDMS connectivity, and collaboration.
Domino is ideal for creating applications that allow "sometimes connected", multiple users to cooperatively process unstructured information.
The term Lotus Notes is now used to refer to the underlying client-server architecture that the product is built upon.

In the earlier Lotus Notes product, a powerful engine served up information to the Notes desktop client. Lotus realized that this server engine could be used to deliver the same information to other "front ends". They developed a task, running on the (then) Notes server, that delivered Notes database information to a web browser. When this functionality was being beta tested the code name was Domino. Beta testers liked the name so much that Lotus decided to rename the complete server product "Lotus Domino". (They also wanted to move away from the image of Notes as a proprietary system, hence the marketing advantage of renaming the server product ). Domino is now a fully standard, Java -compliant, Webserver.

Domino.. An Application development platform.

Domino comes with some pre-written functionality.
It contains a mail file to handle all your incoming and outgoing email, calendar/scheduling, task tracking, etc.
There is an address book to store individual and group information (and much much more !).
These are very powerful Domino databases like any other.
In addition, there are many database templates that can be used to create useful applications "out of the box".
Discussion databases, document tracking databases, simple workflow applications etc.
However, to get the best return on your investment, Domino should be thought of as an integrated development platform for creating Groupware / Workflow applications.

Cooperatively processing unstructured information.

What do I mean by "cooperatively process unstructured information" ?
To clarify, let's draw a distinction with a single user using a traditional database to process "structured information" .
Imagine a clerk at a bank processing a customer's account record.
In a traditional database, the account record might hold Account Number, Account balance, Last Name, First Name, Social Security Number, Date created, Date last accessed. Every Account record will hold the same information, in the same format and will hold only that information.
When a clerk needs to modify the account, the record will be locked to prevent other users simultaneously updating it, the change will be made, and the account will be released.

In contrast, an example of "sometimes connected, multiple users cooperatively processing unstructured information" might be artists working on the design for a new product. This is where Notes shows its strength...
The design team will all want to add their own input, they will want to discuss each others contributions. They will want to add their own sketches of the product. They will want to make simultaneous changes to the design. They will want to take the plans home with them to modify at three AM when inspiration hits. Back in the office they will want their changes to be incorporated into the master database.

So how do you design a system to store and process unstructured information ?

First, you forget all you know about "databases".

Let's go back to the beginning..
Imagine you want to store some pieces of information in the most general way possible.
You set up some containers to hold the information. Let's call these containers "buckets".
In front of you, imagine you can see a number of these buckets.
Into the first bucket you throw some pieces of information. Some more pieces go into the second bucket and so on.
That's about as general as you can get.

Now you have a problem. To retrieve the information from one of your buckets you need to be able to identify the bucket.
You could use one of the pieces of information that you put in the bucket. For example, a person's name or phone number. But the Name might change, and you would lose track of the bucket. Instead we will generate a random number to identify the bucket.

Next problem.. Having located the correct bucket, we need to find the right piece of information. We need to be able to identify the items of information we threw in the bucket. So, to each piece of information we will assign a label called the "item name". One item name may be "LastName" another might be "PhoneNumber".

Please Note: we are not talking about a traditional, relational database !

Specifically....

We now have a system to store our unstructured information. We will call this "collection of buckets", a database. Since a "bucket" is not a very good image, let's call our storage area a "Note" (These Notes will also be referred to as documents, a term more familiar to our users). In each "Document" we have "items"; and each item has its own "item name". ( and an item may be a list of values) .This iterative structure of one element containing other elements has been referred to as the Notes container model.

We'll call our product "Notes" and what's more, since our "database" is a storage area for these Notes, we will call it a "Notes Storage Facility" and the file that we create on the computer will given an extension of NSF.

Notice that we have not yet said anything about the user interface.
The uniqueness of the Lotus Notes architecture lies in its storage paradigm. The possibility exists to develop many front end interfaces: Notes clients, Web browsers, Java applets, VB programs etc.

How does the user access the information in these "documents" using the Domino client ?

In the following sections, I refer to "Lotus Notes" when refering to the desktop client..

We want the user to be able to see the value of items stored in a document.
We will create a display "form". The form will be the mechanism we use to show the user what is in a document.
The areas on the form that display the values stored in the items on a document will be called fields.
One field on the form may be called "AccountNumber". If the document has an item named "AccountNumber", then the value of the item AccountNumber on the document will be displayed in the field AccountNumber on the form. If the document does not contain an item named AccountNumber then the field AccountNumber on the form will be left blank. It can help to imagine the form as a template through which we are viewing the document. If a field on the form, lines up with an item on the document then the item value will be seen.

How do we handle the fact that we do not know what data type the item on a document will be? (Whether it will be a number or a date or a string ?) We will always convert the item to text. How about the fact that there may be many items with the same name ? We will display the value stored in the first item we come across on the note with that name.

So now we have forms to display documents. But what form will we use to display a document. As we have said, two documents may contain completely different items. Which form do we use ? As yet there is no connection between the form and the document.

OK. We will create a special item on a document with the name "Form". The value of that item will determine which form we will use to display the document. So if the value of the item "Form" is "Company", we will use the form called Company.
If the document does not have a "form" item we will use whichever form we have designated as the default form for the database.

Something I only just realized myself.. Just because the value of the item "Form" on a document has the value "Company" does not make this a "company" record. The Notes data model contains no concept of a "record type". All documents in a database are equivalent as far as Notes in concerned.

Having created a form to help display the documents, where does Notes store this form design information.
It stores the design information in other Notes documents. In effect we now have "user documents", storing data and "design documents" storing design elements.
The "user documents" can be accessed by users, the "design documents" can be accessed only by database designers.

The layout information is stored in a rich text field on these design documents. When we look at our data, Notes reconstructs the form layout from the instructions it finds in this rich text field. As it follows the instructions to build the form layout, it might find reference to another form: a subform. Notes will then locate the subform design document and include the instructions found there.

How do we locate a particular document ?

We have documents stored in a database and a mechanism (the form) to look at the items stored on each document.
Next we need some way to locate a specific document.

Imagine we have a database of documents, some of which store information about employees in the company.
We want to see a list of these documents.
First ... We need to select only those documents that hold employee information. (Others might hold, say , Department information.)
In a traditional database, employee information will be stored in "employee records" and we would select to see only those Employee records. However.. Notes has no concept of "record type". How do we identify employee documents ? If we have been consistent, all our employee documents will have an item on them called "Form" with a value of "Employee". So we can create a condition:

SELECT form="Employee"

( Please note.. This means that the value of the item "form" is "Employee". It DOES NOT mean that we are searching for the documents with a "form type" of Employee !!!)

Alternatively, assume that every employee document also has an item "salary". The salary item is always filled in and only exists on employee documents. We could then have a selection condition:

SELECT salary > 0.

This condition would also select the employee documents.

It is important to realize that, as far as Notes is concerned, these two ways of selecting the documents are equivalent. In a selection condition there is nothing special about the item "Form". The Form item is just one more entry on the document. It's value does not determine the document's "record type". In fact the item "Form" might not even be present on some of the "employee" documents.

Having located the correct set of documents, how can we find the particular one we are interested in.
Assume I want to find the document that contains my personal employee information.
We make a list of the employee documents using one of the selection criteria above. From each document in the set we list the information that the user might need to identify the employee. Let's pick the value of the EmployeeName item, the SocialSecuriyNumber item, the Department item. The values appearing in the listing are referred to as column values.

This listing of item values from a set of documents in a column format is a Notes View.

A Notes view may list any set of documents. To Notes, all documents are equivalent, (there being no record type).
A view can therefore contain any mixture of documents. We could show documents containing employee information, departmental information etc. in the same view. If an item called "Name" appeared in both documents, then the View would show "employee" and "department" information jumbled together.
We have seen that when a form has a field that does not correspond to an item on a document,the field is left blank.
Similarly, when a view has a column that does not correspond to an item on a document, the column value is left blank.

Where does Notes store the information used to construct a view ?
It is another Notes document ( Every thing in Notes is stored in documents ) . This single document contains an index of all the documents that are to be included in the view. (The elements in this index are the unique IDs of the documents). The view document also contains column formatting information (i.e. is the column a category. Is it to be displayed as an icon.etc.).
Other view information includes: Are the documents to be displayed as a response hierarchy, i.e. parent-child order, or as a flat record structure.

When a user requests to see a view displayed, Notes opens this "view" document, determines whether it must recalculate which documents will appear in the view ( i.e. it updates the index list) and then is uses the column description items to display the view. This explains why a view can be "out of date". Some views are refreshed only when a user requests an update.

Response Documents and "Response to Responses"
I said that Notes has no form types, but what about "Responses" and "Response to responses".
When you design a form, you have three options: Documents created with the form can be main documents, response documents or response to a response documents.
I guess that when IRIS developed Notes they decided that users should be offered some structure when creating documents.
There are many cases where one document is very obviously related to another.
For example: Having entered company information on one document, you might want employee information, entered on a second document, to be tied to the first.
As you recall, each document entered into Notes is assigned a unique ID. The method used to tie the response document to the main document is to add a single field to the response document called $REF. The $REF field holds the unique ID of its parent.
That's it !
The only thing that makes a response document a response document is the presence of the $REF field !
(i.e. There is no "response" document type).
This has some very powerful repercussions:
1) If you delete the $REF field from a "response" document, it is no longer a response document ( @isresponsedoc returns false).
2) If you add a $REF field to a document ( with the correct value) it becomes a response to some other document.
3) If you change the value of the $REF field it will become a response to a different document. ( This is what happens if you "cut" a response document and "paste" it while pointing to a different main document.) If you have been wondering how to do this in code lookup the "MakeResponse" method in the NotesDocument class.
4) Parent documents do not know whether they have child documents.
5) Child documents do not know whether there parents still exist. (The parent documents may have been deleted, creating orphan responses).
6) The database has no knowledge of the child-parent structure the documents contained in it, without examining the individual documents.

How do "responses" differ from "response to responses" ?
Imagine you have a document hierarchy: A main document with some associated response documents.

So, if you are creating a discussion database, you get a nested response tree by using "response to response" documents..

Notice that although we have introduced a way of "relating" one document to another, this is very different from the associations in a relational database. In a relational database two records are related because the first record contains a foreign key field that has the same value as a key field in the second record. The Notes data model does not relate documents by field values. Notes documents are "related", if at all, by document ID.

I'll be continuing this later......

You may be interested in this (rather old) paper from Lotus and the Advisor magazine:: The Architecture of Lotus Notes

If you have read this far, and you have a question or you are unclear about some aspect of the Lotus Notes Data model, then please email me: GStalley@NotesDesign.com . I'll try to hunt down an answer and include it here.

Footnotes

Identifying a bucket...
In the case of Lotus Notes, This number, used to uniquely identify the document, is called the Notes unique id. It contains two parts; the first identifies the database ( in fact it is the replica ID of the database) and the second part is an ID for the specific document (bucket !) within the database.