Notes Architecture - How forms relate to documents.

The Concern...
Many professionals with a DBMS background attempt to filter their understanding of Notes through a DBMS paradigm. We recommend that you resist this temptation...View Notes for what it is: a technology with its own unique paradigm. Should you try to make Notes fit the DBMS model, you'll become frustrated.
(Lotus Notes Application Development - Solving Business Problems and Increasing Competitiveness - Larson & Shale)

When I first developed applications in Lotus Notes, I experienced problems because I misunderstood how Notes documents relate to forms and views. This concept is central to Notes. It is one of the core differences between Notes and other database development platforms. I have found that many database problems can be traced to this misunderstanding.

The problem arises when you think that the fields on a form determine what gets stored on a document.
The fields on a form DO NOT always govern what is on the associated document.

It is true that in the majority of cases, a user creates a document by opening a form, filling in the fields and then saving the document. Information entered on a displayed form is stored in the document; Each "field" on the document is given the same name as the "field" on the Notes form. The form acts as a template: Information placed on it is passed through to the underlying document.
Similarly, when a document is retrieved, data stored in it is viewed "through" a form.

The word field is used to refer to three different ideas, which is where the confusion arises:

1) The position on the screen where the user is entering data.
2) The name we gave to an input area defined on the Notes form
3) The name given to the location where data is stored in the document

Some different terminology might clarify things.
( The following became clearer to me after reading "Using Fields and Items in LotusScript", (Chapter 26 in Lotus Notes and Domino Server 4.6 Unleashed )

The Lotus Notes database extension "NSF" stands for Notes Storage Facility.
When Lotus Notes was being developed, a document was referred to as a "Note". ( hence the name ? ).
Pretty much everything in a Lotus Notes database is stored in one Note or another . This Includes the design elements and the data.
Each piece of information on a Note is stored in an element called an "Item". Think of each Note as a bucket into which is thrown a number of these items.
In relational databases a value is stored in a field, on a record, in a table ; In Domino, a value is stored in an item, on a Note, in a Domino Notes Storage Facility (database).
So what, you may say ! Well items on a Note behave very differently to fields on a record and the distinction can help to keep things straight.

Imagine each Note in a database as an empty bucket into which ANY information may be thrown.
Into each such bucket you can throw as many pieces of information as you want.
Each piece of information wiill be stored in an item, and each such item has (amongst other properties) a name.
First surprise .. Two different items on a Note can have the same name and yet store totally different values. Lotus Notes won't object ( Try that with two different fields on one record !).

Now, Each Item on a Note is itself a compound object with a number of properties.
To see this, Select any document in a Notes view; Click the right mouse button and choose the "fields" tab from Document properties. As you select each field in turn in the left pane, you will see its properties listed in the right pane.
i.e. Field Name,Data type, Length, Seq Num, Field Flags. Many of these Item properties can be set in Lotusscript. For example, the flags attribute is an interesting one. One of the flags, SUMMARY, is usually present. It tells Notes that this item can be used in a view column. If, through Lotusscript, you set the summary flag on this item to false, then the field on the Note will not be available for use in a view.

As mentioned earlier, each design element is also stored as an item on a Note. A "Design" Note might hold attributes about a form , a view, the database icon, the "About" form, the "Using" form etc.
Another interesting example is a profile form. A profile form ( created with the GetProfile function) can be used in Notes to store "global" information. Each "Note" created when a profile form is stored behaves something like a regular Note (document). But the Profile "Note" does not show up on any view. So is such a profile a "document" or part of the design ?

How do Forms fit in ?

Enter a value into a field on a form and save the form, Lotus Notes will save the information you entered on the form into a Note; the value in the field will be stored in an item on the Note. The Item will be given a data type consistent with the field definition on the form.

The above description explains why Notes programming (i.e Lotusscript) is littered with "Note" this and "Item" that. i.e. NotesItem, NotesRichTextItem etc.

In lotusscript the item that is being stored on a Note is implemented through the NotesItem class.
As mentioned above, a note can have more than one item with the same name. Methods exist, (GetFirstItem) to access these different items.

Items are created when information entered on a form is stored in a document. But, in addition, items may be created directly in code. Thus lotusscript for example, has the methods AppendItem, CopyItem, ReplaceItemvalue to manipulate items and their contents.

Notes does not store any hidden information connecting a note to the form that (may) have created it. The item "Form" on a Note gives Notes an indication of which form to use to display the information in the note, however this may be overridden. If you were to run an agent that changed the contents of the item "Form" to the value "Employee" then Notes would attempt to display the stored information using a form called "Employee". Or, You might open the note in a view that overrides the form value in the note. Or.. The database might not have a form with the name given in the Form item, in which case Notes will use the database's "default" form.

The consequences of the above are many:

On a form you may define the field "priority" to be a number. Saving the information on the form, creates an item "priority" on the note.
Then you run an agent that assigns an alphanumeric to the item "priority" on the note such as
FIELD priority := "A";
The agent does not reference the form, it simply throws the item "priority" (as an alphanumeric) onto the note.

The next time you open the form it expects priority to be a number.
When it finds a string with a value "A" all sorts of interesting errors might occur.
(The formula of a calculated field, "Urgency", may depend on the numeric value of the field "Priority".)

Things get really interesting when you import data directly into Notes.
A Note can store pretty much anything. If you misspell the name of an item, or give the item the wrong format, Notes will faithfully follow your directions.

Believe me, I know this can be confusing. If you see inaccuracies in the above, or if you have suggestions for improvement please email me.
If you know of any documents that explain how Notes databases organize design and data internally, I'd love to know of them - Thanks)