Posted by I have spent many years working deep within the Notes Indexing Facility (NIF). I’m not done, because there is more to be done and harvested from the rich functionality well-bundled from day one by the earliest Domino developers. It can be a challenging area because so much variety of data flows through the code and onto the glass. If one is not careful, you will make one user story a delight while another becomes quite less so.
One of the most problematic operations in NIF is its handling of updates. Someone could say “Well, what is the problem? Other database products have no issue updating their indexes, why is Domino any different?” The answer lies in the ease with which views and folders are created in Designer. It is a very common story that views constantly grow in number and complexity.
So, I often reply to the question of the difference this way: “If you went to your MySQL or Oracle or DB2 or MongoDB administrator and asked for 200-1000 indexes on your database, what would they say?” Of course, they would laugh you out of the room, as they should. Yet, that number of indexes is quite normal on a Domino database and it is why they are updated lazily, on a scheduled or as triggered or forced. There are a variety of update/refresh rules and I will not go into them here.
Some of the costliest issues with NIF are due to excessive queueing for updates. Most of the APIs and documentation call it “refreshing” when a user opens a view and wants to see the latest updates, which is completely reasonable. The trouble what when a large number of users (well, threads) do that at once, they each compete for the right to update the view in a “fair read/write” queue – that is, those who only want to read get no preference over writers (we don’t starve updates). The result can be a breach in Service Level Agreement (SLA) timings and opening a view simply takes too long.
Following is a diagram I put together for SPR JCUS8MXLA2. Included in the mix is the “Update Task” representing scheduled, dedicated view updating:
You may or may not know that in the v9.01 feature packs, Domino development rather quietly produced a feature to attack this problem. It is called “inline view updating” or just “inline”. You can toggle it on via
load updall <database name> [-T <viewname>] -inline on/off
What this does is to apply view updates at time of document updating. All refresh requests at view open time can therefore be treated as nops, as they are. Document updating becomes a heavier unit of work, but the view open and read performance trade-off is worth it.
A second effort was launched in Domino V10, which detects and then automatically creates a dedicated update thread to high usage views. It is enabled via NIF_VIEW_USAGE_ENABLED=1 in notes.ini.
I wanted to measure the effect of the first effort, -inline, so this summer at HCL Chelmsford (well, at home due to COVID-19) I’ve had the privilege of working with an intern in core Domino, Joseph Calles. I set Joseph on a quest to provide real timings with -inline on and off. He wrote some scripts that allowed him to vary the read vs update load against a 200K document database with three views and collected stats using “show trans” at the Domino console. Joseph capped the document update and view operations at 10/second, but of course your numbers will scale according to volume, bulk of data and hardware utilized.
No one is claiming these numbers are real-world, the data and processing is automated. But they are consistent relative to the gathered set. And they show some trends people should know about as they consider using -inline. The variance in test load was simple: for each run, the number of all operations was incremented by one per second.
The OPEN_COLLECTION transaction is what bogs down trying to update a view when inline is off. The benefit is very clear to see in the first graph, and it should be noted that without inline, time to open a view increases much faster than linearly. That .. is the log jam I mentioned.
In the second graph, open view and read view entries are combined to provide a single duration. This is close to what a user would experience opening a view and getting his/her first window of entries rendered. Again, the spike with larger numbers of transactions without -inline is clear.
Now, very few SLAs will be violated with 800 millisecond response. Again, the point is the curve. Showing maximum time to open a view shows the erratic, worst-case user experience, which would indeed violate most SLAs (note, times are in seconds in this graph):
Truth in advertising – document update times do increase with -inline on, but you would expect them to. And they don’t increase exponentially but linearly. But don’t confuse this with a decrease in throughput – it is not affected, only time to complete:
This final graph treats all operations – update, finds, view opens and view entry reads equally and averages their combined duration. So, overall duration impact of -inline processing if you will:
But it’s important to remember the original use case – multiple users trying to open and read a view but first “refreshing” (that is, updating). With -inline on, the refresh operation is unnecessary since any view being read is updated during a NSFNoteUpdate operation. So those view open operations are much lighter weight.
To say it again, this behavior is “normal” for transaction-centric database systems, especially the relational variety where data lives in neat rectangles (ok, they have progressed a bit). But in Domino, a document-storing and -processing NoSQL engine that supports so many different indexes constructing index data so many different ways, a lazy update model is appropriate lots of the time. And the -inline feature addresses those cases where it is not appropriate.
So, if you have contentious views – and please, not all views are hotly contended for, I recommend you strongly consider turning on inline view updating, using
load updall <database name> -inline on -T <name of contentious view>
And reap good benefit.
I would beware of turning it on for all views in a database as the document update cost could then surge. You know your most contentious views; tackle those first.
J D Curtis 