The Query Results Processor – part One

Long time no blog.

Well, I’ve been busy putting together content to blog about.  But since 12.0.1 is on the verge of shipping, this is a good juncture to catch people up with what we’ve been up to.

Today’s topic is … the Query Results Processor

For application developers, it is known as (Notes)QueryResultsProcessor  – the (Lotusscript)/Java class with which we’ve exposed the feature.  We call it QRP because it’s easier to say and .. well, secret acronyms are cool, if you know it, you’re in the club. Just kidding.

Selectively using the word RESULTS because it’s NOT the same as document sets, (Notes)DocumentCollections or the output of Domino Query Language.  There is no human-readable data from any of the ways people find collections of documents using the Domino APIs.  The output set of documents has always needed to be marshaled onto an existing view or manually traversed to render human-readable and machine-navigable data.

So, starting in 12.0.0, we delivered a way to fetch data across any (and I mean ANY) of a set of document collections from any (yes, again, ANY) database.  We built sorting (a chief requirement) as well as Domino specialties of categorization and aggregation.  We also wanted to allow for the power of Formula Language’s computation and marshaling of data across Domino’s NoSQL data model to allow people to make disparate data (even historically differing designs of the same underlying data) readable, sortable and navigable in a central set of results – and we called it the facility QRP.

To finish the user story, we needed to define the nature of the human-readable output data.  We decided on 2 formats – JSON (v12) and Domino views (v12.0.1) – because JSON is the lingua franca of web applications and views have such immense API support as THE way to provide navigable data in Domino.  This blog entry will focus on the view output option because evidence is that it is the most popular and immediately deployable.

One of the early decisions was to expose this feature as a set of API calls and NOT a language.  As a veteran of SQL parsers and execution engines from the inside, I know well its capabilities and complexities.  To provide an analog, what we’ve built is support for SQL table clauses, albeit with Formula Language expression support and utterly different design support which inflexible SQL databases strain to even understand.

Now, if we had built support for SQL syntax, people would expect it to function like SQL does, and Domino is the original NoSQL document store database system, so we weren’t about to try that.  PLUS, we didn’t want people to have to code DQL or any other language to that level of complexity.  It’s not that we might “language-ify” QRP functionality eventually, it’s that we wanted to create API calls that we believe make things clearer and more deliberate.  But as you’ll see, there is no avoiding complexity given what we were building. 

DO NOT LET THAT SCARE YOU, please! 

You are invited to just consume these blog entries and videos in part and come back multiple times till it’s clear.  We’re convinced in the value of what we’re providing here.  It may seem very late in the game given all the relational capabilities that have been around for decades, but Domino data and processing has always different and distinctive.  If you’re using the product, you know what I mean.

QRP pieces

I have been presenting a slide that shows a QRP view at its simplest – and as anyone who’s ever created a Domino view knows, the “bones” are the 1) view selection criteria (default Select @All) and 2) a set of columns fetching document data in summary, even computed, form for users to see and navigate and the ability to open the source documents with a simple double click.

QRP views are no different.

1. The view selection criteria is provided by either DQL queries or collections created by Java or Lotusscript code (or IDTables for those who know the Domino C API).  If you have Domino application code, you already have code that creates collections.  And it’s not just (Notes)DocumentCollections I am talking about – any of the *Collections created by various back end class calls will work.

So, the simplest QRP view would have a single input collection that is added by calling addCollection or addDominoQuery (the DQL call).  You must name your input collection for later reference and serviceability reasons, but you will see the great power this gives you in subsequent blog entries. Note – document security is in force when input collections are produced. Only those documents accessible by the current user produce data in QRP views.

• The set of columns are created left to right (as the view renders).  Of course it can be just one column, but views usually have many.  We support sorting, categorization, aggregation, hiding and formula specification on the single call used to create a column.

The simplest QRP view would have a single column created using the addColumn call. It requires only a programmatic name which you might know from the beanie tab in view column properties designer.  If nothing else is specified on the addColumn call, this name will be treated as a field name when data is fetched. But that can be overridden as well. To complete the description of this call, the optional parameters are:

1. View column title – which can be padded or even all spaces to reserve real estate in the view header.
2. Formula – Formula Language to compute values across all input collection.  If you desire to write separate Formula Language selectively for input collections, you can do it using addFormula (see below), which will override anything specified on the addColumn call.
3. Sort order – ASCENDING, DESCENDING and UNORDERED (which is the default).
4. Hidden setting – true/false for rendering the column or not
5. Categorized setting – true/false for grouping result entries under single categorized values

And that’s it.  QRP processing then takes that input, creates and populates a view using the executeToView call.

Now for the $50,000 (44,356€) question – why would you even do that?  You already have views to do whatever you need.

Well, do you?  At this point it is a law of nature that Domino applications spawn views constantly.  And those views are never quite what is needed next time.  We have a database at HCL whose design is 25 years old.  It has about 700 views.  Only 5% of these have been opened in the last 5 years.  The indexes have been deleted a long time ago but the views still show up as things you might want to open just out of nostalgia.  At the cost of rebuilding then discarding again.

We’re not alone like that; every time I talk to people about their “view space” the same phenomenon exists.

QRP views, or just “Results views” as we’ve come to call them, are throwaway.  They exist to be intentionally discarded.  Rather than hard design documents, they are simply query results (which I would contend all views are, but that’s another topic).  But as results, they are extremely navigable and searchable even within their context.  More on that later.