Graph Updating

Whenever a graph source is processed (i.e. essentially when saving an item or a part), the graph (=nodes + triples) generated by all the mappings matching that source get collected into a graph set.

So, at the end of this collection process, we have regenerated the whole partial graph from the specified source. This set now must be synchronized with the existing graph stored in the database, which also includes the graphs of all the other sources, and the nodes and triples directly added by users.

When the Graph Is Updated

Just like for the index of data pins, the graph is updated in real time, whenever a user saves an item or a part in the editor (or deletes it). Saving the same data again changes nothing in the graph.

The graph of each source is identified by its SID, which by convention always starts with the GUID of the source item or part. So, the graph of a part includes all the nodes and triples having a SID starting with the part’s ID.

Because of this, when an item or a part is saved, the graph set built from it fully replaces the graph previously generated from the same source. This happens even when the set is empty: for instance, if a user removes all the events from an events part, all the nodes and triples previously generated from them get removed.

Update Algorithm

For each source, the update happens in a single database transaction, with these steps:

  1. URIs: all the URIs in the set are resolved into numeric IDs, adding the missing ones.

  2. implicit nodes: nodes used by triples but not existing in the graph are added. For instance, if a mapping emits a triple x:persons/petrarch a crm:E21_person, and the node crm:E21_person does not exist, it gets added. Such nodes have source type 4 (implicit), the SID of the triple, and their URI as their label; when used as predicates, they are tagged as property.

  3. nodes: new nodes are added, and nodes already generated from the same source are updated. Nodes created implicitly by another source, and now explicitly generated by this one, become owned by this source. Nodes owned by other sources or created by users are never updated.

  4. triples: triples are identified by their subject, predicate, object (either a node, or a literal with its type and language), and SID. New triples are added, those no longer generated are deleted, and those still generated keep their ID (their tag and other details are updated when changed).

  5. nodes no longer generated by the source are released. Such a node is deleted only when no triple uses it anymore (unless it is a class or a property). When it is still used by triples of other sources, or by triples created by users, it is kept, and handed over to the source of one of these triples (as an implicit node), so that it gets removed when that source no longer uses it. This ensures that removing a node never deletes in cascade the triples of other sources or users.

  6. node classes: the classes of each node (including the inherited ones, as defined by rdfs:subClassOf) are stored in a dedicated table, which is updated for the nodes affected by the changes (subjects of changed rdf:type triples, and instances of classes whose rdfs:subClassOf triples changed).

Deleting

When an item or a part is deleted, all the triples of its graph are deleted, and its nodes are released according to the same rules seen above for updating (step 5). So, nodes still used by other sources or by users are kept.

⚠️ Before Cadmus 17.1.0, deleting a source deleted all its nodes, with all the triples using them in cascade, even when these triples belonged to other sources or were added by users.

Users Edits

Users can freely add nodes and triples to the graph via the graph editor. Nodes and triples added by users have no SID, so they are never changed by updates; and nodes used by them are never deleted by updates, as explained above.

Users can also edit nodes and triples generated by mapping. In this case, they keep their SID, so the next time their source is saved they get regenerated by the mapping, and the user’s changes are overwritten. In other terms, the graph generated from a source always reflects the source data: to change it, users should change the data. Conversely, to add knowledge which is not in the source data, users should add their own triples.

Concurrency

In a multi-user environment, several users may save at the same time, and their sources may generate the same entities (e.g. two parts referring to the same new place). The update is designed to be safe in this scenario:

  • concurrent updates of the same source are serialized;
  • concurrent insertions of the same URIs, unique UIDs (see UID builder), and nodes never produce duplicates or errors.

Technical Notes

⚙️ The components involved in the update are:

  • GraphUpdater, which is used whenever an item or a part gets saved (UpdateAsync). It runs the mappings matching the source, and passes the resulting graph set to the repository. The set also lists the ID of its source (GraphSet.SourceIds), so that the repository knows which source it represents even when the set is empty. The updater can also Explain how a source gets mapped, without changing the graph. An updater instance is not thread-safe: use one per update.
  • the graph repository (IGraphRepository), which provides these key methods:
    • UpdateGraphAsync, which receives the graph set, and synchronizes it with the database graph as explained above. This is typically used in consequence of a save operation.
    • DeleteGraphSetAsync, which deletes the graph of the source with the specified ID (the GUID prefix of its SIDs). This is typically used in consequence of a delete operation.

Synchronous versions of these methods are also available. The mappings to run for each source type are cached for some minutes (by default, 5), so changes to mappings made by other processes (e.g. by the CLI tool) are seen when the cache expires.

In the PostgreSQL implementation, concurrency is handled via transaction-level advisory locks.