Custom Integrations

Custom integrations with third-party services.

Last updated: 15 March 2022Author: Jillur Quddus

Please note that the OntoPop backend open-source software project, which includes the event-driven data pipelines and APIs, is undergoing extensive redesign and refactoring as part of OntoPop Community 3.x in order to improve performance, security, extensibility and maintainability. As a result, the documentation on this page will be significantly updated. Please refer to the OntoPop Roadmap for further information.

WebProtégé

Overview

WebProtégé is a cloud-based application that enables users to collaboratively manage and edit W3C Web Ontology Language (OWL) ontologies. It is a free, open source and popular ontology development environment demonstrated by the fact that it currently hosts over 68,000 OWL ontology projects with more than 50,000 user accounts. This page describes the custom out-of-the-box OntoPop connector designed to automate the export of an ontology from an existing WebProtégé project to a designated Git repository and branch as an OWL file in real-time.

For further information regarding WebProtégé, please visit https://protege.stanford.edu/products.php.

Design

WebProtégé Limitations

At the time of writing, WebProtégé lacks an API or similar web service that enables consumer applications and systems to programmatically query ontologies managed in existing WebProtégé projects, and to export them as OWL files. Thus the only current method available to integrate WebProtégé with external software services (such as other data modelling or visualization tools) is to manually login and export ontologies from WebProtégé.

In the current absence of a WebProtégé API (or similar web service), the custom OntoPop WebProtégé connector is designed to automate the export of an ontology from an existing WebProtégé project to a designated Git repository and branch as an OWL file in real-time.

Connector Design

The following diagram illustrates the event-driven real-time logical architecture of the custom OntoPop WebProtégé connector.

The following ordered list describes the flow of events and data across the custom WebProtégé connector, from left to right.

1. WebProtégé User

An existing WebProtégé user logs into the WebProtégé cloud-based application and makes edits to an ontology in an existing WebProtégé project, identified by a unique WebProtégé project ID (of the format abc1234-abcd-1234-9876-qwerty98765).

2. WebProtégé Webhook

The WebProtégé cloud-based application offers the ability to configure a webhook payload URL. After each atomic edit of the ontology (for example the creation of a class, the update of an object or class property, and so forth), a HTTP POST request is made to the configured webhook payload URL containing the following JSON request body schema:

An example payload generated by the WebProtégé webhook is provided below.

{
    "projectId": "abc1234-abcd-1234-9876-qwerty98765", 
    "userId": "me@tld.com", 
    "revisionNumber": 2960, 
    "timestamp": 1644414105240
}

Note that the payload generated by the WebProtégé webhook does NOT contain any details on the actual edit itself. It only contains metadata regarding the WebProtégé project, the user ID of the WebProtégé user who made the edit, the revision number and timestamp. Therefore in order to export the ontology corresponding to the specific revision number as an OWL file and in the absence of a WebProtégé API (or similar web service), the custom OntoPop WebProtégé connector uses a web automation framework to programmatically log into WebProtégé before requesting the WebProtégé download URI with the WebProtégé project ID, revision number and export format as request parameters. This fully automated process is detailed further in step 7 below.

3. WebProtégé Subscriber

The first component of the custom OntoPop WebProtégé connector is a subscriber application (an event-driven serverless function written in Node.js) that subscribes to the WebProtégé webhook and to which the WebProtégé webhook HTTP POST request is made containing the WebProtégé webhook payload as detailed in step 2 above. This WebProtégé subscriber application consumes and parses the WebProtégé webhook payload.

4. WebProtégé Subscriber Publication

After the WebProtégé subscriber application consumes and parses the WebProtégé webhook payload, three configurable options are available during deployment that dictate how the downstream OntoPop WebProtégé Exporter service (an event-driven serverless function written in Java 11) is invoked, as follows:

1. Messaging Trigger - the WebProtégé subscriber application publishes a message to the shared messaging system and to a specific topic with the WebProtégé webhook payload contained within the message body. The OntoPop WebProtégé Exporter service is then invoked via its subscription to the message broker and the relevant topic.
2. Scheduler Trigger - the WebProtégé subscriber application generates and inserts a new structured record into a SQL database table with the columns of the record mirroring the parsed contents of the WebProtégé webhook payload. A scheduler service is then configured such that the OntoPop WebProtégé Exporter service is invoked and queries the relevant SQL database table on a periodic basis (for example every 10 minutes) to check for any updates across all of the WebProtégé project IDs defined via the OntoPop Management API. If updates are found, then the record corresponding to the latest revision number is consumed and the OntoPop WebProtégé Exporter service continues execution.
3. HTTP Trigger - the WebProtégé subscriber application makes a HTTP POST request directly to the OntoPop WebProtégé Exporter service with the WebProtégé webhook payload contained within the subsequent HTTP POST request body. The OntoPop WebProtégé Exporter service is then invoked via this HTTP trigger.

During deployment of the WebProtégé subscriber application and OntoPop WebProtégé Exporter service, only one of these trigger options may be configured. Based on your deployment environment, non-functional requirements and OntoPop use case requirements, if you wish to process each and every atomic edit made in WebProtégé in real-time, then you should configure the Messaging Trigger. However if you do not require each and every atomic edit made in WebProtégé to be processed, but instead only wish to process the very latest updates on a periodic basis, then you should configure the Scheduler Trigger.

Note that given the nature of ontology development, tens or even hundreds of edits may be made to an ontology in WebProtégé in a single user session which will need to be considered if you configure a Messaging Trigger or HTTP Trigger and the subsequent platform requirements to cater for the increase in throughput. Therefore it is generally recommended to configure a Scheduler Trigger unless you require each atomic real-time update to be processed.

The HTTP Trigger can also be used for processing atomic edits in real-time. However if you do require real-time atomic processing, then it is recommended to configure a Messaging Trigger over a HTTP Trigger for improved system resilience in production environments. However you may wish to configure a HTTP Trigger for development and testing purposes.

5. WebProtégé Exporter

As detailed in step 4 above, the OntoPop WebProtégé Exporter service is invoked via either a messaging, scheduler or HTTP trigger. In all cases, once invoked, the OntoPop WebProtégé Exporter service will consume the original WebProtégé webhook payload.

6. Ontology Management API

After the OntoPop WebProtégé Exporter service is invoked and has consumed the original WebProtégé webhook payload, it will use the OntoPop Management API to check whether any of the ontologies that OntoPop is monitoring for changes has an webProtegeProjectId attribute equal to the WebProtégé project ID that has been parsed from the WebProtégé webhook payload. If so, it will proceed to export the relevant ontology from WebProtégé as described in step 7 below.

7. WebProtégé Export and Download

Assuming that the WebProtégé project ID refers to an ontology currently being monitored by OntoPop for changes, as described in step 6 above, then the OntoPop WebProtégé Exporter service, in the absence of a WebProtégé API (or similar web service), will use a web automation framework to programmatically log into the WebProtégé cloud application using an OntoPop service account. Once the programmatic login attempt has been successfully authenticated, the OntoPop WebProtégé Exporter service will get the JSESSIONID cookie value of the current authenticated session. Then it will send a HTTP GET request to the WebProtégé download URI at /download with the WebProtégé project ID, revision number and export format (OWL) as GET request parameters, along with the JSESSIONID cookie value. Finally, the OntoPop WebProtégé Exporter service will write the string contents of the HTTP GET response body to local temporary storage as an OWL file.

8. GitHub API

The final stage of the custom WebProtégé connector is to get the OWL file that has been written to local temporary storage in step 7 above and use the GitHub Repositories API to PUT the file to the relevant Git repositories and branches, along with an automated commit message (where the details of the target Git repositories and branches have already been identified in step 6 above - see the OntoPop Management API for further details, particularly in relation to the Management API ontology response body schema).

Note that there may be multiple ontologies that OntoPop is monitoring for changes with the same WebProtégé project ID. Therefore step 8 may write to multiple GitHub repositories and branches.

Note that by writing to the relevant GitHub repositories and branches using the GitHub Repositories API, the GitHub webhook(s) configured on these repositories will then be invoked which in turn will invoke the ontology ingestion service and thus OntoPop's event-driven data pipeline as described in Microservices and Logical System Architecture respectively. Thus this is how the edits made in WebProtégé are automatically ingested into OntoPop via this custom WebProtégé connector.