Connector Deployments

Deploy OntoPop's custom out-of-the-box WebProtégé connector to a self-managed environment.

Last updated: 15 March 2022Author: Jillur Quddus

Please note that the OntoPop backend open-source software project, which includes the event-driven data pipelines and APIs, is undergoing extensive redesign and refactoring as part of OntoPop Community 3.x in order to improve performance, security, extensibility and maintainability. As a result, the documentation on this page will be significantly updated. Please refer to the OntoPop Roadmap for further information.

WebProtégé

Overview

The custom OntoPop WebProtégé connector is designed to automate the export of an ontology from an existing WebProtégé project to a designated Git repository and branch as an OWL file in real-time. This page provides instructions on how to deploy the custom OntoPop WebProtégé connector to a self-managed environment.

To find out more information regarding the custom OntoPop WebProtégé connector including its design and how it works, please refer to WebProtégé in the integration architecture section of this website.

It is recommended that you configure and integrate the steps described in this page into a CI/CD pipeline in order to automate the build, testing and deployment stages.

Prerequisites

1. Please read through the WebProtégé page in the integration architecture section of this website to understand how the custom OntoPop WebProtégé connector works.

2. Referencing the self-managed deployment architecture, please ensure that you have selected, provisioned, configured and deployed to your self-managed environment all the relevant third-party software services required by OntoPop as per the following check list:

   • Secrets Engine - for example HashiCorp Vault
   • Message Broker - for example RabbitMQ
   • RDBMS - for example HSQLDB
   • RDF Triplestore - for example Apache Jena Fuseki
   • Graph Database - for example JanusGraph
   • Search Engine - for example Elasticsearch
     
     

3. Install the required Build Tools as described in Build from Source.

4. Clone the OntoPop source code repository from GitHub as described in Build from Source.

5. Ensure that you have entered the correct configuration into the Spring bootstrap.yml and application.yml property files respectively, relevant to your self-managed environment.

6. Compile and build the OntoPop services and applications as described in Build from Source.

Deployment

WebProtégé Webhook Subscriber

In the following instructions we detail how the WebProtégé webhook subscriber Node.js application can be deployed to a self-managed environment.

1. To invoke the WebProtégé Exporter service via a HTTP POST request (HTTP Trigger) whose request body will contain the WebProtégé webhook payload, set the URL to the WebProtégé Exporter service as an environment variable named WEBPROTEGE_GIT_EXPORTER_URL in your self-managed environment.
2. Alternatively, to invoke the WebProtégé Exporter service via the shared messaging system to which the WebProtégé Exporter service is subscribed (Messaging Trigger), set the RabbitMQ hostname, queue name and credentials as environment variables in your self-managed environment. Specifically these environment variables are named AMQP_HOSTNAME, AMQP_USERNAME, AMQP_PASSWORD and AMQP_QUEUE respectively.
3. Alternatively, to invoke the WebProtégé Exporter service via a scheduler (Scheduler Trigger - recommended), set the RDBMS hostname, port, database name, credentials and dialect as environment variables in your self-managed environment. Specifically these environment variables are named RDBMS_HOSTNAME, RDBMS_PORT, RDBMS_DATABASE, RDBMS_USERNAME, RDBMS_PASSWORD and RDBMS_DIALECT respectively.

If configuring the scheduler trigger (recommended), the WebProtégé webhook subscriber Node.js application uses the Sequelize ORM library to write records to the RDBMS. For further information regarding Sequelize including supported dialects, please visit https://sequelize.org/v6/.

4. Once the appropriate trigger is configured, navigate to $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring/ontopop-app-subscriber-webprotege-webhook and run the Node.js application (which will in turn start a simple HTTP server to listen for WebProtégé webhooks) as follows:

# Navigate to the relevant project folder
$ cd $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring/ontopop-app-subscriber-webprotege-webhook

# Run the Node.js application (which will start a simple HTTP server)
$ node index.js

If you are running the Node.js HTTP server on a machine whose network interfaces are not publicly available (and hence cannot be accessed by GitHub webhooks over the internet) you can install and deploy ngrok to create secure introspectable tunnels to localhost. For example to start a tunnel that exposes a local web server running on port 8080 and bound to localhost, you can run the command $ ngrok http 8080. For further information regarding exposing a local web server to the internet using ngrok, please visit https://ngrok.com/docs.

5. Finally we need to configure WebProtégé to send webhook payloads to the public URL of our WebProtégé webhook subscriber HTTP server. To do so, log into WebProtégé at https://webprotege.stanford.edu and select the relevant project from the project listing. Next select "Project" and "Settings" from the toolbar. Scroll down to "Payload URLs" and enter the public URL of the WebProtégé webhook subscriber HTTP server (including the index.js resource path) along with a request parameter called protocol. This request parameter should be set to http if configuring a HTTP Trigger, amqp if configuring a Messaging Trigger, or sql if configuring a Scheduler Trigger, as illustrated in the following screenshot.

Note that your WebProtégé user must be provisioned with manage privileges for the relevant WebProtégé project in order to access project settings, otherwise this toolbar menu option will be greyed out and disabled. In this case, please contact the owner of the respective WebProtégé project to configure the payload URL.

6. Press "Apply". Now every time an atomic edit is made in WebProtégé for this WebProtégé project, a WebProtégé webhook payload will be generated and a HTTP POST request made to the public URL of the WebProtégé webhook subscriber HTTP server for processing.

WebProtégé Exporter

In the following instructions we detail how the WebProtégé exporter Java Spring Boot application can be deployed to a self-managed environment.

1. Ensure that any environment variables referenced in bootstrap.yml are set in your self-managed environment.
2. Configure the plugins.webprotege.exporter namespace in application.yml, as follows:

3. If you have configured a Messaging Trigger, then you must ensure that the concatenation of the destination and group properties of the webProtegeProjectUpdatedConsumptionChannel binding in the spring.cloud.stream.bindings namespace in application.yml exactly matches the value of the AMQP_QUEUE environment variable configured when deploying the WebProtégé webhook subscriber Node.js application - see above. By default, the name of this queue is webprotege.project.updated.ontopop.

4. Next we need to define two environment variables named WEBPROTEGE_USERNAME and WEBPROTEGE_PASSWORD respectively. These two environment variables must be set with values corresponding to the credentials of an active WebProtégé service account. The WebProtégé exporter service will use these WebProtégé account credentials when programmatically logging into WebProtégé in order to export and download the ontology as an OWL file.

The WebProtégé service account must be provisioned with view privileges to the relevant WebProtégé project ID. Please contact the owner (or a manager) of the respective WebProtégé project in order to share it with the designated WebProtégé service account.

5. Navigate to $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring/ontopop-app-data-ontology-webprotege-exporter-scheduler (if you have configured a Scheduler Trigger) or $ONTOPOP_BASE/ontopop-apps/ontopop-app-data-ontology-webprotege-exporter-messaging (if you have configured a Messaging Trigger or HTTP Trigger) and run the Spring Boot application (which has been packaged as an executable JAR file following the compilation and build of the OntoPop apps from source as described in step 6 in the prerequisites section above) as follows:

# Navigate to the relevant data pipeline function app project folder
$ cd $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring/ontopop-app-data-ontology-webprotege-exporter-scheduler

# Run the packaged Spring Boot application
$ java -Dserver.port=8812 -jar target/ontopop-app-data-ontology-webprotege-exporter-scheduler-2.0.0.jar

The Java VM argument -Dserver.port=8812 defines the port number on which the embedded Apache Tomcat servlet engine will listen for requests (8812 in this case). If you are deploying all the OntoPop applications on the same host, you must specify different port numbers for each different application you start otherwise an exception will be thrown stating that the port number is already in use.

6. If you have configured a Scheduler Trigger, the WebProtégé exporter service will run as per the CRON schedule defined in step 2 above. If you have configured a HTTP Trigger (HTTP POST), then the WebProtégé exporter service can be requested via the /connectors/webprotege/git path (the full URL of which should be configured as an environment variable named WEBPROTEGE_GIT_EXPORTER_URL when deploying the WebProtégé webhook subscriber Node.js application - see above). Finally, if you have configured a Messaging Trigger, then the WebProtégé exporter service will be invoked when a message is published to the relevant RabbitMQ queue.