Connector Deployments

Deploy OntoPop's custom out-of-the-box WebProtégé connector to the Amazon Web Services (AWS) cloud computing platform.

Last updated: 15 March 2022Author: Jillur Quddus

Please note that the OntoPop backend open-source software project, which includes the event-driven data pipelines and APIs, is undergoing extensive redesign and refactoring as part of OntoPop Community 3.x in order to improve performance, security, extensibility and maintainability. As a result, the documentation on this page will be significantly updated. Please refer to the OntoPop Roadmap for further information.

WebProtégé

Overview

The custom OntoPop WebProtégé connector is designed to automate the export of an ontology from an existing WebProtégé project to a designated Git repository and branch as an OWL file in real-time. This page provides instructions on how to deploy the custom OntoPop WebProtégé connector to the Amazon Web Services (AWS) cloud computing platform.

To find out more information regarding the custom OntoPop WebProtégé connector including its design and how it works, please refer to WebProtégé in the integration architecture section of this website.

It is recommended that you configure and integrate the steps described in this page into a CI/CD pipeline in order to automate the build, testing and deployment stages.

Prerequisites

1. Please read through the WebProtégé page in the integration architecture section of this website to understand how the custom OntoPop WebProtégé connector works.
2. In order to compile and build the WebProtégé webhook subscriber Node.js application and WebProtégé exporter Java Spring Boot application in preparation for deployment to AWS Lambda instances, please follow the instructions detailed in Build from Source.
3. We shall use the AWS Command Line Interface (CLI) to deploy the WebProtégé webhook subscriber Node.js application and WebProtégé exporter Java Spring Boot application to AWS Lambda instances. To install the AWS CLI, please follow the instructions below:

The instructions below are for Ubuntu 20.04. Installation instructions for other Linux distributions and other operating systems such as Windows may be found at https://aws.amazon.com/cli.

# Install the required dependencies
$ sudo apt-get update 
$ sudo apt-get install glibc groff less

# Install the AWS CLI from a ZIP file
$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
$ unzip awscliv2.zip
$ sudo ./aws/install

Assuming that the AWS CLI has installed successfully, we can configure it with the Access Key ID and Secret Access Token of an IAM user with privileges to programmatically manage AWS Lambda instances (such as an IAM user provisioned with the AWSLambda_FullAccess AWS managed policy, or similar) as follows:

# Configure the AWS CLI
$ aws configure

    AWS Access Key ID [None]: AKIA123456789
    AWS Secret Access Key [None]: abcdefg987654321hijklmnop
    Default region name [None]: eu-west-2
    Default output format [None]: json

Deployment

WebProtégé Webhook Subscriber

In the following instructions we detail how the WebProtégé webhook subscriber Node.js application can be deployed to an AWS Lambda instance.

1. Create a new (empty) AWS Lambda instance configured with the Node.js 14.x runtime via the AWS Management Console as detailed above. We shall call this AWS Lambda instance webprotege-webhook-subscriber for the purposes of these instructions. Once created, open this new AWS Lambda instance via the AWS Management Console, navigate to Configuration > General configuration, set its memory to 128 MB and set its timeout to 1 min 0 sec.
2. We are now ready to deploy the WebProtégé webhook subscriber Node.js application code contained in the $ONTOPOP_BASE/ontopop-apps/ontopop-apps-aws/ontopop-aws-lambda-app-subscriber-webprotege-webhook project to this AWS Lambda instance. Assuming that you have followed the instructions detailed in the Prerequisites section above, navigate to $ONTOPOP_BASE/ontopop-apps/ontopop-apps-aws/ontopop-aws-lambda-app-subscriber-webprotege-webhook and execute the following commands via your command line:

# Navigate to the relevant project folder
$ cd $ONTOPOP_BASE/ontopop-apps/ontopop-apps-aws/ontopop-aws-lambda-app-subscriber-webprotege-webhook

# Install dependencies using NPM
$ npm install

# Package the index.js file into a ZIP archive file
$ zip -r function.zip .

# Use the AWS CLI to deploy the ZIP file to the relevant AWS Lambda instance
$ aws lambda update-function-code --function-name webprotege-webhook-subscriber --zip-file fileb://function.zip

3. Now that we have uploaded the application code to the WebProtégé webhook subscriber AWS Lambda instance, we need to make it publicly accessible via HTTP. To do this, navigate to the WebProtégé webhook subscriber AWS Lambda instance via the AWS Management Console, select "Add trigger" and then select "API Gateway". Configure API Gateway accordingly by creating a new HTTP API and new custom HTTP POST route (for example /connectors/webprotege/subscriber), and then integrate this new HTTP POST route with the WebProtégé webhook subscriber AWS Lambda instance. The WebProtégé webhook subscriber AWS Lambda is now accessible publicly via HTTP. To identify its HTTPS endpoint, navigate to the WebProtégé webhook subscriber AWS Lambda instance via the AWS Management Console and select Configuration > Triggers (press the refresh button if required). The HTTPS endpoint will look similar to https://abcde12345.execute-api.eu-west-2.amazonaws.com/connectors/webprotege/subscriber (if you have configured a custom domain name in API Gateway, then the custom domain name can be used instead of the AWS hostname).
4. Finally we need to configure WebProtégé to send webhook payloads to the AWS API Gateway HTTPS endpoint noted above. To do so, log into WebProtégé at https://webprotege.stanford.edu and select the relevant project from the project listing. Next select "Project" and "Settings" from the toolbar. Scroll down to "Payload URLs" and enter the AWS API Gateway HTTPS endpoint noted above along with a request parameter called protocol. This request parameter value should be set to lambda if invoking the WebProtégé exporter service directly via the AWS SDK, or sql if configuring a Scheduler Trigger, as illustrated in the following screenshot.

Note that your WebProtégé user must be provisioned with manage privileges for the relevant WebProtégé project in order to access project settings, otherwise this toolbar menu option will be greyed out and disabled. In this case, please contact the owner of the respective WebProtégé project to configure the payload URL.

5. Press "Apply". Now every time an atomic edit is made in WebProtégé for this WebProtégé project, a WebProtégé webhook payload will be generated and a HTTP POST request made to the AWS API Gateway HTTPS endpoint of the WebProtégé webhook subscriber AWS Lambda instance for processing.

WebProtégé Exporter

In the following instructions we detail how the WebProtégé exporter Java Spring Boot application can be deployed to an AWS Lambda instance.

1. Create a new (empty) AWS Lambda instance configured with the Java 11 (Corretto) runtime via the AWS Management Console as detailed in the Setup section above. We shall call this AWS Lambda instance ontology-webprotege-exporter-service for the purposes of these instructions. Once created, open this new AWS Lambda instance via the AWS Management Console, navigate to Configuration > General configuration, set its memory to 1024 MB and set its timeout to 5 min 0 sec.
2. Since we are deploying a Java Spring Boot application that utilizes the Spring Cloud Function project, we need to configure the AWS Lambda instance with details of the main Java class to invoke as well as the name of the Java function that will be executed. To do this, open the AWS Lambda instance via the AWS Management Console, navigate to Configuration > Environment variables and set an environment variable named MAIN_CLASS containing the main Java class to invoke, and another environment variable named spring_cloud_function_definition containing the function name (starting with a lowercase letter) as follows:

3. Next we need to define two further environment variables named WEBPROTEGE_USERNAME and WEBPROTEGE_PASSWORD respectively. These two environment variables must be set with values corresponding to the credentials of an active WebProtégé service account. The WebProtégé exporter service will use these WebProtégé account credentials when programmatically logging into WebProtégé in order to export and download the ontology as an OWL file.

The WebProtégé service account must be provisioned with view privileges to the relevant WebProtégé project ID. Please contact the owner (or a manager) of the respective WebProtégé project in order to share it with the designated WebProtégé service account.

4. Next we need to configure the AWS Lambda instance with the fully qualified class name and method of the function handler. To do this, open the AWS Lambda instance via the AWS Management Console, navigate to Code and select the "Edit" button belonging to the "Runtime settings" section. In the "Handler" box enter org.springframework.cloud.function.adapter.aws.FunctionInvoker::handleRequest, and then press "Save".
5. Assuming that you are integrating OntoPop with the AWS Secrets Manager, we need to provide permission for the AWS Lambda instance to read secrets managed by AWS Secrets Manager. To do this, open the AWS Lambda instance via the AWS Management Console, navigate to Configuration > Permissions and select the execution role name (for example ontology-webprotege-exporter-service-role-abc123). This will take you to the IAM Management Console for this role. Select Add permissions > Attach policies and attach the SecretsManagerReadWrite AWS managed policy to this role (or equivalent custom policy). Now when the AWS Lambda instance is invoked, externalized sensitive properties defined in the OntoPop application context will be loaded from AWS Secrets Manager.
6. If configuring a Scheduler Trigger (recommended), then we can configure an Amazon EventBridge trigger to invoke and run the WebProtégé exporter AWS Lambda instance on a periodic basis governed by a CRON schedule. To do this, open the AWS Lambda instance via the AWS Management Console and select "Add Trigger". From the list of trigger types, select "EventBridge (CloudWatch Events)" and check "Create a new rule". Next enter a rule name, rule description and select "Schedule expression" as the rule type. In the schedule expression text box, enter a CRON or rate expression. For example, to invoke and run the WebProtégé exporter AWS Lambda instance every 10 minutes using a rate expression, you would enter rate(10 minutes). Finally press "Add" to add the new trigger.

For further information regarding AWS CloudWatch Events schedule expressions, please visit Amazon CloudWatch Scheduled Events.

7. We are now ready to deploy the packaged Java Spring application artifact to the AWS Lambda instance. Assuming that you have followed the instructions detailed in the prerequisites section above, navigate to $ONTOPOP_BASE/ontopop-apps/ontopop-apps-aws/ontopop-aws-lambda-app-data-ontology-webprotege-exporter-scheduler and execute the following commands via your command line:

# Navigate to the relevant project folder
$ cd $ONTOPOP_BASE/ontopop-apps/ontopop-apps-aws/ontopop-aws-lambda-app-data-ontology-webprotege-exporter-scheduler/target

# Upload the packaged JAR file to an Amazon S3 bucket
$ aws s3 cp ontopop-aws-lambda-app-data-ontology-webprotege-exporter-scheduler-2.0.0-aws.jar s3://ontopop-apps

# Deploy the function code from Amazon S3 to the relevant AWS Lambda instance
$ aws lambda update-function-code --function-name ontology-webprotege-exporter-service --s3-bucket ontopop-apps --s3-key ontopop-aws-lambda-app-data-ontology-webprotege-exporter-scheduler-2.0.0-aws.jar