Application Deployments

Deploy OntoPop apps to a self-managed environment.

Last updated: 02 March 2022Author: Jillur Quddus

Please note that the OntoPop backend open-source software project, which includes the event-driven data pipelines and APIs, is undergoing extensive redesign and refactoring as part of OntoPop Community 3.x in order to improve performance, security, extensibility and maintainability. As a result, the documentation on this page will be significantly updated. Please refer to the OntoPop Roadmap for further information.

Overview

This page provides instructions on how to build, deploy and run all the OntoPop data pipeline and API Spring Boot applications in a self-managed environment.

It is recommended that you configure and integrate the steps described in this page into a CI/CD pipeline in order to automate the build, testing and deployment stages.

Prerequisites

1. Referencing the self-managed deployment architecture, please ensure that you have selected, provisioned, configured and deployed to your self-managed environment all the relevant third-party software services required by OntoPop as per the following check list:

   • Secrets Engine - for example HashiCorp Vault
   • Message Broker - for example RabbitMQ
   • RDBMS - for example HSQLDB
   • RDF Triplestore - for example Apache Jena Fuseki
   • Graph Database - for example JanusGraph
   • Search Engine - for example Elasticsearch
     
     

2. Install the required Build Tools as described in Build from Source.

3. Clone the OntoPop source code repository from GitHub as described in Build from Source.

4. Ensure that you have entered the correct configuration into the Spring bootstrap.yml and application.yml property files respectively, relevant to your self-managed environment.

5. Compile and build the OntoPop services and applications as described in Build from Source.

Data Pipeline

OntoPop provides Spring Boot application deployments that wrap around each of the event-driven microservices described in the logical system architecture. These applications are provided out-of-the-box to enable quick and easy deployment to a self-managed environment or to a Spring Cloud deployment environment such as Azure Spring Cloud. Assuming that you have followed the prerequisite instructions above, the Spring Boot applications for each of the event-driven microservices that make up the OntoPop data pipeline may be found in the $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring Maven module, which itself contains the following child modules pertinent to the data pipeline:

1. ontopop-app-subscriber-github-webhook - Node.js application that subscribes to GitHub webhooks and forwards the webhook headers and payload to the ontology ingestion service via either a HTTP POST request or by publishing a message to the shared messaging system (to which the ontology ingestion service is subscribed). Note that this is a Node.js application (i.e. not a Spring Boot application) as GitHub webhook requests timeout after 10 seconds after which the HTTP connection is destroyed and the webhook payload lost. Thus we can deploy this lightweight Node.js application that returns a promise (i.e. an immediate response back to GitHub) to avoid the longer cold start-up times incurred by Java-based applications.
2. ontopop-app-data-ontology-ingestor - Spring Boot application deployment wrapper around the ontology ingestion service, invoked by the ontopop-app-subscriber-github-webhook application by either a HTTP POST request or via the shared messaging system to which the ontopop-app-data-ontology-ingestor application is subscribed.
3. ontopop-app-data-ontology-validator - Spring Boot application deployment wrapper around the ontology validation service, invoked via its subscription to the shared messaging system.
4. ontopop-app-data-ontology-loader-triplestore - Spring Boot application deployment wrapper around the ontology triplestore loading service, invoked via its subscription to the shared messaging system.
5. ontopop-app-data-ontology-parser - Spring Boot application deployment wrapper around the ontology parsing service, invoked via its subscription to the shared messaging system.
6. ontopop-app-data-ontology-modeller-graph - Spring Boot application deployment wrapper around the property graph modelling service, invoked via its subscription to the shared messaging system.
7. ontopop-app-data-ontology-loader-graph - Spring Boot application deployment wrapper around the property graph loading service, invoked via its subscription to the shared messaging system.
8. ontopop-app-data-ontology-indexer-graph - Spring Boot application deployment wrapper around the property graph indexing service, invoked via its subscription to the shared messaging system.

GitHub Subscriber

In the following instructions we detail how the GitHub webhook subscriber Node.js application can be deployed to a self-managed environment.

1. To invoke the ontology ingestion service via a HTTP POST request whose request body will contain the GitHub webhook payload, set the URL to the ontology ingestion service as an environment variable named ONTOLOGY_INGESTOR_URL in your self-managed environment.
2. Alternatively, to invoke the ontology ingestion service via the shared messaging system to which the ontology ingestion service is subscribed (recommended), set the RabbitMQ hostname, queue and credentials as environment variables in your self-managed environment. Specifically these environment variables are named AMQP_HOSTNAME, AMQP_USERNAME, AMQP_PASSWORD and AMQP_QUEUE respectively.
3. Navigate to $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring/ontopop-app-subscriber-github-webhook and run the Node.js application (which will in turn start a simple HTTP server to listen for GitHub webhooks) as follows:

# Navigate to the relevant project folder
$ cd $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring/ontopop-app-subscriber-github-webhook

# Run the Node.js application (which will start a simple HTTP server)
$ node index.js

If you are running the Node.js HTTP server on a machine whose network interfaces are not publicly available (and hence cannot be accessed by GitHub webhooks over the internet) you can install and deploy ngrok to create secure introspectable tunnels to localhost. For example to start a tunnel that exposes a local web server running on port 8080 and bound to localhost, you can run the command $ ngrok http 8080. For further information regarding exposing a local web server to the internet using ngrok, please visit https://ngrok.com/docs.

4. Next we need to configure a webhook in the relevant GitHub repository and set its payload URL to the public URL of our GitHub webhook subscriber HTTP server. Navigate to the relevant GitHub repository in a web browser, select Settings > Webhooks > Add webhook and enter the following properties:

The following screenshot provides an example GitHub webhook configuration.

5. Press "Add webhook". Now every time a push event occurs in the relevant GitHub repository, this webhook will be triggered and a HTTP POST request made to the public URL of the GitHub webhook subscriber HTTP server.

Function Apps

In the following instructions we use the ontopop-app-data-ontology-ingestor child Maven module as an example with which to demonstrate how to deploy an event-driven data pipeline function Spring Boot application. However these instructions can be equally applied to deploy any and all of the data pipeline function Spring Boot applications listed above.

1. Ensure that any environment variables referenced in bootstrap.yml are set in your self-managed environment. 2) Navigate to $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring/ontopop-app-data-ontology-ingestor and run the Spring Boot application (which has been packaged as an executable JAR file following the compilation and build of the OntoPop apps from source as described in step 5 in the prerequisites section above) as follows:

# Navigate to the relevant data pipeline function app project folder
$ cd $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring/ontopop-app-data-ontology-ingestor

# Run the packaged Spring Boot application
$ java -Dserver.port=8802 -jar target/ontopop-app-data-ontology-ingestor-2.0.0.jar

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.6.2)

2022-02-09 10:14:10.322 [main] INFO  a.h.o.a.d.o.i.OntologyIngestorApp - No active profile set, falling back to default profiles: default
2022-02-09 10:14:11.394 [main] INFO  o.s.d.r.c.RepositoryConfigurationDelegate - Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2022-02-09 10:14:11.487 [main] INFO  o.s.d.r.c.RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 84 ms. Found 2 JPA repository interfaces.
2022-02-09 10:14:12.441 [main] INFO  o.s.i.c.DefaultConfiguringBeanFactoryPostProcessor - No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2022-02-09 10:14:12.496 [main] INFO  o.s.i.c.DefaultConfiguringBeanFactoryPostProcessor - No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2022-02-09 10:14:12.797 [main] INFO  o.s.cloud.context.scope.GenericScope - BeanFactory id=8b3a5b75-eade-39c8-a32e-7085c64cb4b0
2022-02-09 10:14:13.092 [main] INFO  o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'integrationChannelResolver' of type [org.springframework.integration.support.channel.BeanFactoryChannelResolver] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-02-09 10:14:13.159 [main] INFO  o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.integration.config.IntegrationManagementConfiguration' of type [org.springframework.integration.config.IntegrationManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-02-09 10:14:13.636 [main] INFO  o.s.b.w.e.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8802 (http)
2022-02-09 10:14:13.663 [main] INFO  o.a.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8802"]
2022-02-09 10:14:13.664 [main] INFO  o.a.catalina.core.StandardService - Starting service [Tomcat]
2022-02-09 10:14:13.664 [main] INFO  o.a.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.56]
2022-02-09 10:14:13.749 [main] INFO  o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2022-02-09 10:14:13.750 [main] INFO  o.s.b.w.s.c.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 3393 ms
2022-02-09 10:14:14.039 [main] INFO  o.h.jpa.internal.util.LogHelper - HHH000204: Processing PersistenceUnitInfo [name: default]
2022-02-09 10:14:14.157 [main] INFO  org.hibernate.Version - HHH000412: Hibernate ORM core version 5.6.3.Final
2022-02-09 10:14:14.596 [main] INFO  o.h.annotations.common.Version - HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
2022-02-09 10:14:14.742 [main] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
2022-02-09 10:14:15.171 [main] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
2022-02-09 10:14:15.210 [main] INFO  org.hibernate.dialect.Dialect - HHH000400: Using dialect: org.hibernate.dialect.MySQL57Dialect
2022-02-09 10:14:16.777 [main] INFO  o.h.e.t.j.p.i.JtaPlatformInitiator - HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2022-02-09 10:14:16.788 [main] INFO  o.s.o.j.LocalContainerEntityManagerFactoryBean - Initialized JPA EntityManagerFactory for persistence unit 'default'
2022-02-09 10:14:18.422 [main] WARN  o.s.b.a.o.j.JpaBaseConfiguration$JpaWebConfiguration - spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2022-02-09 10:14:20.420 [main] WARN  c.a.s.c.a.s.AzureServiceBusAutoConfiguration - Can't construct the ServiceBusConnectionStringProvider, namespace: null, connectionString: null
2022-02-09 10:14:20.430 [main] INFO  c.a.s.c.a.s.AzureServiceBusQueueAutoConfiguration - No service bus connection string provided.
2022-02-09 10:14:20.445 [main] INFO  c.a.s.i.s.ServiceBusTemplate - Started ServiceBusTemplate with properties: CheckpointConfig{checkpointMode=RECORD, checkpointCount=0, checkpointInterval=null}
2022-02-09 10:14:20.457 [main] INFO  c.a.s.c.a.s.AzureServiceBusTopicAutoConfiguration - No service bus connection string provided.
2022-02-09 10:14:20.461 [main] INFO  c.a.s.i.s.ServiceBusTemplate - Started ServiceBusTemplate with properties: CheckpointConfig{checkpointMode=RECORD, checkpointCount=0, checkpointInterval=null}
2022-02-09 10:14:21.284 [main] INFO  o.s.c.s.f.FunctionConfiguration$FunctionBindingRegistrar - Functional binding is disabled due to the presense of @EnableBinding annotation in your configuration
2022-02-09 10:14:21.565 [main] INFO  o.s.i.endpoint.EventDrivenConsumer - Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2022-02-09 10:14:21.567 [main] INFO  o.s.i.c.PublishSubscribeChannel - Channel 'ontopop-1.errorChannel' has 1 subscriber(s).
2022-02-09 10:14:21.568 [main] INFO  o.s.i.endpoint.EventDrivenConsumer - started bean '_org.springframework.integration.errorLogger'
2022-02-09 10:14:21.571 [main] INFO  o.s.c.s.binder.DefaultBinderFactory - Creating binder: azure_service_bus
2022-02-09 10:14:21.634 [main] INFO  c.a.s.i.s.ServiceBusTemplate - Started ServiceBusTemplate with properties: CheckpointConfig{checkpointMode=RECORD, checkpointCount=0, checkpointInterval=null}
2022-02-09 10:14:21.712 [main] INFO  o.s.c.s.binder.DefaultBinderFactory - Caching the binder: azure_service_bus
2022-02-09 10:14:21.713 [main] INFO  o.s.c.s.binder.DefaultBinderFactory - Retrieving cached binder: azure_service_bus
2022-02-09 10:14:21.779 [main] INFO  c.a.s.i.core.DefaultMessageHandler - DefaultMessageHandler sync becomes: false
2022-02-09 10:14:21.779 [main] INFO  c.a.s.i.core.DefaultMessageHandler - DefaultMessageHandler syncTimeout becomes: ValueExpression [value=10000]
2022-02-09 10:14:21.780 [main] INFO  c.a.s.i.core.DefaultMessageHandler - Started DefaultMessageHandler with properties: {sendTimeout=ValueExpression [value=10000], destination=ontopop.data.ingested, sync=false}
2022-02-09 10:14:21.782 [main] INFO  o.s.c.s.m.DirectWithAttributesChannel - Channel 'ontopop-1.ingestedPublicationChannel' has 1 subscriber(s).
2022-02-09 10:14:21.785 [main] INFO  o.a.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8802"]
2022-02-09 10:14:21.800 [main] INFO  o.s.b.w.e.tomcat.TomcatWebServer - Tomcat started on port(s): 8802 (http) with context path ''
2022-02-09 10:14:21.822 [main] INFO  a.h.o.a.d.o.i.OntologyIngestorApp - Started OntologyIngestorApp in 18.58 seconds (JVM running for 19.661)

The Java VM argument -Dserver.port=8802 will configure the embedded Apache Tomcat servlet engine to listen for requests on the given port number (in this case 8802). If you are deploying all the OntoPop applications on the same host, you must specify different port numbers for each different application you start otherwise an exception will be thrown stating that the port number is already in use.

API Collections

OntoPop provides Spring Boot application deployments that wrap around each of the API collections described in the logical system architecture. These applications are provided out-of-the-box to enable quick and easy deployment to a self-managed environment or to a Spring Cloud deployment environment such as Azure Spring Cloud. Assuming that you have followed the prerequisite instructions above, the Spring Boot applications for each of the API collections may be found in the $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring Maven module, which itself contains the following child modules pertinent to the OntoPop APIs:

1. ontopop-app-api-ontology-management - Spring Boot application deployment wrapper for the Management API endpoints.
2. ontopop-app-api-ontology-triplestore - Spring Boot application deployment wrapper for the Triplestore API endpoints.
3. ontopop-app-api-ontology-graph - Spring Boot application deployment wrapper for the Graph API endpoints.
4. ontopop-app-api-ontology-search - Spring Boot application deployment wrapper for the Search API endpoints.

API Apps

In the following instructions we use the ontopop-app-api-ontology-management child Maven module as an example with which to demonstrate how to deploy an API collection Spring Boot application. However these instructions can be equally applied to deploy any and all of the API collection Spring Boot applications listed above.

1. Ensure that any environment variables referenced in bootstrap.yml are set in your self-managed environment.
2. Navigate to $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring/ontopop-app-api-ontology-management and run the Spring Boot application (which has been packaged as an executable JAR file following the compilation and build of the OntoPop apps from source as described in step 5 in the prerequisites section above) as follows:

# Navigate to the relevant API collection app project folder
$ cd $ONTOPOP_BASE/ontopop-apps/ontopop-apps-spring/ontopop-app-api-ontology-management

# Run the packaged Spring Boot application
$ java -Dserver.port=8811 -jar target/ontopop-app-api-ontology-management-2.0.0.jar