Elastic Cloud

Provision an Elastic Cloud managed Elasticsearch cluster with Kibana and integrate it with OntoPop.

Last updated: 14 March 2022Author: Jillur Quddus

Please note that the OntoPop backend open-source software project, which includes the event-driven data pipelines and APIs, is undergoing extensive redesign and refactoring as part of OntoPop Community 3.x in order to improve performance, security, extensibility and maintainability. As a result, the documentation on this page will be significantly updated. Please refer to the OntoPop Roadmap for further information.

Overview

Elastic Cloud is a public cloud managed service available on major cloud computing platforms (including Amazon Web Services, Microsoft Azure and Google Cloud Platform) providing fully-managed SaaS offerings from the Elastic stack including Elasticsearch and Kibana. This page provides instructions on how to provision an Elastic Cloud deployment and integrate it with OntoPop.

For further information regarding Elastic Cloud and its features, please visit https://www.elastic.co/subscriptions/cloud.

Elasticsearch is currently the recommended search engine when deploying OntoPop. OntoPop does not currently support Amazon CloudSearch. And though OntoPop does support Azure Cognitive Search, it is not currently recommended. This is because, as described in the Microsoft Azure Deployment Architecture page, it only supports pre-defined fields. In other words, the Azure search index schema must be explicitly defined, along with EDM types, prior to indexing and there is no support for dynamic field creation. Given the nature of ontologies and the fact that new OWL object properties and class properties may be added (either via OntoPop or other industry-standard tools such as WebProtege) during their normal iterative development life cycle, it is not feasible to continuously and explicitly define new search index schema fields after every change to the ontology.

Setup

Elastic Deployment

To provision a fully-managed Elastic Cloud Elasticsearch cluster, navigate to https://cloud.elastic.co to sign up for an Elastic Cloud account. Once signed up and logged in, select "Create Deployment" to create a fully-managed Elastic deployment consisting of a managed Elasticsearch cluster along with Kibana (as a minimum). You will be asked for the following information:

Name - enter a custom name for your deployment, for example mydeployment.
Cloud Provider - select a cloud provider (for example Amazon Web Services) which will be used to host the Elastic deployment. Note that administration of the Elastic deployment is performed entirely via Elastic Cloud and there is no need to create an account on the selected cloud computing platform.
Region - select a region, for example London (eu-west-2).
Hardware Profile - select a hardware profile. It is recommended to select "General Purpose" and, for small ontologies containing hundreds or low thousands of OWL classes, an Elasticsearch hardware profile of 20GB storage with 2GB RAM and up to 4.3 vCPU (in the case of available Amazon Web Services hardware profiles after selecting "Advanced Settings") should suffice. In regards to Kibana, unless you wish to use Kibana as an additional data visualization tool, then a hardware profile of 1GB with up to 8.5 vCPU (in the case of available AWS hardware profiles) will suffice.
Availability Zones - select the number of availability zones based on your deployment and fault tolerance requirements. For development and testing purposes, one (1) availability zone will suffice for both Elasticsearch and Kibana.
Version - OntoPop currently only supports version 7.x of Elasticsearch (see Compatibility Matrix for further details).

Once configured, select "Create Deployment" to create the deployment (it may take Elastic Cloud between 5 - 10 minutes to fully deploy).

Elasticsearch Endpoint

Once created and deployed, navigate to https://cloud.elastic.co/deployments, select the newly provisioned deployment and make a note of the Elasticsearch endpoint (of the format https://mydeployment.es.eu-west-2.aws.cloud.es.io:9243). This will be required when we integrate the managed Elasticsearch cluster with OntoPop below.

Authorization

In order to securely integrate with OntoPop, we must create a new user with a custom role profile limited to only those privileges required by OntoPop. To do this, first we shall create and configure a custom role with selected privileges via Elastic Cloud. Navigate to https://cloud.elastic.co/deployments and select the newly created deployment. In the main menu on the left of the screen, select "Security" and then "Make security changes in Kibana". In Kibana, select "Roles" from the main menu on the left of the screen, then "Create Role" and enter the following information, as illustrated in the screenshot below:

Role Name - enter a custom role name, for example ontopop-read-write.
Index Privileges - enter ontopop* in the "Indices" text box (note the wildcard character). Then in the "Privileges" dropdown, select the following privileges: auto_configure, create, create_doc, create_index, delete, delete_index, index, maintenance, read, view_index_metadata and write.

Once configured, select "Create Role" to create the new role.

Authentication

Finally we must create the new user and attach to it the custom role configured above. To do this, in Kibana select "Users" from the main menu on the left of the screen and then select "Create User". Provide the required information including username (for example myuser) and password. Finally map the custom role created above to this new user in the "Privileges" section. Once configured, select "Create User" to create the new user whilst making a note of the provided username and password for integration with OntoPop below.

Integration

Application Context

As described in Application Context, the configuration for the OntoPop application context may be found in the ontopop-configuration Maven module, at src/main/resources/application.yml. To integrate a fully-managed Elastic Cloud Elasticsearch service with OntoPop, configure the storage.search namespace in application.yml as follows:

Property	Description	Example Value
`service`	The name of the search service used by OntoPop. Set this to `elasticsearch`.	elasticsearch
`elasticsearch.url`	The Elasticsearch endpoint noted above. Note that the URL should be set as an externalized variable and NOT stored as plaintext in application.yml.	https://mydeployment.es.eu-west-2.aws.cloud.es.io:9243
`elasticsearch.username`	The username to use as part of basic authentication requests, as noted above. Note that the username should be set as an externalized variable and NOT stored as plaintext in application.yml.	myuser
`elasticsearch.password`	The password to use as part of basic authentication requests, as noted above. Note that the password should be set as an externalized variable and NOT stored as plaintext in application.yml.	password
`elasticsearch.ssl`	Whether communication with the Elasticsearch instance is secured by SSL. Always set this to `true` when integrating with a fully-managed Elastic Cloud Elasticsearch cluster.	true
`elasticsearch.shards`	If using Elasticsearch, the default number of shards to create when creating a new index.	1
`elasticsearch.replicas`	If using Elasticsearch, the default number of replicas to create when creating a new index.	0