Skip to main content

Installing search API services (with Search API v8)

Important:

All passwords and keys in this section are given for illustration purposes. During a real installation, it is recommended to use more complex and reliable passwords.

1. Before installing​

  1. Consider getting familiar with:

  2. Make sure the necessary preparation steps are completed:

    1. Preparation for installation
    2. Installing License service
    3. Installing API Keys service
    4. Installing Statistics Collection service

  3. Collect the necessary information that was set or retrieved on previous steps:

    ObjectExample valueHow to get value
    Endpoint of the Docker Registry for storing service imagesdocker.registry.example.comSee Fetching installation artifacts
    Kubernetes secret to access the Docker Registryonpremise-registry-credsSee Fetching installation artifacts
    Endpoint of the S3 compatible Installation Artifacts Storageartifacts.example.comSee Fetching installation artifacts
    Bucket name for installation artifactsonpremise-artifactsSee Fetching installation artifacts
    Installation artifacts access keyAKIAIOSFODNN7EXAMPLESee Fetching installation artifacts
    Installation artifacts secret keywJalrXUtnFEMIK7MDENGbPxRfiCYEXAMPLEKEYSee Fetching installation artifacts
    Path to the Search API v8 manifest filemanifests/search-api-v8/1640661259.jsonSee Fetching installation artifacts
    Path to the Catalog APIs manifest filemanifests/catalog-api/1640661259.jsonSee Fetching installation artifacts
    S3 compatible storage endpoint for indexingindexing-storage.example.comYour own infrastructure
    Indexing bucket namesearch-indexingYour own infrastructure
    Access key ID for indexing storageAKIAIOSFODNN7EXAMPLEYour own infrastructure
    Secret access key for indexing storagewJalrXUtnFEMIK7MDENGbPxRfiCYEXAMPLEKEYYour own infrastructure
    License service endpointhttps://licenseSee Installing License service
    API Keys service endpointhttp://keys-service-apiSee Installing API Keys service
    Statistics Collection service endpointhttp://stat-receiverSee Installing Statistics Collection service
    Service tokenCATALOG_APIS_TOKENSee Installing API Keys service

  4. Make sure that the resource requirements specified in the Helm charts are met:

    For more information on how to do this, see the System requirements document.

    Note

    Contents of Helm charts described in this chapter are relevant for the latest API Platform version (see API Platform releases). To find parameters for previous versions, open the required values.yaml on GitHub and select a Platform-<version> tag in the tag switcher on the left.

  5. Choose domain names for the services.

    Example:

    • Domain name for Search API v8: search-api-v8.example.com
    • Domain name for Catalog APIs: catalog-api.example.com

2. Prepare infrastructure​

Configure PostgreSQL​

Note

If you use an alternative from the unified register of Russian programs instead of PostgreSQL, refer to its official documentation for configuration instructions.

  1. Place a PostgreSQL cluster with the domain name catalog-postgresql.storage.example.local in the private network. It is assumed that the cluster works on the standard port 5432.

  2. Connect to the cluster as a superuser (usually postgres).

  3. Create a database user and set a password for them:

    create user dbuser_catalog password '650D7AmZjSR1dkNa';

  4. Create a database owned by this user:

    create database onpremise_catalog owner dbuser_catalog;

  5. Install the required external extension PostGIS for PostgreSQL.

    You can get the extension from the repository where it is available as a ready-made package. For example, for Debian/Ubuntu-based distributions, the installation is performed by the commands:

    sudo apt install postgresql-15-postgis-3
    sudo apt install postgis

  6. Enable the necessary database extension:

    \c onpremise_catalog

    create schema extensions;
    grant usage on schema extensions to public;
    grant execute on all functions in schema extensions to public;
    alter default privileges in schema extensions grant execute on functions to public;
    alter default privileges in schema extensions grant usage on types to public;
    create extension if not exists plpgsql with schema pg_catalog;
    create extension if not exists postgis with schema extensions;

Create API key​

The key is checked before installing Catalog APIs.

Add your first partner and create an API key for them. The list of available services for the key must include the search APIs. See the Managing access to API instruction.

3. Install search API services​

Important:

Search API v8 service is in beta testing. Do not install it at the same time as Search API: the Catalog APIs can only work with one of these services.

Migration from previous Search API version​

If you have search services installed and want to migrate to Search API v8:

  1. Install Search API v8.

  2. Update the Catalog API configuration: in the values-catalog.yaml file, set the search.url parameter to http://search-api-v8:

    search:
      url: http://search-api-v8

  3. Test the services.

  4. If the services are working correctly, uninstall the previous version of Search API:

    helm uninstall search-api

Fresh installation of search services​

Install Search API v8 service​

  1. Create a Helm configuration file. See here for more details on the available settings.

    The example is prefilled with the necessary data collected on previous steps.

    values-search-api-v8.yaml
    dgctlDockerRegistry: docker.registry.example.com

    dgctlStorage:
      host: artifacts.example.com
      secure: true
      bucket: onpremise-artifacts
      accessKey: AKIAIOSFODNN7EXAMPLE
      secretKey: wJalrXUtnFEMIK7MDENGbPxRfiCYEXAMPLEKEY
      manifest: manifests/search-api-v8/1640661259.json
      region: ''

    s3Client:
      enabled: true
      endpoint: indexing-storage.example.com
      accessKeyId: AKIAIOSFODNN7EXAMPLE
      secretAccessKey: wJalrXUtnFEMIK7MDENGbPxRfiCYEXAMPLEKEY
      bucket: search-indexing
      region: ''
      useSsl: false

    worker:
      resources:
        requests:
          cpu: 1400m
          memory: 5500Mi
        limits:
          cpu: 6000m
          memory: 7500Mi

    head:
      head:
        resources:
          requests:
            cpu: 900m
            memory: 4600Mi
          limits:
            cpu: 2000m
            memory: 8Gi
      nginx:
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 250m
            memory: 256Mi
      envoy:
        resources:
          requests:
            cpu: 200m
            memory: 128Mi
          limits:
            cpu: 800m
            memory: 512Mi

    ingressRouter:
      enabled: true
      resources:
        requests:
          cpu: 1
          memory: 400Mi
        limits:
          cpu: 2
          memory: 2Gi

    geodetector:
      enabled: true
      resources:
        requests:
          cpu: 1
          memory: 4000Mi
        limits:
          cpu: 2
          memory: 6Gi

    indexer:
      resources:
        requests:
          cpu: 500m
          memory: 2Gi
        limits:
          cpu: 4000m
          memory: 16Gi

    controller:
      resources:
        requests:
          cpu: 100m
          memory: 50Mi
        limits:
          cpu: 200m
          memory: 1Gi

    filesStorage:
      resources:
        requests:
          cpu: 500m
          memory: 2Gi
        limits:
          cpu: 8000m
          memory: 4Gi

    ingress:
      enabled: true
      className: nginx
      hosts:
        - host: search-api-v8.example.com
          paths:
            - path: /
              pathType: Prefix

    customCAs:
      bundle: ''
      # bundle: |
      #   -----BEGIN CERTIFICATE-----
      #   ...
      #   -----END CERTIFICATE-----
      certsPath: ''

    Where:

    • dgctlDockerRegistry: endpoint of your Docker Registry with the images of the 2GIS On-Premise services in the HOST:PORT format.

    • dgctlStorage: settings for access to the installation artifacts storage.

      • host: endpoint of the S3 compatible installation artifacts storage in the HOST:PORT format.
      • secure: whether to use HTTPS for interacting with the S3 compatible storage. Default value: false.
      • bucket: S3 bucket name.
      • accessKey: access key for accessing the S3 bucket.
      • secretKey: secret key for accessing the S3 bucket.
      • manifest: path to the manifest file in the manifests/search-api-v8/1640661259.json format. This file contains the description of pieces of data that the service requires to operate. See Installation artifacts lifecycle.
      • region: S3 compatible storage region.

    • s3Client: settings for accessing the indexing storage.

      • enabled: enable S3 support for the indexing system.
      • endpoint: endpoint of the S3 compatible storage for indexing.
      • accessKeyId: storage access key ID.
      • secretAccessKey: storage secret access key.
      • bucket: bucket name for storing indexes.
      • region: region of the S3 compatible storage.
      • useSsl: whether to use SSL. Default value: false.

    • worker.resources: computational resources settings for the Worker service. For recommended values, see Computational resources.

    • head.head.resources: computational resources settings for the primary container of the Head service. For recommended values, see Computational resources.

    • head.nginx.resources: computational resources settings for nginx in the Head pod. For recommended values, see Computational resources.

    • head.envoy.resources: computational resources settings for Envoy in the Head pod. For recommended values, see Computational resources.

    • ingressRouter.enabled: enable the IngressRouter component - entry point for external requests.

    • ingressRouter.resources: computational resources settings for the IngressRouter service. For recommended values, see Computational resources.

    • geodetector.enabled: enable the Geodetector component - geolocation detection service.

    • geodetector.resources: computational resources settings for the Geodetector service. For recommended values, see Computational resources.

    • indexer.resources: computational resources settings for the Indexer service. For recommended values, see Computational resources.

    • controller.resources: computational resources settings for the Controller service. For recommended values, see Computational resources.

    • filesStorage.resources: computational resources settings for the FilesStorage service. For recommended values, see Computational resources.

    • ingress: configuration of the Ingress resource. The URL specified in the ingress.hosts[0].host parameter must be accessible from outside your Kubernetes cluster.

    • customCAs: custom certificates settings.

      • bundle: text representation of a certificate in the X.509 PEM public-key format.
      • certsPath: bundle mount directory in the container.

  2. Deploy the service with Helm using the created values-search-api-v8.yaml configuration file.

    helm upgrade --install --version=VERSION --atomic --wait --timeout 7200s --values ./values-search-api-v8.yaml search-api-v8 2gis-on-premise/search-api-v8

    In the --version parameter, specify the required API Platform version. For a list of versions, see the API Platform releases section.

Install Catalog APIs service​

  1. Create a Helm configuration file. See here for more details on the available settings.

    Note:

    You can configure the process of importing new data for Catalog APIs. Use the importer settings group of the configuration file (see below).

    The example is prefilled with the necessary data collected on previous steps.

    values-catalog.yaml
    dgctlDockerRegistry: docker.registry.example.com

    imagePullSecrets: [onpremise-registry-creds]

    dgctlStorage:
      host: artifacts.example.com
      secure: true
      bucket: onpremise-artifacts
      accessKey: AKIAIOSFODNN7EXAMPLE
      secretKey: wJalrXUtnFEMIK7MDENGbPxRfiCYEXAMPLEKEY
      manifest: manifests/catalog-api/1640661259.json
      region: ''
      verifySsl: true

    api:
      postgres:
        host: catalog-postgresql.storage.example.local
        port: 5432
        name: onpremise_catalog
        username: dbuser_catalog
        password: 650D7AmZjSR1dkNa
      ingress:
        enabled: true
        className: nginx
        hosts:
          - host: catalog-api.example.com
            paths:
              - path: /
                pathType: Prefix
        tls: []
        # - hosts:
        #   - catalog-api.example.com
        #   secretName: secret.tls

    search:
      url: http://search-api-v8-ingress-router:9092

    keys:
      url: http://keys-service-api
      token: CATALOG_APIS_TOKEN

    stat:
      url: 'http://stat-receiver'
      enabled: false
      request:
        enabled: false
      search:
        enabled: false

    importer:
      postgres:
        host: catalog-postgresql.storage.example.local
        port: 5432
        name: onpremise_catalog
        username: dbuser_catalog
        password: 650D7AmZjSR1dkNa
        schemaSwitchEnabled: true
      cleaner:
        enabled: true
        versionLimit: 2

    license:
      url: 'https://license'
      requestTimeout: 1s

    customCAs:
      bundle: ''
      # bundle: |
      #   -----BEGIN CERTIFICATE-----
      #   ...
      #   -----END CERTIFICATE-----
      certsPath: ''

    Where:

    • dgctlDockerRegistry: endpoint of your Docker Registry with the images of the 2GIS On-Premise services in the HOST:PORT format.

    • imagePullSecrets: Kubernetes Secrets to access the Docker Registry with the images of the 2GIS On-Premise services.

    • dgctlStorage: settings for access to the installation artifacts storage.

      • host: endpoint of the S3 compatible installation artifacts storage in the HOST:PORT format.
      • secure: whether to use HTTPS for interacting with the S3 compatible storage. Default value: false.
      • bucket: S3 bucket name.
      • accessKey: access key for accessing the S3 bucket.
      • secretKey: secret key for accessing the S3 bucket.
      • manifest: path to the manifest file in the manifests/catalog-api/1640661259.json format. This file contains the description of pieces of data that the service requires to operate. See Installation artifacts lifecycle.
      • region: S3 compatible storage region.
      • verifySsl: whether to enable validation of SSL certificates when connecting to dgctlStorage.host via HTTPS. Default value: true.

    • api.postgres: the PostgreSQL database access settings. Use the values you configured in the PostgreSQL on the previous step.

      • host: host name of the server.
      • port: port of the server.
      • name: database name.
      • username: user name.
      • password: user password.

    • api.ingress: configuration of the Ingress resource. Adapt it to your Ingress installation. The URL specified in the ingress.hosts.host parameter should be accessible from the outside of your Kubernetes cluster, so that users in the private network can browse the URL.

    • search: the Search API v8 access settings.

      • url: URL of the service. This URL must be accessible from all the pods within your Kubernetes cluster.

    • keys: the API Keys service settings.

      • url: URL of the service. This URL must be accessible from all the pods within your Kubernetes cluster.
      • token: service token for sharing usage statistics with the API Keys service (see Installing API Keys service).

    • stat: settings for interaction with the Statistics Collection service.

      • url: URL of the Statistics Collection service.
      • enabled: whether to send key usage statistics.
      • request.enabled: whether to send API request statistics.
      • search.enabled: whether to send search query data.

    • importer: settings of the data import (Kubernetes Importer job). New datasets are imported only if no imports have been done for the given manifest before.

      • postgres: settings of the PostgreSQL server access to import new data for objects.

        • host: host name of the server.
        • port: port of the server.
        • name: database name.
        • username: user name.
        • password: user password.
        • schemaSwitchEnabled: enable/disable working with schemas.
          • true: data is imported to a new schema every time. Switching to older schemas and cleaning them is available.
          • false: creation of new schemas and cleaning the data base is done manually by the administrator.

        For details, see the Update Catalog APIs service section.

      • cleaner: settings of automatically removing old data sets.

        • enabled: automatic removal of old data sets is enabled. For details, see the Update Catalog APIs service section.
        • versionLimit: number of old data sets to keep.

        For additional information about the import process, see Installation artifacts lifecycle.

    • license: the License service settings.

      • url: License service URL address. Example: https://license.
      • requestTimeout: timeout of requests to the License service.

    • customCAs: custom certificates settings.

      • bundle: text representation of a certificate in the X.509 PEM public-key format.
      • certsPath: bundle mount directory in the container.

  2. Deploy the service with Helm using the created values-catalog.yaml configuration file.

    helm upgrade --install --version=VERSION --atomic --wait --timeout 7200s --values ./values-catalog.yaml catalog-api 2gis-on-premise/catalog-api

    In the --version parameter, specify the required API Platform version. For a list of versions, see the API Platform releases section.

    If the configuration file contains importer settings, the command checks if the data already exists in the database and if needed, imports it into PostgreSQL. After that, Helm installs the service.

4. Test deployment​

To test that the search services are working:

  1. Wait until all the pods are in the Ready state:

    kubectl get pod -n <namespace>

  2. Send a GET request:

    curl catalog-api.example.com/3.0/items/geocode?key=API_KEY&q=City

    Where:

    • API_KEY - created key
    • City - name of the city you want to search the information for

If the installation is successful, the service will return a list of search results in JSON format.

What's next?​