OpenSearch managed with Terraform

AWS OpenSearch Service emerges as a powerful solution to meet effective search and analysis of vast amounts of information needs of organizations across various industries.

This article not only delves into the intricacies of Amazon OpenSearch Service but also offers a practical guide to deploying and managing OpenSearch clusters using Terraform. We’ll navigate through the complexities of infrastructure provisioning, security configurations, and user management, all while adhering to best practices and maximizing efficiency.

Whether you are a DevOps engineer looking to automate your OpenSearch deployments or a newcomer, this article offers a comprehensive roadmap to harness the power of OpenSearch in the most efficient way. We’ll explore Terraform code snippets, explain essential configurations, and guide you through the entire deployment process. By the end, you’ll have a clear understanding of how Terraform can simplify and enhance your experience with AWS OpenSearch Service, enabling you to take full control of your search and analytics infrastructure.

What is OpenSearch?

Amazon OpenSearch Service is a versatile platform that simplifies the deployment, operation, and scalability of OpenSearch clusters in the AWS Cloud. OpenSearch offers robust support for legacy Elasticsearch OSS versions up to 7.10, marking the final open-source release of the software. One of the standout features of OpenSearch Service is its flexibility, allowing users to choose between different search engines when creating clusters.

Why Use OpenSearch?

Here’s why OpenSearch is becoming an attractive choice for log analytics, real-time application monitoring and clickstream analysis:

1. Managed vs. Serverless: OpenSearch Service offers both managed and serverless options, each with its own set of features. Managed domains are pre-provisioned clusters that provide granular control over node types and capacity management. In contrast, serverless collections offer automatic scaling and provisioning based on capacity usage, simplifying resource management and cost control.
2. Billing: OpenSearch Service employs an EC2 instance-based billing model, where users pay for compute hours and cumulative storage size. On the other hand, OpenSearch Serverless employs an OCU-hour pricing model, factoring in compute, search/query operations, and S3 storage.
3. Encryption: While encryption at rest is optional for domains, it’s a mandatory feature for collections in OpenSearch Serverless. This ensures that data remains secure in any scenario.
4. Data access control: OpenSearch Service relies on IAM policies for data access control within domains, providing fine-grained control over who can access what. you can allow users to access collections and indexes regardless of their access mechanism or network source. You can provide access to IAM roles and SAML identities.
5. Supported operations: Both OpenSearch Service and OpenSearch Serverless support specific subsets of OpenSearch API operations and cater to different use cases and needs.
6. Dashboards sign-in: Sign-in procedures differ between the two options, with OpenSearch Service requiring a username and password, while OpenSearch Serverless grants automatic login when accessing the AWS console.
7. APIs: Developers can interact programmatically with OpenSearch Service and OpenSearch Serverless using distinct APIs tailored to their chosen deployment option.
8. Network access: Network settings vary between the two, with OpenSearch Service tightly coupling network access for domain and OpenSearch Dashboards endpoints, while OpenSearch Serverless offers more flexibility in configuring network access.
9. Signing requests: OpenSearch Service and OpenSearch Serverless use different client libraries and service names when signing requests, with specific requirements for each.
10. Version upgrades: The responsibility for upgrading to newer OpenSearch versions differs between managed and serverless options, with implications for maintenance and compatibility.
11. Service software updates: Updates to service software are handled differently, with OpenSearch Serverless automatically incorporating bug fixes, features, and performance improvements.
12. VPC Access: Users can configure VPC access in OpenSearch Service, while OpenSearch Serverless offers its own VPC-related options.
13. SAML Authentication: OpenSearch Service enables SAML authentication on a per-domain basis, whereas OpenSearch Serverless employs a different approach at the account level.

OpenSearch managed (domain) vs. serverless:

There are some features that are different from each other.

The difference between OpenSearch managed, and OpenSearch serverless below shows the full picture to help you choose your preferred choice for your needs.

Questions to ask ourselves before we decide which service is most suitable for us:

• Managed or Serverless? Do we want to manage our cluster or use it as serverless? Are we getting more value versus the price difference?
• Managed OpenSearch Users + Access Control :
  • Do we want to manage users inside OpenSearch own database or outside?
  • If we want outside the cluster how do we manage it?
    • IAM users (but you won’t be able to connect to the dashboard if we use OpenSearch domain, we can use it for the CLI access (fluent bit).
    • Cognito, now the question with cognito is how to manage it.
    • We can use SAML auth, Google auth, Facebook auth etc.

Access control 

Fine-grained access control:

Fine-grained access control offers additional ways of controlling access to your data on Amazon OpenSearch Service. For example, depending on who makes the request, you might want a search to return results from only one index. You might want to hide certain fields in your documents or exclude certain documents altogether.

You can read more at: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html

• Cognito for OpenSearch: (We will import it in the Terraform section, you can create it in Terraform if you need to).

This Section is for users only you can configure other Federated identity providers like Facebook, Google, Amazon, Apple and etc.

Before configuring cognito for OpenSearch we need a few requirements.

• Amazon Cognito user pool
• Amazon Cognito identity pool
• IAM role that has the AmazonOpenSearchServiceCognitoAccess policy attached (CognitoAccessForAmazonOpenSearch)

Deploying OpenSearch with Terraform:

While the power of OpenSearch is undeniable, the question arises: how can we streamline its deployment and management within the AWS Cloud? Enter Terraform, an Infrastructure as Code tool that let you define and provision resources in a declarative manner.

Users Setup:

We will create 2 users, 1 master user with admin permissions, 1 limited user with limited permissions.

For that, we need to create an IAM Role.

For the master user, we will use full permission.

For the limited user, we will give him only Read access.

When creating a role, we want to change the “Trust relationship” to:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "cognito-identity.amazonaws.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "cognito-identity.amazonaws.com:aud": "<identity-pool-id>"
      },
      "ForAnyValue:StringLike": {
        "cognito-identity.amazonaws.com:amr": "authenticated"
      }
    }
  }]
}

and for limited user :

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "es:ESHttpGet",
            "Resource": "*"
        }
    ]
}

Now we are going to create two groups and a user for each group inside the Cognito user pool that we created.

Click on groups inside the user pool we created. And create two groups.

admin: in this group, those users with admin permissions for OpenSearch Dashboards will live, please select IAM Role IAMMAdminUserRole.

limited: in this group, those users with limited permissions for OpenSearch Dashboards will live, please select IAM Role IAMLimitedUserRole.

Terraform :

* This is terraform configured for Public access. – If you want private access you will need to configure VPC (see main.tf vpc_options down below and to access it vpc tunnel).

variable.tf

variable "master-iam-user" {
  default = "rotem"
}

variable "region" {
  default = "eu-central-1"
}

variable "domain" {
  default = "opensearch-demo"
}

variable "url" {
  default = "example.com"
}

variable "engine_version" {
  default = "OpenSearch_2.7"
}


variable "cognito_user_pool_name" {
  default = "opensearch-demo"
}
variable "cognito_identity_pool_id" {
  default = "eu-central-1:*********************"
}

main.tf

locals {
  name     = "opensearch-demo"

  # To ensure the name is consistent between what was created and the user data script
  tags = {
    name       = local.name
    Made_by    = "terraform"
    Created_by = "rotem"
  }
}

data "aws_caller_identity" "current" {}

data "aws_acm_certificate" "com" {
  domain      = "*.${var.url}" # *.example.com
  types       = ["AMAZON_ISSUED"]
  most_recent = true
}


data "aws_cognito_user_pools" "selected" {
  name = var.cognito_user_pool_name
}

resource "aws_iam_service_linked_role" "example" {
  aws_service_name = "opensearchservice.amazonaws.com"
}

data "aws_region" "current" {}

data "aws_caller_identity" "current" {}

resource "aws_opensearch_domain" "example" {
  domain_name    = "local.name"
  engine_version = "${var.engine_version}"

  domain_endpoint_options {
    custom_endpoint_enabled = true #if we want custom endpoint.
    custom_endpoint         = "opensearch-custom.${var.url}"
    custom_endpoint_certificate_arn = data.aws_acm_certificate.com.arn
    tls_security_policy             = "Policy-Min-TLS-1-2-2019-07"
    enforce_https                   = true
  }

  cluster_config {
    instance_type          = "t3.medium.search" #t3 is not supported by auto-tune.
    zone_awareness_enabled = true
    zone_awareness_config {
      availability_zone_count = 2
    }
    instance_count = 2
    warm_enabled   = false

    dedicated_master_enabled = false

  }
	# IF YOU WANT VPC access.
  # vpc_options {
  #   subnet_ids = [
  #     module.vpc.public_subnets[0],
  #     module.vpc.public_subnets[1],
  #   ]
  #   security_group_ids = [aws_security_group.example.id]
  # }

  advanced_security_options {
    enabled                        = false
    anonymous_auth_enabled         = true
    internal_user_database_enabled = true
    master_user_options {
      master_user_name     = "admin"
      master_user_password = random_password.password.result
    }

  }
  cognito_options {
    enabled          = true
    identity_pool_id = var.cognito_identity_pool_id
    role_arn         = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/service-role/CognitoAccessForAmazonOpenSearch"
    user_pool_id     = data.aws_cognito_user_pools.selected.ids[0]

  }



  ebs_options {
    ebs_enabled = true
    volume_size = 100
    volume_type = "gp3"
  }

  encrypt_at_rest {
    enabled = true
  }

  node_to_node_encryption {
    enabled = true
  }

  #  off_peak_window_options {
  #    enabled = true
  #    off_peak_window {
  #      window_start_time {
  #        hours   = 07
  #        minutes = 00
  #      }
  #    }
  #  }

  snapshot_options {
    automated_snapshot_start_hour = 07 #10:00 israel time
  }

  tags = local.tags
  depends_on = [aws_iam_service_linked_role.example]
}

resource "aws_opensearch_domain_policy" "opensearch" {
  domain_name = aws_opensearch_domain.example.domain_name

  access_policies = <<POLICIES
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/service-role/opensearch-demo"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-central-1:${data.aws_caller_identity.current.account_id}:domain/opensearch-demo/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": ["arn:aws:iam::021453066858:user/fluent-bit", "arn:aws:iam::021453066858:user/<YOUR MASTER USER>""]
      },
      "Action": [
        "es:ESHttpPut",
        "es:ESHttpPost",
        "es:ESHttpPatch"
      ],
      "Resource": "arn:aws:es:eu-central-1:021453066858:domain/opensearch-demo/*"
    }
  ]
}
POLICIES
}

providers.tf

provider "aws" {
  region = "eu-central-1"
}-

terraform {
  required_version = ">= 1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.47"
    }
  }
 }

 # Update to suit your needs
 # backend "s3" {
 #   bucket = "open-search-demo-tfstate"
 #   region = "eu-central-1"
 #   key    = "state/terraform.tfstate"
 # }
}

Now :  terraform init > terraform plan > terraform apply

Keep in mind that OpenSearch might take some time to fully deploy, so why not take a break, and enjoy a cup of coffee?

Conclusion:

As you embark on this journey, remember that OpenSearch’s versatility and Terraform’s declarative approach enable you to create, configure, and maintain your search and analytics infrastructure efficiently. Whether you opt for managed domains or serverless collections, fine-tune access control, or leverage federated identity providers like Cognito, you have the tools at your disposal to build a robust and secure solution tailored to your needs.

In conclusion, by combining the capabilities of Amazon OpenSearch Service and Terraform, you’re not just managing technology; you’re empowering your organization with the data-driven insights and performance it needs to thrive in today’s data-centric world.