Install a small Elasticsearch development setup on AWS EC2

One of our developer teams is tasked to "renovate" pretty much the entire core business for one of our clients. In the process, we used Elasticsearch, you know, for search!

Our client has a cluster for pre-production/test and production.

For our development team, we started with a virtual machine hosted on premises. But we are in the process of moving all our services to the Cloud. We are telling our clients to move to the Cloud. We have a Cloud first approach for new projects. We want to version and control our infrastructure. It was time to move this service to the Cloud as well.

We have a few client and internal projects using the Amazon Elasticsearch Service: a managed service that makes it easy to deploy, operate, and scale Elasticsearch clusters in the AWS Cloud. It's super fast to set up, highly available, backup is automated, it can be encrypted, etc ... There are a lot of reasons to go for it. And this is why we use it for production and development. But, there is a but.

There is a specific list of versions that are available. At the time of writing, the highest supported version is 6.7, which is really good. The only problem is our client is going to upgrade to 7.2.0 (the latest and greatest). So the managed service is not going to fit the bill.

No problem, we will build a dev instance. We need:

• VPC, subnet, security group, key pair, role, EC2 instance
• Cloudformation: to code the AWS infrastructure needed.
• Ansible: to set up Elasticsearch on the EC2 instance
• Data can be re-generated/uploaded quickly and easily:
  • It does not need to be highly available
  • It does not need to be a multi node cluster
• Performance requirements are very low
  • a couple devs might make concurrent requests

We decided to use an existing VPC and subnet. The instance accessibility will be IP restricted. It will have a dedicated key pair. We will need an instance role that allows us to forward the Elasticsearch logs to Cloudwatch for easy log viewing from the AWS console. And finally a t2.medium instance will be enough to get started. If performance is an issue, we will simply change the instance size. We already have a wildcard TLS certificate for our domain. We're all set.

Cloudformation is going to help us declare the desired state of our Cloud infrastructure.

Ansible is going to help us declare the desired state of the software and configuration on the instance.

This infrastructure code will be versioned in the git repository of the project. In case we run Cloudformation or Ansible against our existing infrastructure, we want updates to be applied. In case there is no change, we want nothing to happen. Idempotency is an important concept and we will have to pay attention to structure our statements to satisfy this requirement, specially in Ansible.

If you want to try it out, you can find all the bits and pieces on github. The readme will guide you through the few commands needed to set it all up.