OpenSearch - Deploying A Change

• First thing you rollout is search service with the existing sirius front end, from the users perspective nothing will really change. It will roll out a new index with the configuration you’ve passed to it. The existing elastic search alias will still point to the old index and configuration.
  
  When your search service has built on jenkins deploy it to adhoc/ pre-production/ production environment.
  Once this elastic search change goes in, changes that start being populated go into the new index and it will also be pushed into the old index.
  
• When the search service change is deployed, check on Cloud 9 that the new elastic search index is there. You can do this with:
  curl https://$SEARCH/_cat/indices to check what indexes are there.
  
• To find out more detail about the index in question you can run
  curl https://$SEARCH/indexname
  
• Then manually trigger the elastic search rebuild, this takes 4.5 hours on production, it is safe to run in the business day. This will fully populate the new elastic search index with 40 million records in Sirius.
  
• Run ECS task: reindex-elasticsearch
  Info on running a task on AWS is found in this runbook: Run Tasks in AWS
  
  To check whether this has worked, after 4-5 hours run the following command to see whether a new indicies is there:
  curl https://$SEARCH/_cat/indices
  
  This will look as follows:
  When the numbers of documents are similar or greater than the original index, the job is probably complete. In the above image there are 45805097 documents.
  IF the existing frontend works with the new alias you can move onto the next step.
  
• You can also check the log file (CloudWatch -> Log groups -> your environment -> reindex-elasticsearch/reindex-elasticsearch) for any errors.
  
• Run ECS task: update elasticsearch alias
  Find more about this in the run AWS tasks repo
  
• Test on environment, when you’re happy it works as expected deploy the Sirius changes
  
• After 2 or 3 days (when we’re sure the new index is working fine) delete the old index. You need to do this in all environments, adhoc pre-production and production.

To delete the elastic search index

• Check what indices already exist:
  curl https://$SEARCH/_cat/indices
  
• Check which indices the elastic search alias is already pointing at (here it’s at the person indices ending 71 meaning we can delete 8f)
  
• Run the delete curl command
  curl -X DELETE https://$SEARCH/*indexname*.
  
  Be careful to make sure it’s pointing at the one you want to delete. Elastic search will allow you to delete the index you’re currently using/pointing at which will break a lot of things.
  
• Check what indexes exist again to check it was removed using:
  curl https://$SEARCH/_cat/indices
  There should be one less than before.
  
  
• If the existing front end won’t work with the new elastic search changes then don’t follow the above repointing steps.
  Instead deploy the elastic search changes and reindex as above.
  Out of hours deploy the sirius front end changes and repoint the alias.

This page was last reviewed on 16 April 2024. It needs to be reviewed again on 16 October 2024 by the page owner #opg-sirius-develop .

This page was set to be reviewed before 16 October 2024 by the page owner #opg-sirius-develop. This might mean the content is out of date.