Chef Backend to Automate HA

[edit on GitHub]

This page explains the procedure to migrate the existing Chef Backend data to the newly deployed Chef Automate HA. This migration involves two steps:

• Back up the data from an existing Chef Backend via knife-ec-backup.
• Restore the backed-up data to the newly deployed Chef Automate HA environment via knife-ec-restore.

Take backup using the knife-ec-backup utility and move the backup folder to the newly deployed Chef Server. Later, restore using the same utility. The backup migrates all the cookbooks, users, data-bags, policies, and organizations.

knife-ec-backup utility backups and restores the data in an Enterprise Chef Server installation, preserving the data in an intermediate, editable text format. It is similar to the knife download and knife upload commands and uses the same underlying libraries. It includes workarounds for unsupported objects by the tools and various Server API deficiencies. The goal is to improve knife download, knife upload, and the Chef Infra Server API to deprecate the tool.

Backup the Existing Chef Backend Data

1. Execute below command to install Habitat:

curl https://raw.githubusercontent.com/habitat-sh/habitat/master/components/hab/install.sh \ | sudo bash

2. Execute the below command to install the habitat package for knife-ec-backup

    hab pkg install chef/knife-ec-backup

3. Execute the below command to generate a knife tidy server report to examine the stale node, data etc. In this command:

• pivotal is the name of the user
• path of pivotal is the path where the user’s pem file is stored.
• node-threshold NUM_DAYS is the maximum number of days since the last checking before a node is considered stale.

    hab pkg exec chef/knife-ec-backup knife tidy server report --node-threshold 60 -s <chef server URL> -u <pivotal> -k <path of pivotal>

For Example:

    hab pkg exec chef/knife-ec-backup knife tidy server report --node-threshold 60 -s https://chef.io -u pivotal -k /etc/opscode/pivotal.pem

4. Execute the below command to initiate a backup of your Chef Server data. In this command:

• with-user-sql is required to handle user passwords and ensure user-specific association groups that are not duplicate.

• --with-key-sql is to handle cases where customers have users with multiple pem keys associated with their user or clients. The current chef-server API only dumps the default key. Sometimes, users will generate and assign additional keys to give additional users access to an account but still be able to lock them out later without removing everyone’s access.

    hab pkg exec chef/knife-ec-backup knife ec backup backup_$(date '+%Y%m%d%H%M%s') --webui-key /etc/opscode/webui_priv.pem -s <chef server

For example:

    hab pkg exec chef/knife-ec-backup knife ec backup backup_$(date '+%Y%m%d%H%M%s') --webui-key /etc/opscode/webui_priv.pem -s https://chef.io`.

• Execute the below command to clean unused data from reports. This is an optional steps

    hab pkg exec chef/knife-ec-backup knife tidy server clean --backup-path /path/to/an-ec-backup

5. Execute the below command to copy the backup directory to the Automate HA Chef Server.

     scp -i /path/to/key backup\_$(date '+%Y%m%d%H%M%s') user@host:/home/user

Restore Backed Up Data to Chef Automate HA

• Execute the below command to install the habitat package for knife-ec-backup

    hab pkg install chef/knife-ec-backup

• Execute the below command to restore the backup.

    hab pkg exec chef/knife-ec-backup knife ec restore /home/centos/backup\_2021061013191623331154 -yes --concurrency 1 --webui-key /hab/svc/automate-cs-oc-erchef/data/webui\_priv.pem --purge -c /hab/pkgs/chef/chef-server-ctl/*/*/omnibus-ctl/spec/fixtures/pivotal.rb

In place Migration (Chef Backend to Automate HA)

As part of this scenario, customer will migrate from chef-backend (5 machines) to Automate HA in-place, i.e. Automate HA will be deployed in those 5 machines only where Chef-backend is running. One extra bastion node will be required which will be managing the deployment of Automate HA on the chef backend infrastructure.

1. ssh to all the backend nodes of chef-backend and run

    chef-backend-ctl stop

2. ssh to all frontend nodes of chef-backend and run

    chef-server-ctl stop

3. Create one bastion machine under the same network space.

4. ssh to bastion machine and download chef-automate cli

https://packages.chef.io/files/current/latest/chef-automate-cli/chef-automate_linux_amd64.zip

5. Extract downloaded zip file

6. Create a airgap bundle using command

    ./chef-automate airgap bundle create 

7. Geneate config.toml file using command

    ./chef-automate init-config-ha existing_infra 

8. Edit config.toml and add the following:

• Update the intance_count
• fqdn : load balance url, which points to frondend node.
• keys : ssh username and private keys
• Make sure to provide Chef backend’s frontend server IPs for Automate HA Chef Automate and Chef Server.
• Make sure to provide Chef backend’s backend server IPs for Automate HA Postgres and OpenSearch machines.
• Sample configuration, please modify according to your needs.

[architecture.existing_infra]
secrets_key_file = "/hab/a2_deploy_workspace/secrets.key"
secrets_store_file = "/hab/a2_deploy_workspace/secrets.json"
architecture = "existing_nodes"
workspace_path = "/hab/a2_deploy_workspace"
ssh_user = "myusername"
ssh_port = "22"
ssh_key_file = "~/.ssh/mykey.pem"
sudo_password = ""

# DON'T MODIFY THE BELOW LINE (backup_mount)
backup_mount = "/mnt/automate_backups"

[automate.config]
# admin_password = ""
# automate load balancer fqdn IP or path
fqdn = "chef.example.com"
instance_count = "2"
# teams_port = ""
config_file = "configs/automate.toml"

[chef_server.config]
instance_count = "2"

[opensearch.config]
instance_count = "3"

[postgresql.config]
instance_count = "3"

[existing_infra.config]
automate_private_ips = ["10.0.1.0","10.0.2.0"]
chef_server_private_ips = ["10.0.1.0","10.0.2.0"]
opensearch_private_ips = ["10.0.3.0","10.0.4.0","10.0.5.0"]
postgresql_private_ips = ["10.0.3.0","10.0.4.0","10.0.5.0"]

9. Deploy using

./chef-automate deploy config.toml <airgapped bundle name>

10. Clean up the old packages from chef-backend (like Elasticsearch and postgres)

11. Doc: Restore Backed Up Data to Chef Automate HA

Using Autoamte HA for Chef-Backend user

1. Download and Install chef-workstation From Bashtion machine or local machine install chef-workstation https://www.chef.io/downloads/tools/workstation

2. Set up workstation

   chef generate repo chef-repo 
   

   cd  chef-repo 
   

   knife configure 
   

   provide chef-server FQDN of Automate HA Chef-Server

   Example :- https://demo-chef-server.com/organizations/demo-org

       knife configure
       Please enter the chef server URL: [https://ip-10-1-0-52.ap-southeast-1.compute.internal/organizations/myorg] https://demo-chef-server.com/organizations/demo-org
       Please enter an existing username or clientname for the API: [ubuntu] org-user
       *****
   
       You must place your client key in:
           /root/.chef/org-user.pem
       Before running commands with Knife
   
       *****
       Knife configuration file written to /root/.chef/credentials
   

3. Ssl fetch command

   knife ssl fetch 
   

4. Ssl check command

   knife ssl check 
   

5. Create role command

   knife role create abc --disable-editing 
   

6. Download cookbook

   knife supermarket download line 
   knife supermarket download apache2
   

7. Upload cookbook

   knife cookbook upload line --cookbook-path cookbooks 
   

8. Bootstrap as node

       knife bootstrap <public_ip_node>  -i <key_to_connect_node> -U ubuntu -N test1 --sudo 
   

   Example :-

       knife bootstrap 15.207.98.155 -i  ~/.ssh/ssh-key.pem -U ubuntu -N node1 --sudo 
       knife bootstrap 3.110.103.65 -i ~/.ssh/ssh-key.pem -U ubuntu -N node2 --sudo