Upgrade Chef Infra Server

[edit on GitHub]

Each new release of Chef Infra Server improves reliability and updates 3rd party components to ensure the security of the server. It is important to keep Chef Infra Server up to date to ensure the secure and reliable operation of Chef Infra in your organization.

Upgrade Matrix

If running a Chef Infra Server 12.17.15 or later you can upgrade directly to the latest releases of Chef Infra Server 14. If you are running a release before 12.17.15 you must perform a stepped upgrade as outlined below.

Requires License

Chef Infra Server 13 and later are governed by the Chef EULA. You are required to accept these terms when using Chef Infra Server for the first time by entering Yes when prompted.

Supported Release

Chef Infra Server 14 and later are supported Chef Software releases. Earlier releases are not supported. For more information about supported Chef Software see the Supported Versions documentation.

Release-Specific Steps

Upgrading to 15.x

Chef Infra Server 15.0 moved from Elasticsearch to OpenSearch as its search index.

The Chef Infra Server 14 upgrade process requires downtime for stopping the server, installing the new package, and then upgrading the server, which will include an automatic Elasticsearch reindexing operation for existing Solr users. We estimate the reindexing operation will take 2 minutes for each 1000 nodes, but the it could take more time, depending on your server hardware and the complexity of your Chef data.

The Chef Infra Server 15 upgrade does not automatically reindex existing external Elasticsearch installations.

The upgrade duration might take more time if you are upgrading from Chef Infra Server 12.x/13.x, as it automatically reindexes your database.

Upgrading to 14.x

Chef Infra Server 14.0 moved from Solr to Elasticsearch as its search index. The Chef Infra Server 14 upgrade process requires downtime for stopping the server, installing the new package, and then upgrading the server, which will include an automatic Elasticsearch reindexing operation for existing Solr users. We estimate the reindexing operation will take 2 minutes for each 1000 nodes, but the it could take more time, depending on your server hardware and the complexity of your Chef data.

The Chef Infra Server 14 upgrade does not automatically reindex existing external Elasticsearch installations.

Upgrading to 14.14

Chef Infra Server 14.14 supports external OpenSearch for indexing. Please follow the migration section below to migrate from Elasticsearch to external OpenSearch.

Steps To Enable External OpenSearch

1. Set the elasticsearch['enable'] attribute to false.
2. Set the opensearch['external'] attribute to true.
3. Set the opensearch['external_url'] attribute to the external OpenSearch URL.
4. Set the opscode_erchef['search_queue_mode'] attribute to batch.
5. Set the opscode_erchef['search_provider'] attribute to opensearch.
6. Set the opscode_erchef['search_auth_username'] attribute to OpenSearch username.
7. Set the opscode_erchef['search_auth_password'] attribute to OpenSearch password.

For example:

elasticsearch['enable'] = false
opscode_erchef['search_queue_mode'] = 'batch'
opscode_erchef['search_provider'] = 'opensearch'
opensearch['external'] = true
opensearch['external_url'] = "http://127.0.0.1:9200"
opscode_erchef['search_auth_username'] = "OPEN_SEARCH_USER"
opscode_erchef['search_auth_password'] = "OPEN_SEARCH_PWD"

Steps To Migrate from Elasticsearch to External OpenSearch

There are two ways to migrate from Elasticsearch to external OpenSearch: migrating your data, or reindexing and reconfiguring your database.

We recommend migrating your data over reindexing and reconfiguring.

Migrate Data

Copy or move your Elasticsearch OSS data and logs directories to the newly installed OpenSearch paths. See OpenSearch’s documentation on upgrading to OpenSearch.

Reindex and Reconfigure

Reindex and reconfigure your database after upgrading to Chef Infra Server 14.13. The duration of this operation will vary depending on your server hardware and the number of node objects on your Chef Infra Server.

Use the Chef Infra Server command-line tool to reindex and reconfigure your database:

chef-server-ctl reindex
chef-server-ctl reconfigure

Upgrading to 14.8

Chef Infra Server 14.8 upgrades PostgreSQL from 9.6 to 13.3. The 14.8 upgrade process requires a one-time downtime to vacuum, upgrade, and re-index the database. The entire upgrade operation takes about one minute for each 1000 nodes (1000 nodes is approximately 286MB). This process may take longer depending on your server hardware and the size of the node objects on your Chef Infra Server.

Upgrading to 14.16

Chef Infra Server 14.16 includes a bug fix for the bifrost database. This bug may create unused authorization IDs in the bifrost database. After upgrading to Chef Infra Server 14.16, the unused authorization IDs must be manually deleted.

To analyze the data that gets deleted and get an estimate of the time needed to delete the data, run:

chef-server-ctl cleanup-bifrost --estimate-only

To delete the unused authorization IDs from the bifrost database, run:

chef-server-ctl cleanup-bifrost

Database Preparation

1. Run VACUUM FULL on the PostgreSQL database if you don’t have automatic vacuuming set up. This process will reduce the size of the database by deleting unnecessary data and speeds up the migration. The VACUUM FULL operation takes around 1 to 2 minutes per gigabyte of data depending on the complexity of the data, and requires free disk space at least as large as the size of your database.

      sudo su - opscode-pgsql
      /opt/opscode/embedded/bin/vacuumdb --all --full
   

   You should then see output like:

      vacuumdb: vacuuming database "bifrost"
      vacuumdb: vacuuming database "oc_id"
      vacuumdb: vacuuming database "opscode-pgsql"
      vacuumdb: vacuuming database "opscode_chef"
      vacuumdb: vacuuming database "postgres"
      vacuumdb: vacuuming database "template1"
   

2. Run ANALYZE on the PostgreSQL database. This process gathers statistics for the query planner to create the most efficient query execution paths. Accurate statistics help the planner choose the most appropriate query plan, and thereby improve the speed of query processing.

   sudo su - opscode-pgsql
   /opt/opscode/embedded/bin/vacuumdb --all --analyze-only
   

   You should then see output like:

   vacuumdb: vacuuming database "bifrost"
   vacuumdb: vacuuming database "oc_id"
   vacuumdb: vacuuming database "opscode-pgsql"
   vacuumdb: vacuuming database "opscode_chef"
   vacuumdb: vacuuming database "postgres"
   vacuumdb: vacuuming database "template1"
   

3. Back up the PostgreSQL database before upgrading so you can restore the full database to a previous release in the event of a failure. See Backup and Restore for more information.

Upgrade Steps

Follow the Chef Infra Server upgrade instructions below.

Upgrading to 12.17.15

As of version 12.14, Chef Infra Server renders passwords inside of the /etc/opscode directory by default. If you are using Chef Infra Server without add-ons, or if you are using the latest add-ons versions, you can set insecure_addon_compat to false in `/etc/opscode/chef-server.rb. and Chef Infra Server will write all credentials to a single location.

For more information on password generation, including a list of supported add-on versions, see Chef Infra Server Credentials Management.

Upgrading to 12.3.0

If you are running a Chef Infra Server release before 12.3.0, please contact Chef Support for guidance on upgrading your Chef Infra Server installation.

Chef Infra Server 14 Upgrade Process

Standalone Server

The Chef Infra Server 14 upgrade process requires downtime for stopping the server, installing the new package, and then upgrading the server, which will include an automatic Elasticsearch reindexing operation for existing Solr users. We estimate the reindexing operation will take 2 minutes for each 1000 nodes, but the it could take more time, depending on your server hardware and the complexity of your Chef data.

1. Run vacuumdb before starting the upgrade:

   sudo su - opscode-pgsql
   /opt/opscode/embedded/bin/vacuumdb --all --full
   exit
   

   You should see output like:

   vacuumdb: vacuuming database "bifrost"
   vacuumdb: vacuuming database "oc_id"
   vacuumdb: vacuuming database "opscode-pgsql"
   vacuumdb: vacuuming database "opscode_chef"
   vacuumdb: vacuuming database "postgres"
   vacuumdb: vacuuming database "template1"
   

2. Back up your Chef Infra Server data before starting the upgrade process using chef-server-ctl-backup. Make a note of where the backup is located (The default is /var/opt/chef-backup). Please note that Chef Infra Server will go offline to perform the backup:

   sudo chef-server-ctl backup
   

3. Confirm that the Chef Infra Server services are operational:

   chef-server-ctl reconfigure
   

4. If you are already running 12.17.15 or greater, proceed to the next step. Otherwise consult the upgrade matrix and perform a stepped upgrade.

   • If you are running Chef Infra Server 12.3.0, upgrade to 12.17.15.
   • If you are running Chef Infra Server 11, first upgrade to 12.3.0, and then to 12.17.15.

   After performing the stepped upgrade to 12.17.15, continue with the next step.

5. Download the desired Chef Infra Server version from the Chef Infra Server Downloads.

6. Stop the Chef Infra Server:

   chef-server-ctl stop || chef-server-ctl kill rabbitmq
   

7. Install the Chef Infra Server package:

   To install with dpkg:

   dpkg -iEG /path/to/chef-server-core-VERSION.deb
   

   To install with the RPM Package Manager:

   rpm -Uvh --nopostun /path/to/chef-server-core-VERSION.rpm
   

8. Upgrade the server and accept the Chef Software license by entering Yes at the prompt:

   chef-server-ctl upgrade
   

   To accept the license and upgrade in one command:

   CHEF_LICENSE='accept' chef-server-ctl upgrade
   

9. If the upgrade failed, see the section below on how to handle an upgrade failure.

   If the upgrade was successful, start Chef Infra Server:

   chef-server-ctl start
   

10. Upgrade any Chef Infra Server add-ons.

11. After the upgrade process is complete, test and verify that the server works.

12. Clean up the Chef Infra Server by removing the old data:

    chef-server-ctl cleanup
    

13. Reindex the database:

    sudo su - opscode-pgsql
    /opt/opscode/embedded/bin/reindexdb --all
    

    You should see output like:

    reindexdb: reindexing database "bifrost"
    reindexdb: reindexing database "oc_id"
    reindexdb: reindexing database "opscode-pgsql"
    reindexdb: reindexing database "opscode_chef"
    reindexdb: reindexing database "postgres"
    reindexdb: reindexing database "template1"
    

14. Run ANALYZE on the PostgreSQL database. This process gathers statistics for the query planner to create the most efficient query execution paths. Accurate statistics help the planner choose the most appropriate query plan, and thereby improve the speed of query processing.

    sudo su - opscode-pgsql
    /opt/opscode/embedded/bin/vacuumdb --all --analyze-only
    

    You should then see output like:

    vacuumdb: vacuuming database "bifrost"
    vacuumdb: vacuuming database "oc_id"
    vacuumdb: vacuuming database "opscode-pgsql"
    vacuumdb: vacuuming database "opscode_chef"
    vacuumdb: vacuuming database "postgres"
    vacuumdb: vacuuming database "template1"
    

You are now finished with the upgrade.

Upgrade Failure Troubleshooting

1. If the upgrade failed and you have a corrupted Chef Infra Server and/or a corrupted database, DO NOT RISK YOUR BACKUP OF THE DATABASE. Take all steps necessary to preserve the backup, including copying it to another disk. Consult with a professional sysadmin for instructions and best practices.

2. Contact customer support.

3. Reinstall the original version of Chef Infra Server you were using before attempting the upgrade process (if you had to perform a stepped upgrade, install your original version of Chef Infra Server before the stepped upgrade, not any versions you upgraded to in the stepped upgrade process). Again, DO NOT RISK YOUR BACKUP OF THE DATABASE. For example, consider using a separate disk from your backup for the new installation.

4. Consult the restore documentation and restore the database from the path to where it was saved:

   chef-server-ctl restore /path/to/tar/archive.tar.gz
   

External PostgreSQL

The following External PostgreSQL upgrade steps are provided as a courtesy only. It is the responsibility of the user to upgrade and maintain any External PostgreSQL configurations.

Upgrade Chef Infra Server

1. Log into the external PostgreSQL machine.

2. Run vacuumdb before starting the upgrade:

   sudo su postgres
   /usr/bin/vacuumdb --all --full
   exit
   

3. Log into the Chef Infra Server machine.

4. Consult the documentation on knife-ec-backup.

   Install knife-ec-backup, if it not already installed. A sample session follows (note that your steps could differ, depending on the versions of your software, the topology of your setup, your OS and distribution, and a range of other factors).

   For example:

   apt-get update
   apt install ruby
   apt install make
   curl -L https://chef.io/chef/install.sh | sudo bash -s -- -P chefdk
   export PATH=$PATH:/root/.chefdk/gem/ruby/2.6.0/bin
   apt-get -y install gcc postgresql libpq-dev
   /opt/chefdk/embedded/bin/gem install knife-ec-backup -- --with-pg-config=/opt/opscode/embedded/postgresql/9.6/bin/pg_config
   

5. Configure knife if it is not already configured. A sample session follows (again, note that your steps could differ, depending on a range of factors).

   For Example:

   $ chef-server-ctl org-create 4thcafe 'Fourth Cafe, Inc.' --association_user janedoe --filename /tmp/4thcafe-validator.pem
   $ chef generate repo chef-repo
   $ cd chef-repo/
   $ mkdir -p .chef
   $ echo '.chef' >> .gitignore
   $ cp /tmp/4thcafe-validator.pem .chef
   $ cp /home/ubuntu/janedoe.pem .chef
   $ cd .chef
   $ knife configure
   WARNING: No knife configuration file found. See https://docs.chef.io/config_rb/ for details.
   Please enter the chef server URL: [https://1.2.3.4/organizations/myorg] https://1.2.3.4/organizations/4thcafe
   Please enter an existing username or clientname for the API: [ubuntu] janedoe
   $ knife ssl fetch
   

6. Backup the database. For Example:

   mkdir /backup
   /opt/chefdk/embedded/bin/knife ec backup /backup
   

7. If you are running Chef Infra Server version 12.17.15 or greater, proceed to the next step below. Otherwise consult the upgrade matrix and perform a stepped upgrade.

   If you are running a Chef Infra Server release before 12.17.15, you cannot upgrade directly to 14.8.X. You must perform a stepped upgrade first.

   • If you are running Chef Infra Server 12.3.0, upgrade to 12.17.15.
   • If you are running Chef Infra Server 11, you must first upgrade to 12.3.0, and then to 12.17.15.

   After performing the stepped upgrade, return here and continue with the next step below.

8. Download the desired version of Chef Infra Server.

9. Stop the Chef Infra Server:

   chef-server-ctl stop || chef-server-ctl kill rabbitmq
   

10. Install the Chef Infra Server package:

    To install with dpkg:

    dpkg -iEG /path/to/chef-server-core-VERSION.deb
    

    To install with the RPM Package Manager:

    rpm -Uvh --nopostun --force /path/to/chef-server-core-VERSION.rpm
    

11. Upgrade Chef Infra Server and accept the Chef Software license by entering Yes at the prompt:

    chef-server-ctl upgrade
    

    To accept the license and upgrade in one command:

    CHEF_LICENSE='accept' chef-server-ctl upgrade
    

12. If the upgrade failed, see the section on upgrade failure troubleshooting.

    If the upgrade was successful, start the Chef Infra Server services and cleanup.

    sudo chef-server-ctl start
    sudo chef-server-ctl cleanup
    

    You are now finished with the Chef Infra Server upgrade. Proceed directly to the Upgrade PostgreSQL section.

Upgrade PostgreSQL

1. Log into the external PostgreSQL machine.

2. Update packages and install your selected PostgreSQL version. Example (Ubuntu/PostgreSQL 13.3):

   sudo apt-get update
   sudo apt-get install postgresql-13
   

3. Check if there are any differences in the config files. Make sure to update the new config files if required. Example (PostgreSQL 13.3):

   diff /etc/postgresql/OLD_POSTGRESQL_VERSION/main/postgresql.conf /etc/postgresql/13/main/postgresql.conf
   diff /etc/postgresql/OLD_POSTGRESQL_VERSION/main/pg_hba.conf     /etc/postgresql/13/main/pg_hba.conf
   

4. Stop the PostgreSQL service.

   sudo systemctl stop postgresql.service
   

5. Log in as the postgres user.

   su postgres
   

6. Ensure that you are in a directory where you can run the pg_upgrade command.

   Example:

   cd /tmp
   

7. Check clusters (notice the --check argument, this will not change any data). Example (PostgreSQL 13.3):

   /usr/lib/postgresql/13/bin/pg_upgrade \
   --old-datadir=/var/lib/postgresql/9.6/main \
   --new-datadir=/var/lib/postgresql/13/main \
   --old-bindir=/usr/lib/postgresql/9.6/bin \
   --new-bindir=/usr/lib/postgresql/13/bin \
   --old-options '-c config_file=/etc/postgresql/9.6/main/postgresql.conf' \
   --new-options '-c config_file=/etc/postgresql/13/main/postgresql.conf' \
   --check
   

8. Migrate the data (without the --check argument). Example (PostgreSQL 13.3):

   /usr/lib/postgresql/13/bin/pg_upgrade \
   --old-datadir=/var/lib/postgresql/9.6/main \
   --new-datadir=/var/lib/postgresql/13/main \
   --old-bindir=/usr/lib/postgresql/9.6/bin \
   --new-bindir=/usr/lib/postgresql/13/bin \
   --old-options '-c config_file=/etc/postgresql/9.6/main/postgresql.conf' \
   --new-options '-c config_file=/etc/postgresql/13/main/postgresql.conf'
   

9. Log out of the postgres user.

   exit
   

10. Swap the ports for the old and new PostgreSQL versions. Example (PostgreSQL 13.3):

    $ sudo vim /etc/postgresql/13/main/postgresql.conf
    # change "port = 5433" to "port = 5432"
    
    $ sudo vim /etc/postgresql/9.6/main/postgresql.conf
    # change "port = 5432" to "port = 5433"
    

11. Start the PostgreSQL service.

    sudo systemctl start postgresql.service
    

12. Log in as the postgres user and confirm that the new PostgreSQL version is correct. Example (PostgreSQL 13.3):

    $ sudo su - postgres
    $ psql -c "SELECT version();"
                                                                       version
    ---------------------------------------------------------------------------------------------------------------------------------------------
     PostgreSQL 13.3 (Ubuntu 13.3-1.pgdg16.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609, 64-bit
    (1 row)
    

13. Run reindexdb. Example:

    $ /usr/bin/reindexdb --all
    reindexdb: reindexing database "bifrost"
    reindexdb: reindexing database "oc_id"
    reindexdb: reindexing database "opscode_chef"
    reindexdb: reindexing database "postgres"
    reindexdb: reindexing database "template1"
    

14. Run ANALYZE on the PostgreSQL database. This process gathers statistics for the query planner to create the most efficient query execution paths. Accurate statistics help the planner choose the most appropriate query plan, and thereby improve the speed of query processing.

    sudo su - opscode-pgsql
    /opt/opscode/embedded/bin/vacuumdb --all --analyze-only
    

    You should then see output like:

    vacuumdb: vacuuming database "bifrost"
    vacuumdb: vacuuming database "oc_id"
    vacuumdb: vacuuming database "opscode-pgsql"
    vacuumdb: vacuuming database "opscode_chef"
    vacuumdb: vacuuming database "postgres"
    vacuumdb: vacuuming database "template1"
    

15. Log into the Chef Infra Server machine.

16. Check the status of Chef Infra Server. PostgreSQL should be connected.

    $ sudo chef-server-ctl status
    -------------------
     Internal Services
    -------------------
    run: bookshelf: (pid 15763) 219163s; run: log: (pid 16559) 228464s
    run: elasticsearch: (pid 15797) 219162s; run: log: (pid 16345) 228507s
    run: nginx: (pid 15901) 219162s; run: log: (pid 16745) 228452s
    run: oc_bifrost: (pid 15909) 219161s; run: log: (pid 16237) 228519s
    run: oc_id: (pid 15915) 219161s; run: log: (pid 16255) 228513s
    run: opscode-erchef: (pid 15948) 219160s; run: log: (pid 16673) 228458s
    run: redis_lb: (pid 15952) 219160s; run: log: (pid 16779) 228445s
    -------------------
     External Services
    -------------------
    run: postgresql: connected OK to 10.0.11.0:5432
    

Upgrade Failure Troubleshooting

1. If the upgrade failed and you are left with a corrupted Chef Infra Server and/or a corrupted database, DO NOT RISK YOUR BACKUP OF THE DATABASE. Take all steps necessary to preserve the backup, including copying it to another disk. Consult with a professional sysadmin for instructions and best practices.

2. Contact customer support.

3. Reinstall the original version of Chef Infra Server you were using before attempting the upgrade process (if you had to perform a stepped upgrade, install your original version of Chef Infra Server before the stepped upgrade, not any versions you upgraded to in the stepped upgrade process). Again, DO NOT RISK YOUR BACKUP OF THE DATABASE. For example, consider using a separate disk from your backup for the new installation.

4. Consult the documentation on knife-ec-restore, and restore the database from the path where it was saved.

   Example:

   /opt/chefdk/embedded/bin/knife ec restore /backup/
   

5. Do not continue upgrading PostgreSQL until you have an uncorrupted Chef Infra Server and an uncorrupted PostgreSQL database.

Chef Backend Install

The Chef Infra Server can operate in a high availability configuration that provides automated load balancing and failover for stateful components in the system architecture.

To upgrade your Chef Backend installation, see High Availability: Upgrade to Chef Backend 2.

Tiered Install

This section describes the upgrade process from a tiered server configuration.

Tiered Upgrade Steps

To upgrade to Chef Infra Server on a tiered Chef Infra Server configuration, do the following:

1. Back up the Chef Infra Server data before starting the upgrade process using knife-ec-backup.

2. Confirm that the Chef Infra Server services are operational:

   chef-server-ctl reconfigure
   

3. Download the desired Chef Infra Server version from the Chef Infra Server Downloads page, then copy it to each server.

4. Stop all front end servers:

   chef-server-ctl stop
   

5. Install the Chef Infra Server package on all servers:

   To install with dpkg:

   dpkg -i /path/to/chef-server-core-<version>.deb
   

   To install with the RPM Package Manager:

   rpm -Uvh --nopostun /path/to/chef-server-core-<version>.rpm
   

6. Stop the back end server:

   chef-server-ctl stop
   

7. Upgrade the server and accept the Chef Software license by entering Yes at the prompt:

   chef-server-ctl upgrade
   

   To accept the license and upgrade in one command:

   CHEF_LICENSE='accept' chef-server-ctl upgrade
   

8. Copy the entire /etc/opscode directory from the back end server to all front end servers:

   scp -r /etc/opscode <each server's IP>:/etc
   

9. Upgrade each of the front end servers:

   chef-server-ctl upgrade
   

10. Run the following command on both the front end, and back end servers:

    chef-server-ctl start
    

11. Upgrade any Chef Infra Server add-ons.

12. After the upgrade process is complete, test and verify that the server works.

13. Clean up the server by removing the old data:

chef-server-ctl cleanup