Anka on AWS EC2 Macs

This guide is also valid for the Anka 3/mac2.metal/Apple processor (M1, M2, etc) EC2 instances.

While you do not need to use the Anka Build Cloud with AWS EC2 Mac instances to run Anka VMs, it is a requirement if you’d like to use our Controller Plugins/Integrations to create VMs. You can read more about the Anka Build Cloud here.

Customers often find that purchasing and managing their own hardware can become a burden. While we do not currently provide an AMI for the Anka Build Cloud, we do for the Anka Virtualization software. This makes it easy to get started with AWS EC2 Mac instances that you can scale as needed using the AWS Management Console or AWS CLI. The AMI is pre-prepared with best practices/configurations for using Anka.

With Anka installed on your AWS EC2 Mac instance, you can:

1. Run ephemeral macOS VMs as well as optimize the instance cost by running more than one at a time.
2. Avoid idle hosts and maximize the usage in the first 24 hour period that Amazon charges you for when starting an EC2 Mac instance.
3. Scale VM capacity up and down as needed using the AWS Management Console or AWS CLI.
4. Prevent jobs from leaving a dirty environment on the host for other jobs to conflict with.
5. Run multiple versions of macOS on the same host.

And much more! Visit our site and Amazon’s blog for more information about AWS EC2 Mac and Anka.

There are three AMI options available for you to use:

1. Use our Marketplace AMI

• macOs pre-configured/optimized + Anka installed
• Provides an hourly billing option for Anka based on the uptime of your EC2 Mac instance
• Marketplace AMIs require Instance metadata service to be enabled.

2. Bring your own license (BYOL) Community AMI

• macOS pre-configured/optimized + Anka installed
• With these AMIs, you will be able to use your own Anka License.

3. Build your own AMI

Note: You must request a dedicated mac* host in order to run EC2 Mac instances. There is a known delay requesting, stopping, and starting EC2 Mac instances as the dedicated host must clean itself each time an instance stops on it.

Marketplace AMI

In order to get started using our Marketplace AMIs you have four options:

1. Basic License: intel | arm64
2. Enterprise License: intel | arm64
3. Enterprise Plus License: intel | arm64

Other than the hourly price, there is a list of features that differ between the two.

You can find a full list of products available on the AWS marketplace by visiting https://veertu.com/aws-marketplace. Or, once subscribed, you can find and launch instances from the marketplace AMIs on the Manage Subscriptions page.

Marketplace AMIs are charged on an hourly basis and don’t need an Anka License.

You can create custom AMIs from the Marketplace AMI and the license for Anka will continue to work and attach to your existing marketplace subscription. However, if we remove the base AMI you took a snapshot from, it will show an expired license and stop working. We instead recommend using our base AMIs instead of creating your own.

Usage

To get up and running with our AWS EC2 Mac instances using our Marketplace AMI, you’ll need to navigate to one of the Marketplace AMI Product URLs listed above and go through the process of subscribing. Take a look at the official AMI Subscription documentation to understand how to subscribe.

Once subscribed, you can start launching AMIs. Marketplace AMIs require Instance metadata service to be enabled.

• (Optional) Automatically join to the Anka Build Cloud Controller using User Data:

  This step requires that you first set up the Anka Build Cloud on a Linux server/docker container in AWS (but not on the EC2 Mac instances we provide AMIs for).

  IMPORTANT: Amazon confirmed that Terminating from the AWS console/API does not properly send SIGTERMs to services and wait for them to stop. This prevents our cloud-connect script from automatically disjoining with ankacluster disjoin before AWS pulls the plug. Therefore, we recommend executing the sudo launchctl unload -w /Library/LaunchDaemons/com.veertu.aws-ec2-mac-amis.cloud-connect.plist command before termination of the instance.

  User Data ENVs

  For user-data, don’t use ;, && or any other type of separator between envs. You also cannot use multiline strings in the ENVs. However, you can replace newlines with \n and our service will get the ENV with the multiple line text

  If you pass in user-data with the exports all on one line, and have non ANKA_ ENVs you’re setting, the cloud-connect.bash service we run on instance start/boot will source/execute them. We recommend you split exports and user-data onto separate lines to avoid this.

  ANKA_CONTROLLER_ADDRESS (string)

  Full URL for the Anka Build Cloud Controller.

  • REQUIRED
  • Must be in the following structure: http[s]://[IP/DOMAIN]:[PORT].

  ANKA_JOIN_ARGS (string)

  Allows you to pass in any “Flags” from ankacluster join --help.

  • Optional

  ANKA_REGISTRY_OVERRIDE_IP + ANKA_REGISTRY_OVERRIDE_DOMAIN (string)

  Allows you to set the registry IP address and domain in the /etc/hosts file.

  • Optional
  • Use 1: if your corporate registry doesn’t have a public domain name, but does have a public IP.
  • Use 2: if you want the EC2 mac mini to pull from a second registry that’s hosted on EC2 instead of a local corporate one (AWS -> AWS is much faster).

  ANKA_LICENSE (string)

  If not already licensed, the cloud-connect service will license Anka using this ENV’s value.

  • Optional
  • Only used with Community AMI.
  • Only available in >= 2.5.4 AMIs.
  • You can also update invalid/expired licenses with this (requires a reboot).
  • Starting in AMIs with a macOS version greater than 12.2.1: The Fulfillment ID output from anka license activate, which is used for releasing cores, is logged to your Cloud Controller > Logs section in the “AWS Cloud Connect Service”.

  ANKA_USE_PUBLIC_IP (boolean)

  This will determine whether the instance/node is joined using the public ipv4. Otherwise, it will default to the local/private ipv4.

  • Optional

  ANKA_CONTROLLER_API_CERT / _KEY / _CA | ANKA_REGISTRY_API_CERT / _KEY / _CA (string)

  The script which handles joining to your controller has a few calls to the controller as well as the registry APIs. If you’re protecting your APIs with TLS and Certificate Authentication, you can set the certs to use with these ENVs.

  • Optional
  

  ANKA_PULL_LATEST_CLOUD_CONNECT (boolean)

  This will issue a git fetch and then, if there are changes pending on our aws-ec2-mac-amis repo, issue git pull to collect the latest version of the Cloud Connect scripts. This is useful if there is a bug in our scripts and you can’t update to a newer AMI yet.

  • Optional
  • Only available in 3.2.0/13.0 or greater AMIs.

  ANKA_UPGRADE_CLI_TO_LATEST (boolean)

  This will force an upgrade of Anka Virtualization CLI to its latest version.

  • Optional
  • Only available in 3.3.4/13.4.1 or greater AMIs.
  • This could be dangerous; please don’t rely on it unless newer AMIs are not available.

  ANKA_PULL_TEMPLATES_REGEX (string)

  This will pull templates from the registry which match the given regex (egrep) pattern.

  • Optional
  • Only available in 3.3.4/13.4.1 or greater AMIs.
  • Only the latest tag will be pulled.
  • Pulls will happen before the node is joined.
  • If your regex starts with -, be sure to escape it with \-.

  ANKA_PULL_TEMPLATES_REGEX_DISTRIBUTE (boolean)

  This will distribute, using the controller API, templates that match the given regex (egrep) pattern to the current node.

  • Optional & Dependent on ANKA_PULL_TEMPLATES_REGEX.
  • IMPORTANT: Be sure that your regex will match the template NAME and not the id. Otherwise this will fail.
  • Only available in 3.4.0 or greater AMIs.
  • Only the latest tag will be pulled.
  • Because distribution is asynchronous, the node will be joined in “drained” mode.

  ANKA_DRAINED_ON_JOIN (string)

  This will join the node in Drain Mode.

  • Optional
  • Only available in 3.3.10 or greater AMIs.
  • Requires Controller 1.32.0 or greater.

  ANKA_EXECUTE_SCRIPT (string)

  This will execute one of the available scripts included with the cloud-connect service. You must specify the full script name.

  • A list of available scripts is available here.
  • Optional
  • Only available in Anka 3.4.0 or greater AMIs.
  • If the script fails, the node will still join.
  • Most scripts support ENVs being passed in through user-data, so be sure to review them to see what’s possible. Dev note: the cloud-connect will not set/see any ENVs without ANKA_ prefix.

Manual Preparation (optional)

Amazon EBS volumes can be very slow even when you max iOPS, etc. Because of this, anka create and other processes can take very long times or outright fail (Apple’s installer is sensitive to disk IO). AWS indicates that you have to pre-warm EBS volumes that are restored from snapshots (which our AMIs are). To do this, follow the instructions outlined here: brew install fio && sudo fio --filename=/dev/r$(df -h / | grep -o 'disk[0-9]') --rw=read --bs=1M --iodepth=32 --ioengine=posixaio --direct=1 --name=volume-initialize Finally, pre-warmed volumes stay warmed – no need to run dd after periods of inactivity on the AWS instance. NOTE: This command is not able to run from user-data.

By default all of our AMIs have a cloud-connect agent which on boot will join your AWS instance to the Anka Build Cloud controller automatically with user data ENVs you set. This is issuing ankacluster join under the hood. Once joined, the agent which runs and communicates with the Anka Build Controller does its best to determine the proper IP to use for the node. On AWS the interfaces are loaded at different times and orders and often you’ll end up with an IP assigned to the node which cannot be used for communication. To solve this, you’ll want to set ANKA_JOIN_ARGS with --host {IP HERE} in the user data for the AWS instance. You can find all available flags/options for the join command here.

Logs

• /var/log/resize-disk.log
• /var/log/cloud-connect.log

Licensing

The Marketplace AMI does not require a license. You are charged hourly for the usage through the AWS marketplace. Anka marketplace AMIs are available with Anka Basic and Anka Enterprise Tier features. For more details on Basic and Enterprise Tier, check out our documentation.

Anka Build Cloud automated setup scripts

We have a script that will set up both a Linux instance with the Anka Build Cloud Controller & Registry. You can find it under our Getting Started repo’s AWS folder.

1. Clone the getting-started repo

   git clone https://github.com/veertuinc/getting-started.git
   cd getting-started
   

2. Execute ./AWS/prepare-build-cloud.bash

   • Running this script will create everything necessary inside of AWS to run the Anka Build Cloud. This includes a security group, elastic IP, etc.

The script can be run locally from your local macOS laptop with an existing AWS credential, region set, etc. These scripts have not been tested on linux.

Community AMI

Our BYOL Community AMIs are useful if you’d like to bring your own existing Anka license. They both have all of the same configuration changes, optimizations, and Anka inside. The difference is that Anka is unlicensed.

You can find a list of currently available Community AMIs below:

AMIs will be published under account IDs 930457884660 (veertu official) and 679593333241 (marketplace account from amazon). Do not trust AMIs from other accounts.

Usage

To get up and running with our AWS EC2 Mac instances using our BYOL Community AMI, you’ll need to:

1. Have an AWS mac1 (intel) or mac2 (arm/apple/m1) dedicated host ready.

2. Have an Anka license.

3. Choose the Community AMI when starting an instance:

• (Optional) Automatically join to the Anka Build Cloud Controller using User Data:

  This step requires that you first set up the Anka Build Cloud on a Linux server/docker container in AWS (but not on the EC2 Mac instances we provide AMIs for).

  IMPORTANT: Amazon confirmed that Terminating from the AWS console/API does not properly send SIGTERMs to services and wait for them to stop. This prevents our cloud-connect script from automatically disjoining with ankacluster disjoin before AWS pulls the plug. Therefore, we recommend executing the sudo launchctl unload -w /Library/LaunchDaemons/com.veertu.aws-ec2-mac-amis.cloud-connect.plist command before termination of the instance.

  User Data ENVs

  For user-data, don’t use ;, && or any other type of separator between envs. You also cannot use multiline strings in the ENVs. However, you can replace newlines with \n and our service will get the ENV with the multiple line text

  If you pass in user-data with the exports all on one line, and have non ANKA_ ENVs you’re setting, the cloud-connect.bash service we run on instance start/boot will source/execute them. We recommend you split exports and user-data onto separate lines to avoid this.

  ANKA_CONTROLLER_ADDRESS (string)

  Full URL for the Anka Build Cloud Controller.

  • REQUIRED
  • Must be in the following structure: http[s]://[IP/DOMAIN]:[PORT].

  ANKA_JOIN_ARGS (string)

  Allows you to pass in any “Flags” from ankacluster join --help.

  • Optional

  ANKA_REGISTRY_OVERRIDE_IP + ANKA_REGISTRY_OVERRIDE_DOMAIN (string)

  Allows you to set the registry IP address and domain in the /etc/hosts file.

  • Optional
  • Use 1: if your corporate registry doesn’t have a public domain name, but does have a public IP.
  • Use 2: if you want the EC2 mac mini to pull from a second registry that’s hosted on EC2 instead of a local corporate one (AWS -> AWS is much faster).

  ANKA_LICENSE (string)

  If not already licensed, the cloud-connect service will license Anka using this ENV’s value.

  • Optional
  • Only used with Community AMI.
  • Only available in >= 2.5.4 AMIs.
  • You can also update invalid/expired licenses with this (requires a reboot).
  • Starting in AMIs with a macOS version greater than 12.2.1: The Fulfillment ID output from anka license activate, which is used for releasing cores, is logged to your Cloud Controller > Logs section in the “AWS Cloud Connect Service”.

  ANKA_USE_PUBLIC_IP (boolean)

  This will determine whether the instance/node is joined using the public ipv4. Otherwise, it will default to the local/private ipv4.

  • Optional

  ANKA_CONTROLLER_API_CERT / _KEY / _CA | ANKA_REGISTRY_API_CERT / _KEY / _CA (string)

  The script which handles joining to your controller has a few calls to the controller as well as the registry APIs. If you’re protecting your APIs with TLS and Certificate Authentication, you can set the certs to use with these ENVs.

  • Optional
  

  ANKA_PULL_LATEST_CLOUD_CONNECT (boolean)

  This will issue a git fetch and then, if there are changes pending on our aws-ec2-mac-amis repo, issue git pull to collect the latest version of the Cloud Connect scripts. This is useful if there is a bug in our scripts and you can’t update to a newer AMI yet.

  • Optional
  • Only available in 3.2.0/13.0 or greater AMIs.

  ANKA_UPGRADE_CLI_TO_LATEST (boolean)

  This will force an upgrade of Anka Virtualization CLI to its latest version.

  • Optional
  • Only available in 3.3.4/13.4.1 or greater AMIs.
  • This could be dangerous; please don’t rely on it unless newer AMIs are not available.

  ANKA_PULL_TEMPLATES_REGEX (string)

  This will pull templates from the registry which match the given regex (egrep) pattern.

  • Optional
  • Only available in 3.3.4/13.4.1 or greater AMIs.
  • Only the latest tag will be pulled.
  • Pulls will happen before the node is joined.
  • If your regex starts with -, be sure to escape it with \-.

  ANKA_PULL_TEMPLATES_REGEX_DISTRIBUTE (boolean)

  This will distribute, using the controller API, templates that match the given regex (egrep) pattern to the current node.

  • Optional & Dependent on ANKA_PULL_TEMPLATES_REGEX.
  • IMPORTANT: Be sure that your regex will match the template NAME and not the id. Otherwise this will fail.
  • Only available in 3.4.0 or greater AMIs.
  • Only the latest tag will be pulled.
  • Because distribution is asynchronous, the node will be joined in “drained” mode.

  ANKA_DRAINED_ON_JOIN (string)

  This will join the node in Drain Mode.

  • Optional
  • Only available in 3.3.10 or greater AMIs.
  • Requires Controller 1.32.0 or greater.

  ANKA_EXECUTE_SCRIPT (string)

  This will execute one of the available scripts included with the cloud-connect service. You must specify the full script name.

  • A list of available scripts is available here.
  • Optional
  • Only available in Anka 3.4.0 or greater AMIs.
  • If the script fails, the node will still join.
  • Most scripts support ENVs being passed in through user-data, so be sure to review them to see what’s possible. Dev note: the cloud-connect will not set/see any ENVs without ANKA_ prefix.

Manual Preparation (optional)

Amazon EBS volumes can be very slow even when you max iOPS, etc. Because of this, anka create and other processes can take very long times or outright fail (Apple’s installer is sensitive to disk IO). AWS indicates that you have to pre-warm EBS volumes that are restored from snapshots (which our AMIs are). To do this, follow the instructions outlined here: brew install fio && sudo fio --filename=/dev/r$(df -h / | grep -o 'disk[0-9]') --rw=read --bs=1M --iodepth=32 --ioengine=posixaio --direct=1 --name=volume-initialize Finally, pre-warmed volumes stay warmed – no need to run dd after periods of inactivity on the AWS instance. NOTE: This command is not able to run from user-data.

By default all of our AMIs have a cloud-connect agent which on boot will join your AWS instance to the Anka Build Cloud controller automatically with user data ENVs you set. This is issuing ankacluster join under the hood. Once joined, the agent which runs and communicates with the Anka Build Controller does its best to determine the proper IP to use for the node. On AWS the interfaces are loaded at different times and orders and often you’ll end up with an IP assigned to the node which cannot be used for communication. To solve this, you’ll want to set ANKA_JOIN_ARGS with --host {IP HERE} in the user data for the AWS instance. You can find all available flags/options for the join command here.

Logs

• /var/log/resize-disk.log
• /var/log/cloud-connect.log

Licensing

When you first license Anka, keep track of the fulfillment ID as you’ll need this to release the cores and use the license on a fresh machine.

The Anka Develop license type will not work on AWS EC2 Macs.

Stopping and starting the instance does not impact the Anka licenses validity, even if you start the instance on a different dedicated machine.

Before terminating an instance, you will need to remove the Anka license first and then contact Veertu support ([email protected]) to clear the fulfillments

Anka Build Cloud automated setup scripts

We have two scripts that will set up both a Linux instance with the Anka Build Cloud Controller & Registry as well as an EC2 Mac instance (Anka Node) to run VMs. This relies on our Community AMI and you will need to have an Anka License. You can find them under our Getting Started repo’s AWS folder.

1. Clone the getting-started repo

   git clone https://github.com/veertuinc/getting-started.git
   cd getting-started
   

2. Execute ./AWS/prepare-build-cloud.bash

   • Running this script will create everything necessary inside of AWS to run the Anka Build Cloud. This includes a security group, elastic IP, etc.

3. Execute ./AWS/prepare-anka-node.bash

   • Requires that you first run prepare-build-cloud.bash.

   • Running this script will create everything necessary inside of AWS to run an EC2 Mac instance. You’ll be prompted for the Anka license to use if the ANKA_LICENSE env variable is not set.

Both scripts can be run locally from your local macOS laptop with an existing AWS credential, region set, etc. These scripts have not been tested on linux.

Build your own AMI

Building your own AMI is easy! You can review our AMI scripts to see how we do it.

Some important notes about creating your own AMI:

• Be sure that the minimum EBS volume specs are gp3, 6000IOPS, and 256 throughput. Anka VM creation is sensitive on slow disks and will likely fail.
• If using the Anka Build Cloud: This step requires that you first set up the Anka Build Cloud on a Linux server/docker container in AWS (but not on the EC2 Mac instances we provide AMIs for).

Answers to Frequently Asked Questions

• VMs can enable access to the 169.254.169.254 instance Metadata by routing it from the host with: networksetup -setadditionalroutes Ethernet 169.254.169.254 255.255.255.255 $(sudo defaults read /Library/Preferences/SystemConfiguration/com.apple.vmnet.plist Shared_Net_Address)