I have an eks cluster with a nodegroup based on a mixed instance distribution launch template with the following configuration:
region: us-west-2
instance_distribution: [p2.xlarge, p3.2xlarge, p2.8xlarge]
max_price: 0.9
on_demand_percentage_above_base_capacity: 0
on_demand_base_capacity: 0
spot_instance_pools: 2
When looking to scale the autoscaling from 0 to 1, the following issue was encountered by the cluster autoscaler:
Launching a new EC2 instance. Status Reason: Could not launch Spot Instances. SpotMaxPriceTooLow - Your Spot request price of 0.9 is lower than the minimum required Spot request fulfillment price of 0.918. Launching EC2 instance failed.
At the time, the spot price of p3.2xlarge happened to be 0.918.
It appears that a request for a spot p3.2xlarge was made instead of making a request for an on-demand instance of p2.xlarge (even though the on-demand price of 0.9 for p2.xlarge was less than the spot price of 0.918 for p3.2xlarge). I expected an on-demand p2.xlarge instance to be allocated up instead of requesting a p3.2xlarge spot instance. Is it because I configured on_demand_percentage_above_base_capacity: 0?
More generally, I want to be able to configure the cluster to get spot instances of p2.xlarge and when not possible, request for on-demand. What is the best configuration to achieve my desired functionality?
Is the configuration of on_demand_percentage_above_base_capacity strictly enforced? If on_demand_percentage_above_base_capacity is set to 1 and my first instance is an on-demand instance, will my next few scaling requests be forced to yield spot instances only or is it more like a weightage with guidelines (e.g. if there are no spots available it will still fall back to on-demand rather than failing to fulfill the request)?