I am trying to build a new GitHub runner AMI using an existing Packer template that last built an image a little over a year ago. The template is using the Amazon EBS builder with AWS Session Manager (SSM) for its SSH connection:
source "amazon-ebs" "ubuntu" { ami_name = "github-runner-ubuntu-jammy-amd64-${local.version_s}" instance_type = "t3.medium" region = "us-east-2" source_ami = data.amazon-ami.ubuntu-2204.id ssh_username = "ubuntu" ssh_interface = "session_manager" pause_before_ssm = "3m" aws_polling { delay_seconds = 30 max_attempts = 50 } temporary_iam_instance_profile_policy_document { Version = "2012-10-17" Statement { Effect = "Allow" Action = ["ssm:DescribeAssociation","ssm:GetDeployablePatchSnapshotForInstance","ssm:GetDocument","ssm:DescribeDocument","ssm:GetManifest","ssm:GetParameter","ssm:GetParameters","ssm:ListAssociations","ssm:ListInstanceAssociations","ssm:PutInventory","ssm:PutComplianceItems","ssm:PutConfigurePackageResult","ssm:UpdateAssociationStatus","ssm:UpdateInstanceAssociationStatus","ssm:UpdateInstanceInformation" ] Resource = ["*"] } Statement { Effect = "Allow" Action = ["ssmmessages:CreateControlChannel","ssmmessages:CreateDataChannel","ssmmessages:OpenControlChannel","ssmmessages:OpenDataChannel" ] Resource = ["*"] } Statement { Effect = "Allow" Action = ["ec2messages:AcknowledgeMessage","ec2messages:DeleteMessage","ec2messages:FailMessage","ec2messages:GetEndpoint","ec2messages:GetMessages","ec2messages:SendReply" ] Resource = ["*"] } } skip_create_ami = false deprecate_at = local.deprecate_s tags = merge( var.global_tags, var.ami_tags, { OS_Version = "ubuntu-jammy" Release = "Latest" Base_AMI_Name = "{{ .SourceAMIName }}" }) snapshot_tags = merge( var.global_tags, var.snapshot_tags, ) launch_block_device_mappings { device_name = "/dev/sda1" volume_size = "15" volume_type = "gp3" delete_on_termination = "true" }}The source AMI used is grabbing the latest version of ubuntu-jammy-22.04-amd64 (at this time, it's: ubuntu-jammy-22.04-amd64-server-20240801). The temporary IAM instance profile policy document is following the AmazonSSMManagedEC2InstanceDefaultPolicy managed policy. The Session Manager also currently has CloudWatch logging, S3 logging, and KMS encryption all disabled. This is using Packer version 1.11.2 and AWS Session Manager plugin version 1.2.650.0.
When running the packer build command: PACKER_LOG=1 packer build -debug -only *.amazon-ebs.* -var=ami_environment=test packer/ubuntu/packer.pkr.hcl, I am getting connection issues. This is what I am seeing in the logs:
==> github-runner-ubuntu.amazon-ebs.ubuntu: Attaching policy to the temporary role: packer-66bcc822-537d-ca74-a3aa-08fa262d2023==> github-runner-ubuntu.amazon-ebs.ubuntu: Launching a source AWS instance... github-runner-ubuntu.amazon-ebs.ubuntu: Instance ID: i-0a358f5e6c0259ef2==> github-runner-ubuntu.amazon-ebs.ubuntu: Waiting for instance (i-0a358f5e6c0259ef2) to become ready...2024/08/14 11:25:06 packer-plugin-amazon_v1.3.2_x5.0_linux_amd64 plugin: 2024/08/14 11:25:06 [INFO] Not using winrm communicator, skipping get password...==> github-runner-ubuntu.amazon-ebs.ubuntu: Waiting 3m0s before establishing the SSM session...2024/08/14 11:28:07 packer-plugin-amazon_v1.3.2_x5.0_linux_amd64 plugin: 2024/08/14 11:28:07 Found available port: 8494 on IP: 0.0.0.02024/08/14 11:28:07 packer-plugin-amazon_v1.3.2_x5.0_linux_amd64 plugin: 2024/08/14 11:28:07 ssm: Starting PortForwarding session to instance i-0a358f5e6c0259ef2==> github-runner-ubuntu.amazon-ebs.ubuntu: Using SSH communicator to connect: localhost2024/08/14 11:29:16 packer-plugin-amazon_v1.3.2_x5.0_linux_amd64 plugin: 2024/08/14 11:29:16 [INFO] Waiting for SSH, up to timeout: 5m0s ==> github-runner-ubuntu.amazon-ebs.ubuntu: Waiting for SSH to become available...2024/08/14 11:29:16 packer-plugin-amazon_v1.3.2_x5.0_linux_amd64 plugin: 2024/08/14 11:29:16 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 127.0.0.1:8494: connect: connection refused... (many more connection refused errors)2024/08/14 11:34:12 packer-plugin-amazon_v1.3.2_x5.0_linux_amd64 plugin: 2024/08/14 11:34:12 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 127.0.0.1:8494: connect: connection refused2024/08/14 11:34:16 packer-plugin-amazon_v1.3.2_x5.0_linux_amd64 plugin: 2024/08/14 11:34:16 [DEBUG] SSH wait cancelled. Exiting loop.==> github-runner-ubuntu.amazon-ebs.ubuntu: Timeout waiting for SSHThe packer build is ran using a GitHub action, but I am also able to reproduce the issue when running the command from my local machine using my AWS credentials. When debugging, I am able to see the EC2 instance get created in the console and was able to confirm that the IAM role attached is using the temporary policies from the template. The security group has no inbound rules set and has outbound open to all traffic. From what I have read in the documentation, I shouldn't need to include any inbound rules when using SSM. I also cannot connect to the instance from the AWS console using Session Manager. I am seeing this error below:
I have tried increasing the pause_before_ssm and even tried updating the temporary policy to allow full access to everything, but I am still seeing the same error.
Could there be some VPC/networking changes preventing Packer from connecting to the instance? Or does the instance need inbound rules after all? Or is there some other configuration I am overlooking?