Unable to scp to and from AWS EC2 cluster

I am unable to scp to and from ec2.I type the following command: scp -i ~/.ssh/my_key.pem ubuntu@ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com:~/models.pickle ~/

I get the following warning:Warning: Identity file /home/ubuntu/.ssh/my_key.pem not accessible: No such file or directory.Permission denied (publickey).

I also tried both chmod 400 and chmod 600.

I am able to ssh into my cluster though.

I checked for the name of the pem file and it matches the one for my EC2 cluster too.