I am looking for a reliable, cost effective and safe way to auto-scale mongodb. We have a cloud application that peaks during business hours and then is mostly unused after hours. We also have a batching engine that wakes itself up and does some operations. I cannot find a good solution that allows for auto-scaling (I don't want to pay for 3 T3-2XLarge while everyone is sleeping on saturday night). I have the following idea, can someone review and let me know where the flaws are:
1) set up 7 (or some odd number > 3) ec2 instances, each one knows it's pecking order, i think these are all replica sets and shards, but i could use your advice on this
2) put 4 of them to sleep (shut them off). The remaining 3 are the active members
3) instrument each one to detect load (with a Ruby script or python script)
4) if there is too much load and there are only 3 boxes, spin 2 more up, now we have 5. which 2 spin up are based on the pecking order, box 3 knows who box 4 is and knows to spin up 4 and wait for it to complete
5) the 2 new boxes are aware that they've been asleep for too long so they ask for a backup and restore the db. this is done perhaps as part of box 3 knowing when box 4 is done starting up, perhaps it backs and and registers the backup to s3 or dropbox
6) once restored, the new boxes register themselves with the cluster (they've previously been registered, they've just been seen as dead for a while), so perhaps "register" is the wrong vocabulary word
7) one by one, the original 3 resize themselves, by deregistering themselves from the cluster and script-resize themselves
8) after all 3 have resized and after sync, kill the 2 temporary ones
I said 7 initially, this is just an idea, maybe the cluster knows how to scale itself vertically and horizontally. the hangup i have is how to deal with backup and restore, and how to auto-add and remove boxes to a cluster, and also if mongo supports, say 7 boxes, with 4 of them dead for long periods of time. anyone have any thoughts or can you give me a push in the right direction if anyone has done this already?
Thanks, Kevin