AWS, AWS, and More AWS

It’s been a summer of AWS for me.

I first started playing with AWS when I did a big data course a couple of summers ago. At the time we used it to learn some key tools like Hadoop and Spark that just aren’t practical on a laptop. (Though, to be honest, I learned more about Hadoop on a Raspberry Pi cluster than from AWS.)

Since then, I’ve used it more and more in client projects and this summer’s is a big one: Move a client’s entire infrastructure from a self-hosted set of servers to AWS, then layer on some analytics and reporting once all the data and applications are hosted in a single virtual private cloud and easily accessible to each other. I’m using all the obvious suspects: S3 for storage, EC2 for compute where we need it (though one server that had been dedicated entirely to storage will just go away), and moving one database from Postgres to Amazon RDS.

There are some interesting network quirks as well: Access from anywhere is one of the key requirements and drivers for the project, so a solid VPN providing access to some (but not all) resources is being developed. There is still a “home office” where they’d like to have faster access to large files, so I’ll be building a storage gateway with cached local copies to make that happen. And of course, multiple VPN connections between the two.

Along the way, I’ve made myself more of an AWS professional. Got the Architect Professional, Big Data Specialty, and Developer Associate certs. Will I continue? Probably not. The remaining places to do drive me towards SysOps/DevOps (not a place I want to focus right now) or networking and security (ditto). More likely to continue my focus on what I’m doing right now, looking for bigger projects with a heavy focus on the kinds of big data solutions and tools that I’ve built elsewhere and used.