![\"New](\"https:\/\/static.gosquared.com\/images\/liquidicity\/17_12_04_reinvent\/17_12_04_reinvent_01@2x.png\")
<\/p>\n<\/p>\n
We spent last week in Las Vegas at AWS re:Invent, Amazon Web Services\u2019 global cloud computing conference.<\/p>\n
As always there was a huge<\/em> number of new services and features announced this year. It can be hard to get your head around all the new stuff AWS throws out every year, so here\u2019s our pick of the ones that caught our eye and made us the most excited for the future.<\/p>\n This obviously isn\u2019t an exhaustive list – it\u2019s mostly those services most relevant to us. If you want the full list, AWS has a big page of everything announced at re:Invent<\/a> which we recommend you check out. But without further ado, here is a list of our favourite things:<\/p>\n We\u2019re super-excited for the introduction of Fargate<\/a> for Elastic Container Service. Fargate essentially completely removes the concept of host instances from running containers on ECS<\/a>. You simply tell AWS \u201crun these containers for me\u201d and Fargate takes care of all the infrastructure invisibly and under the hood. This especially makes things like Auto Scaling a lot easier where you don\u2019t have to worry about whether you\u2019re using your host instance resources efficiently.<\/p>\n AWS is constantly iterating on the performance of their EC2 instance types, so the M5 instances<\/a> come as no surprise – they effectively replace the previous generation of M4 instances, with better performance and lower price. The H1 instances<\/a> come with a ton of local disk storage and are ideal for performing big batch jobs against large amounts of data.<\/p>\n Probably the biggest change to EC2\u2019s pricing model since the Spot market was first introduced a few years ago, Spot Instances can now be launched instantly<\/a> via the One of the key features of Spot Instances is that they may get shut down at any moment if AWS needs the extra capacity for On-Demand instances. For a while now it\u2019s been possible to receive a warning that your instance is about to be shut down, but until now the only option at that point has been to save all your data somewhere external before the instance is terminated. Now, Spot Instances support hibernation<\/a>, effectively saving the machine\u2019s full state to EBS so it can be seamlessly resumed when the Instance is re-launched, rather than being forced to stash your data somewhere external and retrieve it once the instance is restored.<\/p>\n We love T2 instances at GoSquared. We\u2019ve written about them<\/a> in the past. They\u2019re an ideal instance type for low-CPU or very spiky workloads. Out of the box you get a certain amount of CPU usage for a T2 instance, which allows you to maintain a specific low utilisation on average, or the occasional burst of high-CPU for intensive workloads. The new T2 Unlimited<\/a> feature allows you to extend the amount you\u2019re allowed to burst by borrowing future CPU credits up to a day in advance, or charging on a millisecond-by-millisecond surplus basis.<\/p>\n Aurora is AWS\u2019s game-changing cloud-first implementation of MySQL and PostgreSQL. It\u2019s their fastest-growing product ever, and with the features it provides that\u2019s hardly surprising. And this year they\u2019ve added yet more features that may change the way we think about cloud-based relational databases:<\/p>\n Aurora has always supported scaling database reads by adding several read replicas – much like the old-school MySQL replication system, you can spool up as many replicas as you like and spread your read requests across them all to scale indefinitely. Previously however it wasn\u2019t possible to scale any writes to the database. With Aurora Multi-Master<\/a> that\u2019s now possible – you can spin up multiple write instances as well as read replicas to make Aurora fully horizontally-scalable.<\/p>\n This announcement didn\u2019t even make it onto AWS\u2019s announcement page for new products at re:Invent, but we think this is game-changing too. Aurora has supported cross-region replication for read replicas for a while, but in 2018 the service will be adding support for multi-region multi-master configurations. That means the database can scale reads and writes across multiple regions and provide availability that was previously unimaginable with just standard primary\/secondary replication.<\/p>\n Aurora Serverless<\/a> is a new option in the works for the MySQL version of Aurora which takes away a lot of the thinking required when predicting workloads for MySQL. Instead of having to choose a particular instance type and scale for a certain amount of capacity and headroom, Aurora Serverless effectively removes all the thought of scaling databases up and down based on usage and replaces it with a model similar to how ELBs currently work (there are still instances under the hood, but you only pay in terms of \u201ccapacity units\u201d used)<\/p>\n Aurora wasn\u2019t the only service to get a multi-region upgrade at re:Invent this year. DynamoDB now has support for Global Tables<\/a>, effectively replicating writes and eventually-consistent reads across multiple regions. Combine this with recent announcements such as increased runtime limits for Lambda@Edge<\/a> and it\u2019s now easier than ever to move your data closer to your users.<\/p>\n Amazon MQ<\/a> is a managed Apache ActiveMQ service that takes care of all the ops work needed to maintain a highly-available broker service. It\u2019s not a fully-managed scalable serverless service like SQS or DynamoDB; instead you still choose an instance type for deployment, which makes it a little more like ElastiCache or RDS. Still, since we use RabbitMQ internally here at GoSquared for a lot of its brokering\/routing power, so we\u2019re interested to see where Amazon MQ goes and how it can help take away that ops work for us.<\/p>\n Lambda functions can now consume up to 3008MB<\/a> of memory, which is great if you\u2019re using them for memory-intensive workloads such as big data analysis or processing large images.<\/p>\n Lambda has always had an invisible under-the-hood limit on the number of concurrent functions you can have running at any one time in your account. This limit has previously always been opaque and the only way to know that you\u2019ve run into it is when your functions suddenly stop working. Lambda now has new controls<\/a> over this limit, allowing you to view concurrency metrics and limit concurrency on a per-function level (so no single function can run away exhaust your account-level limit, for example)<\/p>\n We use Lambda all over the place here at GoSquared. One thing we\u2019ve always found tricky is how best to develop and test our functions. In an ideal world we\u2019d write completely exhaustive test suites, so we know that every function works perfectly before we roll it out. But in the real world, there\u2019s always something that your tests can\u2019t catch. Now with Lambda\u2019s new Traffic Shifting feature<\/a>, it\u2019s possible to gradually shift traffic from old to new versions of functions, making it easier to spot errors without having to do a full all-or-nothing deploy.<\/p>\n Much like the new Lambda deployment options, API Gateway now allows for better control of deployments<\/a> with gradual rollouts rather than all-or-nothing releases which allow little room for errors not covered by tests.<\/p>\n AWS has been making big strides in the field of AI and machine learning recently. We\u2019ve been involved in the announcements for a few of these so we\u2019re super-excited for what they entail:<\/p>\n DeepLens<\/a> is AWS\u2019s first deep-learning-enabled cloud-connected camera. It\u2019s primarily geared up to facilitate the teaching of machine learning. While it\u2019s not the sort of thing we\u2019ll be using directly here at GoSquared, we\u2019re impressed by AWS\u2019s ambition for this device to enable the future of machine learning education.<\/p>\n Comprehend<\/a>, Translate<\/a> and Transcribe<\/a> are AWS\u2019s new services for natural language processing, machine translation, and automatic speech recognition. These do pretty much exactly what it says on the tin, as a scalable and serverless API. We\u2019re excited to see what we can do with these to enable future features in Assistant.<\/p>\n SageMaker<\/a> is a new service which makes it easier than ever to get up and running with custom ML models. Previously there has been a lot of knowledge required in order to get up and running with training, optimising, deploying and scaling ML models, but SageMaker takes away a lot of that undifferentiated heavy lifting and makes it much easier to focus on what matters: working with models and data.<\/p>\n AWS has had features for logging activity in your AWS account for a long while: CloudTrail, network flow logs, and DNS logs have been available for a while. But unless you know what you\u2019re looking for, it\u2019s hard to spot suspicious activity. GuardDuty<\/a> takes on this role, automatically plugging into all your available logs and flagging any suspicious activity on your account.<\/p>\n AWS WAF is great for mitigating suspicious HTTP activity on ELBs, filtering out requests that match certain rules and preventing them from hitting your infrastructure. But you have to set up and maintain those rules for yourself and know what you\u2019re looking for. Now AWS have enabled sellers on their Marketplace to set up managed rules<\/a>, effectively rule templates which get continually updated, so you don\u2019t have to worry about keeping them maintained yourself.<\/p>\n When doing large-scale analysis across lots of data in S3 it\u2019s important to be efficient. But if for example you only need certain columns of data from a CSV file, or certain items in large JSON objects, your only option was previously to fetch the entire file and then filter out the data you\u2019re interested in. S3 Select<\/a> is an interesting new way of running that filtering entirely on the S3 side, drastically cutting down on the amount of data you need to shift back and forth. And even better, it\u2019s also available for data that\u2019s been cycled out to Glacier, which means dramatic saving on the amount of data you need to restore to S3\u2019s standard storage classes.<\/p>\n\n
Fargate<\/h2>\n
New Instance Types: M5, H1<\/h2>\n
New Spot Instance Options<\/h2>\n
RunInstances<\/code> command exactly like On-Demand instances and without a bid price, with all the pricing benefit previously afforded by Spot Instances. It\u2019s now just as easy to launch a Spot instance as an On-Demand; the only difference is that Spots are often around a tenth of the price (and they occasionally get shut down if the price fluctuates). There\u2019s also been a few changes under the hood to how the pricing system works so it\u2019s less likely to fluctuate wildly and shut down a bunch of your instances at once.<\/p>\nSpot Hibernation<\/h2>\n
T2 Unlimited<\/h2>\n
Aurora<\/h2>\n
Multi-Master<\/h3>\n
Multi-Region Multi-Master<\/h3>\n
Aurora Serverless<\/h3>\n
DynamoDB Global Tables<\/h2>\n
Amazon MQ<\/h2>\n
Bigger Lambdas<\/h2>\n
More Control over Lambda concurrency<\/h2>\n
New Lambda deployment options<\/h2>\n
API Gateway deployment options<\/h2>\n
AI<\/h2>\n
DeepLens<\/h3>\n
AI Services: Comprehend, Translate, Transcribe<\/h3>\n
SageMaker<\/h3>\n
GuardDuty<\/h2>\n
Managed WAF rules<\/h2>\n
S3 and Glacier Select<\/h2>\n
But wait, there’s more!<\/h2>\n