How to Evaluate PaaS Services

Friday, December 30, 2016 by Rainer Stropek

From Algorithms to Libraries to Services

Besides working on our flagship product time cockpit, we help customers and partners to build their own cloud-based Software-as-a-Service (SaaS) solutions. Currently, we are involved in a SaaS project in which IoT plays an important part. Looking back on the work I have been doing in this project for the last few weeks, it strikes me how much my work as a developer and software architect has changed since we made the move to cloud and Platform-as-a-Service (PaaS).

Decades ago, software development for me meant primarily figuring out algorithms and optimizing them because of resource constraints. Over time, class libraries became more and more powerful. The open source movement and the web brought a myriad of ready-made components and frameworks. They reduced the need for inventing new algorithms to those parts of a software product that is really unique. For anything else (e.g. UI components, cross-cutting concerns) you can always find an existing, decent library. Software development became to a notable extent complexity and dependency management. At least in my domain of work, software architecture became increasingly important compared to figuring out complex algorithms.

Nowadays, my job has changed again. Finding libraries and frameworks is still important but another question has become determining: Is there a PaaS service for my problem? PaaS allows us to focus on the core added value that our products provide. We can invest more time into what makes them special instead of dealing with commodity topics. Specifically, PaaS means for us...

...no setting up and maintaining servers
...somebody else makes sure the services are up and running
...we get SLAs (ideally financially backed)
...somebody cares for updating the service without us having to do anything
...cost savings because the service provider benefits from scaling effects

However, using PaaS is also a great commitment. If the PaaS service goes down, our own service is usually negatively influenced. In the worst case, it can mean a downtime for our users. If you embed a component in your software and run it yourself, you can act if something bad happens. If you are using PaaS, you have to trust the service provider that they fix the issue fast. Therefore, it is crucial to carefully evaluate services before betting the farm on them. In this blog post I share some tips about how we evaluate PaaS offerings before we make them part of our own solutions.

Evaluating PaaS Offerings

1. First, you have to find the services!

The cloud is full of great services and it is really hard to keep up to date. My most important strategies for that:

Spend time reading news sites, newsletters of major PaaS providers, follow great bloggers, etc. BTW - I love feedly for that.
Visit industry conferences
Participate in user group meetings and other community events

Remember: Communities are not a one-way street.

You can get valuable knowledge from them but you also have to share your own insights. Start a blog, share code on GitHub, speak at community events, host user group meeting - it's fun and makes sense.

2. Does it offer the functionality we need?

Once discovered, we have to find out whether the service offers what we need. Though this one seems obvious and simple, it often isn't.

Quality of documentation isn't always perfect (e.g. incomplete, covering just basic use cases, outdated). You have to invest time and money building prototypes to evaluate the offered functionality.
If some details are missing, you have to study roadmaps or even get in direct contact with the service provider. Do they plan to develop the missing features? Did the already publicly commit to delivering them? Credible ETA available?

Tip: Many PaaS providers publicly discuss requests for new features on platforms like Uservoice (e.g. Microsoft Azure Feedback), in GitHub issues (e.g. Telerik Kendo UI) or simply in Twitter conversations. Use these channels as a source of information and to give active feedback.

Does the service offer extensibility mechanisms so that we can add the missing features on our own?
Can/should we adapt our requirements so that the PaaS offering becomes usable? Is the advantage of using a ready-made service big enough to justify a tradeoff?

Never decide about using a PaaS offering based just on marketing material. Study the documentation closely or - even better - verify your assumptions with prototypes.

3. Do we trust the provider?

Using a 3rd party PaaS offering is different to using an external component. You do not only depend on the supplier's ability to write great code. You also rely on their skills regarding running a secure and reliable service. Here are the indicators that influence my trust in service providers:

Does the company have a successful track record providing PaaS offerings?
Do they "dogfood" (i.e. use the service for their own business like Microsoft using DocumentDB for millions of users of OneNote)?
How active is the community around services of the provider (e.g. user groups, blogs, tweets, books, forums, etc.)?
Size and financial situation
Do I personally know people that I trust who work there or who have been using the service successfully? What do they say?
Does the provider communicate openly (e.g. blogs, participate in open source projects, public issue management, public feedback portals, white paper with behind-the-scenes information, etc.)?
Does the provider have external certifications (e.g. ISO)?
Does the provider make it easy to start business (e.g. no up-front, long-term commitment necessary, free or nearly free pricing tiers for getting started, etc.) and easy to stop it (e.g. self-service data export features)?

4. Is the serviced offered where I need it?

Especially in Europe, regional aspects matter.

Does the provider offer the service in European data centers?
Does the provider at least care for Europe (e.g. publish information about European data protection conformity) or do they just focus on other parts of the world?
Does the provider offer services in the same data centers that you are in (e.g. at mLab, you can select the public cloud - Azure, AWS, Google - you want them to host a MongoDB for you)? That might be important for security and performance reasons.
Are your terms of service regarding the location of data storage, data processing, support, etc. compatible with the terms of the provider (support following the sun vs. no access to data from outside our own region)?

This is not just a technical matter. Especially in large companies, you might need to get your lawyers involved on this point.

5. Is the pricing model compatible with our business model?

Building a sustainable business model for SaaS in the cloud is all about keeping your costs per subscription under control. Finally, you have to earn more money per subscription than you spend for development and running your stuff in the cloud.

Evaluate the price of PaaS offering you consider by checking whether it is compatible with your pricing and business model.

Here are some questions you should ask yourself:

If your customer base grows, does the costs of the service grow faster than your revenue?
Does the costs per subscription eat up too much of your margin per subscription?
Can you get long-term fixed prices (consider exchange rate risks, too) that are compatible with the pricing terms you offer your customers?

6. Is the service built on industry standards?

Every PaaS offering comes with a certain lock-in effect. You should ask yourself how difficult it would be to move away from the service. Don't get me wrong: I am not saying that you have to avoid lock-in by all means. If the service you select works perfectly fine, it will be of great benefit for your business. However, being prepared for the unexpected is never bad. What if the provider or the service goes out of business. Globally acting service providers in the cloud are likely to not go away from one day to the other. Yet, large companies sometimes stop PaaS offerings because e.g. they don't make enough money from it. At least you should think about what this would mean for your SaaS solution.

Industry standards can help. They make it easier to switch.

Here are some examples illustrating what I mean:

Azure DocumentDB databases can be used as the data store for apps written for MongoDB. If this service would be stopped (I definitely do not expect that to happen any time soon), you could switch to another MongoDB PaaS offering.
Azure Service Bus supports the AMQP protocol. There are other services in the cloud supporting this protocol, too. Therefore, you have options just in case...
Azure Active Directory (AAD) supports the OpenID Connect (OIDC) prototcol. If AAD does not work for you any longer, you can replace it with another OIDC-compatible component.

7. Does the provider offer SLAs?

If you have to sign service level agreements (SLAs) with your customers, you might want SLAs from the PaaS offerings you use internally, too. Just like with the costs issue, it is a question of compatibility. Will you get enough compensation from a service provider if their failure led to you having to pay your customers a fine for a downtime? If not, can you and do you want to take over the remaining risk? Could it become life-threatening for your business?

Financially backed SLAs from PaaS providers are not only a kind of reinsurance. They have other advantages, too. Here are some examples:

SLAs from underlying cloud services are great for your marketing brochures. "Our service runs on database clusters with an SLA of 99.95%", this probably sounds good for your customers.
Maybe the SLA from your service provider is designed so that you will never get much money from it (e.g. compensation capped at 25% of the service costs, no matter how long an outage lasts). However, it puts pressure on the service provider. It probably hurts, if they have to pay 25% of their revenue to all their customers because of an instable service.

Don't decide on high SLA numbers in marketing brochures. You have to read the fine-print, too!

8. Are data protection terms sufficient?

Nowadays, data is a valuable asset for every company in the SaaS and PaaS business. Companies want to have access to as much data as possible to fine-tune their algorithms e.g. using machine learning. At the same time, data protection becomes increasingly important for customers. You have to make sure that the PaaS provider's view on data protection fits to your own approach. You will probably not use an ad-funded PaaS offering if your customers are sensitive concerning the use of their PII (personally identifiable information).

9. How new is the service?

Using a brand-new PaaS offering is a two-edged sword. On the one hand, the service might provide exceptionally new abilities that you can use to differentiate SaaS solution from your competitors. On the other hand, the service might never really take off. In that case, it could happen that the provider shuts down the service or significantly changes it (e.g. technical aspects, pricing, availability limitations). Here are some of my thoughts when considering brand new services:

Do I trust the provider (see above)?
How important is the service for my business? I would be hesitating making a premature service a major cornerstone of my SaaS solution.
Can I get long-term availability contracts and SLAs?