Understanding web site security updates - who should be doing them, and why?

This is a subject which raises itself regularly when you work in open source web sites, so we thought that a blog post on the subject might be useful for website owners. For a non-techie this might appear quite complex; perhaps it is, so please pay attention!

Firstly however let's look at the big picture.

If your web site was built using open source software it is because that offers a very wide range of functionality at a good price (often free), built and donated for free by a large community of developers, and which your web designer is then able to implement, style, customise, modify and otherwise massage to fit your unique web site requirements.

Had you gone the "other" route and chosen "closed source" or proprietary software, you would probably have either had to:

1. Pay more for it.
2. Accept less features.
3. Both of the above.
4. Get trapped in a relationship with a software vendor that you can never leave - without dumping your whole web site.

Now this is not to say at all that "open source good, closed source bad" or anything, but the world is increasingly finding that for SME businesses the value proposition of open source is very good. And so of course do the SMEs and individuals using it, particularly Joomla and WordPress, perhaps the two most popular CMS (Content managed sites) in the world.

But getting all those advanced web features at a price that you could afford does come at a price, and the price is particularly this:

With open source software everyone (who knows anything about web programming) can view the source code that runs the web site. This means that if there are any security vulnerabilities in it, they can be easily seen and then exploited by bad guys with a sufficient skill set. Now even closed-source sites get hacked, quite regularly, but with open source it is a little easier, because you can easily see the code. Another feature of open source, and the reason why you got all those great features and extensions for your web site, is that many of the add-ons are contributed by a wide community of developers. This is why the open source sites develop so fast and adopt the latest features so quickly, when compared to proprietary software. However the broad range of functionality submitted also implies a degree of variety in the coding practices, and a lack of uniformity in how security is built into each component.

Goodies and Baddies

So in the ongoing game of cat and mouse between the hackers and the web site developers, we have two sides:

The hackers, who for whatever reason seek to damage and deface your or anyone else's web sites. Sometimes web sites are selected specifically for what they represent or supposed ethical issues: the Pentagon, Vodafone, or this month's target of choice. At other times they are simply targeted at random, sometimes even by "bots"; automated programmes endlessly scanning the net, and looking for signals that a web site is vulnerable or out of date. How the sites are selected is of little consequence, if yours is found and attempt to exploit it is made, then it needs to be secure. The only thing you can be fairly sure of is that as your web site becomes more successful and popular, and more important to your business, the number of attempted attacks will usually increase. There is some safety in anonymity, but very little benefit to you in a web site that no visitor, customer, or hacker ever visits. The downside of this is that even if your new web site doesn't appear to be much of a target yet, it probably will become one if your work to increase its visitor numbers is successful.

On the other side of the battle are the software developers and web site owners, trying to develop good sites, good code, new features, and keep their businesses trading online and secure. Their part in this is to keep the sites as up to date and secure as they can, find a way of monitoring and blocking exploit attempts.

A key part of how this is managed is that each CMS software community has a team of people looking after security. Typically they will either find a weakness themselves, or be informed of one by a third party who identifies it, or perhaps when a web site is hacked, and the details of "how" are passed through to that security team, or bug squad. This triggers a bout of frantic activity, where the bug squad tests the flaw, develops an understanding of it, and then develops a fix to close the loophole. This fix then has to be migrated out to the whole community, if all the web sites are to be made secure. These "fixes" or updates do not come necessarily one at a time, but for web sites made up of a number of components, any number of them may be updated at any frequency. So this would include updates to the core version of the web site software (Joomla, WordPress, Drupal, Magento or any other). It could equally well apply to any extensions that have been added to the web site, perhaps a shopping cart, contact form, community chat area, photo sharing gallery, classified ad listing, directory, or any of the other add-ons that web site owners might choose to use.

So what you have is not a simple series of updates, but a fast moving and constantly changing mosaic of software updates, to be managed and applied to keep the bad guys out.

The Server

So let's take a short breath before plunging in again, next to look at the server on which your web site is hosted. If it is an open source web site then the chances are strong that it is hosted on an open source web server. This is because the web sites written often in the open source PHP programming language live most naturally on what is called a "LAMP" server, standing for:

• L - Linux the operating system (like Windows) that runs the machine. There are many flavours of Linux, (names like Debian, Redhat, Suse, Fadora, Ubuntu).
• A - Apache the web server software that handles incoming requests for your web pages, and sends the pages back to the visitor's computer.
• M - MySQL the database where the content of your web site is stored.
• P - PHP the language in which the web site is written, which gets the requested data from the (MySQL) database, and puts it together as a web page.

These technologies form the backbone of the web, serving up the majority of the world's web sites. They are supported by other related server software such as:

• OpenSSL (helps manage securely encoded pages like payment pages)
• FTP - Web developer sends his work up to the web server with this.
• C-panel - a control panel for managing all of the other software mentioned here.
• SSH - another way that web masters manage the web space.
• Email programmes - most web servers will have several.

And so it goes on, there are lots more pieces of software working together to make up your web server. Sometimes particular pieces of server software need to be installed during the build. These would be added into the server stack rather than the web site itself, but they provide services that the web site needs. Examples of this can include:

Your web site has a particular imaging requirement, such as images to be dynamically resized or manipulated by the web site itself. This may require something like "GD Library", or similar, which would need to be compiled into the server.

As web sites get larger they often benefit from improved caching, retaining copies of ready-made pages to be served when requested, which can save the server a lot of work making the pages up, and so reduce work on the server. For these to work efficiently sometimes needs a server extension to be added, and complied into the Apache/PHP stack.

And so it goes on. Aren't you glad you have a web developer and server manager to wrestle with this rather than trying to do it yourself?

External APIs

We'll bring in one more group of software, this is external APIs, where you call another web site for services, but then show the results on your own site. This can expose some amazing features for your site, and the concept of the "Mashup", involves bringing one or more external sources together to create something that is altogether new - perhaps greater than the sum of the parts. Typically used in SME web sites, these could include:

• Google Maps API, displays Google map on your site, with additional features such as map customisation, proximity search, route navigation, and so on.
• Social Widgets: Twitter, Facebook, LinkedIn and all the rest give you widgets, from simple "follow me" buttons, to large boxes displaying your friends or Social activity. These are provided by the API from that web property, and can give really powerful social functionality to your web site.
• Chat - live chat boxes are typically provided by a third party web site using an API.
• eCommerce sites often use external APIs, for example providing post code / address lookups, shipping quotes, live exchange rates, and more.
• CRM: If you have a CRM system, it can use an API to interact with your web site visitors and help provide a richer and better tracked user experience.
• Financial widgets: exchange rate and share price tickers showing on your site.
• Accounts packages - Sage, QuickBooks and others often have API features, which allow developers to import or export financial data, orders, and stock levels to a web site.
• Vertical Markets: such as estate agents, hotel bookings, yacht brokers, affiliate marketing schemes, which share data between multiple web sites.

And on and on, with the growth of the web there is almost no limit to the variety of extensions that might be included on your site.

So we have several major groups of software now:

1. The software which came on the server and runs the server itself.
2. Add-ons that may have been included in the server software to support particular web site features.
3. The software that was installed during the build of your web site.
4. Add-ons and extensions to that web site software.
5. Third party APIs called to provide particular rich features.

Any of the above could potentially constitute a security flaw, and even after months or years of successful running it is possible that a new flaw could be created or discovered, or the interaction between any two pieces of software could create a new and unintended flaw.

Who should update each one?

So now that we understand the types of component that work together to deliver your web site, it is time to look at who should be updating them, and the impact that this might have on your web site.

Updates to standard server software:

Normally on a shared server or VPS (virtual server) this would be maintained by your hosting company. This is often invisible to you and is done as-and-when required, though some updates might possibly require a quick re-start, many do not. If this is required a good host will give you plenty of notice, do it at a quiet time, and have a strategy in place to minimise any interruption of service. An update to a live server should be pre-tested on a separate environment before being rolled out to live, but should the roll-out have unintended consequences, the hosting company should have a plan to quickly roll-back to the last stable configuration, re-evaluate the update, and re-apply it once the problem has been ironed out.

However, where a server is used by a wide variety of smaller web sites, updates cannot always be carried out on a regular basis. This is because updates to the core server languages sometimes remove features that used to be required by a web site, perhaps because the feature has been replaced, or turns out to be insecure. So an update can cause a number of web sites to break instantaneously. This is one reason why shared hosting companies can be reluctant to apply updates, because they know that although it is in the interests of their clients security, they will also receive a raft of complaints from those very clients and web designers who are not interested in putting further money and time into keeping their web sites up to date when server software changes.

As your web site grows you will eventually want to move to your own server, either a virtual server or an actual physical server box. In this case the application of updates is down to whatever arrangement you have with a developer or server administrator to update your machine. You need to understand the terms of the dedicated server agreement, and whether the people you paid to initially provision and install the server also have an agreement in place with you to monitor it, and apply security updates as and when they become available.

One final note on dedicated servers. With a shared server it is in the providers interest to keep it up to date and secure, as they have many clients on it, and once it gets exploited they could receive complaints and issues from a large number of clients and web sites. However once your site is on a dedicated server, they provider has less interest in doing updates. Any exploits will affect only you and your business, no one else, so unless you have an agreement in place for updates and monitoring, there is no particular reason why the provider should take it upon themselves to do them for you.

Updates to non-standard server software:

This software was added to the server itself by your developer or the server managers during the process of developing the web site, in order to provide particular features that the web site code needed. With a simpler web site there will probably be none of these, but it is important to understand which libraries may have been added, in order that updates to them may be monitored. It is best to create a list of added libraries when the web site goes live, and that way whoever monitors the server will be able to check these periodically and see whether updates have been made available. Whether this is the provider of the server, or your web developer depends on the agreement between them, and you the web site owner.

With a shared server it is often not possible to add libraries to the core server software versions, so these changes are more likely to have been added when you have your own VPS (virtual server) or dedicated server.

Software core to your web site:

When the web site was built the first thing the developer did was to upload the current "core" version of the CMS you are using, be it Joomla, WordPress, Magento, Drupal or whatever you are using. Each of these has its own system for updates, and notifications which are sent out to subscribers whenever a new version becomes available. Some have several ways of doing it, some as part of the core software itself, some with an RSS feed and some using add-on extensions. Here are some options to start with:

• WordPress update notifications: http://codex.wordpress.org/Mailing_Lists#Announcements or http://codex.wordpress.org/Configuring_Automatic_Background_Updates
• Joomla update notifications: http://feeds.joomla.org/JoomlaSecurityNews
• Drupal update notifications: https://www.drupal.org/documentation/modules/update
• Magento update notifications: http://www.magentocommerce.com/wiki/modules_reference/english/mage_adminhtml/notification/index or http://www.magentocommerce.com/download/release_notes

Collective Security:

Most SME businesses host their web sites on servers hosting many others, and in these cases you will be reliant on the security levels of all the other web sites on that server to protect yours. What can happen on a shared server is that one web site, perhaps because of old scripts on that site can be exploited, and this impacts on the performance of the whole server - and your web site. There is quite a strong level of isolation between different accounts on each server, but there is never 100% isolation across different accounts on one machine. For example a DDOS (denial of service) attack on one web site will use up all the server's resources, and affect every other site on that machine. Similarly a number of web sites on a shared IP address will rely on a good "sender reputation" of that IP. If one web site is hacked and starts sending SPAM, then this will affect the sender reputation, and email deliverability, or all accounts sharing that IP address.

The concept of collective security then is that the manager of the server cannot allow a single site on that machine to become out of date, because this "bad neighbour" can jeopardise the performance and security of all the other "good neighbours" on the same server. If your business cannot possibly tolerate this risk then you need to either take on a dedicated or virtual server. Read and understand the T&Cs of your host to understand how they deal with this issue.

The process of "instant and compulsory updates for all sites on a shared server" is a good one, and subject to reaching agreement with every web site owner on the machine, is one that we as a company have come to prefer. The difficulty of course is that sites not accepting the cost of instant updates need to be removed from the machine for the common good, and as you can imagine this can occasionally be contentious.

Automated updates:

Many servers also offer automated updates, using systems such as "Fantastico", or "Softaculous":

• https://netenberg.com/fantastico.php
• http://www.softaculous.com

These can automatically install and in some cases update the core software version automatically. While these may sound like the answer to all your update problems they still need to be understood, because as usual the update when applied can break aspects of the web site. This becomes more likely with a complex and more customised system. Although the web sites all have ways of writing code which are supposed to be "update proof", not all developers understand or use them, and in some cases they cannot facilitate the type of modification required, and the "core code" needs to be modified to achieve the desired result. In these situations any kind of automatic updates cannot be used, and only a developer with the required skills can apply and test security updates.

Web site extensions: While the core version of the web site software has an established system to notify and in some cases automate updates, it becomes a bit more complex to apply the same system to every add-on (shopping cart, chat solution, directory, etc.). Most of the CMS systems have a control panel listing the add-ons that have been installed, and in most cases highlighting any which do not agree with the latest published version. Out of date extensions are highlighted. The onus is then on the developer or owner to apply any updates. While the application of updates can be quite simple, in most cases the click of a button, we have already covered the fact that updates can change or even break features on which you rely, so if you do this yourself you may end up with a broken feature in the web site. With simple sites it is usually a case of just clicking the "update" button, but as usual once your site contains more custom developed features, this is less likely to work, and a manual approach to putting in place these updates will be required.

Third party APIs. This group of code snippets are in the main supplied to you by a large web organisation; Google, Facebook or whoever. In most cases they are free of charge to you the web site owner, some start free but introduce charged options for larger sites and more advanced services, with the free options offered as a way of integrating you and your users ever closer in to the functionality of the web property itself.

Now the relationship here can be interesting, and because you have paid them nothing, they are similarly not obliged to you for anything. This can mean for example that a developer spends many months putting together some advanced functionality for you based upon Google Maps, or "Facebook user authentication" and then a week after your web site goes live, you find these features broken. During the night Google / Facebook has upgraded its API to a new version, and the code on which your web site depended is now all or partly defunct. The unfortunate fact is that you have no recourse against anyone else for this, and because they give you the service for nothing there is no contractual support relationship at all.

It isn't up to Facebook to pay for your next round of code changes - they offered you the code and all the functionality related to it free of charge, and as they say "He who pays the piper calls the tune". Similarly your developer built a working web site with the API version as it was, and then Google / Facebook changed it. Not the developers fault either. The trade-off that you accept when using these free features from the major web properties, is that they can offer you what functions they like, and change them when they like. If you have built a web site around the functions and it breaks, then bad luck. However as we often see this is a trade-off well worth making, because of the rich features that are offered to you by the API, and the benefit to your web site and your users.

So when the code is offered to you for free, it is best to remember that free really means - "You may have to spend money developing around this code, and then spend more when we change it". As long as this is understood then everyone should be happy, but if you start ranting at Facebook or Google when your API-based feature breaks, it will do little good.

The exception to all of the above is if you pay for a feature or code. At this point you are in some kind of contract with the provider and can expect some kind of SLA (service level agreement), as laid out in the small print of the very long user agreement that you or your developer accepted when you signed up to use the API. Although this may technically give you some entitlement to support, we all understand how the contract has been carefully worded, and the web giant still retains the ability to change the API at will, and the cost of any corrective changes to your web site will still be borne by you.

So how to structure a support agreement to make this work.

1. Remember to cover all these areas, and find out which sorts of codes have been used in the development of your web site.
2. Understand which updates your hosting company will do, and which your developer will do.
3. Understand that as the complexity of your web site grows, so will the number and variety of updates required.
4. Updates on more than just a simple "core code" web site will need to be done carefully, and manually.
5. Updates should be done regularly and promptly when they become available, but sometimes there is a reason to hold back while waiting for interlinked components or code to be ready to update at the same time.
6. It is hard to put a flat rate on the time or cost requirement for updates, as neither you nor the developer know how often these will come through in the future, or the implications for the features of the web site once they have been implemented.
7. Where your web site is business critical, it makes sense that security updates, like all other types of updates should be tested in a non-live environment first. For this reason the cost of maintaining a replica version of your site for testing is often well worthwhile. Most good developers will offer this, though lower cost and simpler sites often do not warrant the overhead in cost or complexity.

When planning to build a web site remember:

1. The more features you add to the site, the greater the ongoing cost of updates in the future. This will impact the ongoing costs to your company. When building a site it is often a case of "more is not better", and just developing every feature you can think of will not only increase maintenance costs, but can also slow the site down.
2. Put an indicative budget aside for this and other work after go live.
3. Good developers will help you plan for this eventuality, just ask them and find out.

This cost being understood, sites using open source software and third party APIs do offer a very strong value proposition. The advanced features (usually at a sensible cost) make them very attractive proposition for many businesses, overshadowing the unfortunate but usually relatively low ongoing cost of security updates.

We hope that this helps unravel the often complex subject of web site security, why it needs to be done and by whom. A new web site should have a long and productive life in your organisation, but it does need to be nurtured and looked after, this will hopefully shed some light on how that should be carried out.