We thought we'd share some information on our testing process.
My name is David Blooman, I am the principal tester for responsive news, all questions and queries can be sent to me at @dblooman
When testing the devices, there are 4 main areas to cover, feature phones, smart phones with low level support, smart phones and tablets. The primary tablet market will be 7-inch devices up to 720x1280 pixels with the lowest smart phone being 240x360 pixels.
Currently, we are only approaching devices that are not conforming to desktops, so smart TV's, games consoles and other exotic devices are test, desktops are not.
We do test desktop browsers, Chrome, Firefox, IE, however we typical test the version which our stats point to, or when there was a feature update. As some browsers use auto updating services, it can often be a case of waiting for the next release to switch testing to a new version.
When testing on a desktop, the hardware does play an important part. The pixel density of large desktop screens, which is typically low, means testing on a desktop screen may prove fruitless, given that any CSS issues may be masked. When testing for mobile, images may also appear over blown or scaled incorrectly, given the larger screen.
The final part of our per-requisite list is cutting the mustard - blog post
Minimum Device Testing
- Symbian S60 - N96 (Symbian 9.3)
This phone represents the lowest of our requirements, mobile barlesque, our header, is 240px wide, meaning that our lowest support width is the same. The Nokia N96 is a great test case to identify the impacts to other dependencies within a webpage. This phone is also widely used, given the low cost nature of the OS(open source Symbian), as well as having a strong user base in our stats. The OS also supports opera mini/mobile for comparing a non JS/JS experience
- Android 2.2 Native Browser- Galaxy S
This phone is our minimum Android OS support for smart phones, this is based upon the fact that it cuts the mustard in our tests. While the user base for this OS is dropping due to more recent versions of Android, 4.0 etc. This devices is a good hardware point to test against as well. It features a 1ghz CPU, which was less optimised when 2.2 was launched, meaning that 2.2 on a slower CPU may produce a smoother experience. With manual testing, we have discovered a consistency across Android in that, if it works on 2.2, it has a strong chance of working on later versions. Android 2.2 is the best browser to test against for maximum stability across the Android user base, from 2.1 to 2.3.3
- iOS 6 Native Browser - iPhone4S
This browser has taken most of the iOS user base, being stable and having a lot of the more advanced features of HTML5. The browser produces a huge amount of web traffic globally, so testing this is a must for any large commercial website.
- Blackberry OS 6 - Bold 9700
BBOS6 is not the latest version of the OS, but does feature a near identical browser as 7, with minor improvements seen in 7. Even so, the uptake of 7 is light, so the focus should be on 6. This is for 2 reasons, 1, BBOS6 supports a Webkit browser, 2, the browser cuts the mustard. What is important to note about BBO6 is the Curve range. The curve range is a budget range of phones with lower report rates or resolutions. The 9300 for example, has a strong user base, but it also is a good indicator of how the smartphone experience is impacted by a screen with a small vertical height. For this reason, testing on a 9300 could give you a skewed result, the same for only testing on the 9700. If you are only testing features that have no layouts and CSS changes, the 9700 is the better of the two, but testing on the 9300 will give you an idea of the budget uses experience.
- Nexus 7
Android 4.1 was launched on the most popular Android tablet, the Nexus 7. With 4.1 came Chrome for Android by default, this represents a shift away from the native browser, which many Android OEMs customise. By having a consistent browser across all devices, Google makes testing Chrome easier too. More than that, the Nexus 7 size, resolution and pixel density make it a solid device to test tablet and phone layouts.
- Device of 240 pixels wide screen - N95
- Device of more than 1000 wide screen - Any device
- Touch Screen - Any device will be relevant, this is to test hit states, button placements and general usage while using a device with on screen keyboard.
- Non-touch - In most cases, Symbian and blackberry phones will fall into this category, how the user can navigate the site is of importance, so testing the functionality of the site with such devices is a high priority.
It is recommended that you clear all cookies, cache and reset browser to default before beginning any testing. Not doing this can lead to older versions of the site interfering with testing as you always must look at the latest version of the site. With builds occurring all the time, minor changes will often show little impact, such as back end work. Front end work can mean changes to the site that will fundamentally affect testing, as the look can be radically different on certain devices. You should always consider this when working, there is no such thing as a minor change when testing responsive web pages.
For further testing, the following list identifies the most common Mobile OS's. Below is explanation of the differences between the OS's and browsers, this could help you determine whether you testing need to extend to all the devices.
The Android 2.3 OS has the the largest user base of Android, therefor the most users on that versions browser. We use the Galaxy S as 1, it is one of the most popular devices in our stats, 2, it has a typical size screen and pixel density for that OS, meaning a more balanced approach when doing UI testing.
The Bold 9700 is again, one of the more popular devices, but it also represents a crossover device from OS5 to OS6. It's screen is of a good pixel size for Blackberrys and its internals are one of the more high specifications for a feature phone.
Windows phone is a good way of testing IE10, as a large amount of the code base was used to create the browser on WP8, though mobile has less features.
- iOS 6 - iphone 4
- Android 2.2 - HTC desire
- Android 2.3 - Samsung Galaxy S
- Android 4 - Both stock and Chrome browsers
- Blackberry OS 5 - Bold 9700
- Blackberry OS 6 - Bold 9700
- Blackberry OS 7 - 9380
- Windows Phone 8 - Lumia 920
- Symbian S60 - Symbian Browser
- Symbian S40 - Nokia Browser
- Nexus 7
- iOS 6 - iPad
Browser and OS Analysis
While emulators are very useful in a lab style environment, they often miss things that are only found when using handsets. There is also a lot of nuances that you will discover when you use this many devices. Below is a combination of personal experience, information from other sources and my opinion on the direction when testing responsive design.
iOS. The iphone and ipod range from the 3Gs upwards all support iOS5, which has support for most standards, the browser on iOS5 is the same on every device, therefore testing on 1 device is suitable. The second most supported version of the OS which supports a slightly lower performing browser is iOS4, however this has not performed differently except in hardware (larger screen) Here is a comparison of the 2 versions http://www.blaze.io/mobile/ios5-top10-performance-changes/ - The range of iOS devices represents nearly 20 million page views per month, hence it's high priority status.
Android 2.1 is the lowest version of Android to cut the mustard. Android has varying levels of support, Android 2.2 browser and onwards has been the most consistent, with Android 2.2 and 2.3 being the highest user base. Android 2.1 has the ability to support many smart phone features, but lacks some font embedding and other technologies, such as Flash. This has proven to be one of the more problematic devices, but is considered a baseline phone for smart phone support. This particular version of Android is in decline as the jump from 2.1 to 2.2 is being made, or the user upgrades to a different phone. Android 2.1 represents about 25% of the Android user base with a million page views per month.
Android 2.2 was the last version to be open source, so a lot of devices used this version for Google TVs, phones, music players etc. The browser for 2.3 is also based on this version. Differences in the operating system can have some impact on the browser, but the difference between 2.2 and 2.3 is mostly down to how the particular vendor changes the phone. A change log is listed in the link below http://www.knowitsdifference.com/difference-between-android-2-2-and-android-2-3/ Android 2.2 is a measurable jump in performance over 2.1, the user base is almost 3 times as big with nearly 8 million page views per month. Android 2.3 offers an overall system update that is relevant to user, not as much for browsers though. In our test, there have been less that 1% of difference between the 2, with only 1 or 2 bugs discovered not to replicate on both versions. The 7 million combined users with 20 million page views shows us how important it is to correctly test on Android 2.x versions.
Android 4 is the newest and most advanced version of the OS, it currently supports 2 browsers, Chrome and the Android browser. It seems that Google intend to replace the stock browser with Chrome on upcoming devices, so testing on both until this is that case is prudent. The current version of Chrome uses a new version of Webkit than the stock android browser. Info on browser http://www.android.com/about/ice-cream-sandwich/. What it is important is the that Chrome is a Google product, therefore it may not be present on the open source version of the OS, so for future testing, stats will determine which webkit version and by extension, which browser is the most popular on the platform. Android 4.1 brought a few improvements over 4.0, better HTML5 video support, supports the updated HTML5 Media Capture specification on input elements.WebView now supports vertical text, including Ruby Text and other vertical text glyphs.
Android 4.1 had a few tweaks to the browser, but in reality, the focus is now on Chrome as the stock browser faces heavier competition. Chrome Should be seen as an equal in testing to the stock browser.
Blackberry OS 5 is a pure feature phone experience, it's screen size support is one of the biggest factors, given the high landscape resolution, but low vertical. The browser has been the most consistent for feature phones, supporting all the requirements.
Blackberry OS 6 has patchy support, it uses a webkit browser, but the support for font embedding has been hit and miss. The browser is currently one of the highest usage of the website, meaning that this is important OS for testing. Newer versions of the Blackberry browser seem to have fixed the issues we are having with 6. Browser Info http://crackberry.com/blackberry-6-review - new-browserhttp://crackberry.com/blackberry-6-review#new-browser One of the core issues with Blackberries is the fact that they have unusual screens. The majority of phones sold between 2008 and 2011 have a 480x360 screen; this is intended for reading emails instead of web pages. What should also be tested is what the screen reports itself as, on some devices the resolution is dropped to 360 wide due to hardware limitations. It is important to identify issues with the look and feel on a webpage based upon these types of devices, especially for us with more than 4 million users and 16 million page views.
Windows Phone 7 uses the same code base as Internet Explorer 9, in a lot of cases, phone issues replicate onto the desktop. If there are issues on the mobile OS, testing on IE9 could be the fastest way to identify the issue, the mobile version is essentially a stripped down version of the desktop browser, as such, don't expect high levels of support for technologies. The lack of font embedding is of a particular annoyance, for this reason, IE9 and WP7 fall into smartphone with low level support category. http://en.wikipedia.org/wiki/Windows_Phone_7.5 Windows Phone 8 will launch with a new, IE10 based browser with much better HTML5 support, at time of writing, it still is being developed. Windows phone 7 is now a deprecated platform, with not more updates coming, as usage goes down, so will the need to support this browser.
Windows Phone 8 has an IE10 based browser with much better HTML5 support, at time of writing, it still has some issues. IE 10 has a large bug with viewport tags, this leads to hardware pixels being detected rather than software. There is a fix in the works from Microsoft, no timeline on that though. IE 10 offers a much better support listed here, some of the things IE10 mobile can't support are Inline video, Some new manipulation views APIs for touch panning and zooming, with the exception of –ms-touch-action, Multi-track HTML5 audio (simultaneous), ActiveX and VBScript, Drag-and-drop APIs, File access APIs with the exception of blobs which are supported on Windows Phone 8.
Symbian 10 is also known as Symbian Belle, a new brand from Nokia. This comes with a brand new browser which is Webkit based and supports lots of good HTML5 features. There a few notable issues, such as font embed support, but it brings a few Nokia/Symbian phones into the realm of HTML5, which is a good thing. Check out my post here for more info
Silk Browser. Amazon has forked Android 2.2 to create the Kindle OS with Silk browser. This browser routes all traffic through Amazon servers and can affect some elements of the website. Form submission is something that should be thoroughly tested to ensure the server is able to receive the form data. You may also find that because of the form factor, you will have to make a decision, only use smartphone layouts or use both tablet and smartphone layouts, this is mostly due to the screen width being able to support tablet web pages. But what is most troubling for the tester is that you may have to test both of these layouts on the say device, this is going to really annoy you.
Silk Browser version 2 is a fork of 4.0 with the AOSP browser. This browser has had some changes for the cloud sync, but has the same support as Android 4 stock browser.
The iPad is currently the market leader, with the single device having 60% of the market, the market grows every month though, so the user base is ever growing. While the differences to the iPad browser are subtle to the iPhone browser, it is important to look at the iPad as the biggest device, in terms of screen size, that users will visit with. While this device can support most, if not all, desktop sites with ease, it is important to remember that speed and simplicity are often desired more than functionality. For this reason, users of iPads and other tablets, will use the mobile version of the site, especially if they are using a data service. If it is outside of the scope of your project, you should consider potential markets with the iPad, responsive news' largest device support is 7 inches, however testing is conducted on the iPad.
The next section will include information about the browsers that you will need to test in addition to the native browsers we have already talked about. Just to reiterate, all the operating systems we have talked about so far have their own native browser.
Third party browsers that have big impact in the world include:
UC Browser - Information
Chrome Mobile - Information
Opera Mini/Mobile - Information
Firefox - Information
As we are test for users all over the world, UC browser and Opera Mini may be less important to you. These 2 browsers combined represent a sizeable chunk of Africa and Asia browsing traffic, so if you are in that region, you are probably already testing on it. Firefox is growing, but they are updating rapidly, so testing on the latest stable build is advisable, especially as it is more than likely the user will upgrade. The complete custom build is only available on Android. Chrome is not yet the default browser, it will also sit alongside the native Android browser, at least for now. Testing on both is probably the best approach, similar to Firefox, testing the latest build is best. There is a version of Chrome for iOS, but this uses the same version of Webkit as the version of IOS you are on. The Android version of Chrome is update frequently and is the only place to get the complete Chrome build.
In total, you final device list could have other smaller markets included, our current device list looks something like this :
- iOS 6 - iPhone
- iOS 6 - iPad mini
- iOS 6 - iPad
- iOS 5 - iPad
- iOS 5 - iPod
- Android 2.1 - HTC desire
- Android 2.2 - Samsung Galaxy S
- Android 2.3 - Samsung Galaxy S
- Android 3.2 - Motorola Xoom
- Android 4.0 - Galaxy Nexus (Stock and Chrome)
- Android 4.1 - Galaxy Nexus (Stock and Chrome)
- Blackberry OS 4 - Curve 8520 (We have recently dropped testing this browser)
- Blackberry OS 5 - Bold 9700
- Blackberry OS 6 - Bold 9700
- Blackberry OS 7 - Bold 9860
- Windows Phone 7.5 - Lumia 800
- Windows Phone 8 - Lumia 920
- Symbian S60 - N95
- Symbain S60 - N8
- Symbian S40 - C3
- Symbian S40 - Asha 303
- Kindle E-Reader
- Kindle Fire - Android 2.2 Fork
- Kindle Fire HD - Android 4 Fork
- Android 3.0 - Samsung 7.7
- Android 4 - Samsung 7.0 v2
- Android 4.2 - Nexus 7
- IE 10 - Microsoft Surface
Edit 20/6/2012 - We have recently determined our 3 in market tablets to look at, Kindle fire, BB Playbook and Galaxy Tab 7.7. They are all webkit based browsers, but have some differences in support, as well as volume of sales. The 7.7 uses a pixel ratio of 1, whereas the other 2 dont, this means that the break points will have to be correctly established for tablets in order for users not to receive broken features.
Edit 29/8/2012 - We are no longer testing on BBOS4 anymore, we have found the share to drop below our support level and global usage has dropped to an all time low. We have added other versions of Android and other hardware sets, such as 7 inch tablets.
Edit 13/9/2012 - I have made some big changes to the testing list this month, we have changed the minimum level of support to start at Android 2.2 and BBOS6, this is due to total browser market shifts on the site as well as globally and within the UK.
Edit 13/11/2012 - I have now updated list to include changes to BBOS7, IE 10 on Windows and some Android and iOS support changes.
Edit 16/01/2013 - I have updated Windows phone to indicate our new support list, as well as changing our testing devices