As a special of the annual benchmark tests of the mobile networks in 2014, Focus Infocom conducted a test on the performance of popular apps in the German networks.

Test Setup

Focus Infocom performed the measurements at the beginning of August 2014 in the Rhine-Main area and in the south of North Rhine-Westphalia. The measurement system included a lightweight architecture with the latest Samsung Galaxy S5 smartphones and the SmartAppManager (SAM), Focus Infocom's newly developed technology for the direct control of smartphone apps.

SAM measurement system with new lightweight architecture

For the test, 35,000 test samples were collected by car and public transport over a distance of 1,700 kilometers.

Route of SAM benchmark test 2014

A special feature of the test was that we did not limit the measurements to testing popular apps like WhatsApp, Facebook and CHIP Online, but we also tested the platform's own SMS app as well as standard HTTP web browsing, upload, and download scenarios. This approach allowed us to compare the performance of SMS and WhatsApp as well as comparing App scenarios to standard scenarios.

The direct control of apps is much more complex than standard testing such as HTTP upload and download because an app usually allows for many different use cases. Therefore, the new fourth generation of network testing technology as represented by SAM does not only include test case control, but also a novel, highly flexible QoS KPI architecture.

Test Results

The results of the special measurements have shown that the winner of the Germany-wide benchmark, the Deutsche Telekom, is also top in the “App League”. The figure below shows some results for individual apps:

SAM benchmark results session times
SAM benchmark results success rates
SAM benchmark results mean data rates

For publication on CHIP Online, only a few of the available KPI were evaluated. With the SAM technology, however, it is not only possible to define the use cases for the service tests freely, but also to set the KPI definitions individually as required. The collected data - Android API data, GPS, IP trace, L3 protocol messages and others – allow for detailed evaluations, and provide extensive diagnostic information.

More Information

More information on the structure of the special test, the measurement results and the KPI used can be found in our SAM Benchmark White Paper, which you can request by e-mail. Or download a short presentation