Comments on: Mobile Device Detection Results

By: Steve Kamerman

Steve Kamerman — Mon, 08 Feb 2010 01:50:45 +0000

Hi John (and Luca and Alex), I see this post has attracted the lead developers and large clients . I just wanted to add some insight to this discussion, albeit a little late. As performance is the primary objective of Tera-WURFL, I have run repeated tests on all the WURFL APIs, both from the WURFL project and third-parties. I run my DB of 45000 unique UAs through the APIs to determine the uncached and cached performance. One secondary goal for Tera-WURFL is fast determination of mobile vs. non-mobile, so the DB includes both. I would like to know if during your tests of the WURFL PHP APIs you are reinstatiating the library between every iteration or simply running a new UA through the existing object. Due to the complex structure of the new WURFL PHP API, there is a significant overhead involved in instatiation (upwards of 40 PHP class files may be included, not counting the cache).
As the author of Tera-WURFL and a big supporter of the WURFL project, I spend a lot of time thinking about the process of device detection and the future of the mobile Internet experience. Here is what the focus should be right now: we need very accurate results *and* high performance! As Luca implied, it is worth the cost of a slower initial detection since the subsequent requsts will be served very quickly. Accoring to the data I have gathered from very large clients, the number of unique *mobile* user agents that hit your site are in an exponential decay. Some of my clients are getting tens of thousands of unique UAs within the first couple weeks, then the number of new UAs starts dropping drammatically. My recommendation is that you keep the accuracy of the WURFL project and evaluate your requirements for performance. I am biased towards Tera-WURFL, but the PHP WURFL API is also very good. I would be interested to see how Tera-WURFL would fare in your results. Version 2.1.0 will be released on February 10, 2010 and features a new high-speed mobile/non-mobile engine that shows a detection rate of over 300 unique non-cached UAs per second with a cached lookup speed of about 1000/sec on my laptop. I will probably conduct a test similar to yours and post it on http://www.tera-wurfl.com. Please let me know if you need a larger pool of UAs to test with.

By: Alex Kerr

Alex Kerr — Thu, 04 Jun 2009 15:09:59 +0000

In reply to Andrew Deal above, most people move straight to the MySQL implementation of WURFL, which is called TeraWURFL (unless they move to the new API), you’re right the ordinary WURFL XML parser is way too slow for many. http://www.tera-wurfl.com/

I just checked the DB that TeraWURFL automatically generates and it looks very well indexed to me – and performance matches that. So not sure what you are referring to?

In more general terms I am “betting the farm” on WURFL simply due to the years of input from all sorts of professional sources, and the fact it’s used to run many commercial systems, some very large. This counts for an awful lot in my book (and it’s free). Worth noting too that rival systems like Device Atlas in large part populate their DBs directly from WURFL.

By: Andrew Deal

Andrew Deal — Mon, 25 May 2009 18:00:41 +0000

I too noticed how slow the wurfl API was when i implemented it last year.. I moved to the MySQL version (forgot that exact name) and was appalled at the lack of indexing on the db.

i wish I had an extra 20-30 hours to set up my own db for managing wurfl data in MySQL… that way, I think I could add a lot of value to the tool for all to share.

By: Mikael Henriksson

Mikael Henriksson — Mon, 04 May 2009 16:49:53 +0000

I must fill in here. We are using WURFL. Currently the old API and speed is completely ok. I do device detection on average 25ms per device. Getting the right page etc from the database takes another 25ms.

I was trying out the new API and average detection speed was 0,5ms thanks to some zip-goodness and matching strategy improvements! Looking forward to the new .net api!

By: John Keith

John Keith — Tue, 28 Apr 2009 15:11:36 +0000

Hi Idel,

We currently have 641 UAs marked as mobile, using WURFL as the identifier. WURFL and Device Atlas both work well, with basically one difference around Opera Mini that is well understood and not an actual issue for us.

Our results are freely available here:
http://www.cloudfour.com/mobile/summary.php

– John

By: Idel Fuschini

Idel Fuschini — Tue, 28 Apr 2009 10:01:22 +0000

Hi,
about the 1580 UA you tested, how many are mobile ?
This is important to decide the best API that have you tested.
If I want to decide which API to use, sure I want the fastest but, also I need know which API detects the UA with more accuracy.

Idel

By: John Keith

John Keith — Mon, 27 Apr 2009 15:56:22 +0000

Hi Luca,

I probably will end up adding some memory-based caching to our test environment, though with our low volume it hasn’t been the highest thing on my priority list. Now, of course, it becomes quite interesting to make the comparison. Due to the way we constructed our initial test environment, we ended up with batch processing of unique, non-repeating UAs, but going forward the plan is to perform lookups when the connection is established. I expect that caching makes much more sense now.

The other thing we are trying to understand is what, if any, considerations need to be made by the many small website operators who are adding device adaption to their sites. Our experience with these people is that they often do not have enough control over their server environment, or perhaps lack the technical capability, to implement an effective caching strategy. This is why we are looking at the PHP APIs (most prevalent in our world), and why we are looking at constrained server environments as real world examples.

It’s probably true that beginners need to focus on functionality over performance, which would tend to push the performance aspects to the back burner. But we do see 1) some beginners that are high volume already and 2) people who have become successful with their mobile web application and are facing refactoring work to address subsequent performance issues. Server-side caching opportunities are part of that equation, as are improved techniques for delivering web content to mobile devices (our main focus at Cloud Four).

Cheers,
John

By: Luca Passani

Luca Passani — Sun, 26 Apr 2009 14:37:55 +0000

John, I have asked my team to look into this. I am curious to see the results we get.
After that, we will improve the web patch and re-run the test to see the difference.
If necessary, we can also update the matcher strategies used by the API and go for 100% coverage. My take is that this is probably not necessary, because the bulk of unrecognised UAs is probably to be found among those spiders and bots.

Question: are you telling me that, in your application, you cannot use any caching? why is that?

Luca

By: John Keith

John Keith — Sun, 26 Apr 2009 06:02:31 +0000

For anyone interested, here is a link to the user agents that have come to our mobile concurrency test:

In the order we processed them while testing:
http://www.cloudfour.com/mobile/user_agents.php

Or sorted by UA:
http://www.cloudfour.com/mobile/user_agents.php?sort=UA

Regards,
John

By: Luca Passani

Luca Passani — Sun, 26 Apr 2009 00:21:30 +0000

What wasn’t clear to me until you posted your answer to Chris was that you were talking about the PHP API. This changes a few things (caching works differently with PHP as compared to Java). I imagine that running the tests with MEMCache would not be very significant (all APIs would perform the same).

What is more interesting to me, is understanding why the new API isn’t cutting it for you (as compared to the old one) in terms of device detection.

I would be grateful if you could help us to get to the end of this. The new API has been released only recently, and it may need some fine tuning, yet I am confident that it will blow anything anyone may have had before out of the weeds in terms of precision and control eventually.

Cheers

Luca