Sounds good.... definitely an incentive for me to finish up the media UI for openMobile
I grabbed one artist - ~560 songs so that will be the test set. I'm also curious to see how many support more then ID3 tags but that will be for another test.
Does revFE use the mediaInfo library or something else for tags?
Windows and CF load the first time, then hibernation is used the rest of the time, CF is instant if it was left on before hibernation. I think that is what should be benchmarked.
HiJackX1 UAMCB w/ The Tobiathin Core Android/Win 7 hybrid system!
4x 10inch Tablet
1x Win 7 / Rear Entertainment PC
ft/ Web Server Streaming
There are a lot of other variables to take into account when doing these kinds of tests, I'd say the only reasonable way to compare is for the user to install each FE with the settings/skin and hardware that will be used, then compare the load times that way. On any of my machines for instance, just the loading of the .NET framework will add at least 7 seconds on its own (Because I don't let it pre-load on boot). The more hardware is configured in the FE, the longer it'll likely take to load. The higher quality the images on the chosen skin, the longer it will take to load (any FE). The more skin support a FE has, the longer it will likely take to load as well (which means any FE without a lot of skin features will likely load faster).
SOFTWARE IS INSTANT ON WHEN RESUMING FROM HIBERNATION - thats the whole point of hibernation. This is a benchmark of one of many different aspects of software-cf has a slow load time no excuses - i haven't finished the tests yet but i'm pretty sure it will be at the front of the music index times - every software has its strong and weak areas.
#2 Not true at all - read my post about FE's making proper use of threading above
#3 True... I'll post a table of Kb's of images loaded for each main menu screen from the different front ends-but from an initial look their all within about 15% of each other. Im comparing default skins - not any of the extras.
#4 Not really sure what you mean by "skin support".....although I feel like i disagree
hardware -- all that may affect how long it'll take to load.
#4 - By skin support I mean the following, back on the first versions of RR, it would load up increadibly fast (< 1s), because it only supported labels and buttons. Then once you start adding scrolling objects, lists, sliders, secondary images (album art, indicators), flash, etc, there's a lot more processing required to load a single screen than before.
My average playlist has 4000 songs. I think 2000 to 4000k songs would be great for testing.
I think how each FE implements GPS and internet/wi-fi plugins should also be tested (I mean, how it impacts start up times?
I have the impression that every time I do a fresh install, when I add GPS/Internet functionality, everything that was neat, fast and clean suddenly becomes slower ( and/or unreliable - but that's for a different thread). It's not like we have a lot of options there, so a "naked" FE testing is not very useful, IMHO.
While I thank you for your opinion i'm not really sure what your suggesting instead. Once you start factoring in load times of plugins, you are gauging not just how well the front end is designed but how well each of the plugins are designed. Then you need to take into account that there are several variations of each of the different plugins. It ends up being hundreds of benchmarks between the different FE's...and it then you open up all the arguments of "x plugin does xyz thats why its slower then y plugin"...etc.
Guino: Something that justchat_1 does, is he loads up the GUI and lets that display before loading everything else. Being as humans are only so fast, by the time you click through and hit a button to turn on your HD radio it will either already be loaded or (if he is doing it right) the command will be queued to be run once it's loaded, making the HD radio seem slow and not necessarily the FE. His is the only FE that does this, and as a design point it is outside the scope of this topic besides stating that it is there.
It's not just skin features, but also the design of the skinning system itself. An example: I wrote a RR skin loader for my frontend which initially only supported only three features; buttons, lists and labels. Each skin window took on average three to four times longer to load than an equivalent RevFE skin window. I haven't done any truly in depth testing to figure out why this is, but my initial investigations indicated it may be due to the fact the xml parsing is significantly faster than the text parsing that is required to load RR skins. Not to mention the fact that multiple types (Sliders and Lists for being the primary example I've run into) have very similar formats. Until I started using RegEx's, it was even slower, and I can't imagine it being any faster in RR due to the fact that VB's string comparisons are deathly slow in comparison to c++'s, though I haven't gotten a chance to browse through the old RR source and see how it's done.
If RR were to pre-load every screen like RevFE does, I'm sure it would take forever to load (I assume this was in line with the decision not to pre-load). If RevFE did like RR and NOT pre-load the screens, it would take less than a second to load. My point to all this is that while a simple "This FE loads in this time, and this one in this time..." is partially effective from an end user standpoint, it is not an entirely equivalent measure. there is a lot more going on under the surface which may be necessarily to take into account to provide any truly objective timing of frontends. It will become a key arguing point about your numbers.
Note that I said timing of a frontend, and not just "load time". On the surface, you measured exactly what you said you were going to measure and I commend you for that. I'm just nitpicking where this is going What would be REALLY interesting (albeit impossible) would be to run a profiler on each program's source.