Thread: FrontEnd Benchmarks (Load times and Indexing speed)

Originally Posted by justchat_1

I was expecting that to be the first post to be honest lol.....but hey anyone who wants to check the data feel free - I think we would all benefit from data from multiple setups.

That said the reason it loads so fast (and wont get any slower as more plugins are added) is due to the architecture. Most of the front ends load and initialize every plugin before the first screen is shown. This means they get slower and slower as you add functionality.

Open Mobile (and i though nGhost either does or was planning to) does it differently...only the first screen is initialized on the main thread....everything else initializes in the background. I really don't understand why commercial applications (aka streetdeck and centrafuse) have no idea how to use threads properly. RevFE I know loads each plugin on a separate thread and im sure if it did prioritized loading it could be in the same range or even better (given its c++ roots).

I'm just busting your stone by the way. It did get me to play with open mobile though, so I suppose it was effective advertising.

Originally Posted by csfile

thanks for this...i hope CF3 is quicker than CF2

CF has alot of stuff to load up.... Plus in my opinion load times do not matter. Most of us use hibernation so once Windows and CF load the first time, then hibernation is used the rest of the time, CF is instant if it was left on before hibernation. I think that is what should be benchmarked.

Originally Posted by HiJackZX1

CF has alot of stuff to load up.... Plus in my opinion load times do not matter. Most of us use hibernation so once Windows and CF load the first time, then hibernation is used the rest of the time, CF is instant if it was left on before hibernation. I think that is what should be benchmarked.

that is very tru.

wen my pc resumes from hibernation i hear music playing a while before my screen turns on.
thats all i care about

There are a lot of other variables to take into account when doing these kinds of tests, I'd say the only reasonable way to compare is for the user to install each FE with the settings/skin and hardware that will be used, then compare the load times that way. On any of my machines for instance, just the loading of the .NET framework will add at least 7 seconds on its own (Because I don't let it pre-load on boot). The more hardware is configured in the FE, the longer it'll likely take to load. The higher quality the images on the chosen skin, the longer it will take to load (any FE). The more skin support a FE has, the longer it will likely take to load as well (which means any FE without a lot of skin features will likely load faster).

Originally Posted by HiJackZX1

CF has alot of stuff to load up.... Plus in my opinion load times do not matter. Most of us use hibernation so once Windows and CF load the first time, then hibernation is used the rest of the time, CF is instant if it was left on before hibernation. I think that is what should be benchmarked.

I thought that was a joke the first time i read it....ALL SOFTWARE IS INSTANT ON WHEN RESUMING FROM HIBERNATION - thats the whole point of hibernation. This is a benchmark of one of many different aspects of software-cf has a slow load time no excuses - i haven't finished the tests yet but i'm pretty sure it will be at the front of the music index times - every software has its strong and weak areas.

Originally Posted by guino

There are a lot of other variables to take into account when doing these kinds of tests, I'd say the only reasonable way to compare is for the user to install each FE with the settings/skin and hardware that will be used, then compare the load times that way. On any of my machines for instance, just the loading of the .NET framework will add at least 7 seconds on its own (Because I don't let it pre-load on boot). The more hardware is configured in the FE, the longer it'll likely take to load. The higher quality the images on the chosen skin, the longer it will take to load (any FE). The more skin support a FE has, the longer it will likely take to load as well (which means any FE without a lot of skin features will likely load faster).

#1 True - the .net framework loading on startup is a good point that I missed.
#2 Not true at all - read my post about FE's making proper use of threading above
#3 True... I'll post a table of Kb's of images loaded for each main menu screen from the different front ends-but from an initial look their all within about 15% of each other. Im comparing default skins - not any of the extras.
#4 Not really sure what you mean by "skin support".....although I feel like i disagree

Originally Posted by justchat_1

#2 Not true at all - read my post about FE's making proper use of threading above
#4 Not really sure what you mean by "skin support".....although I feel like i disagree

#2 - Threading can only do so much, you still have to load code/objects/data for controlling the hardware -- all that may affect how long it'll take to load.
#4 - By skin support I mean the following, back on the first versions of RR, it would load up increadibly fast (< 1s), because it only supported labels and buttons. Then once you start adding scrolling objects, lists, sliders, secondary images (album art, indicators), flash, etc, there's a lot more processing required to load a single screen than before.

My average playlist has 4000 songs. I think 2000 to 4000k songs would be great for testing.

I think how each FE implements GPS and internet/wi-fi plugins should also be tested (I mean, how it impacts start up times?

I have the impression that every time I do a fresh install, when I add GPS/Internet functionality, everything that was neat, fast and clean suddenly becomes slower ( and/or unreliable - but that's for a different thread). It's not like we have a lot of options there, so a "naked" FE testing is not very useful, IMHO.

Originally Posted by guino

#2 - Threading can only do so much, you still have to load code/objects/data for controlling the hardware -- all that may affect how long it'll take to load.
#4 - By skin support I mean the following, back on the first versions of RR, it would load up increadibly fast (< 1s), because it only supported labels and buttons. Then once you start adding scrolling objects, lists, sliders, secondary images (album art, indicators), flash, etc, there's a lot more processing required to load a single screen than before.

Thats what I thought you meant.... Well yes and no-yes that obviously does effect performance. But no I see that as a design issue... poorly coded FE's that hack together a variety of sources to get a working GUI will suffer a performance penalty. Other front ends that design their skinning engines from the ground up don't suffer the same penalty. I'll be more then happy to post a table of supported skin features (although that will take some input from the various FE forums to gather) but I think its at the users discretion if they think having flash support is worth an extra 5 seconds load time (just an example).

RipplingHurst:
While I thank you for your opinion i'm not really sure what your suggesting instead. Once you start factoring in load times of plugins, you are gauging not just how well the front end is designed but how well each of the plugins are designed. Then you need to take into account that there are several variations of each of the different plugins. It ends up being hundreds of benchmarks between the different FE's...and it then you open up all the arguments of "x plugin does xyz thats why its slower then y plugin"...etc.

Guino: Something that justchat_1 does, is he loads up the GUI and lets that display before loading everything else. Being as humans are only so fast, by the time you click through and hit a button to turn on your HD radio it will either already be loaded or (if he is doing it right) the command will be queued to be run once it's loaded, making the HD radio seem slow and not necessarily the FE. His is the only FE that does this, and as a design point it is outside the scope of this topic besides stating that it is there.

It's not just skin features, but also the design of the skinning system itself. An example: I wrote a RR skin loader for my frontend which initially only supported only three features; buttons, lists and labels. Each skin window took on average three to four times longer to load than an equivalent RevFE skin window. I haven't done any truly in depth testing to figure out why this is, but my initial investigations indicated it may be due to the fact the xml parsing is significantly faster than the text parsing that is required to load RR skins. Not to mention the fact that multiple types (Sliders and Lists for being the primary example I've run into) have very similar formats. Until I started using RegEx's, it was even slower, and I can't imagine it being any faster in RR due to the fact that VB's string comparisons are deathly slow in comparison to c++'s, though I haven't gotten a chance to browse through the old RR source and see how it's done.

If RR were to pre-load every screen like RevFE does, I'm sure it would take forever to load (I assume this was in line with the decision not to pre-load). If RevFE did like RR and NOT pre-load the screens, it would take less than a second to load. My point to all this is that while a simple "This FE loads in this time, and this one in this time..." is partially effective from an end user standpoint, it is not an entirely equivalent measure. there is a lot more going on under the surface which may be necessarily to take into account to provide any truly objective timing of frontends. It will become a key arguing point about your numbers.

Note that I said timing of a frontend, and not just "load time". On the surface, you measured exactly what you said you were going to measure and I commend you for that. I'm just nitpicking where this is going What would be REALLY interesting (albeit impossible) would be to run a profiler on each program's source.