FrontEnd Benchmarks (Load times and Indexing speed) - Page 2

justchat_1 · 09-21-2009, 11:01 AM

Quote: Originally Posted by HiJackZX1

CF has alot of stuff to load up.... Plus in my opinion load times do not matter. Most of us use hibernation so once Windows and CF load the first time, then hibernation is used the rest of the time, CF is instant if it was left on before hibernation. I think that is what should be benchmarked.

I thought that was a joke the first time i read it....ALL SOFTWARE IS INSTANT ON WHEN RESUMING FROM HIBERNATION - thats the whole point of hibernation. This is a benchmark of one of many different aspects of software-cf has a slow load time no excuses - i haven't finished the tests yet but i'm pretty sure it will be at the front of the music index times - every software has its strong and weak areas.

Quote: Originally Posted by guino

There are a lot of other variables to take into account when doing these kinds of tests, I'd say the only reasonable way to compare is for the user to install each FE with the settings/skin and hardware that will be used, then compare the load times that way. On any of my machines for instance, just the loading of the .NET framework will add at least 7 seconds on its own (Because I don't let it pre-load on boot). The more hardware is configured in the FE, the longer it'll likely take to load. The higher quality the images on the chosen skin, the longer it will take to load (any FE). The more skin support a FE has, the longer it will likely take to load as well (which means any FE without a lot of skin features will likely load faster).

#1 True - the .net framework loading on startup is a good point that I missed.
#2 Not true at all - read my post about FE's making proper use of threading above
#3 True... I'll post a table of Kb's of images loaded for each main menu screen from the different front ends-but from an initial look their all within about 15% of each other. Im comparing default skins - not any of the extras.
#4 Not really sure what you mean by "skin support".....although I feel like i disagree

guino · 09-21-2009, 02:18 PM

Quote: Originally Posted by justchat_1

#2 Not true at all - read my post about FE's making proper use of threading above
#4 Not really sure what you mean by "skin support".....although I feel like i disagree

#2 - Threading can only do so much, you still have to load code/objects/data for controlling the hardware -- all that may affect how long it'll take to load.
#4 - By skin support I mean the following, back on the first versions of RR, it would load up increadibly fast (< 1s), because it only supported labels and buttons. Then once you start adding scrolling objects, lists, sliders, secondary images (album art, indicators), flash, etc, there's a lot more processing required to load a single screen than before.

RipplingHurst · 09-21-2009, 02:25 PM

My average playlist has 4000 songs. I think 2000 to 4000k songs would be great for testing.

I think how each FE implements GPS and internet/wi-fi plugins should also be tested (I mean, how it impacts start up times?

I have the impression that every time I do a fresh install, when I add GPS/Internet functionality, everything that was neat, fast and clean suddenly becomes slower ( and/or unreliable - but that's for a different thread). It's not like we have a lot of options there, so a "naked" FE testing is not very useful, IMHO.

justchat_1 · 09-21-2009, 07:36 PM

Quote: Originally Posted by guino

#2 - Threading can only do so much, you still have to load code/objects/data for controlling the hardware -- all that may affect how long it'll take to load.
#4 - By skin support I mean the following, back on the first versions of RR, it would load up increadibly fast (< 1s), because it only supported labels and buttons. Then once you start adding scrolling objects, lists, sliders, secondary images (album art, indicators), flash, etc, there's a lot more processing required to load a single screen than before.

Thats what I thought you meant.... Well yes and no-yes that obviously does effect performance. But no I see that as a design issue... poorly coded FE's that hack together a variety of sources to get a working GUI will suffer a performance penalty. Other front ends that design their skinning engines from the ground up don't suffer the same penalty. I'll be more then happy to post a table of supported skin features (although that will take some input from the various FE forums to gather) but I think its at the users discretion if they think having flash support is worth an extra 5 seconds load time (just an example).

RipplingHurst:
While I thank you for your opinion i'm not really sure what your suggesting instead. Once you start factoring in load times of plugins, you are gauging not just how well the front end is designed but how well each of the plugins are designed. Then you need to take into account that there are several variations of each of the different plugins. It ends up being hundreds of benchmarks between the different FE's...and it then you open up all the arguments of "x plugin does xyz thats why its slower then y plugin"...etc.

malcom2073 · 09-21-2009, 07:56 PM

Guino: Something that justchat_1 does, is he loads up the GUI and lets that display before loading everything else. Being as humans are only so fast, by the time you click through and hit a button to turn on your HD radio it will either already be loaded or (if he is doing it right) the command will be queued to be run once it's loaded, making the HD radio seem slow and not necessarily the FE. His is the only FE that does this, and as a design point it is outside the scope of this topic besides stating that it is there.

It's not just skin features, but also the design of the skinning system itself. An example: I wrote a RR skin loader for my frontend which initially only supported only three features; buttons, lists and labels. Each skin window took on average three to four times longer to load than an equivalent RevFE skin window. I haven't done any truly in depth testing to figure out why this is, but my initial investigations indicated it may be due to the fact the xml parsing is significantly faster than the text parsing that is required to load RR skins. Not to mention the fact that multiple types (Sliders and Lists for being the primary example I've run into) have very similar formats. Until I started using RegEx's, it was even slower, and I can't imagine it being any faster in RR due to the fact that VB's string comparisons are deathly slow in comparison to c++'s, though I haven't gotten a chance to browse through the old RR source and see how it's done.

If RR were to pre-load every screen like RevFE does, I'm sure it would take forever to load (I assume this was in line with the decision not to pre-load). If RevFE did like RR and NOT pre-load the screens, it would take less than a second to load. My point to all this is that while a simple "This FE loads in this time, and this one in this time..." is partially effective from an end user standpoint, it is not an entirely equivalent measure. there is a lot more going on under the surface which may be necessarily to take into account to provide any truly objective timing of frontends. It will become a key arguing point about your numbers.

Note that I said timing of a frontend, and not just "load time". On the surface, you measured exactly what you said you were going to measure and I commend you for that. I'm just nitpicking where this is going

What would be REALLY interesting (albeit impossible) would be to run a profiler on each program's source.

justchat_1 · 09-21-2009, 08:22 PM

Couldn't have said it any better....

[Off Topic]
Funny you mentioned profiling everyones source...i've been sitting with a profiler for the last two days now optimizing here and there. I also did the first bit of benchmarking on media loading (of course i'll have to do it again so all tests are in the same session) but results were interesting:
Street Deck (2.0) Centrafuse (2.1) Open Mobile (0.5.0.3)
66.40 16.90 69.64
33.78 7.10 52.25
32.36 6.97 52.74

RideRunner I haven't gotten working yet (ill be posting a help request soon) and revFE im waiting for one minor tweak. Needless to say I know the area thats getting profiled next

[Back on topic]
I'm noticing some differences between what each of the front ends actually do when they index. It would be nice to get together a list of what fields they each read and store...along with how they handle cover art (folder images only, id3 cover extraction only or both).

rsrblade · 09-21-2009, 11:18 PM

Just to throw another wrench into the topic...One has to keep in mind that each individual systems' hardware may also help or hinder FE load times; (single vs dual vs quad core, amount, type, speed of memory, internal sound vs external (usb), screen res, etc). It would be next to impossible to obtain anything other than generic benchmarks due to these obstacles. Great idea, be not totally feasible, imho.

justchat_1 · 09-21-2009, 11:25 PM

Quote: Originally Posted by rsrblade

Just to throw another wrench into the topic...One has to keep in mind that each individual systems' hardware may also help or hinder FE load times; (single vs dual vs quad core, amount, type, speed of memory, internal sound vs external (usb), screen res, etc). It would be next to impossible to obtain anything other than generic benchmarks due to these obstacles. Great idea, be not totally feasible, imho.

This has been stated a few times - the goal was exactly that - a generic benchmark.

Centrafuse is not going to suddenly go to a 1sec load time on a 5400rpm hard drive or less ram.... It's believed that the spread will be similar on other hardware just generally longer or shorter but a single core benchmark is coming soon -so well know for sure.

kev000 · 09-22-2009, 06:16 PM

no nGhost in the benchmark?

Having the .NET runtime preloaded is a big benefit to openMobile. the .NET runtime does add to the system startup time though. I'm actually surprised that revFE was around 2 seconds. I thought it might be faster.

nGhost2 can operate in 3 modes: passive, aggressive, and progressive. In passive mode, no images are cached in memory. This makes nghost lighter, but also slower as each screen has to load from disk all the images. In contrast, aggressive mode loads ALL images into memory at startup. This makes screen switching FAST and but it also starts up slightly slower. Progressive mode is the default. It loads screens into memory when you go to them. The result is, nGhost get's progressively faster yet saves memory by only caching the screens you visit.

nGhost also loads all plugins at the start. But there are only a handful of plugins or two that are likely being loaded.

nGhost3 will start 1 plugin and you'll launch the other's yourself (or you'll preconfigure nGhost to start up whatever plugins you like. hmmm... session saving...

). All plugins will run out of process yet maintain an IPC link back to the "server" --a background daemon that manages plugins.

Anyway, it'd be interesting to see how fast media scans are of the various plugins. nGhost can certainly improve this. Currently, all scanning is done out of process via nscan. It runs in the background allowing you do browse music while it searches for new stuff. it uses multi-threading on each scan, but with some pretty awful mutex locking (IMHO). I'd probably be better as a fork()'d process instead of a thread...

malcom2073 · 09-22-2009, 06:37 PM

To be honest, I think Linux opens up a whole other can of worms in this discussion. Linux version, the setup of the distro, install options, up to dateness of the drivers all can have a huge effect on the load time of different programs. Who's to say what distro, kernel version, driver revision, etc you are going to benchmark this stuff to?

kev000 · 09-22-2009, 06:52 PM

Quote: Originally Posted by malcom2073

To be honest, I think Linux opens up a whole other can of worms in this discussion. Linux version, the setup of the distro, install options, up to dateness of the drivers all can have a huge effect on the load time of different programs. Who's to say what distro, kernel version, driver revision, etc you are going to benchmark this stuff to?

it doesn't really matter, as long as you are comparing apples to apples and the base operating system version is documented (ie, "nghost was tested on stock ubuntu 9.04".

What'd be even more interesting is to compare revFE and OpenMobile on windows AND Linux and compare their startup times.

10 bucks says at least revFE starts up faster on Linux than it does on windows. OpenMobile probably wont' because mono is much slower on Linux than .NET is on windows.

justchat_1 · 09-22-2009, 08:48 PM

Quote: Originally Posted by kev000

it doesn't really matter, as long as you are comparing apples to apples and the base operating system version is documented (ie, "nghost was tested on stock ubuntu 9.04".

What'd be even more interesting is to compare revFE and OpenMobile on windows AND Linux and compare their startup times.

Its a completely different file system and memory management...so no comparing windows to linux but I will be posting a comparison of nGhost, OpenMobile and revFE all done on linux to compare those 3. I want to get the windows stuff finished first....but if you wanted to run a comparison of the 3 before I get to it I would be happy to add it to the first post.

kev000 · 09-23-2009, 11:46 AM

i tried out openmobile on my work machine (ubuntu 9.04). I got some pretty inconsistant results. Sometimes I would get a blue-screen, sometimes i would get black sections that the buttons are supposed to be, and sometimes I would get the full screen.

Name: screenshot1.jpg
Views: 87
Size: 34.1 KB

Name: screenshot1.jpg
Views: 87
Size: 34.1 KB

Name: screenshot2.jpg
Views: 83
Size: 18.6 KB

I like how you fade in/out the window. You'll have to show me how you do that.

justchat_1 · 09-23-2009, 03:16 PM

Quote: Originally Posted by kev000

i tried out openmobile on my work machine (ubuntu 9.04). I got some pretty inconsistant results. Sometimes I would get a blue-screen, sometimes i would get black sections that the buttons are supposed to be, and sometimes I would get the full screen.

Attachment 56556
Attachment 56557

I like how you fade in/out the window. You'll have to show me how you do that.

very strange that you would get the full screen and the black boxes. There was a case issue with one of the skin images that didn't show up till it was running on linux. Just rename halfButton.png to HalfButton.png. Its been fixed in the svn for awhile now but I haven't gotten 0.5 to the release point yet so its going to be a few. If you do end up trying to do real benchmarks remind me to send u the version that doesn't fade in the controls so its easier to tell when things are loaded.

justchat_1 · 09-23-2009, 04:01 PM

Media Index Benchmark has been added to the first post:
Since the Open Mobile edition used in this test hasn't been released the music database was added to a sample POC application available for public testing. Available here for anyone that want's to do fact checking:
http://www.mp3car.com/vbulletin/open...c-preview.html

Also available are the trace logs from revFE...pm me if anyone wants to verify those.

User Name		Remember Me?
Password