FrontEnd Benchmarks (Load times and Indexing speed)

justchat_1 · 09-20-2009, 07:32 PM

Load Times:
Ok as discussed in another thread there seemed to be some disagreement about various front end load times and one of the biggest problems was that different hardware was used. So to help out I have benchmarked load times of the various windows front ends on exactly the same setup.

The test rig:
Windows 7
Dell Inspiron 1520
500GB 7200rpm hard drive
t7500 2.2Ghz Core2Duo
2.0GB Ram

The Test:
Each front end was installed cleanly with no extra plugins...was configured and then started and run 3 times to ensure all the initial config was complete. All other processes on the system were stopped and the program was started by a custom test app. When the UI was visible and complete (Ride runner took an extra second to load button captions) I pressed F10 and the timer would stop. Each of the three runs was listed along with their average.

The Results: (All times in seconds)

Now obviously with different hardware the results will vary but I would imagine the spread would be the same on most setups.

Media Index Speeds:
The next item that was highly debated was media index speeds. This was a test to see how long it would take to index your media collection.

The test rig:
-Same as above-

The Test:
Each Software was cleanly installed ensuring it started with a brand new database. It was then directed to index a collection of 500 mp3s and the amount of time from start to finish was recorded. For two of the applications, the speeds were pretty quick so the applications were traced while running to ensure measured times were accurate. Each application was then directed to empty its database and the test was repeated another two times. As a final test each front end was told to index 10 corrupt files in an attempt to crash them. All the front ends successfully skipped the corrupted files without a problem.

The Results:

Analysis:
For two of the front ends the first run was significantly longer then the second two. This is believed to be due to overhead from database creation and expansion. File indexing and pre-fetching may have also helped decrease times for the second and third runs.

The software behind the indexing:
FE Tag Reader Database
StreetDeck Windows Media Player Windows Media Player Database
Centrafuse Bass.Net Jet DB
RoadRunnerMM MediaInfo SQLite (external exe)
RevFE MediaInfo SQLite (dll)
OpenMobile Optimized TagLib-Sharp SQLite (managed code)

csfile · 09-20-2009, 07:50 PM

thanks for this...i hope CF3 is quicker than CF2

malcom2073 · 09-20-2009, 07:57 PM

While you're at it... you should benchmark the overall speed of the frontends. Screen changes, media loading, id3 scanning etc.

justchat_1 · 09-20-2009, 08:31 PM

I can try hooking the windows messages and checking for UI idle but any of the front ends that do screen changes in the background would show times of 0 so that would be a tough one.

Media loading is possible...im guessing you mean amount of time between when you click a song and hear sound out of the speakers? or something else?

ID3 tagging is a good one too...what size collection do you think I should use? I have about 500,000 songs and don't really want to wait for that to be indexed 3 times per front end lol.

50 songs? 500? 2000?

Machinehead · 09-20-2009, 08:46 PM

Interesting coincidence that Open Mobile the front end the thread starter developed is the fastest loading of the 4 tested, or am I just a conspiracy theorist....

malcom2073 · 09-20-2009, 09:01 PM

Quote: Originally Posted by Machinehead

Interesting coincidence that Open Mobile the front end the thread starter developed is the fastest loading of the 4 tested, or am I just a conspiracy theorist....

It also has nowhere near the features or functionality of the other ones yet. RevFE loads in under 100ms without any functionality (plugins) too

justchat_1 · 09-20-2009, 09:18 PM

Quote: Originally Posted by Machinehead

Interesting coincidence that Open Mobile the front end the thread starter developed is the fastest loading of the 4 tested, or am I just a conspiracy theorist....

I was expecting that to be the first post to be honest lol.....but hey anyone who wants to check the data feel free - I think we would all benefit from data from multiple setups.

That said the reason it loads so fast (and wont get any slower as more plugins are added) is due to the architecture. Most of the front ends load and initialize every plugin before the first screen is shown. This means they get slower and slower as you add functionality.

Open Mobile (and i though nGhost either does or was planning to) does it differently...only the first screen is initialized on the main thread....everything else initializes in the background. I really don't understand why commercial applications (aka streetdeck and centrafuse) have no idea how to use threads properly. RevFE I know loads each plugin on a separate thread and im sure if it did prioritized loading it could be in the same range or even better (given its c++ roots).

malcom2073 · 09-20-2009, 09:26 PM

Quote: Originally Posted by justchat_1

RevFE I know loads each plugin on a separate thread and im sure if it did prioritized loading it could be in the same range or even better (given its c++ roots).

Interestingly enough, the threading of the plugins is advantageous to them while they run, not while they start. I load them synchronously, this is intentional. That's why I wanted you to test the other things I listed, because that's where RevFE excels. I'll bet it also uses more ram than any of the other FE's. That reminds me... do a ram and hard disc comparison too!

justchat_1 · 09-20-2009, 09:59 PM

Quote: Originally Posted by malcom2073

Interestingly enough, the threading of the plugins is advantageous to them while they run, not while they start.

Very true but my point was that it could be advantageous to both. If they are loaded synchronously theres no reason the main ui cant be rendered at the same time instead of waiting for them to load.

Get back to me on my second post so I know how many songs to use and such for the other benchmarks... and yea i'll do a memory use comparison but drive space would be tough since it really depends on what skin is used and such. I'll always have you beat on that though :P

malcom2073 · 09-20-2009, 10:07 PM

Quote: Originally Posted by justchat_1

Very true but my point was that it could be advantageous to both. If they are loaded synchronously theres no reason the main ui cant be rendered at the same time instead of waiting for them to load.

Get back to me on my second post so I know how many songs to use and such for the other benchmarks... and yea i'll do a memory use comparison but drive space would be tough since it really depends on what skin is used and such. I'll always have you beat on that though :P

Of course you will, .net binaries are always super tiny

I'd say 500 songs is a good test to get an average speed.

For media loading, I mean both the time inbetween clicking a new file and hearing sound, as well as the time to load a playlist (of a set size, say 20-50 songs).

justchat_1 · 09-20-2009, 11:01 PM

Sounds good.... definitely an incentive for me to finish up the media UI for openMobile

I grabbed one artist - ~560 songs so that will be the test set. I'm also curious to see how many support more then ID3 tags but that will be for another test.

[off topic]
Does revFE use the mediaInfo library or something else for tags?

Machinehead · 09-21-2009, 09:48 AM

Quote: Originally Posted by justchat_1

I was expecting that to be the first post to be honest lol.....but hey anyone who wants to check the data feel free - I think we would all benefit from data from multiple setups.

That said the reason it loads so fast (and wont get any slower as more plugins are added) is due to the architecture. Most of the front ends load and initialize every plugin before the first screen is shown. This means they get slower and slower as you add functionality.

Open Mobile (and i though nGhost either does or was planning to) does it differently...only the first screen is initialized on the main thread....everything else initializes in the background. I really don't understand why commercial applications (aka streetdeck and centrafuse) have no idea how to use threads properly. RevFE I know loads each plugin on a separate thread and im sure if it did prioritized loading it could be in the same range or even better (given its c++ roots).

I'm just busting your stone by the way. It did get me to play with open mobile though, so I suppose it was effective advertising.

HiJackZX1 · 09-21-2009, 09:58 AM

Quote: Originally Posted by csfile

thanks for this...i hope CF3 is quicker than CF2

CF has alot of stuff to load up.... Plus in my opinion load times do not matter. Most of us use hibernation so once Windows and CF load the first time, then hibernation is used the rest of the time, CF is instant if it was left on before hibernation. I think that is what should be benchmarked.

trinybwoy · 09-21-2009, 10:35 AM

Quote: Originally Posted by HiJackZX1

CF has alot of stuff to load up.... Plus in my opinion load times do not matter. Most of us use hibernation so once Windows and CF load the first time, then hibernation is used the rest of the time, CF is instant if it was left on before hibernation. I think that is what should be benchmarked.

that is very tru.

wen my pc resumes from hibernation i hear music playing a while before my screen turns on.
thats all i care about

guino · 09-21-2009, 10:36 AM

There are a lot of other variables to take into account when doing these kinds of tests, I'd say the only reasonable way to compare is for the user to install each FE with the settings/skin and hardware that will be used, then compare the load times that way. On any of my machines for instance, just the loading of the .NET framework will add at least 7 seconds on its own (Because I don't let it pre-load on boot). The more hardware is configured in the FE, the longer it'll likely take to load. The higher quality the images on the chosen skin, the longer it will take to load (any FE). The more skin support a FE has, the longer it will likely take to load as well (which means any FE without a lot of skin features will likely load faster).

User Name		Remember Me?
Password