The 'dirty laundry' posts frodo mentioned have been edited / deleted from this thread so we can get back to the topic at hand. I apologize to frodo and anyone who might have read them before they were removed.Originally Posted by nicgalla
Can you give me some details about the hardware and operating system of your pc? Sorry if I've asked you this before, but I've read and committed to memory a huge mass of information regarding this project, and memorizing your hardware / software config is akin to me memorizing what color socks each of you have on.
I have 2000 odd files taking up around 10 gigs on my 1.2 ghz celeron laptop running a very pristine install of WinXP with 256mb ram and a 5400 rpm 20gb hitachi hdd. It takes about 1.2 seconds for the database refresh code to list all of them and compare the count to the number in the database. I can launch M/E and before I can get to the Media Browser window the database is reconciled and ready to go. When I've added or removed files it takes closer to 90 seconds to page through the database and do what is needed.
Obviously it will take longer if you have:
a) a slower cpu
b) a slower hard drive
c) more files to list
d) less memory
I'm not totally against turning off auto refresh as a user option, but I'd first like to make the refresh process as bullet-proof as possible while acting in the role I designed it to work in. Granted we have more condensed collections and most likely faster machines that we test on, I'd rather spend time finding out why folks are seeing the database build crash and burn and fix those problems before I enable them to just bypass the problem at the expense of lost functionality to other / future users of MediaEngine who don't understand or consider the implications of enabling the option. 9 out of 10 of the problems I've located inside the database refresh code traced back to a specific file with some unforseen structure in the filename and no ID3 tag, or non-standard ID3V2.X tag layouts. Since we can't ask everyone to just send us their collections, this necessitates a difficult (but imo worthwhile) session of testing and troubleshooting. We went to great pains to make it work before release on some very diverse and sizeable collections (ask James aka cloudy about his collection), but you just can't anticipate and compensate for every possible problem you're going to run into when you enlarge your test base by several orders of magnitude.
Another area that I've focused on is the file sync function, and I've heard next to no feedback about it other than one bug related to the sync drive being offline when you start the sync process. This function will sychronize your database while it copies files from your source drive to your media folder (or at least it's supposed to). In the near future I would like to make this process run automatically and unattended so it can be used in conjunction with 802.11 wireless networks as well as flash memory cards and usb drives. This will for the most part negate the need to reconcile the database count at startup, but not completely.
>>It all boils down to knowing when the user has added, deleted, or modified the files in their collection.<<
To put it in a nutshell, we wanted this process to run automatically without user intervention for very good reasons. Anyone who's done something similar has contemplated the possible scenarios and made a tradeoff decision based on the data and their personal opinion. The way we did it is a the result of a decision based on our personal opinions of how we felt it should work, taking into consideration where we would like the project to go in the future and what we've experienced in the past. I hope you can all understand my sticking to my guns on this. I won't deny that there's some pride at stake, but there are many, many other factors involved. I'm very much interested in hearing differing viewpoints. Collecting a bunch of 'my database takes forever too' posts in this thread is not the productive conversation that I'm going to spend my attentions on.
So please post some of those thoughts (the simple solutions) so we can hear your viewpoint and add them to the thought process.Originally Posted by nicgalla
A side note: for what it's worth ID3V2.X tags should theoretically read faster since they are stored at the beginning of the mp3 file. V1.X tag data is stored at the end, necessitating a seek inside the file. Although this takes a relatively miniscule amount of time, it adds up.