Page 2 of 2 FirstFirst 12
Results 11 to 20 of 20

Thread: Feature Request (Do not automatically update db)

  1. #11
    I'm sorry, and you are....? frodobaggins's Avatar
    Join Date
    Jan 2003
    Location
    Ruston, LA
    Posts
    8,846
    Quote Originally Posted by nicgalla
    this is no dirty laundry, but an interesting discussion.

    On my system it takes about thirty seconds to refresh the database. I have 6170 files arranged in 422 folders. Every file has only ID3V1 tag.
    Unfortunately ME2.0 always refresh the db, even if you haven't touched your library.
    I use hibernation so it's not a big problem but I think there are many simple solutions to improve this.
    I was referring to the other stuff, not the request I made.
    [H]4 Life
    My next generation Front End is right on schedule.
    It will be done sometime in the next generation.
    I'm a lesbian too.
    I am for hire!

  2. #12
    Maximum Bitrate phat_bastard's Avatar
    Join Date
    Sep 2002
    Location
    St.Louis, MO
    Posts
    549
    Quote Originally Posted by nicgalla
    On my system it takes about thirty seconds to refresh the database. I have 6170 files arranged in 422 folders. Every file has only ID3V1 tag.
    Unfortunately ME2.0 always refresh the db, even if you haven't touched your library.
    I use hibernation so it's not a big problem but I think there are many simple solutions to improve this.
    The 'dirty laundry' posts frodo mentioned have been edited / deleted from this thread so we can get back to the topic at hand. I apologize to frodo and anyone who might have read them before they were removed.

    Can you give me some details about the hardware and operating system of your pc? Sorry if I've asked you this before, but I've read and committed to memory a huge mass of information regarding this project, and memorizing your hardware / software config is akin to me memorizing what color socks each of you have on.

    I have 2000 odd files taking up around 10 gigs on my 1.2 ghz celeron laptop running a very pristine install of WinXP with 256mb ram and a 5400 rpm 20gb hitachi hdd. It takes about 1.2 seconds for the database refresh code to list all of them and compare the count to the number in the database. I can launch M/E and before I can get to the Media Browser window the database is reconciled and ready to go. When I've added or removed files it takes closer to 90 seconds to page through the database and do what is needed.

    Obviously it will take longer if you have:
    a) a slower cpu
    b) a slower hard drive
    c) more files to list
    d) less memory

    I'm not totally against turning off auto refresh as a user option, but I'd first like to make the refresh process as bullet-proof as possible while acting in the role I designed it to work in. Granted we have more condensed collections and most likely faster machines that we test on, I'd rather spend time finding out why folks are seeing the database build crash and burn and fix those problems before I enable them to just bypass the problem at the expense of lost functionality to other / future users of MediaEngine who don't understand or consider the implications of enabling the option. 9 out of 10 of the problems I've located inside the database refresh code traced back to a specific file with some unforseen structure in the filename and no ID3 tag, or non-standard ID3V2.X tag layouts. Since we can't ask everyone to just send us their collections, this necessitates a difficult (but imo worthwhile) session of testing and troubleshooting. We went to great pains to make it work before release on some very diverse and sizeable collections (ask James aka cloudy about his collection), but you just can't anticipate and compensate for every possible problem you're going to run into when you enlarge your test base by several orders of magnitude.

    Another area that I've focused on is the file sync function, and I've heard next to no feedback about it other than one bug related to the sync drive being offline when you start the sync process. This function will sychronize your database while it copies files from your source drive to your media folder (or at least it's supposed to). In the near future I would like to make this process run automatically and unattended so it can be used in conjunction with 802.11 wireless networks as well as flash memory cards and usb drives. This will for the most part negate the need to reconcile the database count at startup, but not completely.

    >>It all boils down to knowing when the user has added, deleted, or modified the files in their collection.<<

    To put it in a nutshell, we wanted this process to run automatically without user intervention for very good reasons. Anyone who's done something similar has contemplated the possible scenarios and made a tradeoff decision based on the data and their personal opinion. The way we did it is a the result of a decision based on our personal opinions of how we felt it should work, taking into consideration where we would like the project to go in the future and what we've experienced in the past. I hope you can all understand my sticking to my guns on this. I won't deny that there's some pride at stake, but there are many, many other factors involved. I'm very much interested in hearing differing viewpoints. Collecting a bunch of 'my database takes forever too' posts in this thread is not the productive conversation that I'm going to spend my attentions on.

    Quote Originally Posted by nicgalla
    ... but I think there are many simple solutions to improve this.
    So please post some of those thoughts (the simple solutions) so we can hear your viewpoint and add them to the thought process.

    A side note: for what it's worth ID3V2.X tags should theoretically read faster since they are stored at the beginning of the mp3 file. V1.X tag data is stored at the end, necessitating a seek inside the file. Although this takes a relatively miniscule amount of time, it adds up.
    '01 Chevy Xtreme Stepside (pics)

    FIC K7MNF-64 / Athlon 3200 | 256 DDR | 120 GB 7.2k WD
    WinXP Pro | MediaEngine | 12.1 VGA TFT Touch|Creative CIMR-100
    350w Vector | 320w mATX PS | Hellroaring BIC95150 | ButterflySDC

  3. #13
    Low Bitrate
    Join Date
    Jul 2003
    Location
    Milan, Italy
    Posts
    58
    Dear Phat,
    tonight I'll reply you adeguately to your interesting post.

    Of course don't see me as the one who knows everything.

    Could you post a simple flow chart of the db refresh routine?

    I have reported my configuration in the signature, if you need more details tell me.

    There are certain ID3 tag routine that open the file and read it all before finding the ID3. Does ME do this or use a seek method?

    Well I'll consider to move to ID3V2, I used ID3V1 for compatibility.
    -NicGalla-
    custom-made aluminum case | MK2.8+ PSU | EPIA 800 | 256 Mb DDR | 120 Gb 7.2k WD | Wireless Chieftec mouse & keyb | Trust USB Keypad & USB webcam | Majestic TFT 7" 16:9 | WinXP & ME2 | connected to factory head-unit in a PT Cruiser 2.0L

  4. #14
    Maximum Bitrate phat_bastard's Avatar
    Join Date
    Sep 2002
    Location
    St.Louis, MO
    Posts
    549
    Quote Originally Posted by nicgalla
    Dear Phat,
    tonight I'll reply you adeguately to your interesting post.

    Of course don't see me as the one who knows everything.

    Could you post a simple flow chart of the db refresh routine?
    This would take more time than I have right now. I can show you the source code, but if you don't understand VB and SQL it won't mean anything to you.

    Aside from that, I'm looking for ideas outside the realm of the refresh process as well, i.e. have I just conceptualized the problem wrong.

    The very simplest part of the refresh code does this (condensed):

    1) list and count all the files in your media directory recursively

    2) get a count of all the records in the database

    3) compare the two numbers

    4) if they don't match, compare the two arrays of data to see if there are files that need to be added, removed or updated in the database.

    If I'm understanding your statement correctly, it's taking your machine 30-ish seconds to perform steps 1-3. That is a long time to wait, I wouldn't want to either. Then if there are files to add / update / remove it will take even longer. Additionally, I would have liked to do a more complete analysis of the aggregate data that's built in those two steps to handle the scenario of say deleting 10 files and then adding 10 new files, but that would obviously use more processor cycles and more time. That's why I'm encouraging people to use the file sync function.

    Quote Originally Posted by nicgalla
    I have reported my configuration in the signature, if you need more details tell me.
    Duh. I guess i just scan past the sigs any more, sorry. The common thread I see emerging is that the epia 800 is pretty slow these days. I've talked to about four or five others using them and they're all complaining about the db refresh being slow. You know you want a M10000 (and i don't even get any kickbacks from VIA for saying that) Can you play dvd video ok on that?

    Quote Originally Posted by nicgalla
    There are certain ID3 tag routine that open the file and read it all before finding the ID3. Does ME do this or use a seek method?
    I think I stated that it seeks inside the file. The exact code if memory serves is something like:
    Seek filehandle, filelen - 127
    for v1 tags. v2 tags I only read the header and it's at the top of the file, so that's not really relevant to speed.

    Quote Originally Posted by nicgalla
    Well I'll consider to move to ID3V2, I used ID3V1 for compatibility.
    That statement wasn't meant to cause you to re-compile your whole collection, but just for example when I 'acquire' new music I usually use winamp to make sure all the tags are complete, and are all v2 tags.
    '01 Chevy Xtreme Stepside (pics)

    FIC K7MNF-64 / Athlon 3200 | 256 DDR | 120 GB 7.2k WD
    WinXP Pro | MediaEngine | 12.1 VGA TFT Touch|Creative CIMR-100
    350w Vector | 320w mATX PS | Hellroaring BIC95150 | ButterflySDC

  5. #15
    Low Bitrate
    Join Date
    Jul 2003
    Location
    Milan, Italy
    Posts
    58
    The simplest solution is to make an option in Control Panel to turn off automatic DB refreshing. There is already a manual button in the control panel to refresh it, you could make a keypress to reach the funcion easily. Or best, a keypress that after 2 seconds you pressed (to prevent accidental pressings) it starts AutoSync. Apart this, it's not a problem to use AutoSync for me.
    (small note: have you added drive check in autosync? if I trigger it and my wireless network is not in range ME quits and reloads)

    Or you can bypass the count number and always compare the data structure, maybe it can improve speed on slower system.

    you can post the code if you like, I know VB quite well since it's the only language I can code with

    I have compared ME2.0 rebuilding db time with my own written VB program that I was developing before discovering ME.

    6710 files, 422 folders on 100 mbps ethernet network drive. PC athlon 3200+
    ME 2.0: 3 minutes 40 seconds
    my program (custom DB, not SQL): 4 minutes 20 seconds

    so there is nothing in ME to optimize this routine, it's already fast. It seems that also Windows API make use of the the recursive code. What about File System Object (FSO)?

    Yeah EPIA 800 is quite slow but M10000 is not so faster, there's only 25% improvement. It's comparable to a Pentium II 300, which allows DVD playing but with little stuttering.
    The reason I chose it is all for heat and power consumption. Why don't they sell Centrino's spare motherboards?
    However the Western Digital drive is one of the 7.2k drives with best access time, beaten only by IBM Deskstar GXP120 series or newer Hitachi 7K250.
    -NicGalla-
    custom-made aluminum case | MK2.8+ PSU | EPIA 800 | 256 Mb DDR | 120 Gb 7.2k WD | Wireless Chieftec mouse & keyb | Trust USB Keypad & USB webcam | Majestic TFT 7" 16:9 | WinXP & ME2 | connected to factory head-unit in a PT Cruiser 2.0L

  6. #16
    Maximum Bitrate phat_bastard's Avatar
    Join Date
    Sep 2002
    Location
    St.Louis, MO
    Posts
    549
    Quote Originally Posted by nicgalla
    The simplest solution is to make an option in Control Panel to turn off automatic DB refreshing. There is already a manual button in the control panel to refresh it, you could make a keypress to reach the funcion easily. Or best, a keypress that after 2 seconds you pressed (to prevent accidental pressings) it starts AutoSync.
    Good idea with the AutoSync button. I'll be a lot happier when it's fully automatic tho and my stuff starts to sync whenver I'm within a block of my house. I haven't spent much time with this, but I will soon.

    Your AutoSync bug will be resolved in the next bugfix release. The only way I'm putting an option in to disable auto db refresh is if I hear some well thought out feedback about AutoSync, and I hear frodo tell me he's not having his database builds crash on him. Does it work properly for you other than the drive offline drive bug you reported? After using it does your database refresh run the short pass or does it take a long time. If it runs a full parsing of the db records then something's not happening in AutoSync that should. I'm sorry I haven't been able to test this out, but MediaEngine is not the only thing happening in my life at the moment. Some day I'd like to actually get my carpc installed in my vehicle, but the chance of that happening any time soon is fairly slim.

    Quote Originally Posted by nicgalla
    Or you can bypass the count number and always compare the data structure, maybe it can improve speed on slower system.
    The only problem I see with this is that I think you might not have considered what I would need to build the 'data structure' from? Here's what's taking a long time for you; the findfile api and the recursion code that loads your directory listing in an array. If I build some kind of stateful salt or checksum value based on the aggregate data of your directory structure, it would have to come from the listing, wouldnt you think? I appreciate the statement, but it doesn't seem very well thought out. If I'm wrong, I'd love to hear you elaborate on your idea.

    Quote Originally Posted by nicgalla
    I have compared ME2.0 rebuilding db time with my own written VB program that I was developing before discovering ME.

    6710 files, 422 folders on 100 mbps ethernet network drive. PC athlon 3200+
    ME 2.0: 3 minutes 40 seconds
    my program (custom DB, not SQL): 4 minutes 20 seconds

    so there is nothing in ME to optimize this routine, it's already fast. It seems that also Windows API make use of the the recursive code. What about File System Object (FSO)?
    I use the FindFile() and related API calls to do the listing, the recursion is very simple vb code working on the data that the API returns to it. Using FSO would probably be slower since it's just a wrapper class for the API calls as far as i know.
    '01 Chevy Xtreme Stepside (pics)

    FIC K7MNF-64 / Athlon 3200 | 256 DDR | 120 GB 7.2k WD
    WinXP Pro | MediaEngine | 12.1 VGA TFT Touch|Creative CIMR-100
    350w Vector | 320w mATX PS | Hellroaring BIC95150 | ButterflySDC

  7. #17
    FLAC
    Join Date
    Jun 2003
    Location
    Memphis - TN
    Posts
    1,272
    I know you were saying that thse who have an issue with it may have older hardware. I have to speak for those who have an EPIA. They are not the fastest boards in the world, but were designed for its purpose, small. I myself have all the fastest equipment I can get besides the board (fast ram, 7200rpm hd, ect) But auto updating the database is what turned me against ME. I know its planned as a fix in the next release, but I felt I should throw in my .02.
    I dont have a slow PC, I have a large database. I also would like to see the UPdated DB as a button.
    2011 Nissan Frontier SL
    AMD X3 2.2 | M4-ATX | 16Gb SSD | 2GB DDR3-1333 | MSI GF615M-P33 MB
    OBDPROS USB | BU-303 GPS l LILLIPUT TS | Car2PC adapter | XM Direct | USB Dual band N with custom mag-mount antenna.

  8. #18
    Maximum Bitrate phat_bastard's Avatar
    Join Date
    Sep 2002
    Location
    St.Louis, MO
    Posts
    549
    Quote Originally Posted by Custommx3
    auto updating the database is what turned me against ME. I know its planned as a fix in the next release, but I felt I should throw in my .02.
    Can you elaborate on this ('what turned me against ME') i.e. how you were using it, etc? As I mentioned earlier, the way we decided to do things involved certain trade-offs in immediacy in favor of functionality and simplicity based on what we thought of the user base and how we perceived people would use MediaEngine.

    I don't want your .02, I want you to spend a whole dollar telling me about things. This isn't a poll, it's a discussion thread. Talk to me people. I thirst for knowledge.

    Quote Originally Posted by Custommx3
    I dont have a slow PC, I have a large database. I also would like to see the UPdated DB as a button.
    Thats a lot less likely to happen than an AutoSync button.
    '01 Chevy Xtreme Stepside (pics)

    FIC K7MNF-64 / Athlon 3200 | 256 DDR | 120 GB 7.2k WD
    WinXP Pro | MediaEngine | 12.1 VGA TFT Touch|Creative CIMR-100
    350w Vector | 320w mATX PS | Hellroaring BIC95150 | ButterflySDC

  9. #19
    FLAC
    Join Date
    Jun 2003
    Location
    Memphis - TN
    Posts
    1,272
    well, it seems that everytime I start ME2.0 it updates its database. But with all the music I have, it takes forever to update its database. Id prefer it not to auto update its database. I was unaware we could turn off the file sync. and yes I seemed to have overlooked your part regarding the file sync. I wasnt really against it. I just said to myself "this thing takes forever." Im not one to start up my car and wait 5 mins before I can listen to music. I'll try turning the file sync off and see if that will help. I think manually updating the database is the best option for those who do not need to update its database on every run of ME. now thats my .03
    2011 Nissan Frontier SL
    AMD X3 2.2 | M4-ATX | 16Gb SSD | 2GB DDR3-1333 | MSI GF615M-P33 MB
    OBDPROS USB | BU-303 GPS l LILLIPUT TS | Car2PC adapter | XM Direct | USB Dual band N with custom mag-mount antenna.

  10. #20
    Maximum Bitrate phat_bastard's Avatar
    Join Date
    Sep 2002
    Location
    St.Louis, MO
    Posts
    549
    Quote Originally Posted by Custommx3
    well, it seems that everytime I start ME2.0 it updates its database. But with all the music I have, it takes forever to update its database. Id prefer it not to auto update its database. I was unaware we could turn off the file sync. and yes I seemed to have overlooked your part regarding the file sync. I wasnt really against it. I just said to myself "this thing takes forever." Im not one to start up my car and wait 5 mins before I can listen to music. I'll try turning the file sync off and see if that will help. I think manually updating the database is the best option for those who do not need to update its database on every run of ME. now thats my .03
    .03 won't get you in either. From your comment it's apparent you don't understand the difference between the database refresh process, and the AutoSync function in the settings screen. Spend some time getting to know the interface and tweaking all the knobs and buttons, then come back and tell me what you think.

    It seems that 90% of the ppl that try out our software use 10% of the functionality because they don't experiment with things and they especially don't read the manual or ask questions, and that's a shame.
    '01 Chevy Xtreme Stepside (pics)

    FIC K7MNF-64 / Athlon 3200 | 256 DDR | 120 GB 7.2k WD
    WinXP Pro | MediaEngine | 12.1 VGA TFT Touch|Creative CIMR-100
    350w Vector | 320w mATX PS | Hellroaring BIC95150 | ButterflySDC

Page 2 of 2 FirstFirst 12

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •