Announcement

Collapse
No announcement yet.

Feature Request (Do not automatically update db)

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Feature Request (Do not automatically update db)

    Can we have an option to only update the database on user command ?

    Please

    Frodo
    [H]4 Life
    My next generation Front End is right on schedule.
    It will be done sometime in the next generation.
    I'm a lesbian too.
    I am for hire!

  • #2
    Bump
    [H]4 Life
    My next generation Front End is right on schedule.
    It will be done sometime in the next generation.
    I'm a lesbian too.
    I am for hire!

    Comment


    • #3
      Originally posted by frodobaggins
      Can we have an option to only update the database on user command ?

      Please

      Frodo
      I second and third that!

      Comment


      • #4
        Short Answer: No

        Long Answer:

        We built M/E 2.0 on / for platforms that were fast enough to do DVD playback. This in itself implies something of a benchmark for systems. Granted not everyone cares to play DVDs, we saw no point in releasing another version of 1.XX that couldn't do jack sh_t. We felt that most people would either get themselves faster hardware to run it on or find something else. Of course people are still running Windows 98 on carputers, so that in itself says something of people's lack of willingness to step into this decade with their choice of operating system.

        Here are a few things that will make your database refresh faster.

        1) Create a dedicated folder to store your media files in. Don't use your \My Documents\My Music\ folder, and especially don't use your \My Documents\ folder. If your media folder has anything in it besides media files that you intend to play in MediaEngine, you're wasting time listing them out when M/E starts. On a relatively modern cpu (capable of running WinXP or 2k) with a decent hard drive (not the cheapest one you could find), reconciling a database of about 5k files takes less than 2 seconds if there were no files added / removed since last build. If you're not seeing comparable behavior then there are some other problems we need to work out with you on a case by case basis.

        2) Don't try to put your media on a USB or worse, a network connected drive. You'll have a hard time playing back movies of any quality, and you'll get to your destination before the file listing will have completed. Let me emphasize the NETWORK DRIVE portion. This is the mp3CAR.com forum, not the mp3HTPC.com forum last I checked.

        3) Invest in a faster hard drive.

        4) For the love of all that is holy, don't try to use your \windows\ folder as your media folder (yes, we've had people do this). If you don't understand why you shouldn't do this, you shouldn't have a pc in your car.

        5) Try to use the built-in file sync function. If you find problems please report them to me and I'll try to get them resolved asap. I'm going to try to make this work automatically in a future release, but most importantly this function (from version 2.0.27 up) synchronizes your database during the file sync operation.

        6) Be patient. I've had a few ideas that might help me sync the database faster and more accurately. Unfortunately we've been busy trying to make the visualizations work more reliably, and that's taking a lot of time.

        In a nutshell, the database functions in 2.0 have been very well received by most everyone we've talked to. We've spend hundreds of hours trying to make it as transparent and user-friendly as possible, so we're not about to subvert all that work just so you don't have to buy faster equipment for your carpc. We'll all be happier in the long run, and you're not going to be able to run that old motherboard forever anyway.
        '01 Chevy Xtreme Stepside (pics)

        FIC K7MNF-64 / Athlon 3200 | 256 DDR | 120 GB 7.2k WD
        WinXP Pro | MediaEngine | 12.1 VGA TFT Touch|Creative CIMR-100
        350w Vector | 320w mATX PS | Hellroaring BIC95150 | ButterflySDC

        Comment


        • #5
          Phat_B

          I don't see how that really applies to what we are asking. We just want the option for it not to sync if we don't want it to. I don't see why this would be a problem. The sync slows down everything, and it tries to sync on every startup. During database syncing, the program tends to crash or becomes non-responsive.

          I guess my question is, why would it be bad for us to do manual syncing if we so desire ?

          Frodo
          [H]4 Life
          My next generation Front End is right on schedule.
          It will be done sometime in the next generation.
          I'm a lesbian too.
          I am for hire!

          Comment


          • #6
            Originally posted by frodobaggins
            I don't see how that really applies to what we are asking. We just want the option for it not to sync if we don't want it to. I don't see why this would be a problem. The sync slows down everything, and it tries to sync on every startup. During database syncing, the program tends to crash or becomes non-responsive.
            Sounds like you need to spend some quality time generating some well prepared bug reports. Giving in on this one request would be developer suicide to me. Spend some of your time and talent helping us figure out why the database refresh takes so long and crashes, and I'd gladly help you out.

            Originally posted by frodobaggins
            I guess my question is, why would it be bad for us to do manual syncing if we so desire ?
            Aside from the numerous technical reasons, because if I add this feature request for you I will be subverting true problems
            '01 Chevy Xtreme Stepside (pics)

            FIC K7MNF-64 / Athlon 3200 | 256 DDR | 120 GB 7.2k WD
            WinXP Pro | MediaEngine | 12.1 VGA TFT Touch|Creative CIMR-100
            350w Vector | 320w mATX PS | Hellroaring BIC95150 | ButterflySDC

            Comment


            • #7
              I recently download (1) file onto my setup. It took ME when starting up over a min to finish refreshing the database. otherwise it takes about 10 secs. I have a 7200 RPM 120 Gig drive, keep all media (and only media) in my media directory, and am running WINXP on a C3 800. What else can I do to speed this up? You can imagine how long it takes if I add a couple dirs of MP3s to the list. ive got about 30 gig of mp3s at the moment...

              Grand AMplifier Project 1.0:
              http://drive.to/cliffsgrandamgt/
              http://www.astallaslions.com/carmp3/
              --------------------------------
              -Eden 800 mini-itx motherboard
              -5.6 NTSC TFT-LCD
              -80 gig hard drive
              -128 MB RAM
              -MPBS1 DC-DC PSU
              -PB Remote /w Girder
              -Media Engine & Windows XP
              -Playstation II also added
              -Steering Wheel Buttons modified for Control.

              Comment


              • #8
                Originally posted by Cliff
                I recently download (1) file onto my setup. It took ME when starting up over a min to finish refreshing the database. otherwise it takes about 10 secs. I have a 7200 RPM 120 Gig drive, keep all media (and only media) in my media directory, and am running WINXP on a C3 800. What else can I do to speed this up? You can imagine how long it takes if I add a couple dirs of MP3s to the list. ive got about 30 gig of mp3s at the moment...
                Yes, whether you add 1 file or a thousand, it will still check the timestamps on all your existing files with the ones in the database to make sure none of them have changed or been deleted. I'm working on a way to make it do less, but it also depends on people reading the suggestions I post and telling me what their opinions & results are.

                Another thing that's been frequently stated that means nothing to me, how large your collection is in _gaBytes. It's about how many files there are. If you have 100 gigs worth of full length divx movies your database is gonna build loads faster than if you have 100 gigs of 96kbps mp3s with ID3V1 tags.

                Did you try using the file sync function like I suggested? It's going to be the key to not seeing the database refreshing at all in the near future.
                '01 Chevy Xtreme Stepside (pics)

                FIC K7MNF-64 / Athlon 3200 | 256 DDR | 120 GB 7.2k WD
                WinXP Pro | MediaEngine | 12.1 VGA TFT Touch|Creative CIMR-100
                350w Vector | 320w mATX PS | Hellroaring BIC95150 | ButterflySDC

                Comment


                • #9
                  OK, let us tidy this up a bit.

                  For anyone reading this thread:

                  It seems that me and phat_bastard have a few issues that we are resolving in
                  private that are unrelated to this issue or to points brought up in this thread.
                  We are solving it in private and I apologize for our dirty laundry.

                  Frodo
                  [H]4 Life
                  My next generation Front End is right on schedule.
                  It will be done sometime in the next generation.
                  I'm a lesbian too.
                  I am for hire!

                  Comment


                  • #10
                    this is no dirty laundry, but an interesting discussion.

                    On my system it takes about thirty seconds to refresh the database. I have 6170 files arranged in 422 folders. Every file has only ID3V1 tag.
                    Unfortunately ME2.0 always refresh the db, even if you haven't touched your library.
                    I use hibernation so it's not a big problem but I think there are many simple solutions to improve this.
                    -NicGalla-
                    custom-made aluminum case | MK2.8+ PSU | EPIA 800 | 256 Mb DDR | 120 Gb 7.2k WD | Wireless Chieftec mouse & keyb | Trust USB Keypad & USB webcam | Majestic TFT 7" 16:9 | WinXP & ME2 | connected to factory head-unit in a PT Cruiser 2.0L

                    Comment


                    • #11
                      Originally posted by nicgalla
                      this is no dirty laundry, but an interesting discussion.

                      On my system it takes about thirty seconds to refresh the database. I have 6170 files arranged in 422 folders. Every file has only ID3V1 tag.
                      Unfortunately ME2.0 always refresh the db, even if you haven't touched your library.
                      I use hibernation so it's not a big problem but I think there are many simple solutions to improve this.
                      I was referring to the other stuff, not the request I made.
                      [H]4 Life
                      My next generation Front End is right on schedule.
                      It will be done sometime in the next generation.
                      I'm a lesbian too.
                      I am for hire!

                      Comment


                      • #12
                        Originally posted by nicgalla
                        On my system it takes about thirty seconds to refresh the database. I have 6170 files arranged in 422 folders. Every file has only ID3V1 tag.
                        Unfortunately ME2.0 always refresh the db, even if you haven't touched your library.
                        I use hibernation so it's not a big problem but I think there are many simple solutions to improve this.
                        The 'dirty laundry' posts frodo mentioned have been edited / deleted from this thread so we can get back to the topic at hand. I apologize to frodo and anyone who might have read them before they were removed.

                        Can you give me some details about the hardware and operating system of your pc? Sorry if I've asked you this before, but I've read and committed to memory a huge mass of information regarding this project, and memorizing your hardware / software config is akin to me memorizing what color socks each of you have on.

                        I have 2000 odd files taking up around 10 gigs on my 1.2 ghz celeron laptop running a very pristine install of WinXP with 256mb ram and a 5400 rpm 20gb hitachi hdd. It takes about 1.2 seconds for the database refresh code to list all of them and compare the count to the number in the database. I can launch M/E and before I can get to the Media Browser window the database is reconciled and ready to go. When I've added or removed files it takes closer to 90 seconds to page through the database and do what is needed.

                        Obviously it will take longer if you have:
                        a) a slower cpu
                        b) a slower hard drive
                        c) more files to list
                        d) less memory

                        I'm not totally against turning off auto refresh as a user option, but I'd first like to make the refresh process as bullet-proof as possible while acting in the role I designed it to work in. Granted we have more condensed collections and most likely faster machines that we test on, I'd rather spend time finding out why folks are seeing the database build crash and burn and fix those problems before I enable them to just bypass the problem at the expense of lost functionality to other / future users of MediaEngine who don't understand or consider the implications of enabling the option. 9 out of 10 of the problems I've located inside the database refresh code traced back to a specific file with some unforseen structure in the filename and no ID3 tag, or non-standard ID3V2.X tag layouts. Since we can't ask everyone to just send us their collections, this necessitates a difficult (but imo worthwhile) session of testing and troubleshooting. We went to great pains to make it work before release on some very diverse and sizeable collections (ask James aka cloudy about his collection), but you just can't anticipate and compensate for every possible problem you're going to run into when you enlarge your test base by several orders of magnitude.

                        Another area that I've focused on is the file sync function, and I've heard next to no feedback about it other than one bug related to the sync drive being offline when you start the sync process. This function will sychronize your database while it copies files from your source drive to your media folder (or at least it's supposed to). In the near future I would like to make this process run automatically and unattended so it can be used in conjunction with 802.11 wireless networks as well as flash memory cards and usb drives. This will for the most part negate the need to reconcile the database count at startup, but not completely.

                        >>It all boils down to knowing when the user has added, deleted, or modified the files in their collection.<<

                        To put it in a nutshell, we wanted this process to run automatically without user intervention for very good reasons. Anyone who's done something similar has contemplated the possible scenarios and made a tradeoff decision based on the data and their personal opinion. The way we did it is a the result of a decision based on our personal opinions of how we felt it should work, taking into consideration where we would like the project to go in the future and what we've experienced in the past. I hope you can all understand my sticking to my guns on this. I won't deny that there's some pride at stake, but there are many, many other factors involved. I'm very much interested in hearing differing viewpoints. Collecting a bunch of 'my database takes forever too' posts in this thread is not the productive conversation that I'm going to spend my attentions on.

                        Originally posted by nicgalla
                        ... but I think there are many simple solutions to improve this.
                        So please post some of those thoughts (the simple solutions) so we can hear your viewpoint and add them to the thought process.

                        A side note: for what it's worth ID3V2.X tags should theoretically read faster since they are stored at the beginning of the mp3 file. V1.X tag data is stored at the end, necessitating a seek inside the file. Although this takes a relatively miniscule amount of time, it adds up.
                        '01 Chevy Xtreme Stepside (pics)

                        FIC K7MNF-64 / Athlon 3200 | 256 DDR | 120 GB 7.2k WD
                        WinXP Pro | MediaEngine | 12.1 VGA TFT Touch|Creative CIMR-100
                        350w Vector | 320w mATX PS | Hellroaring BIC95150 | ButterflySDC

                        Comment


                        • #13
                          Dear Phat,
                          tonight I'll reply you adeguately to your interesting post.

                          Of course don't see me as the one who knows everything.

                          Could you post a simple flow chart of the db refresh routine?

                          I have reported my configuration in the signature, if you need more details tell me.

                          There are certain ID3 tag routine that open the file and read it all before finding the ID3. Does ME do this or use a seek method?

                          Well I'll consider to move to ID3V2, I used ID3V1 for compatibility.
                          -NicGalla-
                          custom-made aluminum case | MK2.8+ PSU | EPIA 800 | 256 Mb DDR | 120 Gb 7.2k WD | Wireless Chieftec mouse & keyb | Trust USB Keypad & USB webcam | Majestic TFT 7" 16:9 | WinXP & ME2 | connected to factory head-unit in a PT Cruiser 2.0L

                          Comment


                          • #14
                            Originally posted by nicgalla
                            Dear Phat,
                            tonight I'll reply you adeguately to your interesting post.

                            Of course don't see me as the one who knows everything.

                            Could you post a simple flow chart of the db refresh routine?
                            This would take more time than I have right now. I can show you the source code, but if you don't understand VB and SQL it won't mean anything to you.

                            Aside from that, I'm looking for ideas outside the realm of the refresh process as well, i.e. have I just conceptualized the problem wrong.

                            The very simplest part of the refresh code does this (condensed):

                            1) list and count all the files in your media directory recursively

                            2) get a count of all the records in the database

                            3) compare the two numbers

                            4) if they don't match, compare the two arrays of data to see if there are files that need to be added, removed or updated in the database.

                            If I'm understanding your statement correctly, it's taking your machine 30-ish seconds to perform steps 1-3. That is a long time to wait, I wouldn't want to either. Then if there are files to add / update / remove it will take even longer. Additionally, I would have liked to do a more complete analysis of the aggregate data that's built in those two steps to handle the scenario of say deleting 10 files and then adding 10 new files, but that would obviously use more processor cycles and more time. That's why I'm encouraging people to use the file sync function.

                            Originally posted by nicgalla
                            I have reported my configuration in the signature, if you need more details tell me.
                            Duh. I guess i just scan past the sigs any more, sorry. The common thread I see emerging is that the epia 800 is pretty slow these days. I've talked to about four or five others using them and they're all complaining about the db refresh being slow. You know you want a M10000 (and i don't even get any kickbacks from VIA for saying that) Can you play dvd video ok on that?

                            Originally posted by nicgalla
                            There are certain ID3 tag routine that open the file and read it all before finding the ID3. Does ME do this or use a seek method?
                            I think I stated that it seeks inside the file. The exact code if memory serves is something like:
                            Seek filehandle, filelen - 127
                            for v1 tags. v2 tags I only read the header and it's at the top of the file, so that's not really relevant to speed.

                            Originally posted by nicgalla
                            Well I'll consider to move to ID3V2, I used ID3V1 for compatibility.
                            That statement wasn't meant to cause you to re-compile your whole collection, but just for example when I 'acquire' new music I usually use winamp to make sure all the tags are complete, and are all v2 tags.
                            '01 Chevy Xtreme Stepside (pics)

                            FIC K7MNF-64 / Athlon 3200 | 256 DDR | 120 GB 7.2k WD
                            WinXP Pro | MediaEngine | 12.1 VGA TFT Touch|Creative CIMR-100
                            350w Vector | 320w mATX PS | Hellroaring BIC95150 | ButterflySDC

                            Comment


                            • #15
                              The simplest solution is to make an option in Control Panel to turn off automatic DB refreshing. There is already a manual button in the control panel to refresh it, you could make a keypress to reach the funcion easily. Or best, a keypress that after 2 seconds you pressed (to prevent accidental pressings) it starts AutoSync. Apart this, it's not a problem to use AutoSync for me.
                              (small note: have you added drive check in autosync? if I trigger it and my wireless network is not in range ME quits and reloads)

                              Or you can bypass the count number and always compare the data structure, maybe it can improve speed on slower system.

                              you can post the code if you like, I know VB quite well since it's the only language I can code with

                              I have compared ME2.0 rebuilding db time with my own written VB program that I was developing before discovering ME.

                              6710 files, 422 folders on 100 mbps ethernet network drive. PC athlon 3200+
                              ME 2.0: 3 minutes 40 seconds
                              my program (custom DB, not SQL): 4 minutes 20 seconds

                              so there is nothing in ME to optimize this routine, it's already fast. It seems that also Windows API make use of the the recursive code. What about File System Object (FSO)?

                              Yeah EPIA 800 is quite slow but M10000 is not so faster, there's only 25% improvement. It's comparable to a Pentium II 300, which allows DVD playing but with little stuttering.
                              The reason I chose it is all for heat and power consumption. Why don't they sell Centrino's spare motherboards?
                              However the Western Digital drive is one of the 7.2k drives with best access time, beaten only by IBM Deskstar GXP120 series or newer Hitachi 7K250.
                              -NicGalla-
                              custom-made aluminum case | MK2.8+ PSU | EPIA 800 | 256 Mb DDR | 120 Gb 7.2k WD | Wireless Chieftec mouse & keyb | Trust USB Keypad & USB webcam | Majestic TFT 7" 16:9 | WinXP & ME2 | connected to factory head-unit in a PT Cruiser 2.0L

                              Comment

                              Working...
                              X