Announcement

Collapse
No announcement yet.

Open Sync

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Open Sync

    While this will be included as part of an OpenMobile plugin, it is fully open source and is designed to be used stand alone with any operating system or front end.

    What is it:
    Open Sync is a protocol, a server and a client library. All of which are open source and fully documented. It is designed to sync files between a "sync server" and a carPC. This server can be located on a local network, a remote network or even across the internet.

    Design Goals:
    • Fully extensible -can have extensions and functionality added without any change to its design
    • Secure - The protocol supports salted password exchange and optionally even SSL transactions to ensure your data remains safe even when traveling through unprotected connections.
    • Zero Configuration - This entire setup can be deployed with no end user configuration required. The client automatically finds the server on the network and syncs files without any setup required. It uses a variety of methods including UPnP and UDP broadcasts to ensure this works locally, over wifi, through firewalls and even over the internet. No mapping drives or editing config files required.
    • Platform Neutral - The server and client library are written in C# to ensure they can be run on Windows, Linux or OSX. The protocol is open source to allow anyone to create platform specific implementations in their favorite language.
    • Fast - the protocol automatically scales to the capabilities of the current connection. While on wifi, files are transferred with no overhead to ensure maximum transfer speed. Over a cellular connection, various compression algorithms (including gzip and LZMA) are used to minimize bandwidth usage and maximize transfer speed. The LZMA extension for example, can transmit ~500kb/s over a 100kb/s connection.

    Expected public release: Dec 1, 2010
    Anyone interested in developing their own implementations or helping with the design are welcome to contribute.

    Important Links:
    openMobile - An open source C# Front End (why choose openMobile?)
    - Always Recruiting Developers -
    Like what you see? Donations are always welcome

  • #2
    It has to be asked, but why would this protocol be a choice for developers over any of the already mature open source file sync protocols?
    "stop with the REINSTALLS, what do you think we got some lame-o installer!!!" - mitchjs
    RevFE
    My Shop

    Comment


    • #3
      Originally posted by malcom2073 View Post
      It has to be asked, but why would this protocol be a choice for developers over any of the already mature open source file sync protocols?
      Name some and I can tell you the exact differences but mainly, none of them can do what this can.

      CarPCs are a special use case... you could be using a local wifi connection or you could be using a tethered cell phone. A bloated language like syncML would not only destroy your transfer cap it would be very very slow. Other alternatives like KDEs OpenSync are not designed for binary transfer (like music). Even LDAP suffers from being too generic, unstandardized and most importantly it doesn't support compression.
      openMobile - An open source C# Front End (why choose openMobile?)
      - Always Recruiting Developers -
      Like what you see? Donations are always welcome

      Comment


      • #4
        I guess I'm curious as to what it can do that unison or rsync cannot?

        SSL is a plus, but I'm never scared that someone is watching me and/or sniffing packets of music I'm transferring. Good for others I guess.
        Tidder

        Try RevFE
        The best resurrected frontend I've ever used, period.


        I Wish I could ban people

        Comment


        • #5
          rsync seems like a sensible mature tool that does all this, but better.

          Among other things, it also does differential file copying as well as compression; for the initial copy, rsync will be roughly the same speed as anything else. For subsequent copies, it would blow this out of the water, surely? This is PhD thesis material, analyzed in massive depth beyond "I anticipate a five times compression ratio"...

          Password exchange is so passe. For that matter, passwords are so passe :-(
          You can also use public key authentication with rsync, meaning passwords are unnecessary. VNC has fallen from grace in the circles I travel in, thanks in large part to difficult-to-verify password trust - you're implementing something like VNC's password mechanism, but weaker.

          I'll give you server-discovery, but bonjour and friends are open [hell, even nmbd can do that part], so you could do that trivially. Also it would presumably make more sense to have the car automatically pull rather than to try to push to it, which I assume would be the point of discovery.

          I also do not entirely concur that C# is platform-neutral right now. RSync runs on *anything*. C# and mono, not so much. But we've had that argument before and I suspect neither of us is going to budge viewpoint.


          I mean, overall... I always think it's good to try and implement this stuff. But at some point, if you really want to use this for practical stuff rather than academic implementing interest, just use the existing mature tools.

          Gary
          OBDGPSLogger, for logging OBDII and/or GPS data
          OBDSim, an OBDII/ELM327 software simulator
          mp3car forums: obdgpslogger, obdsim

          Comment


          • #6
            Originally posted by Tidder View Post
            I guess I'm curious as to what it can do that unison or rsync cannot
            First 4 that come to mind:
            • require no configuration (instead of batch scripts you have to edit anytime the server ip changes)
            • safely works over the internet
            • Work on systems other then unix (windows syncing with a linux server)
            • Can be extended with optional extensions like playlist syncing or FE plugin updates
            openMobile - An open source C# Front End (why choose openMobile?)
            - Always Recruiting Developers -
            Like what you see? Donations are always welcome

            Comment


            • #7
              So I can load it on two computers and sync things without *any* configuration? Impressive, how does it know what I want synced?

              rsync and unison both have windows, mac, and linux builds, and some nice GUIs. (DeltaCopy for rsync under windows, linux there are a handful, mac I don't know; unison has it's own)

              Interested in the extensions and plugins.
              Tidder

              Try RevFE
              The best resurrected frontend I've ever used, period.


              I Wish I could ban people

              Comment


              • #8
                Originally posted by chunkyks View Post
                rsync seems like a sensible mature tool that does all this, but better.

                Among other things, it also does differential file copying as well as compression; for the initial copy, rsync will be roughly the same speed as anything else. For subsequent copies, it would blow this out of the water, surely? This is PhD thesis material, analyzed in massive depth beyond "I anticipate a five times compression ratio"...

                Password exchange is so passe. For that matter, passwords are so passe :-(
                You can also use public key authentication with rsync, meaning passwords are unnecessary. VNC has fallen from grace in the circles I travel in, thanks in large part to difficult-to-verify password trust - you're implementing something like VNC's password mechanism, but weaker.

                I'll give you server-discovery, but bonjour and friends are open [hell, even nmbd can do that part], so you could do that trivially. Also it would presumably make more sense to have the car automatically pull rather than to try to push to it, which I assume would be the point of discovery.

                I also do not entirely concur that C# is platform-neutral right now. RSync runs on *anything*. C# and mono, not so much. But we've had that argument before and I suspect neither of us is going to budge viewpoint.


                I mean, overall... I always think it's good to try and implement this stuff. But at some point, if you really want to use this for practical stuff rather than academic implementing interest, just use the existing mature tools.

                Gary
                The only thing rsynx does that I haven't done yet is differential compression. Its actually not as difficult as you would think (especially since the rsync algorithm can be used) it just is phd level material to optimize. In either case it would have little value here since file changes will be rare and files added or removed will be far more common (unless you plan on editing your pictures, music and videos).

                Salted password exchange is the basis for the linux password algorithm so I would hardly call it weak or passe. Untrusted public keys are just as insecure as untrusted passwords... in either case generating a public key and generating a password are just as easy to do...the difference is public key exchange requires either manually copying it to the client (which makes it exactly the same as a password) or insecurely grabbing it from the server. If implemented in the latter fashion preshared keys (passwords) are actually more secure.

                Yea great bonjour can discover services but not this service. Implementing all of bonjour for something that can be done in a few lines adds bloat without value. Not to mention the protocol would have to be modified to support this which would rule out existing libraries. We use UPnP for network discovery which is a far more prevalent protocol (yes i realize different use cases).

                C# and mono runs on anything rsyanc runs on - unix and its variants (including osx) and windows. I know of no rsync implementation for any other platform. Regardless, its an open protocol that can be implemented on whatever platform in whatever language you want.

                edit:
                Basically, not all of us run a linux console in our cars and operate our cars via keyboard. The goal here is a system that just works and adapts to the conditions a carPC faces. Combinations of scripts and various programs not designed for this use case is just taking the riderunner approach to things. If you have to leave your favorite front end because your server ip changed or wifi dropped and you need to tether a phone-the carPC concept has failed.
                openMobile - An open source C# Front End (why choose openMobile?)
                - Always Recruiting Developers -
                Like what you see? Donations are always welcome

                Comment


                • #9
                  i love this idea. If possible can you allow maximum size allowed over non wifi connections?
                  ie. allow maximum 500MB over 3g connection per month, or something like that?
                  So that you won't go totally over your connection limit.

                  Comment


                  • #10
                    Originally posted by justchat_1 View Post
                    C# and mono runs on anything rsyanc runs on - unix and its variants (including osx) and windows. I know of no rsync implementation for any other platform. Regardless, its an open protocol that can be implemented on whatever platform in whatever language you want.

                    edit:
                    Basically, not all of us run a linux console in our cars and operate our cars via keyboard. The goal here is a system that just works and adapts to the conditions a carPC faces. Combinations of scripts and various programs not designed for this use case is just taking the riderunner approach to things. If you have to leave your favorite front end because your server ip changed or wifi dropped and you need to tether a phone-the carPC concept has failed.
                    So what I gather is, existing sync protocol XYZ doesn't have a discovery ability so that is why the world needs OpenSync? I like the idea of no configuration and works-out-of-the-box. I'm just worried this little side project will go the way of OpenOBD... It's fun having a diverse number of side-projects but eventually, you realize that you are just one man and one man can only do so much before he's overwhelmed.

                    So, with that in mind... I'd just write a frontend to rsync that uses the SDP of the UPNP spec and be done.

                    Also, there are a lot of reasons why a system wouldn't want to use mono. I like mono, but OEMs are likely thinking twice about it after that whole thing with Tomtom and Fat32. Also there also is the size excuse which Fedora now uses as an excuse not to include mono or anything that uses mono in its default installation.
                    Former author of LinuxICE, nghost, nobdy.
                    Current author of Automotive Message Broker (AMB).
                    Works on Tizen IVI. Does not represent anyone or anything but himself.

                    Comment


                    • #11
                      Originally posted by tripzero View Post
                      So what I gather is, existing sync protocol XYZ doesn't have a discovery ability so that is why the world needs OpenSync? I like the idea of no configuration and works-out-of-the-box. I'm just worried this little side project will go the way of OpenOBD... It's fun having a diverse number of side-projects but eventually, you realize that you are just one man and one man can only do so much before he's overwhelmed.

                      So, with that in mind... I'd just write a frontend to rsync that uses the SDP of the UPNP spec and be done.
                      Very true.... but for me, Open Sync is developed for the same reason as open installer (which is now fully mature) - its easier to create a universal solution that does exactly what i need. If i use rsync, then i still need to create a wrapper for it for each platform, handle configuration and telling it what to sync with who and handle service discovery to figure it all out. Then I have to deal with packaging it for all the platforms and figuring out why one implementation runs differently then another. It was actually faster just to write this from scratch, ensure it had a fast extensible design and know its compile once run everywhere. If i want to add sync features that rsync doesn't support at a later time I can do that easily and know it will work everywhere. For new file transfers we can actually achieve 2-3x the throughput as rsync in low bandwidth situations. Just figured this may have advantages outside OM so wanted to let the entire community take advantage if they were interested.

                      Originally posted by tripzero View Post
                      Also, there are a lot of reasons why a system wouldn't want to use mono. I like mono, but OEMs are likely thinking twice about it after that whole thing with Tomtom and Fat32. Also there also is the size excuse which Fedora now uses as an excuse not to include mono or anything that uses mono in its default installation.
                      Tom Tom was using a copyrighted technology from microsoft...completely different from mono which implements an ECMA standard. The only part of mono that microsoft could even attempt to come after was its winforms implementation (which we don't use) and that was handled by an agreement between novell and microsoft where microsoft agreed not to sue.

                      Fedora is trying to cut bloat sure, but mono is the last thing that needs to be removed to get bloat down in that distro. I think support for 20 year old hardware should be more likely.
                      openMobile - An open source C# Front End (why choose openMobile?)
                      - Always Recruiting Developers -
                      Like what you see? Donations are always welcome

                      Comment


                      • #12
                        Originally posted by justchat_1 View Post
                        Fedora is trying to cut bloat sure, but mono is the last thing that needs to be removed to get bloat down in that distro. I think support for 20 year old hardware should be more likely.
                        IDK, with the amount of patent infringement lawsuits being thrown around, I'm not sure anything is safe. I'm sure MS could think of some patent that they own that they can press on mono if they wanted to. I mean, if you can patent and sue someone over the "Slide to unlock" feature on a smartphone, you can probably sue any piece of software for anything. The thing with oracle and google over java creates more FUD. I argue against the anti-mono folks all the time, but at the end of the day, anyone can file a lawsuit against anyone for any reason and that usually triggers some response in the form of lawyers, cold hard cash and/or judgments.

                        The fat32 thing had nothing to do with copyright because Tomtom was using the open source implementation of vfat as opposed to anything MS actually owned the copyright on. It was a patent infringement issue.

                        As for the fedora thing, it's actually quite humorous that a distro that takes up 5 dvds is worried about space... I think it's well known that the Fedora folks hate mono and that's their primary reason for not including it.
                        Former author of LinuxICE, nghost, nobdy.
                        Current author of Automotive Message Broker (AMB).
                        Works on Tizen IVI. Does not represent anyone or anything but himself.

                        Comment

                        Working...
                        X