Page 2 of 7 FirstFirst 1234567 LastLast
Results 11 to 20 of 68

Thread: What data to collect?

  1. #11
    Variable Bitrate
    Join Date
    Jul 2006
    Location
    Brisbane, Australia
    Posts
    346
    I think you should be using the standard NMEA sentences as it is multiplatform so there will be no more talk of the Windows v's Unix warring factions.

    The other plus is that this is a documented industry standard. Also as discussed, there is a risk that if this data is filtered by the user, you will find that you want to use some data that you have not got a year from now.

    The other advantage from using NMEA is that users could start contributing right now before the server is in place simply by enabling the logging in Xport becasue as you say almost everybody on the Windows Platform uses C's software...

    Once the server is in place, they could start uploading all the data they have.

    I already have Xport logging turned on so I have an evidence trail to refute a speed camera fine if it ever turns up in the mail so I could give you quite a deal of data straight away if you ever extend your reach to Australia!.

    You might also look into how the US government manages place names. In Australia, there is a central government controlled database of place names that appear on topographic maps (towns, buildings, mountains, creeks and rivers etcc and some road names. This database includes the lat and lon of all these places and I have the file sitting on my Car PC where it is accessible from my topgraphic mapping application (and the 20 Gb of topo maps I have installed).

    So guys, stop bickering about this, turn on logging and start collecting data straight away so there is some real data available for upload as soon as your server comes on line.

    Oh and there is no reason why your server can't support multiple input file formats just like we do here http://www.4x4earth.com.au/
    RodW
    2007 Toyota Hilux with a CarPC..

    Worklog: http://www.mp3car.com/vbulletin/work...ota-hilux.html
    OziExplorer GPS Embedded in RR: http://www.mp3car.com/vbulletin/sb-s...iexplorer.html

  2. #12
    Constant Bitrate
    Join Date
    May 2008
    Posts
    223
    Quote Originally Posted by malcom2073 View Post
    I think we should collect the same data that is contained in the NMEA sentences (eg all of it), but compress it to a binary format for upload to the server to save server space.
    Hmm...compress...binary format...save server space. How about we just zip the files? NMEA is just straight up text and should be highly compressible, plus you don't lose any data.

  3. #13
    North of the land of Hey Huns
    Auto Apps:loading...

    Join Date
    Jun 2004
    Location
    Westminster, MD
    Posts
    2,127
    Sicarius: we could compress, or we could offload some of the work the people working with the data have to do onto the data collector, you'd probably see about the same amount of space savings. That and you could always zip up the binary data, see if it's any smaller then

    NMEA is expensive to process in large amounts cycle-wise
    "stop with the REINSTALLS, what do you think we got some lame-o installer!!!" - mitchjs
    RevFE
    My Shop

  4. #14
    Constant Bitrate
    Join Date
    May 2008
    Posts
    223
    Another question to answer is how many points per unit time do you want? Most people are going to be producing 1 point per second but mine is going to be putting out 10 per second.

    If you're offloading to the client to pre-process before submitting you could also do some smart stuff like decreasing the report rate the slower you're moving, culling entirely when stationary.

  5. #15
    North of the land of Hey Huns
    Auto Apps:loading...

    Join Date
    Jun 2004
    Location
    Westminster, MD
    Posts
    2,127
    We do want to know when someone is stationary, that could be used to figure (for example) stop signs, stoplights vs stop signs, what turn lanes have lights, etc. Like I said I don't think removing any data at all is a good idea. Space and bandwidth are relatively cheap, but going back and re-collecting a couple months worth of data from a couple hundred people because we forgot something... not so much.


    I also think that getting as many samples per second should be key. As long as the samples are timestamped they can be of value. Until we start getting into the actual processing of this data, we really have no idea what is useful and what isn't, so I think what data to cull out is a discussion for much later.
    "stop with the REINSTALLS, what do you think we got some lame-o installer!!!" - mitchjs
    RevFE
    My Shop

  6. #16
    Variable Bitrate
    Auto Apps:loading...
    thekl0wn's Avatar
    Join Date
    Apr 2005
    Location
    PoCo, Indiana
    Posts
    284
    The sentences are standards... Therefore, I vote they be stored intact, line-by-line, in sequence on the server. At that point, any further information can be propagated from there. Surely, the server will have a mildy idle time where it can perform daily "builds" of the data.

    A user uploads the formatted GPS output (like in the .txt file attached above) to the FTP. A parser application runs, which opens each new file, and inserts each sentence into a table, and then moves the text file into a save folder under a formatted name.

    This is the most basic of the tasks, but it will also be an incredibly processor intensive one. All other data could easily (though, not fast) be propagated from a raw data table.
    Play with it, 'til it's broke.

  7. #17
    is back. FKA Robert Wray
    Auto Apps:loading...
    Fiberoptic's Avatar
    Join Date
    Jul 1978
    Location
    Baltimore, MD
    Posts
    1,418
    Blog Entries
    143
    Quote Originally Posted by Sicarius View Post
    Another question to answer is how many points per unit time do you want? Most people are going to be producing 1 point per second but mine is going to be putting out 10 per second.
    I think 10Hz GPS outputs might be very valuable. What hardware are you using to collect that stream?

  8. #18
    Constant Bitrate
    Join Date
    May 2008
    Posts
    223
    Quote Originally Posted by Fiberoptic View Post
    I think 10Hz GPS outputs might be very valuable. What hardware are you using to collect that stream?
    The Wintec WBT-300 / G-Rays I.

  9. #19
    Admin. Linux loser.
    Auto Apps:loading...
    Bugbyte's Avatar
    Join Date
    Sep 2004
    Location
    Corning, NY
    Posts
    7,359
    Blog Entries
    2
    @ FO and ecog - How about we do some experiments with NMEA data?

    Why don't we compare a couple of datasets and compare the results of:

    1. binary
    2. zipped
    3. raw

    Pick some track data from a GPS that is 1 minute, 10 minutes, 20 minutes, 30 minutes and 1 hour long.

    Figure the data sizes for all of the options and let's graph them to get an idea of how big the data storage requirements would be. Remember to take into consideration backups of the data both in time and in size/cost.

    Then, we should see what kind of processing has to take place on the data to turn it into openstreetmap track information and run it against the datasets. You'll have to do this in the future to process the data and it would be a good idea to see if that would scale.

    Then, we can calculate storage size and processing times for 1, 10, 100, 1000, and 10,000 users.

    No sense starting it off one way and then finding out it won't scale.
    Quote Originally Posted by ghettocruzer View Post
    I was gung ho on building a PC [until] just recently. However, between my new phone having internet and GPS and all...and this kit...Im starting to have trouble justfiying it haha.
    Want to:
    -Find out about the new iBug iPad install?
    -Find out about carPC's in just 5 minutes? View the Car PC 101 video

  10. #20
    is back. FKA Robert Wray
    Auto Apps:loading...
    Fiberoptic's Avatar
    Join Date
    Jul 1978
    Location
    Baltimore, MD
    Posts
    1,418
    Blog Entries
    143
    For now we have created this account with the following permissions:

    -upload files to the server
    -view all the uploaded files in that directory
    -Can't delete files
    -can't download files

    IP:12.167.132.204
    ftp port 21:
    username: datadump
    password: datadump


    File Format:
    TicketID_DDMMYYYY_HHMMSS.txt
    Get your ticket id: http://www.mp3car.com/ticket.php
    The forum username on the ticket.php page is optional. It will only be used down the road for credit purposes (who uploaded how many tracks, etc....)

    More on data format:
    -Files should contain NMEA sentences
    -files should be in text format
    -a collection of text files in zipped format will be OK
    -Files without the proper format will be given the lowest processing priority

Page 2 of 7 FirstFirst 1234567 LastLast

Similar Threads

  1. Replies: 444
    Last Post: 10-21-2014, 05:50 PM
  2. Replies: 7
    Last Post: 04-06-2009, 04:14 PM
  3. Can you guys please help me??? OBD Renault Clio help needed!
    By madtoonbull in forum Engine Management, OBD-II, Engine Diagnostics, etc.
    Replies: 7
    Last Post: 02-19-2009, 10:36 AM
  4. Pinout Color Codes / Tables
    By gummybear in forum General Hardware Discussion
    Replies: 4
    Last Post: 05-12-2005, 03:05 AM
  5. VOICES Traffic Data
    By stevieg in forum V.O.I.C.E.S
    Replies: 85
    Last Post: 05-31-2004, 08:24 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •