Page 1 of 7 1234567 LastLast
Results 1 to 10 of 68

Thread: What data to collect?

  1. #1
    SuperMod - OBDII GPS Logger forum
    Auto Apps:loading...

    Join Date
    Mar 2009
    Location
    Los Angeles
    Posts
    924

    What data to collect?

    From this thread, one of the listed requirements is:

    Passively, crowd source collected map data
    This is the most graspable item on the list [as a developer], so I wanted to start a thread to gather what exactly the shopping list of data to passively collect is.

    As a trivial starting point, I would say:
    1) The raw sentences collected from the gps receiver. Actually parsing them on the server is probably unnecessarily expensive, so I am inclined to try to convert them to...
    2) position/heading/velocity. Individual clients could convert the data to something more useful [such as these values] before uploading them. It'd reduce strain on the server, and most clients will presumably already have the capability to do this.

    Suggestions?

    Gary (-;
    OBDGPSLogger, for logging OBDII and/or GPS data
    OBDSim, an OBDII/ELM327 software simulator
    mp3car forums: obdgpslogger, obdsim

  2. #2
    is back. FKA Robert Wray
    Auto Apps:loading...
    Fiberoptic's Avatar
    Join Date
    Jul 1978
    Location
    Baltimore, MD
    Posts
    1,419
    Blog Entries
    143
    I am not an expert on programming, algorithms, or the NEMA stream but my concern was that the parser might strip and delete too much raw data, which would limit the pool that an algorithm has to work with.
    Would it really take that much server load to send all the streams up and parse them server side? They could be processed once based on the currently accepted method.
    If the parsing theory changes the source data would always be there.
    If this concept isn't penny wise and pound foolish, I would be willing to fund all the CPU power we need to give algorithm developers the best chance for success.

    At the same time we don’t need to pay money to store and process junk.

  3. #3
    Variable Bitrate
    Auto Apps:loading...
    thekl0wn's Avatar
    Join Date
    Apr 2005
    Location
    PoCo, Indiana
    Posts
    284
    IMO, raw data would be best. Ad hoc info could be gathered from the same application, and the server(s) could be optimized to process data at idle time.

    Then again, I'm only about an hour into researching any of this...
    Play with it, 'til it's broke.

  4. #4
    Admin. Linux loser.
    Auto Apps:loading...
    Bugbyte's Avatar
    Join Date
    Sep 2004
    Location
    Corning, NY
    Posts
    7,359
    Blog Entries
    2
    Are there formats in addition to NMEA data that we can look at to see what data is collected? Maybe we should make a list of possible data to collect, check the goals of the project (free, open, routable data?) and select from that list plus any other data.

    For example, Thunderstick has a thread open right now where he is thinking about trying to build OCR recognition for speed limit signs in the US. If something like that were available, you would be able to estimate the posted speed limits on some roads.

    Given that we may not know all of the types of data that may come along, when formulating or picking a standard for the data, we ought to give some thought to providing the flexibility to extend the data, or integrate it with other data like video or audio. (Example: your GPS reports your position and as you drive, it pulls up text for historic markers along the roadside and reads them. Or, takes photos or vido and geotags them and uploads them to a server for real, real-time traffic info).

    Sorry, I think we're talking about the actual data to collect right now. I think I've veered into the standards area.

    How about we start this way: What is the goal of project and collecting the data?
    Quote Originally Posted by ghettocruzer View Post
    I was gung ho on building a PC [until] just recently. However, between my new phone having internet and GPS and all...and this kit...Im starting to have trouble justfiying it haha.
    Want to:
    -Find out about the new iBug iPad install?
    -Find out about carPC's in just 5 minutes? View the Car PC 101 video

  5. #5
    Variable Bitrate
    Auto Apps:loading...
    thekl0wn's Avatar
    Join Date
    Apr 2005
    Location
    PoCo, Indiana
    Posts
    284
    Glad to see my brain wasn't the only one working in overdrive on this... Then again, it's what I get paid to do, and my brain seems to operate based in SQL.

    But basically, all data I can forsee would be centered around a few key items:
    1.) Position -- XYZ, or Longitude, Latitude, and elevation
    2.) Direction
    3.) Speed/Velocity
    4.) DateTime
    5.) User

    From that, all other information I can think of off the top of my head can be added separately via either user input, or some form of data acquisition from other sources. Things like road names, speed limits, addresses, destinations, detours, images, and any other number of details like BugByte suggested.

    As I said, my brain works in SQL, and it's used to old-school database layouts using header/detail tables, so basically I see data coming in, and either new XYZ coordinates (table_1) being put in or their info updated. Each time that XYZ is "hit", the datetime/direction/speed/user is recorded (table_2). From there, the most basic information can be gathered from the data. Heck, at this point 3D maps could be generated!

    Personally, I think that's a good starting point for discussion of the data itself... Now, onto data acquisition from the hardware... Something I know nothing of at the moment.
    Play with it, 'til it's broke.

  6. #6
    Mp3Car Staff
    Auto Apps:loading...
    ecog's Avatar
    Join Date
    Aug 2005
    Posts
    92
    Blog Entries
    11
    How about we start this way: What is the goal of project and collecting the data?
    Lol I like the way you think bugbyte.

    So the goal would be to collect data in one place, process it and dump it to the OSM database.

    A secondary goal would be for the community to have GPS data to play around for their own map projects or other applications that have dependency on map data.

    I have been looking at NMEA streams and what kind of data they record (a good article here http://www.gpsinformation.org/dale/nmea.htm). From what I saw in the gps streams I have recorded (from a BU353 puck) and read online, hardware providers most of the time follow the NMEA 0183 standard.

    I have also been looking at OSM's data and what they need. From a pure data perspective their gpx trax files consist of:
    -GPS Tracks can contain multiple Track Segments each of which contains track points
    -The Tracks have name tags
    -The Track points have lat and long coordinates and a time stamp

    I attached a sample download from their database(file called gpxTrax_sampleData.zip )

    I have been playing around with the whole process just to get a feel for what you need to do to actually get some data on a map, label the data, render it and upload it to OSM for other people to see.......It's actually kinda complicated so far and I'm stuck on labeling data (using JOSM).

    Here's what I've done so far:
    -Record some GPS data (attached as "GPSSim(4).txt")
    -convert it to gpx
    -import it into JOSM
    -play around with it and produce a gpx file with more information
    -render it and use it in OSM

    The cool thing I noticed is that the NMEA stream has a ton of cool information that could be of use beyond OSM map generation (like maybe nav routing down the road?).

    So, anyway here's what I've been thinking:

    -dump a whole bunch of raw NMEA data to a server. Which then we can use to extract the important stuff (Lat, Long, Timestamp and maybe Elevation, and Speed).
    -Figure out how to automatically process this data to be useful to the OSM project (in the process weed out bad data, label it and organize it)
    -Build a database with this data that in turn can be used for...I'm sure somebody's going to figure something useful to do

    Since the initial goal of the project is to help out the OSM guys, collecting the data is the first step and making it useful to the OSM project is the second step. The "making it useful" might get a little tricky as it seems it needs a human to interact with the data, but there has to be a way to automate it.

    As to the data collection, we have an FTP server up and running. Potentially down the road we'll have a database, but we don't really know what kind of data we need right now, and it's a lot easier to record and dump a file at this stage of the game.

    What do you guys think? Let me know, I have a whole bunch of ideas brewing, but I thought I'd start with a little rant and some ideas
    Attached Files Attached Files

  7. #7
    Admin. Linux loser.
    Auto Apps:loading...
    Bugbyte's Avatar
    Join Date
    Sep 2004
    Location
    Corning, NY
    Posts
    7,359
    Blog Entries
    2
    How about accuracy? Knowing the number of satellites or the accuracy helps when you process the data - say if two track went down the same 4 lane highway, one might show off in the median while another is more accurate and is accurate enough to tell which lane it is in.
    Quote Originally Posted by ghettocruzer View Post
    I was gung ho on building a PC [until] just recently. However, between my new phone having internet and GPS and all...and this kit...Im starting to have trouble justfiying it haha.
    Want to:
    -Find out about the new iBug iPad install?
    -Find out about carPC's in just 5 minutes? View the Car PC 101 video

  8. #8
    Mp3Car Staff
    Auto Apps:loading...
    ecog's Avatar
    Join Date
    Aug 2005
    Posts
    92
    Blog Entries
    11
    How about accuracy? Knowing the number of satellites or the accuracy helps when you process the data - say if two track went down the same 4 lane highway, one might show off in the median while another is more accurate and is accurate enough to tell which lane it is in.
    That's why I was thinking collect the raw NMEA sentences and figure out what we need later on.

    The OSM guys don't need that info, but it would help if we were to do our own routing engine based on the OSM data.

  9. #9
    is back. FKA Robert Wray
    Auto Apps:loading...
    Fiberoptic's Avatar
    Join Date
    Jul 1978
    Location
    Baltimore, MD
    Posts
    1,419
    Blog Entries
    143
    Quote Originally Posted by Bugbyte View Post
    How about accuracy? Knowing the number of satellites or the accuracy helps when you process the data - say if two track went down the same 4 lane highway, one might show off in the median while another is more accurate and is accurate enough to tell which lane it is in.
    i completely agree. I would think your readings would get more accurate with every pass.

    You might even be able to determine where urban or physical signal "canyons" are to present a map of GPS dark areas. I am not sure what you would do with that, but it might be interesting.

    Collection all the raw data might also give you an insight into the health of the GPS satellite network.

  10. #10
    North of the land of Hey Huns
    Auto Apps:loading...

    Join Date
    Jun 2004
    Location
    Westminster, MD
    Posts
    2,127
    I think we should collect the same data that is contained in the NMEA sentences (eg all of it), but compress it to a binary format for upload to the server to save server space. It doesn't need to be collected and uploaded AS sentences, but I think we should save all the data. There is nothing worse than collecting a bunch of data, only to realize you left something out that could've been very helpful. Has there been any discussion on a data format as of yet? Fixed length field, TLV or something else? Being as GPS data is very reliably consistent, I think it would save space to go with fixed length field values, just my two cents.
    "stop with the REINSTALLS, what do you think we got some lame-o installer!!!" - mitchjs
    RevFE
    My Shop

Page 1 of 7 1234567 LastLast

Similar Threads

  1. Replies: 444
    Last Post: 10-21-2014, 06:50 PM
  2. Replies: 7
    Last Post: 04-06-2009, 05:14 PM
  3. Can you guys please help me??? OBD Renault Clio help needed!
    By madtoonbull in forum Engine Management, OBD-II, Engine Diagnostics, etc.
    Replies: 7
    Last Post: 02-19-2009, 11:36 AM
  4. Pinout Color Codes / Tables
    By gummybear in forum General Hardware Discussion
    Replies: 4
    Last Post: 05-12-2005, 04:05 AM
  5. VOICES Traffic Data
    By stevieg in forum V.O.I.C.E.S
    Replies: 85
    Last Post: 05-31-2004, 09:24 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •