Thread: What data to collect?

I am not an expert on programming, algorithms, or the NEMA stream but my concern was that the parser might strip and delete too much raw data, which would limit the pool that an algorithm has to work with.
Would it really take that much server load to send all the streams up and parse them server side? They could be processed once based on the currently accepted method.
If the parsing theory changes the source data would always be there.
If this concept isn't penny wise and pound foolish, I would be willing to fund all the CPU power we need to give algorithm developers the best chance for success.

At the same time we don’t need to pay money to store and process junk.

IMO, raw data would be best. Ad hoc info could be gathered from the same application, and the server(s) could be optimized to process data at idle time.

Then again, I'm only about an hour into researching any of this...

Are there formats in addition to NMEA data that we can look at to see what data is collected? Maybe we should make a list of possible data to collect, check the goals of the project (free, open, routable data?) and select from that list plus any other data.

For example, Thunderstick has a thread open right now where he is thinking about trying to build OCR recognition for speed limit signs in the US. If something like that were available, you would be able to estimate the posted speed limits on some roads.

Given that we may not know all of the types of data that may come along, when formulating or picking a standard for the data, we ought to give some thought to providing the flexibility to extend the data, or integrate it with other data like video or audio. (Example: your GPS reports your position and as you drive, it pulls up text for historic markers along the roadside and reads them. Or, takes photos or vido and geotags them and uploads them to a server for real, real-time traffic info).

Sorry, I think we're talking about the actual data to collect right now. I think I've veered into the standards area.

How about we start this way: What is the goal of project and collecting the data?

Glad to see my brain wasn't the only one working in overdrive on this... Then again, it's what I get paid to do, and my brain seems to operate based in SQL.

But basically, all data I can forsee would be centered around a few key items:
1.) Position -- XYZ, or Longitude, Latitude, and elevation
2.) Direction
3.) Speed/Velocity
4.) DateTime
5.) User

From that, all other information I can think of off the top of my head can be added separately via either user input, or some form of data acquisition from other sources. Things like road names, speed limits, addresses, destinations, detours, images, and any other number of details like BugByte suggested.

As I said, my brain works in SQL, and it's used to old-school database layouts using header/detail tables, so basically I see data coming in, and either new XYZ coordinates (table_1) being put in or their info updated. Each time that XYZ is "hit", the datetime/direction/speed/user is recorded (table_2). From there, the most basic information can be gathered from the data. Heck, at this point 3D maps could be generated!

Personally, I think that's a good starting point for discussion of the data itself... Now, onto data acquisition from the hardware... Something I know nothing of at the moment.

How about we start this way: What is the goal of project and collecting the data?

Lol I like the way you think bugbyte.

So the goal would be to collect data in one place, process it and dump it to the OSM database.

A secondary goal would be for the community to have GPS data to play around for their own map projects or other applications that have dependency on map data.

I have been looking at NMEA streams and what kind of data they record (a good article here http://www.gpsinformation.org/dale/nmea.htm). From what I saw in the gps streams I have recorded (from a BU353 puck) and read online, hardware providers most of the time follow the NMEA 0183 standard.

I have also been looking at OSM's data and what they need. From a pure data perspective their gpx trax files consist of:
-GPS Tracks can contain multiple Track Segments each of which contains track points
-The Tracks have name tags
-The Track points have lat and long coordinates and a time stamp

I attached a sample download from their database(file called gpxTrax_sampleData.zip )

I have been playing around with the whole process just to get a feel for what you need to do to actually get some data on a map, label the data, render it and upload it to OSM for other people to see.......It's actually kinda complicated so far and I'm stuck on labeling data (using JOSM).

Here's what I've done so far:
-Record some GPS data (attached as "GPSSim(4).txt")
-convert it to gpx
-import it into JOSM
-play around with it and produce a gpx file with more information
-render it and use it in OSM

The cool thing I noticed is that the NMEA stream has a ton of cool information that could be of use beyond OSM map generation (like maybe nav routing down the road?).

So, anyway here's what I've been thinking:

-dump a whole bunch of raw NMEA data to a server. Which then we can use to extract the important stuff (Lat, Long, Timestamp and maybe Elevation, and Speed).
-Figure out how to automatically process this data to be useful to the OSM project (in the process weed out bad data, label it and organize it)
-Build a database with this data that in turn can be used for...I'm sure somebody's going to figure something useful to do

Since the initial goal of the project is to help out the OSM guys, collecting the data is the first step and making it useful to the OSM project is the second step. The "making it useful" might get a little tricky as it seems it needs a human to interact with the data, but there has to be a way to automate it.

As to the data collection, we have an FTP server up and running. Potentially down the road we'll have a database, but we don't really know what kind of data we need right now, and it's a lot easier to record and dump a file at this stage of the game.

What do you guys think? Let me know, I have a whole bunch of ideas brewing, but I thought I'd start with a little rant and some ideas

How about accuracy? Knowing the number of satellites or the accuracy helps when you process the data - say if two track went down the same 4 lane highway, one might show off in the median while another is more accurate and is accurate enough to tell which lane it is in.

How about accuracy? Knowing the number of satellites or the accuracy helps when you process the data - say if two track went down the same 4 lane highway, one might show off in the median while another is more accurate and is accurate enough to tell which lane it is in.

That's why I was thinking collect the raw NMEA sentences and figure out what we need later on.

The OSM guys don't need that info, but it would help if we were to do our own routing engine based on the OSM data.

Originally Posted by Bugbyte

How about accuracy? Knowing the number of satellites or the accuracy helps when you process the data - say if two track went down the same 4 lane highway, one might show off in the median while another is more accurate and is accurate enough to tell which lane it is in.

i completely agree. I would think your readings would get more accurate with every pass.

You might even be able to determine where urban or physical signal "canyons" are to present a map of GPS dark areas. I am not sure what you would do with that, but it might be interesting.

Collection all the raw data might also give you an insight into the health of the GPS satellite network.

I think we should collect the same data that is contained in the NMEA sentences (eg all of it), but compress it to a binary format for upload to the server to save server space. It doesn't need to be collected and uploaded AS sentences, but I think we should save all the data. There is nothing worse than collecting a bunch of data, only to realize you left something out that could've been very helpful. Has there been any discussion on a data format as of yet? Fixed length field, TLV or something else? Being as GPS data is very reliably consistent, I think it would save space to go with fixed length field values, just my two cents.