What data to collect? - Page 2

thekl0wn · 07-06-2009, 09:37 AM

The sentences are standards... Therefore, I vote they be stored intact, line-by-line, in sequence on the server. At that point, any further information can be propagated from there. Surely, the server will have a mildy idle time where it can perform daily "builds" of the data.

A user uploads the formatted GPS output (like in the .txt file attached above) to the FTP. A parser application runs, which opens each new file, and inserts each sentence into a table, and then moves the text file into a save folder under a formatted name.

This is the most basic of the tasks, but it will also be an incredibly processor intensive one. All other data could easily (though, not fast) be propagated from a raw data table.

Fiberoptic · 07-06-2009, 09:47 AM

Quote: Originally Posted by Sicarius

Another question to answer is how many points per unit time do you want? Most people are going to be producing 1 point per second but mine is going to be putting out 10 per second.

I think 10Hz GPS outputs might be very valuable. What hardware are you using to collect that stream?

Sicarius · 07-06-2009, 12:35 PM

Quote: Originally Posted by Fiberoptic

I think 10Hz GPS outputs might be very valuable. What hardware are you using to collect that stream?

The Wintec WBT-300 / G-Rays I.

Bugbyte · 07-06-2009, 02:42 PM

@ FO and ecog - How about we do some experiments with NMEA data?

Why don't we compare a couple of datasets and compare the results of:

1. binary
2. zipped
3. raw

Pick some track data from a GPS that is 1 minute, 10 minutes, 20 minutes, 30 minutes and 1 hour long.

Figure the data sizes for all of the options and let's graph them to get an idea of how big the data storage requirements would be. Remember to take into consideration backups of the data both in time and in size/cost.

Then, we should see what kind of processing has to take place on the data to turn it into openstreetmap track information and run it against the datasets. You'll have to do this in the future to process the data and it would be a good idea to see if that would scale.

Then, we can calculate storage size and processing times for 1, 10, 100, 1000, and 10,000 users.

No sense starting it off one way and then finding out it won't scale.

Fiberoptic · 07-08-2009, 12:03 PM

For now we have created this account with the following permissions:

-upload files to the server
-view all the uploaded files in that directory
-Can't delete files
-can't download files

IP:12.167.132.204
ftp port 21:
username: datadump
password: datadump

File Format:
TicketID_DDMMYYYY_HHMMSS.txt
Get your ticket id: http://www.mp3car.com/ticket.php
The forum username on the ticket.php page is optional. It will only be used down the road for credit purposes (who uploaded how many tracks, etc....)

More on data format:
-Files should contain NMEA sentences
-files should be in text format
-a collection of text files in zipped format will be OK
-Files without the proper format will be given the lowest processing priority

Bugbyte · 07-08-2009, 12:12 PM

What's the address of the server?

malcom2073 · 07-08-2009, 12:21 PM

Why are you not allowing download of the uploaded data?

Heather · 07-08-2009, 01:20 PM

There are three key issues in data collection:

1. Authenticity - is the data accurate, real, or is it false, which requires a unique identifier so that data that is false can be stripped out or weighted.

2. Accessibility - the data needs to be accessible to all, so that people can use the data to create better algorithms, better maps, utilize the map data in front ends, etc.

3. Anonymity - everyone needs to be able to upload data and be confident that their identity will not be public. Therefore, data will only be accessible publicly once we determine how we will hide, stripped out or encrypt, the unique identifier necessary for authenticity. Please help Goce determine the best method for this, as once this is solved, the data will be accessible.

thekl0wn · 07-08-2009, 01:59 PM

I guess the question I pose is this: Is the entire goal aimed at adding to the OSM project, or are we going to have our own database to develop from?

Heather · 07-08-2009, 02:01 PM

Good question - no we will have our own database and we can give it to OSM, as well as others that want to make use of it. Also, all of the OSM data is available for download from their site.

Fiberoptic · 07-08-2009, 02:03 PM

Quote: Originally Posted by thekl0wn

I guess the question I pose is this: Is the entire goal aimed at adding to the OSM project, or are we going to have our own database to develop from?

At this point they are both goals because OSM doesn't support passive data collection, processing and the algorithms that might make this possible.

thekl0wn · 07-08-2009, 02:13 PM

So, what form of database is this going to be? (I ask because my db experience is limited to MS-SQL & SQL-Anywhere) I'm just curious as to the level of participation I could offer... Other than just providing data.

ecog · 07-08-2009, 02:47 PM

Quote:

So, what form of database is this going to be?

I'm thinking a Mysql DB? It's open source and reliable, especially for data dumps.

If this project gets big enough for the database to be a problem, it will be a good problem to have and we can always port to something else.

For data collecting purposes though the DB will be fairly simple. Once the data processing starts we'll might need to get more elaborate and think about it some more.

ecog · 07-08-2009, 03:41 PM

Quote: Originally Posted by Bugbyte

@ FO and ecog - How about we do some experiments with NMEA data?

Why don't we compare a couple of datasets and compare the results of:

1. binary
2. zipped
3. raw

Pick some track data from a GPS that is 1 minute, 10 minutes, 20 minutes, 30 minutes and 1 hour long.

Figure the data sizes for all of the options and let's graph them to get an idea of how big the data storage requirements would be. Remember to take into consideration backups of the data both in time and in size/cost.

Then, we should see what kind of processing has to take place on the data to turn it into openstreetmap track information and run it against the datasets. You'll have to do this in the future to process the data and it would be a good idea to see if that would scale.

Then, we can calculate storage size and processing times for 1, 10, 100, 1000, and 10,000 users.

No sense starting it off one way and then finding out it won't scale.

Just some stats from a GPS NMEA stream:

-text file is encoded in ASCII format
-Track length 18.3mi
-Trip time is about 30 min
-Number of NMEA sentences with a 0.5s sample rate from a BU-353 is 7651
-Size of the txt file is 460KB
-Compressed file size is 82KB

more in a little bit.....

Fiberoptic · 07-09-2009, 09:54 AM

Clarified temporary upload details in previous posts. and here.

User Name		Remember Me?
Password