What data to collect?

**Bugbyte** · 07-08-2009, 12:12 PM

What's the address of the server?

**malcom2073** · 07-08-2009, 12:21 PM

Why are you not allowing download of the uploaded data?

**Heather** · 07-08-2009, 01:20 PM

There are three key issues in data collection:

1. Authenticity - is the data accurate, real, or is it false, which requires a unique identifier so that data that is false can be stripped out or weighted.

2. Accessibility - the data needs to be accessible to all, so that people can use the data to create better algorithms, better maps, utilize the map data in front ends, etc.

3. Anonymity - everyone needs to be able to upload data and be confident that their identity will not be public. Therefore, data will only be accessible publicly once we determine how we will hide, stripped out or encrypt, the unique identifier necessary for authenticity. Please help Goce determine the best method for this, as once this is solved, the data will be accessible.

**thekl0wn** · 07-08-2009, 01:59 PM

I guess the question I pose is this: Is the entire goal aimed at adding to the OSM project, or are we going to have our own database to develop from?

**Heather** · 07-08-2009, 02:01 PM

Good question - no we will have our own database and we can give it to OSM, as well as others that want to make use of it. Also, all of the OSM data is available for download from their site.

**Fiberoptic** · 07-08-2009, 02:03 PM

Originally Posted by thekl0wn

I guess the question I pose is this: Is the entire goal aimed at adding to the OSM project, or are we going to have our own database to develop from?

At this point they are both goals because OSM doesn't support passive data collection, processing and the algorithms that might make this possible.

**thekl0wn** · 07-08-2009, 02:13 PM

So, what form of database is this going to be? (I ask because my db experience is limited to MS-SQL & SQL-Anywhere) I'm just curious as to the level of participation I could offer... Other than just providing data.

**ecog** · 07-08-2009, 02:47 PM

So, what form of database is this going to be?

I'm thinking a Mysql DB? It's open source and reliable, especially for data dumps.

If this project gets big enough for the database to be a problem, it will be a good problem to have and we can always port to something else.

For data collecting purposes though the DB will be fairly simple. Once the data processing starts we'll might need to get more elaborate and think about it some more.

**ecog** · 07-08-2009, 03:41 PM

Originally Posted by Bugbyte

@ FO and ecog - How about we do some experiments with NMEA data?

Why don't we compare a couple of datasets and compare the results of:

1. binary
2. zipped
3. raw

Pick some track data from a GPS that is 1 minute, 10 minutes, 20 minutes, 30 minutes and 1 hour long.

Figure the data sizes for all of the options and let's graph them to get an idea of how big the data storage requirements would be. Remember to take into consideration backups of the data both in time and in size/cost.

Then, we should see what kind of processing has to take place on the data to turn it into openstreetmap track information and run it against the datasets. You'll have to do this in the future to process the data and it would be a good idea to see if that would scale.

Then, we can calculate storage size and processing times for 1, 10, 100, 1000, and 10,000 users.

No sense starting it off one way and then finding out it won't scale.

Just some stats from a GPS NMEA stream:

-text file is encoded in ASCII format
-Track length 18.3mi
-Trip time is about 30 min
-Number of NMEA sentences with a 0.5s sample rate from a BU-353 is 7651
-Size of the txt file is 460KB
-Compressed file size is 82KB

more in a little bit.....