Crowdsourced Maps Project - mods help/thoughts

**Heather** · 10-06-2009, 12:13 PM

I have been in contact with OSM for almost 6 months now, starting with an OSM meet up I attended in Baltimore. They have been going through some staffing changes and slow to respond. This is why they did not attend the AFK Fest which had originally been intended as an mp3Car-OSM event.

I have had multiple emails with the founder of OSM letting him know that we seek to help him. In particular we have asked for information from them regarding 1. what data they would like and 2. how they would like it. So far the only thing that we know is that mp3Car will most likely need to be able to verify that the data is real before we dump it on them. But i do know that we can do a data dump, as they have done this in the past.

So - i think that while we wait for feedback, let's start uploading for two reason -1. to show OSM that we intent to provide data that will help them and 2. to have deta to show them, which may allow them to provide better feedback in terms of what data they would like and how they would like it.

I emailed the founder last week with a link to this forum and lambosprit's application. I will wait another few days for a response and then follow up with OSM again. Sean will be working with lambospirit this afternoon to test out the new version and document setup, and Goce is documenting how you can access the data via ftp.

FYI - if you are wondering why mp3Car is pushing this, I can explain. I have a personal interest in helping put a dent in map data fees and navigation costs. As I have seen the invoices first hand, I am aware that these costs and upfront license requirements inhibit companies like Centrafuse from being able to create an excellent product. Agreed no need to reinvent the wheel, this data should help all others out there like OSM. However, I also think that innovation comes from where you least expect, and if forum members are interested in actually creating the maps and collaborating on routing engines...mp3Car will try to help in whatever way possible including using contacts that we have to bring in external resources. We do not see that this will drive profit, but rather grow the community which is something that we are firmly committed to doing.

**JohnWPB** · 10-06-2009, 03:12 PM

I would like to see a template/example of the data for each point to be collected.

I can put together a script that will be included with the upcoming release of the DFX skin, that can collect that data. When the skin is run for the very first time, a pop-up will explain what it is, and they can opt in or out of collecting the data.

I will have the plugin require that a 4 satellite GPS lock is available, to help keep the data as accurate as possible. If it is less than 3 satellites, no log entry will be made.

I will then need to create a script that they can run periodically that will upload the data to a server ect.

I know this is a RR solution only, but being it has the largest number of users in the forums, is a good way to start gathering the data for sure.

I just need to know all the detail before I can start to put this together.

**durwood** · 10-06-2009, 07:15 PM

Originally Posted by justchat_1

there we go...now everyones on the same page.

As far as data storage and interoperability - whos in contact with the OSM developers? Can we find out if .GPX files are easiest for them or if they would prefer batch adding via an API? Also, can we batch add under a single username (ex: mp3car) or will they want unique usernames?

http://wiki.openstreetmap.org/wiki/Upload
http://wiki.openstreetmap.org/wiki/Api06#GPS_Traces

**justchat_1** · 10-06-2009, 08:10 PM

Originally Posted by durwood

http://wiki.openstreetmap.org/wiki/Upload
http://wiki.openstreetmap.org/wiki/Api06#GPS_Traces

If you scrolled up-you would see thats not a viable solution:
http://wiki.openstreetmap.org/wiki/A...Authentication

Bugbyte & Heather:
Good info...will wait to hear back on that.

Some suggestions for the first pass (to hopefully take the best first step we can);
For the xml submitted:
* Change sequential numbering to a relative timestamp (makes it alot easier to validate-potentially useful for other things down the road)
* Add hdop and vdop to the submitted data (again-for error filtering)

For the serverside stuff:
* Use http not ftp for submission -> it deals with packet loss a lot better. Allow a raw xml POST, or a zipped file upload.
* On upload everything should be dumped to an sql database for storage
* A separate thread then processes all of the data in the queue...checking data validity and filtering as necessary to ensure an accurate submission...and finally packing it up and dumping it to OSM

Other things to consider:
* Many users want to ensure there is no way they can be tracked using this system, and be completely anonymous
* At the same time we need to ensure the data is valid and coming from a reliable source
* Would some type of one way unique id be beneficial in ensuring source validity? Possibly a salted hash?
* Would an ip based blacklist be enough?

Thoughts?

**Heather** · 10-07-2009, 10:37 AM

Originally Posted by justchat_1

For the serverside stuff:
* Use http not ftp for submission -> it deals with packet loss a lot better. Allow a raw xml POST, or a zipped file upload.
* On upload everything should be dumped to an sql database for storage
* A separate thread then processes all of the data in the queue...checking data validity and filtering as necessary to ensure an accurate submission...and finally packing it up and dumping it to OSM

Ok - let me talk to Goce

Originally Posted by justchat_1

Other things to consider:
* Many users want to ensure there is no way they can be tracked using this system, and be completely anonymous
* At the same time we need to ensure the data is valid and coming from a reliable source
* Would some type of one way unique id be beneficial in ensuring source validity? Possibly a salted hash?
* Would an ip based blacklist be enough?

Thoughts?

I thought that we are generating a unique identifier....so that we can weed out bad data providers. We discussed this and thought that banning an IP is really not sufficient.
Am i wrong?

In terms of privacy - The unique identifier should only be known by the uploader - they should hang onto it... mp3Car will not maintain a database of unique identifier and name so that we do not know who is driving where. I am unsure if this is what is happening based on the description of the app.
can you comment lambosprit?

Also - don't we have the program cutting off the tracks at some random place between x and y miles from where the tracks began. Lambosprit explained here: Crowdsourced Map Data Collection App - How it works. in sticky above.

**lambosprit** · 10-07-2009, 04:32 PM

Originally Posted by Heather

Ok - let me talk to Goce

If this is the way you want it done just tell me and I'll amend my app.

Originally Posted by Heather

I thought that we are generating a unique identifier....so that we can weed out bad data providers. We discussed this and thought that banning an IP is really not sufficient.
Am i wrong?

Nope. the app generates its own unique id when its installed. Its only known to the user so provides anonymity. However if people want to see how many miles, points etc they've logged then they could search using it or reveal it if they wish. Its up to us to decide how we want to show that.

Originally Posted by Heather

In terms of privacy - The unique identifier should only be known by the uploader - they should hang onto it... mp3Car will not maintain a database of unique identifier and name so that we do not know who is driving where. I am unsure if this is what is happening based on the description of the app.
can you comment lambosprit?

Thats exactly whats happening.

Originally Posted by Heather

Also - don't we have the program cutting off the tracks at some random place between x and y miles from where the tracks began. Lambosprit explained here: Crowdsourced Map Data Collection App - How it works. in sticky above.

yes. again for privacy we delete a random number of points (with a user definable minimum) from the beginning and end of each track. And yes I did explain that in the sticky.

**justchat_1** · 10-07-2009, 05:19 PM

Originally Posted by lambosprit

Nope. the app generates its own unique id when its installed. Its only known to the user so provides anonymity. However if people want to see how many miles, points etc they've logged then they could search using it or reveal it if they wish. Its up to us to decide how we want to show that.

Originally Posted by lambosprit

In terms of privacy - The unique identifier should only be known by the uploader - they should hang onto it... mp3Car will not maintain a database of unique identifier and name so that we do not know who is driving where. I am unsure if this is what is happening based on the description of the app.
can you comment lambosprit?

Thats exactly whats happening.

In terms of real world privacy that really doesn't offer any. This still allows anyone with your "uniqueid" access to everywhere you've been. It also could potentially be subpoenaed and used in court. Are either of those cases likely? probably not-but I do remember hearing that being a concern from some forums members.
More importantly for me, is the complete lack of data protection a client side uniqueid provides. Its a sad but true fact that you put a system with potential to be used for malicious purposes and someone will try. Especially given the open specs, someone could easily add 100,000 bogus tracks to the database in a few minutes...and then do the same thing again the next day.

Originally Posted by lambosprit

yes. again for privacy we delete a random number of points (with a user definable minimum) from the beginning and end of each track. And yes I did explain that in the sticky.

Well according to that sticky:

It ignores a random number of points when the app is started or comes out of hibernation/suspend.
It deletes a random number of collected points when the app is started or comes out of hibernate/suspend (this is because most of my tracks where to and from work so every end of journey showed where I started previously, not good for privacy!)

I was just saying if there will be other implementations of this idea it would help if that was more specific (i was thinking it was some fixed range). So your saying its a user selectable range of points removed from the start and end of each trip?

**lambosprit** · 10-07-2009, 06:09 PM

Originally Posted by justchat_1

I was just saying if there will be other implementations of this idea it would help if that was more specific (i was thinking it was some fixed range). So your saying its a user selectable range of points removed from the start and end of each trip?

No its random but with a min setting. so the user can say that atleast 10 points are ignore but it could be more. This was done because of suggestions from another user about privacy, as was the client side user id.

My opinion is that we have to get people on board and to do that they need to be sure of their privacy. At the moment nobodies uploading let alone putting junk in.

**Heather** · 10-08-2009, 05:35 PM

On 2 Oct 2009, at 08:05, Heather Sarkissian wrote:
> Steve
>
> I would like to have the mp3Car forums help OSM by providing data to
> improve your existing maps.

Cool!

> The mp3Car forum is full of car computer users from around the world
> who can easily collect NMEA and other data and automatically submit
> it to us. We are working with our forum members to automate the
> process of collecting the data from car computers. We will next
> move on to collection from iPhones.
>
> Could you let me know:
> 1. If you can use NMEA data to help generate the maps and

It's part of the problem - we use it as an input but at best it gives
you geometry - no metadata.

> 2. What other data might be useful?

PoIs, streetnames, oneway information...

> There is an explanation of the collection application that one of
> our forum members created to automate collection: http://www.mp3car.com/vbulletin/crow...how-works.html
>
> Feel free to post in the forums or you can email me any thoughts
> that you would have in terms of what data OSM needs, how OSM would
> like to receive it, and whether our current application will provide
> this data.

So that app you link to is basically there, it just needs to spit out
GPX files instead of a bespoke XML format.

Yours &c.

Steve

**justchat_1** · 10-08-2009, 06:04 PM

Also, can we batch add under a single username (ex: mp3car) or will they want unique usernames?

^ We still need to know that ^

Also, if you can ask what type of safeguards they want on data integrity? Do we just need to check for device related errors? prevent flooding? or do we need to be able to provide accountability? (ex: if someone submits bad data prevent them from doing it again)