VOICES Traffic Data

stevieg · 05-10-2004, 05:56 PM

Mentioned this to SilentAdmirer a few weeks ago, I've got started on this though at last.

Was looking at making a web application to allow a user to register, configure a number of web sources of traffic data etc

VOICES would then be able to download this data processed into a personalised single format and do whatever with it.

I've been busy last week or so but have a few days to write the basic framework for a start. I was going to host on a box under my desk at work but to make sure the site won't be going anywhere I've shelled out for some extra hosting for the extra bandwidth and MySQL hosting, so that parts all sorted.

I've got at least one working data source (from BBC/goverment web site for UK) and know of a couple of others that need a little more work processing the data; I will need help in finding sources of data so I can then write the code to feed this into the database.

SilentAdmirer, are you still interested in implementing this?

Also is anyone else interested in this once they can see a beta of how it will work?

UPDATE: 11 May 20:17 BST

Right, now have something workable.

Website is http://www.stevieg.org/traffic - You need to register on it don't worry you'll get no spam by entering your email address, you need it for the password to be sent.

Currently three data sources, two types - Route 66 TMC and BBC TPEG. The route 66 data needs more work as the lat/lon it gives are wrong at the moment.

Data is downloadable as XML or CSV. The CSV is the easiest format if you are looking to test the data by simply importing into MapPoint or similar.

Notes:
* This is a beta version so expect a couple of bugs if I haven't seen them yet
* I know the only usable data so far is the UK, but feel free to give me some more data sources that provide the right information, I'll have a go at parsing anything. If you can fix the Route 66 lat/lon, tell me.
* You'll see the URL for the route 66 data is not pointing at the route 66 server - this is because my hosting provider blocks connecting by the server to port 8080, so I've had to put a 'proxy' page in. The data is live.
* Type of incident is not implemented yet, it will be given time though, I've only had a few hours on and off during the day to write all this including the parser back end.
* Once again it's not finished, there are more feature to add obviously.

UPDATE 11 May 22:06 BST
* UK Route 66 data now works correctly, thanks wildcard

UPDATE 26 May 23:18 BST
* Proof of Concept App Available for Download http://www.stevieg.org/traffic/traffic.zip
See http://www.mp3car.com/vbulletin/show...638#post209638 for more info

veetid · 05-10-2004, 07:09 PM

I am in the process of working on something similiar myself. I have written an application currently called "CarConnection", which currently loads up and uses mappoint. It has a little map in the corner and goes to your current location and tracks you as you move (Basic GPS location). It constantly updates your address and I use the postalcode. It has 4 major buttons on the left.

Traffic
Weather
Web Search
Email

I wrote it in C#. Currently I pull traffic images based on zipcode from http://www.sigalert.com. This is only a source for California traffic though. I scrap all my data and images from there. I use msnbc and weather.com for my weather information. I based that logic on MythWeather (a linux weather program for the Myth PVR system) The email client I wrote in C# and its embedded. The Web Search just goes to google and allows you to use the virtual keyboard from VOICES. Anyway... Your idea sounds awesome integrating the pushpins into Mappoint. I can prolly help give you a reliable source for California data. I should have a beta of my software in the next week also, for people who want to play around with it.

rando · 05-10-2004, 07:18 PM

This is a cool idea and one I've been thinking about for a while. It's good to know that other people have already started on it. If only some of this stuff were open source (sigh).

rando · 05-10-2004, 07:19 PM

It's soooo coool, I have to post twice.

mobileh1 · 05-10-2004, 07:59 PM

I'd be intersted in this also, are you willing to share source?

skippy76 · 05-10-2004, 08:54 PM

Obviously adding this to VOICES is the way to go. XML format to VOICES would be the most ideal medium.

I (obviously along with many others) was considerring something like this. The key is indexing sources of data by country: For the UK:

www.theaa.com

Maybe if you can get a website up, with the ability to recommend traffic sites, this would be a good starting point.

mobileh1 · 05-10-2004, 09:12 PM

where it integrates is one thing, the crux of this, for me, is to get traffic data in a format that is workable, say XML, with more than just a highway designation, but a lat/lon.

veetid · 05-11-2004, 12:03 AM

There is a bunch of traffic data scattered, but most of the US data is in the format of intersections. Street names and Interstate/Highways. I have yet to come up with the best method of getting this added into Mappoint via pushpins. Mappoint will accept intersections, but it doesn't do it that well with Interstates, at least for me. Currently I have the weather module completed very nice. It grabs postalcode from Mappoint, based on GPS, and gathers a bunch of weather info, radar map, forecast. At the moment I just use zipcodes to determine regions for California and link to images on www.sigalert.com for my traffic. I have started to work with a method for taking the intersection information and implementing it into Mappoint. Any good suggestions?

When I do get something working I will post the source.

veetid · 05-11-2004, 02:53 AM

One of my ideas is to just geocode my addresses and get a lat/long. I have tested with MapMarker (I have it because we use it at work), but would not be able to redistribute or point to my work servers. I investigated and found some good free sources such as:

http://www.stevemorse.org/jcal/latlon.php

If you look, he gives two results from two different sources. Both are farely reliable and use a simple post that could be used for scraping purposes. Fairly slow lookups, but it could do the trick. Basically scrape a site and gather all the Street Names / Speeds.... Then post these streets to one of these geocode services. Then you would have lat/long/speed for an area. That is all you would need to very easily add pushpins into Mappoint. I am going to see how good it works sometime tomorrow. I know of multiple services for multiple US states that you could easily scrap street address/speeds from. Sound like a decent idea? Any better ones? I wish we had a legit Mapmarker server somewhere to use, but these free geocode services might work out ok.

SilentAdmirer · 05-11-2004, 03:06 AM

@stevieg Of course I am interested. Will start with VOICES again from today evening. Looking forward to integrate the traffic data in VOICES ASAP.

stevieg · 05-11-2004, 04:30 AM

Have started this morning along with a few other things, I will keep the first post updated with more info as I get somewhere

tekta · 05-11-2004, 04:51 AM

sounds great y'all!

skippy76 · 05-11-2004, 06:35 AM

@Stevig: If you need webspace, mysql and PHP access, I can provide you wish an account on my webserver (the same as Satish uses for VOICES). Currently I have about 140 Gb of free space on my server, so space is not a problem.

Obviously the key to a successful interface is to keep the amount of data transferred to a minimum. The client (VOICES) already has the maps, so no maps would need to be transferred. Obviously some method of parssing websites would be needed, and each new website added would need it's own parser.

I know for the UK, Route 66 application has an 'update traffic' button. You may want to sniff / Etherreal the communication with the Route 66 server. I presume it's eithere an FTP or HTTP pull. You could then use the same data format for communication as them. Route 66 software can be found at www.66.com If you need some Ethereal trace I can get that for you.

skippy76 · 05-11-2004, 06:37 AM

Update: I notice that Route 66 now provide traffic info for: "Free up-to-date traffic information for NL, F, I, D, and GB "

Let's just use their interface!!! Free data!!! Route 66 has just entered the US Market as well, so hopefully before long they maybe a good source for US Traffic data.

Andrew

stevieg · 05-11-2004, 07:56 AM

Quote: Originally Posted by skippy76

@Stevig: If you need webspace, mysql and PHP access, I can provide you wish an account on my webserver (the same as Satish uses for VOICES). Currently I have about 140 Gb of free space on my server, so space is not a problem.

Thanks for the offer, if the traffic gets expensive I might take you up on that! At the moment, I've just added MySQL to my existing web hosting.

Quote: Originally Posted by skippy76

Obviously the key to a successful interface is to keep the amount of data transferred to a minimum. The client (VOICES) already has the maps, so no maps would need to be transferred. Obviously some method of parssing websites would be needed, and each new website added would need it's own parser.

Indeed, that's exactly what I'm doing. I've written a couple of parsers already, I'm just re-working them from on-demand to scheduled (every 5 mins) to keep the lot in a single up-to-date db table.

Quote: Originally Posted by skippy76

I know for the UK, Route 66 application has an 'update traffic' button. You may want to sniff / Etherreal the communication with the Route 66 server. I presume it's eithere an FTP or HTTP pull. You could then use the same data format for communication as them. Route 66 software can be found at www.66.com If you need some Ethereal trace I can get that for you.

We're defininately on the same train of thought. I've already done as you suggest for Route 66, which I bought a while back, but the location data doesn't seem to be lat/lon.

Here is the URL it uses for UK: http://rs0124.66.com:8080/tmc?time=0&l=ENG&m=eng8&u=1
and here is the URL for Germany: http://rs0124.66.com:8080/tmc?time=0&l=ENG&m=deu8&u=1

Be good if someone can convert this, I've had a brief look but doesn't seem to be any recognised representation of lat/lon. Quick script I wrote a few weeks ago to make it importable into MapPoint is http://supra.aston.ac.uk/tmc.php but that shows that the lat/lon is wrong.

The first feed that I've got working and plotting the correct locations comes from this URL: http://www.bbc.co.uk/travelnews/xml/tpegml_en/rtm.xml - Which gets and shares data with the UK highways agency - http://www.tih.org.uk/TIH/Index.do