New mp3car passive maps server

Fiberoptic · 07-02-2009, 11:21 AM

I have put up a new server so we can quickly test ideas.
When we get closer to our final configuration we can put the proper os on there and reconfigure.
It is plugged into our 100MB internet pipe.
HW Specs:
Intel Dual core 2.8 Ghz
4gb ram
160 gig software mirror
Offsite internet backups
1000VA UPS backup

Software:
Windows 7
Filezilla FTP server
Idrive remote backup
Antivirus

For now we have created this account with the following permissions:

-upload files to the server
-view all the uploaded files in that directory
-Can't delete files
-can't download files

IP:12.167.132.204
ftp port 21:
username: datadump
password: datadump

File Format:
TicketID_DDMMYYYY_HHMMSS.txt
Get your ticket id: http://www.mp3car.com/ticket.php
The forum username on the ticket.php page is optional. It will only be used down the road for credit purposes (who uploaded how many tracks, etc....)

More on data format:
-Files should contain NMEA sentences
-files should be in text format
-a collection of text files in zipped format will be OK
-Files without the proper format will be given the lowest processing priority

DarquePervert · 07-02-2009, 01:07 PM

Quote: Originally Posted by Fiberoptic

Ok, now we have to figure out what to put on it.

AntiVirus and a Firewall for your protection.

Bugbyte · 07-02-2009, 03:33 PM

Need some specs on how to access data/put data in a folder on the server. Maybe how to query it up, etc.

Fiberoptic · 07-02-2009, 03:40 PM

Chunky and Ecog have the admin password. I think those things are in the works.

Fiberoptic · 07-08-2009, 12:02 PM

added login details.

Fiberoptic · 07-09-2009, 09:52 AM

added details about file format and upload process

thekl0wn · 07-09-2009, 10:49 AM

Is there any interest in a simple user/client-based application which would save the NMEA data to a properly formatted file? And weed out any obviously erroneous data.

My thoughts on this would be that it could add a header to the file which could provide useful info for server processing... And of course be scalable to the project. For issues such as those discussed in the Data Privacy Thread.

I'm actually in the process of writing an application to do this for myself (to get the data into the MP3car-specified format for upload), and figured I could offer it up to everyone else as well. If there's interest, please let me know what the specs/features you would be looking for.

Had I brought my thumb drive today, I could have posted screenshots.

Fiberoptic · 07-09-2009, 10:55 AM

Quote: Originally Posted by thekl0wn

Is there any interest in a simple user/client-based application which would save the NMEA data to a properly formatted file? And weed out any obviously erroneous data.

My thoughts on this would be that it could add a header to the file which could provide useful info for server processing... And of course be scalable to the project. For issues such as those discussed in the Data Privacy Thread.

I'm actually in the process of writing an application to do this for myself (to get the data into the MP3car-specified format for upload), and figured I could offer it up to everyone else as well. If there's interest, please let me know what the specs/features you would be looking for.

Had I brought my thumb drive today, I could have posted screenshots.

This is a great idea. Idealy the app and eventual data collection format would be flexible enought to except more fields, if the user wanted to share them:

1. accelerometer data
2. Wheel turn angle (from can bus)
3. Temprature sensors
4. Other OBD-II sensors
5. TPMS sensors
6. Client hardware paramenters (CPU/ OS/ connectivity type)

The predominent OS from our community is windows, but maybe there is a way to make something cross platform or make an existing app cross platform?

thekl0wn · 07-09-2009, 11:01 AM

As for me developing it... Cross-platform isn't really an option. I'm Windows-based, and biased. However, for these simple applications (and the language I'm using), posting the source code of how I handle the processes is straightforward enough that most anyone with programming experience could translate.

Another thought I had, was that later on, the application could envelope multiple files, and then once the data is formatted, it could create the single compressed file for upload. (if anyone knows of either a commandline zipper/compressor or a .NET component, please let me know)

Brain... Spinning... Hamster... Wheezing...

thekl0wn · 07-09-2009, 12:11 PM

On the page where we get enter our username to get our numbers for uploading, could we go ahead and put out a simple disclaimer? Preferably with a checkbox stating "I agree" and triggering the ok button. You know, a simple note, stating "Any information I provide or use is of testing purposes and not guaranteed to be 100% accurate, but as accurate as possible, and I'm not gonna use the data to stalk people, nor am I gonna endanger people while driving on sidewalks to get cool tracks. Basically, I take all blame for any wrong-doing I do." Well, maybe someone else should write the disclaimer.

Step #1 in arse-covering: Always put the blame on the user!

chunkyks · 07-09-2009, 01:24 PM

Before it gets too late to change this, are we absolutely sure that DDMMYYYY is the best way to go? Among other things, DDMM is an unusual or confusing order for many people outside of europe. A useful short discussion of ISO8601 is here. The most important reasons to use ISO date format in my experience are twofold:
1) Unambiguous. I deal with people from the UK and the US on a daily basis, and an unambiguous date format is the difference between effing nightmare and ... not effing nightmare
2) Sortable. Working through a huge directory listing to find "the nearest trace to a specific date" is a nightmare

Specifying time - what use will that be, other than to separate tracks from the same day? If the time is used to mean anything, then should it be converted to UTC? If not, then a TZ suffix is important.

More on the list of "nightmare things I have to deal with on a regular basis"

Gary

PS Apologies if this is in the realm of the bikeshed, but date formats are a particular bugbear to me.

Fiberoptic · 07-09-2009, 01:30 PM

Quote: Originally Posted by chunkyks

Before it gets too late to change this, are we absolutely sure that DDMMYYYY is the best way to go? Among other things, DDMM is an unusual or confusing order for many people outside of europe. A useful short discussion of ISO8601 is here. The most important reasons to use ISO date format in my experience are twofold:
1) Unambiguous. I deal with people from the UK and the US on a daily basis, and an unambiguous date format is the difference between effing nightmare and ... not effing nightmare
2) Sortable. Working through a huge directory listing to find "the nearest trace to a specific date" is a nightmare

Specifying time - what use will that be, other than to separate tracks from the same day? If the time is used to mean anything, then should it be converted to UTC? If not, then a TZ suffix is important.

More on the list of "nightmare things I have to deal with on a regular basis"

Gary

PS Apologies if this is in the realm of the bikeshed, but date formats are a particular bugbear to me.

The theory behind the seconds was to create multiple file names if more than one file was created in the same minute for some reason.

I like the idea of international standards, and good point about converting to UTC.

Anyone else have an opinion on this?

Fiberoptic · 07-09-2009, 01:33 PM

The gist of that iso link is here:

Basically to comply with the 'full' ISO format you need to:
- Use 4-digit years when storing or printing dates.
- Use the order Year-Month-Day for the date, i.e. biggest first.
- Use leading zeroes on the digits 00 - 09 for the month and day numbers.
- Always put the Date BEFORE the Time.
- Use the order hours:minutes:seconds for times.
- Use the 24-hour format for times (not the 12-hour am/pm).
- Use a leading zero on hour, minute and second for digits 00 - 09 inclusive.
- Use UT time scale for dates and times if transferring data internationally.
- When printing dates and times use the '-' and ':' separators.
e.g. 1996-12-31 23:59:59 is the last second of last year.
- When storing dates you can strip off the separators and store as a 32-bit
number, an ASCII string, packed BCD or whatever you like. Sort algorithms
are much easier with this YMD format.
- Times can be treated the same way (sorted or stored), as in the discussion
for dates.

thekl0wn · 07-09-2009, 01:53 PM

Correct me if I'm wrong, but didn't I read in the NMEA information that the timestamps used in sentencing is UTC? This is an important bit of info for parsing the files.

And I'm all-for the YYYYMMDD and HHMMSS formats! As for naming conventions, is the filename for first recorded second?

chunkyks · 07-09-2009, 02:20 PM

Quote:

Basically to comply with the 'full' ISO format you need to:
- Use 4-digit years when storing or printing dates.
- Use the order Year-Month-Day for the date, i.e. biggest first.
- Use leading zeroes on the digits 00 - 09 for the month and day numbers.
- Always put the Date BEFORE the Time.
- Use the order hours:minutes:seconds for times.
- Use the 24-hour format for times (not the 12-hour am/pm).
- Use a leading zero on hour, minute and second for digits 00 - 09 inclusive.
- Use UT time scale for dates and times if transferring data internationally.
- When printing dates and times use the '-' and ':' separators.
e.g. 1996-12-31 23:59:59 is the last second of last year.
- When storing dates you can strip off the separators and store as a 32-bit
number, an ASCII string, packed BCD or whatever you like. Sort algorithms
are much easier with this YMD format.
- Times can be treated the same way (sorted or stored), as in the discussion
for dates.

Sounds right. It's worth noting that : in filenames is bad karma [no matter what the standards may say], so I always store the time packed without separators.

A simpler rule to remember ordering is "it's big endian" - store units biggest-first. Eg, years are bigger than months, and days are bigger than hours.

Yes, grabbing the timestamp exactly as the NMEA sentence mentions it is UTC, but a lot of userland programs will be converting to local time, so beware.

Gary (-;

User Name		Remember Me?
Password