Fusion Brain Data Logging - Page 2

inh · 09-05-2007, 01:46 PM

It's definatly worth it man, I'm the same way. I got all my planned upgrades done for the next release of RRCam, but by then i had a bunch of little ideas, and rewrote half the codebase

So worth it in the end though.

This is going to be skinnable, right? Aside from mean/median/mode of the values, how about a graph of all values, or a graph every x seconds or so? Being able to apply equations to the graphed output would be awesome too, kinda like a spreadsheet app

2k1Toaster · 09-05-2007, 02:09 PM

Quote: Originally Posted by inh

It's definatly worth it man, I'm the same way. I got all my planned upgrades done for the next release of RRCam, but by then i had a bunch of little ideas, and rewrote half the codebase

So worth it in the end though.

This is going to be skinnable, right? Aside from mean/median/mode of the values, how about a graph of all values, or a graph every x seconds or so? Being able to apply equations to the graphed output would be awesome too, kinda like a spreadsheet app

not sure with what you mean apply equations to it.

But yes the graph time span will be changeable. Not in a skin but all through javascript!

Since it is all "client side" it was really my only choice. That and it is easy.

inh · 09-05-2007, 02:19 PM

Cheater!

Whats going through my mind when i say equations is that basically, I would liek to use an accelerometer to measuer acceleration (yes, I know, very creative) and estimate power from that. I've dont it before based off of the datalogging from my cars computer, which spit out a timestamp and the vehicle speed...

Its probably too complicated and niche, best left to spreadsheet apps (which is why csv would be nice =] )

ginigma · 09-05-2007, 07:32 PM

Quote: Originally Posted by 2k1Toaster

"sensor,port,vidpid,yyyymmdd,hh:mm:ss:uu,raw,evalua ted"

so for the first data bit in the pic above, it would be "Ultrasonic Sensor,0,0,20071225,16:30:30:500,3.456,10.65". Good enough?

That works for me. The UI and graphs look great, even though I probably wouldn't use them. I basically will import the data into Excel and do what I need to do there, but I understand that others want the cool stuff.

Looking good...

2k1Toaster · 09-10-2007, 03:16 AM

Ok, here is some more updates:

When you just open the webpage, you see the first image. All the sensors are dynamically retrieved from the XML file. The timesteps right now are milliseconds, seconds, minutes, and hours. The starttime and endtime are not done yet, but you will be able to choose which time to which time.

After seelcting the sensor. Usually the menu scrolls back up, but I clicked it again and scrolled it down. So all the top menus animate themselves up and down:

Changed the timestep, so more data is shown, and the menus default roll up:

Blue ZX3 · 09-10-2007, 03:48 AM

Quote: Originally Posted by 2k1Toaster

I think I may be getting a little carried away like this, but once I get started on an idea, I tend to wander on design and then "just a little change" in the layout and I lose 4 hours to tweaking it to look perfect.

Its a curse! I hope it is worth it though...

Yep, I'm definitely in that same boat, I always add extras while debugging the previous problems, only to end up with x2 the problems I started with and less sleep. LOL

2k1Toaster · 09-10-2007, 02:22 PM

Quote: Originally Posted by Blue ZX3

Yep, I'm definitely in that same boat, I always add extras while debugging the previous problems, only to end up with x2 the problems I started with and less sleep. LOL

Yes! Usually right after a new idea, I go to bed with a program that doesnt compile!

2k1Toaster · 09-11-2007, 02:36 AM

Ok I got the time working now. So now you can select the time (down to the tenth of a millisecond) to start the textual and graphical display as well as the time (again tenth of a millisecond) to stop the display.

I think everything is pretty much done. Any suggestions? I just need to configure the program to output the correct XML format now

.

Here is the newest shot:

thekl0wn · 09-27-2007, 03:33 PM

Slick... Very slick!

gt__1 · 09-28-2007, 03:36 PM

Very interesting I have a project coming up i want to work on for work that has
data logging if you want to see if you can accomodate.

i need to count and stamp when the inputs are active

so for DI 1 count how many times it was pressed at what time and how long

I have 9 inputs i need to count eventually.

the project will go much deeper then that but that is one part

also how many time the button was pressed in each hr of every day.

It is for monitoring a piece of machinery when we would like to know how often it was used
what time of day is busiest, what is the life expectancy of the machine at the current rates,
is the equipment being misused my the user all kinds of data if possible. Then later i would
like to add a camera to monitor the equipment to see what something breaks and how it
happened. This will give the maintenance people alot of info to fix it when it breaks down.

anything else you can think of would help to
thanks

h3rk · 10-13-2007, 10:16 PM

data file size and utility:

Is there a maximum amount of data that can be stored for any one sensor?
Will it 'roll' to keep recent values in event max file size is reached?
Can data be cleared for just one sensor? For a given timeframe?

Is it always going to be working, or must it be turned on prior?

What about hitting record and having it permamently save to file the last few minutes (or similar) of data? Turning on could possibly be triggered by an input (i.e. when braking, it knows to store rotor temperature for a set time or duration of the braking, stuff like that, for 'black box' functionality, or for parents keeping an eye on childrens driving habits: when they go too fast, it starts recording, or data saves in the even of a theft)

2k1Toaster · 10-26-2007, 03:55 PM

Quote: Originally Posted by h3rk

data file size and utility:

Is there a maximum amount of data that can be stored for any one sensor?
Will it 'roll' to keep recent values in event max file size is reached?
Can data be cleared for just one sensor? For a given timeframe?

Is it always going to be working, or must it be turned on prior?

What about hitting record and having it permamently save to file the last few minutes (or similar) of data? Turning on could possibly be triggered by an input (i.e. when braking, it knows to store rotor temperature for a set time or duration of the braking, stuff like that, for 'black box' functionality, or for parents keeping an eye on childrens driving habits: when they go too fast, it starts recording, or data saves in the even of a theft)

The software just logs. It doesnt do any cleanup or erasing or anything of the sort. It reads a value and dumps it to the file. It is a very nice file though.

Sorry I didnt respond to this sooner I just saw it. Also, logging must be turned on and then it logs until it is turned off. You can create an other button to start/stop the logging.

I will try to find a way to store x amount of data points in memory without creating huge bogged down systems because for every sensor it is storing like 1000 data points in RAM! I will see what I can do.

The reason I dug up this post is because I am now finalizing the logging, and well I am now adding a lot more.

It is never finished lol.

I know someone mentioned they wanted comma seperated lists... I just realized that Excel opens XML files natively. I think any version over 2003 does! So you can just open open the log file directly. So in that case, a comma seperated list seems unneccessary... What do you think?

Also do you prefer 1 consolidated log as in the new version, or multiple logs spread out in different files in a structural/folder layout? I like the single file, but I am not sure if it can get too large that way. I need to add some protection for that.

h3rk · 10-26-2007, 06:07 PM

I'm cool with one log, based on the number of sensors any one installation will have, especially if the data can be chooped up with respect to time, kinda like you mentioned.
How are you differentiating each 'recording'? Is there a field with a unique identifier for a recording instance, or something like that?

2k1Toaster · 10-27-2007, 12:14 AM

Quote: Originally Posted by h3rk

I'm cool with one log, based on the number of sensors any one installation will have, especially if the data can be chooped up with respect to time, kinda like you mentioned.
How are you differentiating each 'recording'? Is there a field with a unique identifier for a recording instance, or something like that?

It will be in a folder of the sensor name, and then a timestamp on the file itself. Every day it should create a new file. Well, at least for 100 years, and then your January 1st 2108 recordings will be placed in your January 1st 2008 folder.

I dont think that should be too much of a problem. (Famous last words <cough>y2k</cough>

)

2k1Toaster · 10-27-2007, 01:26 AM

Ok, testing the size of the data and I am now satisfied with it.

It turns out to be 470bytes per data point.

So some figures:
1 hour worth of recording at 1Hz: 1.6Mb (temperature or something)
60 seconds at 180Hz: 4.8Mb (performance testing)
1 full day at 1 data point a minute: 116Mb

I dont think that is too bad. Expecially for all the data stored. I think I am going to actually take some repetitive data out, so it is "back to the basics" with it. Or I'll just add another option in the skin file

So I think those sizes are reasonable, no?