EWF memory usage limit != installed RAM, What's up?

Falcon4 · 05-30-2008, 07:05 AM

Hokay. Well, I'm not using EWF on a CarPC, but I am using it on two Flash-driven PCs (one with a built-in SSD, and one on CompactFlash). So far I've found that pretty much _all_ EWF-related information is here at mp3car.com. So... maybe there are some EWF experts here.

EWF has a big problem with RAM (REG). It only seems to be capable of using up to about 512mb of memory, before it pretty much melts down and crashes rather violently. It hangs up (and crashes) any write operations without so much as a "out of disk space" (more appropriately...) error. But here's the thing: I have 2 GB of RAM. What the fork?

I actually put 2gb of RAM in this thing so I can do huge I/O operations like defragmenting the Flash volume in RAM. But I'm stuck in this loop:
Defrag a little
Commit
Reboot
Defrag a little
Commit
Reboot...

So, there are two solutions to this annoying problem: one, find a way to commit the data while Windows is running - and start a NEW overlay based on that one (which I don't think is far off, is it?). Two, find a way to make EWF stop flipping out whenever it encounters a mere 512mb of uncommitted data. I can live with the unusually long commit time... but dang. Having to reboot every 2 minutes sucks.

Thanks!

SFiorito · 05-30-2008, 11:38 AM

what exactly are you writing to disk? you should really try to minimize the writing to your protected volume or reevaluate what you're using EWF for.

why not just disable EWF, defrag, re-enable, and reboot?

Falcon4 · 05-30-2008, 12:58 PM

Well,simply because of how much RAM I have. Defragging straight on Flash is like trying to murder it. The way defraggers work is designed for hard drives, not flash, so it spends a LOT of time reading data, writing it to a temporary location, updating the file system, reading data, writing it to a new location, updating the file system, so on and so forth... defragging to EWF allows for (theoretically) a single long (but dangerous) write pass after performing all its writing in memory instead of going straight to disk.

Besides, once you go EWF, you then tend to realize just how much you were murdering your Flash in the past (by how long everything takes with it disabled). =P

tigergibb · 05-30-2008, 03:13 PM

why are you defragging flash in the first place when it has almost nil seek time?

you could:

move everything to hard drive
turn off ewf
defrag
turn on ewf
move everything back to flash drive

SFiorito · 05-30-2008, 05:17 PM

ok, didn't realize you were trying to defrag a flash drive. since EWF is protecting from writes you shouldn't be suffering from fragmentation, right?

Falcon4 · 05-30-2008, 10:21 PM

Well, EWF only stores volume writes, it doesn't store on a file basis. So the drive still gets fragmented like usual... even when there are big open spaces to write to, Windows still chooses to split a large file into, from what I saw, about 700 pieces. I have no idea why it decides to do crap like that, but it does, and it hurts.

Seek times are irrelevant... memory usage and loading times suffer from a fragmented drive none the less. And since the Flash in the Eee is soldered to the motherboard, you can't just yank it out and defrag it in another computer. That's how I defragged my CompactFlash-based computer though - I read the whole drive into a file using WinImage, then told it to "defragment image", then rewrote it to the Flash. But when it's soldered to the board...

I just want to know why EWF hangs up and starts crashing programs after about (from what I've now found) 450mb of memory usage on a 2gb system. I would think it would be able to handle more than that. Am I the only one with that problem? Besides, it doesn't just happen when defragmenting - any time the "data in memory" exceeds about 450mb, it starts going haywire.

SFiorito · 05-31-2008, 09:52 AM

ok, i still don't understand....how many files are you writing to the EWF volume and for what purpose? those writes are being stored in memory so as soon as you reboot they're blown away so why bother defragging? again, if you're having to write large amounts of data which you need to be maintained between reboots then you should perform those writes with EWF disabled.

Falcon4 · 05-31-2008, 07:04 PM

... I use EWF for Flash safety, not for preventing writes... I routinely commit changes before each reboot (I even wrote a batch file to clear temp and other garbage, commit-and-disable, then re-enable). I use EWF to minimize the number of write cycles wasted on useless tasks that Windows and NTFS perform (using NTFS for compression). Suggesting that I disable EWF to do everyday things is like pretty much saying not to use EWF at all!

SFiorito · 05-31-2008, 10:47 PM

well, it doesn't sound like EWF fits your requirements. or you should partition your disk. have an EWF protected system partition and a writable partition. or maybe you should consider using the File Based Writer Filter instead? you can protect everything except those files that need to be regularly updated?

Falcon4 · 05-31-2008, 11:00 PM

Wuhh? Partition a Flash disk? Why, so one partition wears out the Flash faster than the other? I don't see the point there... like I said, I'm using EWF so I can CONTROL when Windows actually writes to the Flash, as opposed to treating the Flash as if writes are a good thing.

I thought that was the whole point of using EWF on a CarPC anyway? Since CarPCs typically use Flash (as I would imagine, given the vibrations in a car that would kill a hard drive), I figure you're using EWF just the same way I am... to make the Flash last longer!

I just want EWF to work properly. No more, no less. EWF in its current mode suits my purposes perfectly. However, its memory usage bug is keeping me from easily using it. EWF should be able to use all available memory, and it refuses to. There's my problem. Isn't there a solution?

edit: Unless there's another similar "large write cache" application I'm not aware of - something that can make the write cache expand endlessly until I decide to write it. That would be even better, because I believe the cache can expand beyond about 512mb. EWF is just the closest thing I have to something that can delay writes and do them all at once...

SFiorito · 06-01-2008, 09:33 AM

I've never had a CarPC that had to write such large amounts of data to the protected volume. the only writes that ever occurred were when adding music, and I typically stored that on an external drive (flash or HDD).

You also said you're using an Eee. The flash in those devices are designed to last a lot longer than a CF or cheap thumbdrive.

What are you writing that's so large and needs to be committed to the protected volume so regularly?

Falcon4 · 06-01-2008, 06:21 PM

Actually, the Eee uses nothing more than standard "off the shelf" (so to speak) Flash chips and a typical Flash-to-ATA controller chip found in CompactFlash cards. Not the high end Flash chips that are used in hard-drive-replacement SSD's. It wasn't really designed to be reliable or durable, just cheap. So it's pretty much booting and running off a glorified USB stick.

What am I writing? Well, I'm just using it like the typical computer that it is... installing software, downloading stuff from the internet, so on and so forth. The only reason it needs to be committed so regularly is because EWF refuses to use all available memory (which, contrary to appearances, is what this topic is about).

EWF (or ANY OTHER write caching system that can do what EWF does without the hassles) also speeds up the computer because it doesn't need to wait on the Flash drive to do all its erasing and rewriting hundreds of times in a row (thank you Windows). I can almost immediately notice when EWF isn't enabled on a boot, because things take longer to load and the computer generally feels sluggish. Since EWF takes all writes and puts them in RAM (and makes that data readable from RAM instead of the drive), combined with read caching filling the remaining memory, this system seriously hauls the proverbial *** when EWF is giving it that boost.

It's just the problem that EWF crashes before it uses even a quarter of the available memory!

craptree · 06-04-2008, 06:41 PM

Hey there, I had exactly the same problem as you. EWF is very inefficient, and eats up RAM faster then fire to petrol. Try FBWF, does a smiler thing,but much much much more efficient (for various reasons). I have a guide up on installing this if your interested.

tannin · 06-19-2008, 09:26 AM

Quote: Originally Posted by craptree

Try FBWF, does a smiler thing,but much much much more efficient (for various reasons). I have a guide up on installing this if your interested.

guide linky?
(edit: nvm, search works

)

From reading the docs, FBWF doesn't support file locking. Have you found anything that reacts poorly to this? Some of my software wouldn't work, but I've got a traditional 2.5" drive I can install anything non-system on, if all normal windows stuff is okay with that.

craptree · 06-23-2008, 10:43 AM

http://www.mp3car.com/vbulletin/winn...ws-xp-pro.html

I have not run into problems (yet) with the lack of file locking. I use FBWF on my laptops primary drive, where all programs are installed, including windows.