Marc's Public Blog - All Items

Update: I have a much better solution (and meter), the ECM1240 from Brultech, read about it here.

Ok, the title is a mouthful, but that's why it's been about 6 months since I started and I have results to show for it only now.

My goals were to get:

a daily summary of power use per hour

have enough data to recreate a PG&E bill on a daily basis before we get it (to be able to predict whether we're using electricity in the right amount and at the right time before getting the bill).

more importantly, be able to drill a bill down to the day and the hour (we have data updates every 2mn) and see how much more power the house is using if the dishwasher is on, or the electric toaster oven.

get a real sense of how much our AC is running and how much it costs.

For this, while ordering the Solar Panels (aka PV system) from Cobalt Power, I asked for a monitoring system and the only one they found at the time that would allow monitoring of both the PV production and the PG&E Meter was a third party One Meter from One Brand Electronics.

It looks like this:

the meter that does gathering and resending to my monitoring server

the probe gathering boxes (gets volts and amps)

this shows the CTs that measure current from their induced magnetic field

The voltage probes are simply connected to the lines and those are precise. Unfortunately, I don't care about voltage nearly as much as current sinc ethe voltage tends to be known and the current is what affects your bill.

It's a reasonably easy solution to add to any setup during or after the fact, but the low points with the Brand Electronics One Meter solution are:

by design the current measured is inexact. I see current used on the PV inverter when it's not supposed to use much of anything, and of course I also see random small currents in either direction on the AC when it's off.

while the data gathered is good enough for general trends, longer term use has shown that it can be off by as much as 40Kwh/month on my PG&E Meter, which is a fair amount.

the One Meter interface is really bad. I'm a linux CLI guy, but it is bad. It is impossible to change its basic configuration like what it sends over telnet and at what interval without having its firmware reburnt by onebrand, but get this: you can't even change the IP address it'll talk to either without sending it back.

it has a serial interface for configuration that is quite poor and it'll dump old data you might have missed in a different format that what it sends over telnet to make your life harder.

it is a very slow microcontroller that is so slow that it takes over one minute to send 12 samples of data

it is of course unable to set its own time over ntp and is unaware of DST, so I just put mine on UTC and I fix things on the server side.

last, but not least, it will randomly corrupt itself and send totally bogus data points. Thankfully it however recovers at the next sample. I had to write fairly complex code to analyse the data samples, detect and throw away bad ones.

Ok, so are you sold yet? :)
Well, it's not that bad now since I went through the effort of writing the code to deal with all this. So, if you were to get one, you'd be up and running pretty quickly if you can hack perl, the language I used for my magic script that does all the work (see the bottom of the page for source code).

The above script took a fair amount of time to write since outside of working around quirks in the One Meter output, I wrote in the PG&E billing logic for California and it is able to output per hour per day production and use, as well as equivalency in dollars. Converting into dollars make sense since with TOU (time of use pricing) you can end up with days where your used more than you produced, but the end bill is still negative.

This is a typical example of a summer day (July 8th picked at random):


Hourly Differences
00_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
01_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
02_:  1.0Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  1.0Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  1.0Kwh/$0.1 PG&E|
03_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
04_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
05_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
06_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC| -0.0Kwh/$0.0 PV|  0.8Kwh/$0.1 PG&E|
07_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC| -0.7Kwh/$0.1 PV|  0.1Kwh/$0.0 PG&E|
08_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC| -2.2Kwh/$0.2 PV| -1.3Kwh/$0.1 PG&E|
09_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC| -3.2Kwh/$0.3 PV| -2.3Kwh/$0.2 PG&E|
10-:  0.8Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.8Kwh/$0.1 HouseNoAC| -3.9Kwh/$0.6 PV| -3.1Kwh/$0.4 PG&E|
11-:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC| -4.3Kwh/$0.6 PV| -3.4Kwh/$0.5 PG&E|
12-:  0.8Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.8Kwh/$0.1 HouseNoAC| -4.5Kwh/$0.6 PV| -3.6Kwh/$0.5 PG&E|
13~:  0.9Kwh/$0.3 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.2 HouseNoAC| -4.4Kwh/$1.3 PV| -3.5Kwh/$1.0 PG&E|
14~:  0.8Kwh/$0.2 House|  0.0Kwh/$0.0 AC|  0.8Kwh/$0.2 HouseNoAC| -4.1Kwh/$1.2 PV| -3.2Kwh/$0.9 PG&E|
15~:  0.9Kwh/$0.3 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.3 HouseNoAC| -3.3Kwh/$1.0 PV| -2.4Kwh/$0.7 PG&E|
16~:  0.9Kwh/$0.3 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.3 HouseNoAC| -2.0Kwh/$0.6 PV| -1.1Kwh/$0.3 PG&E|
17~:  0.9Kwh/$0.3 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.3 HouseNoAC| -0.8Kwh/$0.2 PV|  0.1Kwh/$0.0 PG&E|
18~:  0.9Kwh/$0.3 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.3 HouseNoAC| -0.2Kwh/$0.1 PV|  0.7Kwh/$0.2 PG&E|
19~:  1.0Kwh/$0.3 House|  0.0Kwh/$0.0 AC|  1.0Kwh/$0.3 HouseNoAC| -0.1Kwh/$0.0 PV|  0.9Kwh/$0.3 PG&E|
20-:  1.0Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  1.0Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  1.0Kwh/$0.1 PG&E|
21-:  1.1Kwh/$0.2 House|  0.0Kwh/$0.0 AC|  1.1Kwh/$0.2 HouseNoAC|  0.0Kwh/$0.0 PV|  1.1Kwh/$0.2 PG&E|
22_:  1.0Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  1.0Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  1.1Kwh/$0.1 PG&E|
23_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
----------------------------------------------------------------------------------------------------
Wed: 22.6Kwh/ $3.4House|  0.1Kwh/ $0.0AC| 22.5Kwh/ $3.4HouseNoAC| 34.7Kwh/-$6.7PV|-12.1Kwh/-$3.3PG&E|

Split per rate:
$.08 11.0Kwh/$0.93House|  0.0Kwh/$0.00AC| 10.9Kwh/$0.92HouseNoAC| -6.2Kwh/$0.52PV|  4.9Kwh/$0.41PG&E|
$.14  4.6Kwh/$0.67House|  0.0Kwh/$0.00AC|  4.6Kwh/$0.66HouseNoAC|-12.7Kwh/$1.83PV| -8.0Kwh/$1.16PG&E|
$.29  6.2Kwh/$1.83House|  0.0Kwh/$0.00AC|  6.2Kwh/$1.82HouseNoAC|-14.8Kwh/$4.35PV| -8.6Kwh/$2.52PG&E|

Solar panels produced  34.7Kwh during 12.10h, between 06:46:28 and 18:50:28

Yellow shows partial peak rates, where it's good to have the meter run backwards, and red for peak rates when it's even better. Unfortunately the solar panels are facing south east and south instead of south west where they would produce more when electricity is worth more.

That is a more interesting example in September showing that despite having used more electricity in a day than what was produced, the daily bill was still negative thanks to the time of use magic, even it our case where we only get partial benefit from TOU due to the sub optimal south east angle for some of our panels:


Hourly Differences
00_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
01_:  1.0Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  1.0Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  1.0Kwh/$0.1 PG&E|
02_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
03_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
04_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
05_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
06_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
07_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC| -0.1Kwh/$0.0 PV|  0.8Kwh/$0.1 PG&E|
08_:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC| -1.5Kwh/$0.1 PV| -0.7Kwh/$0.1 PG&E|
09_:  0.8Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.8Kwh/$0.1 HouseNoAC| -2.8Kwh/$0.2 PV| -1.9Kwh/$0.2 PG&E|
10-:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.8Kwh/$0.1 HouseNoAC| -3.5Kwh/$0.5 PV| -2.6Kwh/$0.4 PG&E|
11-:  0.8Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.8Kwh/$0.1 HouseNoAC| -3.9Kwh/$0.6 PV| -3.1Kwh/$0.4 PG&E|
12-:  0.8Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.8Kwh/$0.1 HouseNoAC| -4.1Kwh/$0.6 PV| -3.2Kwh/$0.5 PG&E|
13~:  0.8Kwh/$0.2 House|  0.0Kwh/$0.0 AC|  0.8Kwh/$0.2 HouseNoAC| -4.0Kwh/$1.2 PV| -3.2Kwh/$0.9 PG&E|
14~:  0.8Kwh/$0.2 House|  0.0Kwh/$0.0 AC|  0.8Kwh/$0.2 HouseNoAC| -3.6Kwh/$1.0 PV| -2.8Kwh/$0.8 PG&E|
15~:  0.9Kwh/$0.3 House|  0.0Kwh/$0.0 AC|  0.8Kwh/$0.2 HouseNoAC| -2.2Kwh/$0.6 PV| -1.3Kwh/$0.4 PG&E|
16~:  0.9Kwh/$0.3 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.3 HouseNoAC| -1.0Kwh/$0.3 PV| -0.1Kwh/$0.0 PG&E|
17~:  0.9Kwh/$0.3 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.3 HouseNoAC| -0.1Kwh/$0.0 PV|  0.8Kwh/$0.2 PG&E|
18~:  0.9Kwh/$0.3 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.3 HouseNoAC| -0.1Kwh/$0.0 PV|  0.8Kwh/$0.2 PG&E|
19~:  0.9Kwh/$0.3 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.3 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.3 PG&E|
20-:  0.9Kwh/$0.1 House|  0.0Kwh/$0.0 AC|  0.9Kwh/$0.1 HouseNoAC|  0.0Kwh/$0.0 PV|  0.9Kwh/$0.1 PG&E|
21-:  2.6Kwh/$0.4 House|  1.1Kwh/$0.2 AC|  1.5Kwh/$0.2 HouseNoAC|  0.0Kwh/$0.0 PV|  2.7Kwh/$0.4 PG&E|
22_:  3.2Kwh/$0.3 House|  1.1Kwh/$0.1 AC|  2.0Kwh/$0.2 HouseNoAC|  0.0Kwh/$0.0 PV|  3.2Kwh/$0.3 PG&E|
23_:  3.8Kwh/$0.3 House|  1.8Kwh/$0.2 AC|  2.0Kwh/$0.2 HouseNoAC|  0.0Kwh/$0.0 PV|  3.8Kwh/$0.3 PG&E|
----------------------------------------------------------------------------------------------------
Thu: 29.0Kwh/ $4.0House|  4.4Kwh/ $0.4AC| 24.7Kwh/ $3.6HouseNoAC| 27.6Kwh/-$5.2PV|  1.5Kwh/-$1.2PG&E|

Split per rate:
$.08 15.8Kwh/$1.34House|  3.0Kwh/$0.25AC| 12.9Kwh/$1.09HouseNoAC| -4.4Kwh/$0.37PV| 11.5Kwh/$0.97PG&E|
$.14  6.0Kwh/$0.87House|  1.1Kwh/$0.17AC|  4.9Kwh/$0.70HouseNoAC|-11.5Kwh/$1.66PV| -5.4Kwh/$0.78PG&E|
$.29  6.1Kwh/$1.79House|  0.0Kwh/$0.00AC|  6.1Kwh/$1.78HouseNoAC|-10.9Kwh/$3.19PV| -4.8Kwh/$1.40PG&E|

Solar panels produced  27.6Kwh during 10.20h, between 07:34:28 and 17:48:28

Now, a per hour text output is useful to see how we did on a given day, but it does not let you see your power usage in the last 10mn after you turned something big on or off, or how all of last week looked, or see a month's trend at a glance.

This is where cacti comes in. I spent a fair amount of time looking for a graphing solution that would keep all my data and let me zoom on any portions I wish. Quite frankly, the fancy widget that google uses in google finance would have been what I was really hoping for, but baring that, cacti came like a reasonable alternative.

So, I came up with a compound graph that looked reasonable, and the option to see all the graph items separately.

So since this is about solar panels, one of my questions to Cobalt Power was why my system was spec'ed for 5Kwh at peak and why I was typically seeing 4.5-4.7Kwh at best. The reason is that 2/3rd of the panels are pointing south-south-east and the remaining 3rd is pointing due south.
What this means is that our system doesn't actually peak at one time and then drop off. The graph clear shows that the PV production takes longer to peak and then stays at that lower peak a bit longer before dropping off more sharply. So, the production at the end of the day is still the same, but just with a slightly longer and slightly lower peak. Too bad the peak is on the morning side as opposed to the afternoon side when the electricity is worth more.
You'll also notice the sharp start which is when the sun goes high enough to reach the roof over the neighbours' houses across the street. You'll also notice the jigsaw drop on the way down as the sun starts to get hidden behind some branches in the high trees blocking our west view.

This graph and the graphs below are selected in zoom mode, so you can use the mouse cursor to draw a rectangle on a time region and the graph will refresh on the time slice you selected.

Below is a closeup of all the colors showing AC working hard in bursts while the black PG&E meter line goes from sometimes negative to always positive the green area of solar panel energy offset goes down to 0. Blue then shows the basic house energy usage with the red peaks on top showing AC tripling the house electricity use when it's running.

In other words: the red line is the AC use, the blue is the rest of the house use, red line goes on top of house use gives the red area which is total house use. You then remove green which is production from the solar panels and you end up with the black line which is what the PG&E meter actually sees. Confused yet? :)

Here's a typical day without AC use since we weren't home:

Here's a very warm day with AC (keep in mind that you can zoom in by selecting a time slice with the mouse):

and a glance at just AC use during a few hot days in July:

Here is a link to an historical view of this graph at multiple time sample levels. Try the zoom function (magnifying mirror next to the graph) on the yearly graph at the bottom and you'll be able to zoom on random time in the past.

Another view is each probe graphed separately for a less cluttered view. Note that the AC view is pretty useless right now as it's only showing noise picked up by the coils, but if you click on it, you'll get the multiple timerange view as above and can view more interesting months, like the July zoom above.

What else?
Well, I had written some somewhat complex code to actually keep up of production for each tier and find out if I went into tier 2 to 5 (which pay or cost more per unit of energy), but because the One Meter gives me somewhat inaccurate readings, especially on the PG&E side, the numbers were just too far off to compute a bill that was close enough to the actual bill I was getting. So, I unfortunately had to give that part up.
It would otherwise have been nice to know in advance if my use or production for a given tier was going to exceed tier 1 and possibly adjust electricity use accordingly, but I'll need a more accurate measuring device than the One Brand one.

All this ended up being a fair amount of work, which with not that much extra work could have been a service that all solar companies sell. I think a few do, but I'm not sure if it's as complete as what I did, or if it is, please let me know so that I can compare with their work.

Anyway, if it's useful to you, here's the parsebrandpower script. Please note that if you were planning on taking it and selling service based on my work, it is protected by the GPL 3 copyright which you must understand and apply if you are going to use it. I also request that you contact me and let me know if you're going to use the code.
For more details on cacti integration, see my Gatewaying 1-wire, XPL (Oregon Scientify Weather), Brultech ECM1240 Power Data, and Brand OneMeter Data to cacti page.

π 2009-11-26 01:01 in Cats

We had a nice quiet little thanksgiving, Jennifer was resting after a few bad nights due to her minor nose surgery (opening the passageways so that she's less congested and can breathe better).

We had some turkey legs and so did chat :)

See more images for Five Thanksgiving

π 2009-11-23 01:01 in Cars, Ncars

While hacking, I have been watching old seasons of WRC, World Rally Championship. Unfortunately, I'm kind of doing it backwards: I started with 2008 and I now know the future when watching the older seasons.
Nonetheless, watching the 2008-2009 season and the close to photo finish by Sebastien Loeb, was nice.

Now, I'm back in time watching 2002, where some of the videos are from Speed channel before those morons dropped WRC, and others from a Japanese channel. The Japanese version is hilarious to listen too, and they totally win for having very appropriate great techno music tracks in the background. Others should have learned from them :)
It's cool to see Sébastien Loeb already win a tarmac stage in Germany all the way back then.

Awesome WRC highlights video #1

Awesome WRC highlights video #2

link to collection of WRC highlights videos

π 2009-11-22 01:01 in Public

My thinkpad 500GB hard drive had been slowly failing (which I knew courtesy smartd, and , so I got an advanced replacement from Hitachi and copied the failing drive minus its two already failed sectors (courtesy of GNU dd_rescue).

While we're at it, I recommend that people run smartd with the following in their smartd.conf:
DEVICESCAN -R 194 -R 231 -I 9 -W 5 -a -o on -S on -s (S/../.././02|L/../../6/03) -m root -M exec /usr/share/smartmontools/smartd-runner

and that you put this in your crontab for later analysis and figuring out what's being going on with your drives, even after the fact:
2 1 * * * root DIR=/var/log/smart; mkdir -p $DIR ; FILE=$DIR/`date '+\%F'`.log; for i in /dev/sd?; do echo $i; smartctl -a $i; echo; echo; done > $FILE

The harder part was figuring out which files were going to be partially lost due to those sectors. After racking my brain on how I could convert those sectors into filenames and getting nowhere, I realized that was a very simple way of finding out: just read all the files and log the errors.
A day or so later I had read the entire filesystem and narrowed it down to the two files that were damaged. Sometimes the low tech option is the best.

While I was doing that, I was trying some new tools and two new boot CDs I had built, something any self respecting system should carry: UBCD and/or UBCD4Win. UBCD4Win is basically a windows live CD you build from your windows install media, and UBCD has a lot of dos boot floppies (disk check, bios, rescue and more), and a nice version of parted magic, recovery linux live distro that runs from RAM.
The only small downside is where gparted, at least on one occasion, has a small bug that kind of shredded an ext3 partition of mine, but it worked fine the rest of the time, even for resizing ntfs partitions.

The cool part was when I tried to rescue to the ext3 partition for practise (I had backups), and saw Ted Tso online in my gtalk friend list (I'm trying to remember when I added him, maybe when I helped him out with his G1 at a linux conference), and he nicely helped me out trying to fake out resize2fs and then debugfs but in the end, the filesystem was kind of mangled though.

Anyway, gnu ddrescue, ubcd and ubcd4win rule (and mine now contain extra tools, including image for linux and image for windows).

π 2009-11-21 01:01 in Clubbing

It had been a little while since I saw Tijs. He was back for a his US Kaleidoscope tour. I didn't have a good reason not to go check it out :)

In a nutshell, it was not to the level of his former "Tiësto in concert" tours. It was not bad presentation-wise, but replacing live lyric singers doing their tunes live by a video screen with somewhat poor resolution.
Ok, that and the fact that most of the tunes he played, weren't so good or to my liking, but eh, it was still a good evening :)

kaleidoscope tour/the intro video was nice

the video screens were nice-ish, but the resolution quite poor and the rendering not so good

looks like old 200 lines interlaced CRTs we used to have :)

it was nice to see the singers during their song, but as nice as live appearances

yes, he thinks he looks cool in big. I suppose that's somewhat true :)

Tiësto finished with his signature tune, Adagio for Strings:

See more images for DJ Tiësto @ Cow Palace

π 2009-11-21 01:01 in Linux

Ok, it's not often that I recommend a commercial product (or it's actually even the first time for that matter), but I would like to recommend Image for Windows, Linux, and Dos.

Why?
Well, doing linux backups and full restores that I can boot is not an issue for me. I realize that some might have a bit harder of a time than I would doing a full linux restore from a boot floppy or CD, and making it bootable, but for the seasoned linux admin, it's not actually that hard.
Doing the same with windows is harder though. To this day I haven't found a good way to do a full file level backup of a windows partition that I can restore and then boot from (especially with ntfs and special file permissions, ACLs, and forks).
As far as I know, windows users just end up using a ghost like solution (i.e. block level backup and not file level backup).

Ghost is however a royal pain in the ass. First, there is not one ghost, but 2 mostly incompatible ones: Symantec Ghost (.gho images) and Norton Ghost (.v2i). Most people know the consumer version of Ghost, aka Norton Ghost which creates .v2i images.
What most people don't realize is that restoring those images if you get a full failure and need to recover from boot media, is actually not that simple. Some ghosts versions do not allow you to recover from the network at all, requiring a USB hard drive that's at least as big as what you're backing up (not always convenient or feasible). If you use the network/corportate Symantec ghost however, and spend enough time figuring things out, you can restore a Norton .v2i ghost over the network, but boy it's not easy.
Read more about Symantec Ghost vs Norton Ghost.

I personally gave up when norton ghost 14 gave me some bullshit error message that I couldn't backup a brand new laptop and compress the image. A search of the error code showed no reasonable fix, and that was enough to entice me to look for a new solution.

I looked at the linux solutions briefly, but didn't find anything that wasn't just a fancy dd, i.e. a full partition backup of all the blocks, whether they are used or not. This is not what I wanted, it's stupid a wasteful.

This is when I stumbled on a reasonably priced shareware program: image for windows (linux and dos). I ended up going with that for the following reasons:

it has full backup and restore support of all supported filesystems (unsupported ones just cause a full partition backup) from windows, linux, and dos

Filesystem aware backups and restores of FAT, FAT32, NTFS, EXT2, EXT3, ReiserFS, and XFS. What this means is that only used blocks will be backed up and in the case of windows, it can skip the backup of the hibernation file and the windows paging file.

the restore supports built in filesystem resize for all those filesystems (no small feat). The only caveat is that if you're restoring a 30GB partition, you need 30GB of partition space before you can resize down to 10GB if you only have let's say 5GB of data.

you can backup or restore to vmware virtual disks without being inside vmware. This is pretty cool since it lets you restore and boot an image without having a machine to restore on.

the windows client has a driver to backup a running system and read open/locked files (trivial on linux, but not on windows).

there is simple but good enough automated backup support for windows (linux is of course easy by using cron, but I really don't need this for linux since rsync to an nfs server will work fine too).

network restores are easy by using either a custom made bootcd with windows (using UBCD4win) or simply booting any linux rescue CD (or the program can make one for you), mounting whatever network share over nfs or smb and doing your restore that way.

the PDF docs are pretty good, the programs have good flags for automation from the command line

the email support is just about the best support I've ever had for a commercial program. There is no SLA or phone number to call, but I've had answers most often within an hour or less 7 days a week so far.

there is a basic program to open up backup images and extract individual files from linux and windows.

the shareware is not crippled in a way that the program is not usable: you can do full backups and restores to properly evaluate the software (you do have to restore backups after 30 days, which is more than reasonable).

Downsides:

it's not open source, no, it's not free as in beer, but neither is ghost and at least this one works and has both a native linux and windows client. Considering that the open source versions don't come close in features, I'm more than ok with using a reasonably priced closed source/commercial app here.

No Mac support, but I'm guessing Mac can be backed up about as easily as linux machines, with rsync and some rescue media to copy the files back over, so support is not as necessary.

You can read more about it and download it from here (windows version includes full linux client too, or linux client can be gotten directly here)

Separately, while I was doing my search, here are other options I found:

G4L (Ghost for Linux) and Ping (Ping is not Ghost) didn't seem to be anything more than fancy DDs that were not filesystem aware, and of course they would not backup a running windows system

partimage also looked interesting, but it doesn't quite seem finished, especially for ntfs support.

clonezilla looked more interesting. It uses ntfsclone which is a pretty cool ntfs partition copier that skips unused blocks (and for bonus points, it uses spartse blocks to make a disk image you can otherwise mount over loopback). Along with ntfsresize, it's not bad for backing up a non running ntfs partition. If you don't need to backup a running windows partition, clonezilla ain't bad.

π 2009-11-21 00:00 in Computers

I've actually liked the thinkpads I've gotten over the years. I've had a total of 5 now: thinkpad 365XD (800x600), thinkpad A21p (1600x1200), thinpad T42p (1600x1200), thinkpad Z61p (1920x1200), and I'm now testing a thinkpad W500 (1920x1200).

As you can tell, I'm addicted to high resolutions. Afterall my desktop is running 3840x2400, so even 1920x1200 feels small to me :)

I still have the last 3. The T42p is still a good laptop, even if it's a bit short on memory and CPU horsepower now.

The Z61p is still a good machine where I'm writing this blog post. Its somewhat bigger size is nice because I was able to fit an extra big hitachi 500GB hard drive that doesn't fit in other laptops. Its downfall though is a stupid bios that prevents more than 3GB of Ram from being mapped (I have 4GB I paid for and I'm a bit pissed about that).

The W500, I still have to test further, but it looks promising. It supports at least 8GB of RAM, CPU is of course a bit faster, and it comes with 802.11N wireless while being a bit thinner than the T42p (not that I really care).

Here's the small family:

don't pay attention to the OS on the left, it was being removed :)

While the W500 is not a huge upgrade from the Z61p, it's a more standard laptop at work and still a good incremental upgrade.

See more images for Thinkpad Evolution

π 2009-11-19 01:01 in Dining, Ndining

Jennifer had her 35th BD today, and we went to Papillon to celebrate. It was yummy :)

See more images for Jennifer's 35th Birthday

π 2009-11-19 01:01 in Exercising

We had a 5K run today. I was on the fence as to whether I should have done it since I haven't been running a lot recently, especially after having been sick, but my coworker went and that was enough incentive for me to go too so that I could hopefully stay ahead of him.

In the end, I did meet that goal, but my 5K time was about 24mn, which is bad compared to peak times of 22:18, but it's at least under the 25mn benchmark.

π 2009-11-09 01:01 in Cats

Roomba Chat takes care of eating spiders and moving insects, or other things that look like food. Roomba takes care of the dirt and hair.
Right now, both are recharging :)

both roombas are charging

See more images for Roomba Chat

π 2009-11-09 01:01 in Google

Al Gore came to google to promote his new book. It was an interesting talk and nice to see him in person.

See more images for Al Gore Visit

π 2009-11-08 00:00 in Computers, Linuxha, Solar

π 2009-11-04 01:01 in Public

Comcast still tries to charge me $70/month for a few digital channels I watch and they won't unbundle, as well as a DCH-3200 descrambler box that they force me to use by needlessly scrambling QAM64/QAM256 encrypted channels, but at least the process of getting the bill lowered to something somewhat more reasonable is less painful today.

I went to www.askcomcast.com/CA as per the bill as opposed to calling the poke my eyes out in pain 800-Comcast, and 10mn later clock time, it was done.


Rotcelyn(Wed Nov 04 2009 18:43:29 GMT-0800 (PST))
Hello, Marc.

Marc_(Wed Nov 04 2009 15:43:39 GMT-0800 (PST))
do you have access to my account?

Rotcelyn(Wed Nov 04 2009 18:43:57 GMT-0800 (PST))
I need to ask for your account number.
I see here that you want to have new prices set for your account, correct?

Marc_(Wed Nov 04 2009 15:44:17 GMT-0800 (PST))
XXXX XX XXX XXXXXXX

Rotcelyn(Wed Nov 04 2009 18:44:28 GMT-0800 (PST))
Thank you so much for that.
I see here that you want to have new prices set for your account, correct?

Marc_(Wed Nov 04 2009 15:45:29 GMT-0800 (PST))
this is a long running problem where I am forced to use your descrambling box against my will since I have QAM64/QAM256 capable TVs and DVRs

I've already talked to your executive staff several times about this and basically they just lower my price since I only watch 7 of your channels, 
3 of which now needlessly force me to use and pay for the DCH 3200 tuner/descrambler

That price adjustement just timed out so I'd like you to re-enable it :)

Marc_(Wed Nov 04 2009 15:46:02 GMT-0800 (PST))
so basically if you can reset the old price, that would be fine
It's still more than what I should pay for 7 channels, but it'll do

Rotcelyn(Wed Nov 04 2009 18:46:59 GMT-0800 (PST))
I understand your concern. Please give me few moments to review your account.

Marc_(Wed Nov 04 2009 15:47:06 GMT-0800 (PST))
sure thing, take your time

Rotcelyn(Wed Nov 04 2009 18:47:14 GMT-0800 (PST))
Thank you so much for that.

Rotcelyn(Wed Nov 04 2009 18:49:16 GMT-0800 (PST))
Thank you for waiting.

Marc_(Wed Nov 04 2009 15:49:41 GMT-0800 (PST))
take your time, I multitask too :)

Rotcelyn(Wed Nov 04 2009 18:50:23 GMT-0800 (PST))
Marc, the best thing I can do to your account is to apply $20 off for 12 mos. So, for 12 mos, you will only have to pay $41.99 for the cable service.

Rotcelyn(Wed Nov 04 2009 18:50:31 GMT-0800 (PST))
Will this work for you?

Marc_(Wed Nov 04 2009 15:50:51 GMT-0800 (PST))
that would be fine, thank you

Rotcelyn(Wed Nov 04 2009 18:50:56 GMT-0800 (PST))
You are most welcome.
Please give me few moments to process this for you.
There will be processing fee for $1.99, one time.
I have completed your request for today.
Your confirmation number is 1000403345242022.

Marc_(Wed Nov 04 2009 15:52:54 GMT-0800 (PST))
thank you

Rotcelyn(Wed Nov 04 2009 18:53:11 GMT-0800 (PST))
After 24 hours, you can check your account online to see for the changes in the statements.
You are most welcome.

Marc_(Wed Nov 04 2009 15:53:23 GMT-0800 (PST))
thanks for your help, your courtesy and have a great day

Rotcelyn(Wed Nov 04 2009 18:53:26 GMT-0800 (PST))
Recap and breakdown of what I have processed:
- Digital Starter $41.99.
- Transaction Fee $1.99
Everything is set.

Rotcelyn(Wed Nov 04 2009 18:53:55 GMT-0800 (PST))
As our valued customer, I would like to make sure I have resolved your concern.
It�s been a pleasure serving you today.
Is there anything else I can help you with today?

Marc_(Wed Nov 04 2009 15:54:31 GMT-0800 (PST))
you already did, thanks

Rotcelyn(Wed Nov 04 2009 18:54:51 GMT-0800 (PST))
You are most welcome.
I hope you won't forget to click END SESSION and take 3 surveys for me. Thank you.

Rotcelyn(Wed Nov 04 2009 18:55:24 GMT-0800 (PST))
Thank you for choosing Comcast as your service provider. We truly appreciate your business! If you need further assistance, you can chat with one of our 
Customer Support Specialists 24 hour a day, 7 days a week at http://www.comcastsupport.com.  You have a great day ahead!
Should you like to watch full TV shows and movies online, go to fancast.com.

Ok, it's still sad that they crypt channels and give you bogus high prices unless you complain, but all in all that was better than before :)