"The 800 MHz PC-6400 DDR2 runs at 5-6-6 timings"
i would set your memory back to default timings before attempting to overclock your processor. the best thing to do is to overclock one thing at a time. when you raise your fsb you're overclocking your processor and your memory at the same time. so as you increase your fsb you should adjust your memory divider to keep it as close to stock at possible (error on the low side rather than the high side.)
now your cpu ratio is the your multiplier. so a setting of 9 here and a setting of 333 on your fsb would give you a 2997 mhz clock speed on your processor.
cpu vid is the voltage adjustment for your processor
i'm assuming by fsb start you mean fsb strap. in the past this used to determine your trd or common performance level as its called by asus. i believe in most cases with the latest bios you can adjust the trd independently, however this may not be true in your case. it also controls what memory dividers are available to you.
cpu GTLREF is pretty tricky to explain. this was taken from Tom's Hardware
"CPU/NB GTL Voltage Reference: CPU: Auto, 0.63x, 0.61x, 0.59x, and 0.57x. NB: Auto, 0.67x, and 0.61x. CPU Gunning Transceiver Logic (GTL) voltages are nothing more than reference levels that the CPU uses when determining if a data or address signal is either high (1) or low (0). Precision voltage dividers generate these voltages and are usually specified as a percentage of VTT. In the case of 0.67, this would be 67% of VTT. For example, if VTT is 1.20v then a CPU GTL Voltage Reference of 0.67x would result in a GTLREF value of 0.67 x 1.20V = ~0.80V.
These reference values are particularly important when overclocking quad-core CPUs, especially when venturing above about 450FSB. The ability to tune these values per die can mean the difference between 475FSB and 500FSB. Unfortunately, ASUS' implementation of this functionality is rather incomplete as manipulation of only a single GTLREF value is possible (and in a somewhat imprecise manner). The real power in GTLREF tuning comes in the ability to provide each quad-core die with separate reference values. (Recall that current quad-cores are an MCM consisting of two dual-core dies on a single package.)"