Can you hook something that is normally powered by a 12v 250mA AC wall adapter directly up to a battery? Like lets say this one which is 12v and 4.5amps.....
http://www3.towerhobbies.com/cgi-bin...?&I=LXSK06&P=7

Will the extra amperage affect what I'm hooking it up to?

The amperage of batteries and power supplies is a maximum, not a constant. It will be fine.

another way to look at it may be to imagine two gasoline engines.
One may be a lawn mower, the other may be a truck engine.
They both could be feeding off one single gas tank via two hoses.
the lawn mower will be sipping gas.
The truck engine will be guzzling gas.

anyhoo, your wall adapter is like the smaller motor sipping fuel, and it will mathematically last for 18 hours before you need to recharge the battery.

That's fine.

The only time you need to worry about using a power supply with a bigger current rating is with unregulated supplies.

Batteries are usually measured in 'amp-hours' or milliamphours, which means that if you draw a current of one amp for an hour, you'd use one amphour, so if you had a 4.5ah battery and drained it to zero capacity (something you don't want to do too often), then you could run a 250ma load for 18 hours without recharging.

This.

If you notice on the picture it says 4.5Ah. That is a measure of charge, not the maximum current. Maximum current will be well over 250mA.

It should run for about 18 hours.

Also, to the OP - remember, you really, really, shouldn't take a lead-acid battery (like the one you have there) down past 70% of its capacity.

This means you should never actually drain it fully, or even half way.

You'll otherwise end up replacing the battery rather soon as the plates sulphate over.

I'm not sure what you mean by that?
Regulation is to the voltage, not the current.
Devices will always only draw what they need, just like a gasoline engine will draw from the gas tank what it needs, and no more.

What I mean is that if an unregulated supply is rated at 12v at 4A, if you only draw 100mA the voltage will be quite a bit more.

Not an issue with most devices, but some particularly sensitive things might fry.

OK , I buy that. ^^
Here is my overall saying about voltage:
Voltage is Sloppy.
If you're within a few volts (3 or 4) , plus or minus, nearly every device will do fine.
It's a rare device that wouldn't, is what I've seen.