There are some GCC options which might be interesting. Please check their exact meaning in the docu (I mean the info pages, use info gcc to see them) and try out which ones produce the best code.
Note: These are for gcc-3.4 or later.

First, I recommend passing -pipe to gcc to use pipes instead of temporary files.
Furthermore, use -fno-rtti and -fno-exceptions. The compiler will write an error for one file which requires expections; leave away the latter option for that one.
And, always use -Wno-multichar to prevent all those warnings about multi-char constants.

Try a combination of the following general optimization flags:
-O2, -O3, -Os, -foptimize-sibling-calls, -funit-at-a-time -finline-functions, -fomit-frame-pointer (the latter may be implied by -O2)

Also, use appropriate machine/architecture-specific settings, for i386 platform have a look at:
-march=athlon-xp (or pentium3, pentium4, etc.)
-mfpmath={387|sse}, -mmmx, -msse, -msse2, -m3dnow
-ffast-math (implies: -fno-math-errno, -fno-trapping-math)
-minline-all-stringops

More voodoo can be played with the following but I recommend not to touch them:
-falign-jumps=, -falign-loops=, -falign-functions=

You can also try the feedback optimizer which requires actually running the program between the two compilation passes:
First pass: -fprofile-arcs
Second pass: -fbranch-probabilities

Some curious people may try -fmem-report -ftime-report -Q and other options on the same info page.

I currently use these gcc flags when compiling POVRay:

 make CXXFLAGS="-W -Wall -O2 -ffast-math -march=athlon-xp -Wno-multichar
 -finline-functions -funit-at-a-time -fno-rtti -fno-exceptions
 -fmessage-length=$COLUMNS -I/usr/local/numerics/include/"

Alternatively, for povray-3.6, one can specify the options at configure time and then get away with a plain make for the build.

 ../povray-3.6/configure --without-svga COMPILED_BY="fill in here" \
  CFLAGS="-W -Wall -O2 -ffast-math -march=athlon-xp -Wno-multichar  \
          -finline-functions -funit-at-a-time"
  CXXFLAGS="-W -Wall -O2 -ffast-math -march=athlon-xp -Wno-multichar  \
            -finline-functions -funit-at-a-time -fno-rtti"

These flags are neither guaranteed to produce especially fast nor espcially small code, it's just what I am using...

Especially when using the Intel Pentium4 processor, you can get even better results using Intel's C++ compiler for Linux. It is available for free for non-commercial use (but a quite big download, expect >60Mb). Unfortunately, the option syntax of GCC and ICC (Intel's C++ compiler) are still not fully compatible, so simply running make probably won't do the trick. Just pass all source files at once to icc, enable inter-object optimizations, check how to use the profiler and wait... (for compilation to complete).