[Techtalk] removing installed programs

[Noir]

Ok, this might sound a bit stupid. But here it goes:

When we want to un-install a program in RHL we usually use rpm -e & the 
rest is taken care of. But say we have installed a program from the source 
(gz, Z, bzip2, tar, make, makefile, etc) and now the whole file is 
scattered around the file system. Say, some files are in /usr/local/bin || 
/usr/bin && some are in /etc/ (conf files) && yet some are in 
/etc/program_name/....|| /usr/share/doc/program_name/

So, if we want to delete/ remove the program altogether how will we do it?

I know we can use "locate" then "unlink/ rmdir" in perl or bash or even 
hand hack. But am looking for s/t automated like rpm -e..

Thanks in advance for any kind of help!