A lot of people out there buy a shared hosting package and expect to be able to run a site with hundreds of hits per second on such a package. Joomla is not static HTML. You may be able to service hundreds, if not thousands of users per second with static HTML pages, but Joomla needs quite some CPU power and if you plan a site with more than about 5 hits per second, you should directly look for a dedicated server, even more hits, and you should think about a load balancing system, several servers and a MySQL cluster. There is an interesting thread on this issue here on the Joomla-forum.
A small site is a powerful site! This also applies to images. Take a look at your images and see if you really need them in a resolution of this size. Wouldn't a picture with half the dpi look as good? Often people don't notice that they are using images the size of lower Manhattan because they set the size in the HTML code to something way smaller and their browser has the images in his cache, but others first have to load a huge image and then it gets reduced to the size of a stamp. This takes a lot of time for them and they don't want to wait that long. So see to a reasonable size of your images both in pixel and in bytes.
Another thing that's often slowing down the performance are missing images. If you have a reference to a non-existent file, the server will need a lot of time to notice that and answer with a 404. In some cases, your server is configured that way that you are redirected to the front page when the server can't find the file and in that case the browser receives a complete new page each time he requests a missing file. Imagine just one page with an image that is used at a dozen places in the layout and is not present. This can slow down the delivery to up to half a minute! Again, the web analyzer will tell you precisely what images are too big and how many images with size you have (wrongly) on your site!
If you can't afford a new server and you are on your performance limit, think about buying a PHP accelerator. There are again numerous available, like Zend or APC. Just look in this forum for further information. They can do little wonders about your performance.