Generally speaking, fruit trees take approximately seven years to get out of their juvenile stage. At that time they are maturing in order to reach the adult stage in which they produce blossoms, which in turn gets pollinated, and then produces fruit. They only way of getting around that time frame, would have been if the key lime had been grafted onto a older rootstock--which is usually what commercial nursery growers do. Just hold on for a couple more years!
since you got rid of it it doesn't make much difference now but if it was only a foot tall after four years i'd say poor lighting and lack of proper nutrition
Unless it is a budded plant it takes 5 years minimum to produce flowers. Graft takes only 2-3 years. Transplant it into polythene bag of thick Gage, if you need the pot for another use.
Key lime trees grow to 3 to 5 feet. A one foot tree over this period of time seems to indicate a lack of nutrition in the soil or lack of light Key limes need slightly acidic soil and 8 to 12 hours of sunlight per day to be happy
