While it would be nice to have a long course meters set-up (50 meter long pool) you can run a perfectly nice comprehensive swim team program with a six lane, 25 yard long pool. (you don't want fewer lanes than that because it limits your ability to hold dual meets)
A deep end with a diving area would be nice, but new diving boards are pretty much uninsurable these days unless it's a school or univeristy pool. So instead of an L or T-shaped pool with a diving area, go with an L-shaped will with a 25 yard x 6 or 8 lane set-up with water depth of 3'6" at one end, and about 6-7' (or whatever the minimum is for competitive diving blocks these days) at the deeper end as the main body of the pool, with the small part of the pool being a shallow/zero entry kids area with some spray areas for the kids to play in.
If you've got a little bit left in the budget, you can install a small waterslide that ends in the deeper part of the kids area. (For whatever reason, they're much easier to get insurance for than the diving boards are)