FYI most vacuum boosted braking systems in street cars and pressure reserve ABS systems will allow the pedal to go soft once the car is shut off after a few pedal pumps. You have enough vacuum in the reservoir to provide a few vacuum assisted stops should the engine die when the car is moving, but after that it's only mechanical advantage, which in a street car is pretty huuuuge compared to a 'real' racing car (read: LIGHT) without boosted brakes, where the driver doesn't mind 150-200 lbf pedal force.
At least all street cars I have ever done this with. Every one of my cars does this with the engine off. Firm for one or two pumps, then goes squishy. Start car, shut off, same story.
A brake pedal that starts firm and then goes squishy is (nearly) always going to be a fluid boiling issue.
I will now do my favorite thing in the world, and refer you to an excellent book by my favorite, Carroll Smith, Engineer to Win, pages 189-192. Page 192 in particular will interest you because he specifically tells his drivers to do a cool-down lap after their runs to help keep the braking system from heat-soaking and boiling the fluid in the calipers (which is bad for the fluid) and to allow everything else to cool down a little.
What you need to do if the pedal is going squishy once everything is all hot and bothered from the fluid in the calipers boiling is to FIRST:
DETERMINE THE TEMPERATURE THE CALIPERS ARE ACTUALLY REACING WHEN THE PEDAL STARTS GOING SOFT, so go out and drive it hard, bring it into the pits, and let everything heat soak and you think the pedal is going squishy, and buy or borrow an infra-red temp gun (or get those trick adhesive color chart temp indicators that AP Racing makes - you are talking about spending $10k on a racing ABS system [which is not your culprit in this case] so I can assume you can afford a nice temp gun for $150) and see what the temp is actually coming to.
THEN!:
1) Go to fresh new brake fluid with a higher boiling point.
If none is available, then:
2) Improve cooling to your brake CALIPERS, which can be better ducts in the car side, a water spray duct-air-cooling system or possibly even different wheels with more air-space or pumping action....
3) Insulate the calipers from the hot bits in the system better - Page 190 in Smith's book shows a nice little diagram of how to make ceramic / carbon / Bakelite piston insulators that isolate the pad from the caliper pistons.
When you put new fluid in, make sure to get all the old fluid out, and all the new in. Bleeding the system with one of the colored fluids (like the blue colored ATE 'Super Blue') before putting in Wilwood 570 or the other 600* fluids makes it really easy to see when you've got all the old fluid out. Only use fresh fluid from a new sealed bottle, if you don't use the whole bottle, DON'T USE IT NEXT TIME for the brakes - it's ok to save for use in the clutch, but toss it if you are in danger of using it for the brakes. Once it's been opened, it'll start absorbing water vapor from the air and it's boiling point goes down down down.
ALSO as was mentioned above, this is a forum of student designers, not nearly all of whom are weekend racers, you will get lots of oppinions by posting here, so of your original options:
<STRIKE>1. Change to expensive Bosch ABS Racing System.
2. To find some other less expensive Racing aftermarket ABS System.
3. Find the way to cool down the fluid before it gets to the original ABS</STRIKE>
ABS is NOT YOUR PROBLEM!!! It's boiling the brake fluid in the calipers. BUT you won't know that for sure until you get some temperature measurements.
Lastly, that's all just my oppinion, since I'm a college student with aging Saab 900 Turbos, who has never owned a Cayman to use for track days.
Just because your car says Porsche on it doesn't mean you're getting Porsch LMP prototype racer technology, especially since it doesn't say GT3 or GT2 on the side - it says Cayman, a very tidy, very attractive sports coupe meant for the street. You will invariable find limits in the stock hardware if you are doing extensive track driving in hot weather on very high speed endurance road courses.
I can't think of a single factory stock braking system that would be up to the task of extended periods of driving at Watkin's Glen. That's why there's a racing aftermarket supply industry to sell you that $10k racing ABS pump and controller in the first place.
Best,
Drew