Auxiliaries + perfect infinitives

Auxiliaries + perfect infinitives

Use the perfect infinitive of the verbs in brackets with a suitable auxiliary verb:

Example:

1. I've never seen a London policeman.~

You (see) one! You've been in London a week already!

1. I've never seen a London policeman.~

You must have seen one! You've been in London a week already!

Question:

2. We (start) yesterday (this was the plan); but the flight was cancelled because of the fog, so we're still here, as you see.

My answer:

2. We could have started yesterday (this was the plan); but the flight was cancelled because of the fog, so we're still here, as you see.

But the correct answer is were to have started. Could someone explain it to me?

Thanks in advance.