The correct answer is D. No error.
The sentence is grammatically correct. The word "after" is used to indicate a sequence of events. The word "directly" is used to indicate that the action happened immediately after the previous action. The word "to" is used to indicate the destination of the action.
The sentence can be rewritten as follows:
After leaving his office, he went directly to a restaurant.
The sentence can also be rewritten as follows:
He went directly to a restaurant after leaving his office.