Before we draw an architecture, we write the test set. It is the fastest way to find out what the problem really is.

Start with fifty examples

Fifty real cases, labelled by the people who will use the system, beat a thousand synthetic ones.

Let the scores pick the architecture

Once you can measure, the debate about which model to use mostly settles itself.