Finding early signs of dementia in the back of the eye may be a way to catch the disease early and start preventive treatment, a study says.