You have to write the book that wants to be written. And if the book will be too difficult for grown-ups, then you write it for children.
You are not the opinion of somebody who doesn't know you.