From my point of view, English is a really difficult language, well if you really want to use it properly. English is my second language, and now I have become a English trainer for the people learning English.
1. There are so many rules (grammatical stuff) in the language system, and you have to mind yourself to use them accordingly.
2. there are so many syllables for most words when you speak. You have to speed up your pronunciation if you want to achieve that fluency. but be aware native speakers have the oral muscle built, you have not. you may tumble easily when you try to follow the conversation flow with native English speakers even though you have very good English knowledge.
3. English language has never experienced any collation in the history. a lot of words come from other languages, and some grammatical rules are lack of logical reason, and are truly nuisances.
4. too many vocabularies. look at the dictionary, and you will see how many things you should remember. and it is common for even those native English speakers to encounter difficulties in reading articles. well, they may guess the meaning anyway, but there must be some words they are not sure anyway... but in other languages, as a scholar or well-educated person, it is very rare for you to have some words that you can't recognise.
So if someone says English is an easy language, I alway dismiss such comments because I know they have never experienced the difficulties to learn the language, or they may only get the "grocery shopping" language skills.
When you cross a river, it should take you the same strength from one shore to another. there is no such thing that learning English is easy whereas leaning Japanese is difficult. as long as you want to cross the language barriers that are the things in the middle, you will experience the same...
Hope this help