Yes, it's true. I feel luke-warm for Harry Potter and his wizarding world at best. Which isn't to say it's a bad book series. It's fine. Please, feel free to enjoy them. But they never gave me the magic escapism that I think they provided for a lot of people.
Maybe it was because I was slightly too old when the books were first published. I read The Sorcerer's Stone in the 8th grade, and quickly read the next two, which were already out. But then I had to wait a long time for the 4th book, and by the 5th I was already getting a little old for stories about teenage wizards (and pretty sick of Harry's self-absorption). When the 6th and 7th books were published, I could tell I was only reading them to hear the end of the story, not because I was that interested in any of the characters or conflict. The long gaps between books also made it hard for me to keep track of characters. I wonder if I had started reading the series after it had been completed whether I would have been able to hold onto the thread of the story better than I did.
I think another issue was I never found a character to fully inhabit and experience the story. You'd think Hermione would be my girl, but her type-A, grade-driven personality is way too similar to mine. I see in Hermione the things I dislike about myself, so she's the last figure I want to escape into.
Harry was boring and rude. Ron was fine, if a bit goofy. And the rest of the cast were supporting players who get very little face-time, certainly not enough to develop them into characters that really interested me. Even poor Hagrid got the brush-off after the first three books as Rowling focused more on the Death Eaters conspiracy. Snape is the one exception. I'd much prefer to read the series from his POV, but it was hard to get too into his character when he was always being framed as Harry's nemesis.
I think Rowling's real genius was the world-building. The idea of a parallel wizard universe is a great concept and I can see why it appeals to so many people. But a good world with only basic character sketches works better for fanfiction or theme parks than it does as an story that stands on its own.
Just my opinion of course. Feel free to disagree.
Course, I also dislike Tolkien, so maybe my disinterest in Harry Potter has more to do with my issues with the fantasy genre in general.