Speaking at a news conference, Gilbert strongly refuted theories that James has had input on previous hires, the firing of coach Mike Brown and general manager Danny Ferry's recent departure.
"Truth is the truth," Gilbert said.
New GM Chris Grant confirmed the club has talked with Izzo but would not provide any details.
Grant said the team has no timetable to hire a new coach and insisted the Cavs "just want to find the right guy."
A person familiar with the situation told The Associated Press on Tuesday that Gilbert and Izzo discussed terms of a possible contract that would pay the coach up to $6 million a year for four or five seasons. Use of one of Gilbert's jets has not been discussed.
The person spoke on condition of anonymity because the discussions were to be confidential.
The (Cleveland) Plain Dealer first reported Monday that Gilbert and Izzo talked about the framework of a deal, worth about $6 million a season for up to five years.
Izzo makes more than $3 million a season and is under contract through 2016.
According to the latest version of Izzo's deal — which was last signed by him on Sept. 4, 2009 — he has to pay the school $500,000 within 30 days of terminating his employment if he takes a job in pro or college basketball.
A message was left Tuesday by the AP for Izzo, who didn't confirm or deny the Cavs-related reports in a radio interview Monday night.
"I've taken the no-comment-on-all-of-it policy, which is a little bit unlike Tom Izzo," he said on WILS radio in Lansing, Mich. "But I've been very consistent with that. And maybe to close it up, I would just say there are so many things out there that are so far-fetched right now.
"I'm the Michigan State coach and that's what I'm going to do right now. Who knows what the future brings? But it sure isn't at all like it's maybe being speculated."
Izzo led the Spartans to the Final Four this year for the sixth time in 12 seasons. The late John Wooden at UCLA and Duke's Mike Krzyzewski are the only other coaches who have accomplished that feat.