BOSTON (WHDH) -- Lawyers for James “Whitey” Bulger were in court Tuesday asking a federal appeals court to remove a judge who’s scheduled to preside over Bulger’s murder trial.

Bulger’s attorneys told the 1st U.S. Circuit Court of Appeals that U.S. District Judge Richard Stearns cannot be impartial and therefore should be removed as the judge to oversee Bulger’s trial.

Stearns was a federal prosecutor and ran the criminal division in the U.S. Attorney’s Office in the 1980s, when Bulger was accused of committing numerous crimes, including several murders. Bulger said he was given immunity by the Feds during that period.

Because of that, J.W. Carney, Bulger’s defense attorney, said Stearns should step down.

“The trial must be overseen by a judge who is not connected to the most infamous period in federal law enforcement history in Boston,” said Carney.

Surprisingly, some of Bulger’s alleged victims said Carney has raised some valid points. Thomas Donahue, whose father was allegedly murder by Bulger, doesn’t want anything to get in the way of the trial.

“I actually didn’t want him to step down, or any judge to step down, but at this point right now, if there’s going to be all this then he should step down. Right now it’s early enough where it’s not going to interfere with the time or any delays,” said Donahue.

Steven Rakes, whose liquor store was allegedly extorted by Bulger, disagrees.

“I don’t believe Stearns necessarily had any information directly involved, that’s why we need Mr. Stearns on board,” said Rakes.

“If James Bulger was allegedly committing dozens and dozens of crimes including murders why wasn’t he prosecuted,” said Carney.

The three-judge panel took the matter under advisement. It is not known when they’ll make a decision on Judge Stearns.

Bulger’s trial is scheduled to begin on June 6. Carney said Bulger will testify at the trial and explain how he was given immunity from prosecution by federal authorities.