A federal appeals court says the S.E.C.’s use of an in-house judge violates defendants’ rights.

A divided federal appeals court panel on Wednesday threw a wrinkle into how the Securities and Exchange Commission prosecutes some enforcement actions by declaring that its administrative proceedings can violate a defendant’s constitutional rights.

The U.S. Court of Appeals for the Fifth Circuit, in a 2-1 decision, ruled that the S.E.C. violated a hedge fund manager’s Seventh Amendment right to a jury trial when it let an in-house judge decide a civil fraud case. Such administrative proceedings are common among regulatory agencies, which use them to decide some enforcement actions.

The ruling from the Fifth Circuit — one of the nation’s most conservative federal appellate courts — is another legal challenge to the S.E.C.’s growing reliance on administrative law judges, rather than the filing of civil complaints in federal court. But for the moment, the ruling’s impact is limited to federal courts in the court’s jurisdiction, which covers Texas, Louisiana and Mississippi.

“The Seventh Amendment guarantees petitioners a jury trial because the S.E.C.’s enforcement action is akin to traditional actions at law to which the jury-trial right attaches,” Circuit Judge Jennifer Walker Elrod wrote in the majority opinion.

Judge Elrod, who was appointed to the court by former President George W. Bush, said the S.E.C. did not have the authority to bring such a case before an administrative court because it did not involve solely “public rights.”

The majority opinion said the in-house judge’s ruling against George Jarkesy in a securities fraud case should be vacated and sent back to the regulator. The court said the S.E.C. must act in accordance with the appellate court’s ruling, presumably requiring the case to be refiled in federal court.

In a dissenting opinion, Judge W. Eugene Davis wrote that the majority misinterpreted the Supreme Court’s definition of what constitutes “public rights.” Judge Davis said Congress permits agencies to litigate cases before in-house judges if it involves “public rights,” such as protecting investors and “furthering public interests.”

An S.E.C. spokesman said the agency was “assessing the decision to determine appropriate next steps.”

This month, the Supreme Court said it would take up another Fifth Circuit ruling that raised a challenge to a different aspect of the S.E.C.’s administrative proceeding process. In 2018, the Supreme Court ruled 7-2 that administrative law judges at the S.E.C. had been appointed to office in an unconstitutional manner.