Yuma 18b: They said to him, “We are the representatives of Beis Din, and you are our representative as well as the representative of Beis Din. We impose on you an oath, by the One who made His name dwell in this Temple, that you will not deviate from the instructions we have given you.”
יומא יח ע”ב: אמרו לו: אישי כהן גדול, אנו שלוחי בית דין ואתה שלוחנו ושליח בית דין, משביעין אנו עליך במי ששכן שמו בבית הזה שלא תשנה דבר מכל מה שאמרנו לך.
Before the First Zionist Congress took place in Switzerland in 1897, Herzl sought out 100 rabbis from Poland and Lithuania who would attend the event and lend Jewish legitimacy to the new movement. In the city of Kosova, where Rabbi Shmaryahu Yosef Karelitz, father of the Chazon Ish, served as rav, there was one rav who believed in Zionism. He was invited to the congress and he planned to attend. When R’ Shmaryahu Yosef heard about this, he called in this rav and tried to convince him not to go. In the end, he made him swear that he would not go to the congress, and indeed he did not go. Eventually he recognized his mistake and dropped Zionism altogether.
Someone asked the Chazon Ish, “Why did your father think that swearing would stop him from going to the congress? Didn’t he think he was doing a mitzvah by participating in the Zionist congress that would promote the settlement of Eretz Yisroel? If so, the oath would be, according to him, like swearing not to do a mitzvah, and he would still do the mitzvah.”
The Chazon Ish replied, “We find that before the Yom Kippur service in the Beis Hamikdash, the Beis Din made the Kohein Gadol swear that he would do the Avodah of the Ketores as they instructed him. Rabbi Akiva Eiger on the Mishnayos quotes the Pri Chadash who asks the same question you asked: According to the Tzedukim, their practice of burning the Ketores before entering the Kodesh Hakodashim is the right way to do the mitzvah, so why would a Tzeduki be deterred by the oath? One who swears not to do a mitzvah must still do the mitzvah.
“My answer (Chazon Ish Orach Chaim 126:17) is that deep down, the Tzedukim did not really believe in their own shitah; it was only their yetzer hara that convinced them to sin, but they took an oath seriously, and would not transgress it. So too with Zionism, even though the yetzer hara convinces them with Yishuv Eretz Yisroel and beautiful words, in their hearts they know the truth: Zionism is heresy and a departure from the path of the Torah; therefore the man knew that swearing not to attend the congress is not swearing not to do a mitzvah. It is a real oath, and he would not violate his oath.”
Source: Orchos Rabbeinu v. 4, p. 189
[The Pri Chadash answers that when one swears not to do a mitzvah, it’s true that he should still do the mitzvah, but nevertheless the oath is a שבועת שוא, a vain oath (Shevuos 29a). Since it was well-known that every Kohein Gadol had to swear, this would stop at least those Tzedukim who took the prohibition of a vain oath more seriously than the mitzvah of doing the avodah their way. Others, who might be willing to do the Tzeduki avodah even at the expense of a vain oath, might still do it.
We could also propose two other answers:
1) Perhaps it was known to Chazal that the Tzedukim, having their own interpretation of the Written Torah, held that when one swears not to do a mitzvah, the oath is effective and one may indeed not do the mitzvah.
2) The answer may lie in the introductory sentence, “We are the representatives of Beis Din, and you are our representative as well as the representative of Beis Din.” This meant that they were making the Kohein Gadol the representative of the Jewish people to burn the Ketores, only on condition that he would burn it the way they instructed him. If he chose to do it the Tzeduki way, he would cease to function as Kohein Gadol, and his service would not be a mitzvah even according to the Tzedukim. Thus it would not be allowed under the rule of one who swears not to do a mitzvah. (This answer would only work according to the opinion that the kohanim are the representatives of the people. The Gemara (bottom of 19a) points out that the Mishnah reads better according to that opinion.)]